ICPR 2020

Personalized Models in Human Activity Recognition
Using Deep Learning

Hamza Amrani, Daniela Micucci, Paolo Napoletano
University of Milano-Bicocca
Paper
97.3%
Semi-Supervised
S-CNN accuracy
99.9%
Supervised
S-CNN accuracy
3
Datasets
Cross-dataset evaluation

TL;DR — We combine deep learning with incremental learning to create personalized HAR models that adapt to individual users, achieving 97.3% semi-supervised and 99.9% supervised accuracy — significantly outperforming traditional ensemble methods while being deployable on mobile devices.

Simplified CNN (S-CNN) architecture for personalized human activity recognition

Figure 1. The Simplified CNN (S-CNN) architecture designed for efficient on-device human activity recognition. The network uses 1D convolutions with batch normalization and ReLU activations, progressively increasing filter depth (16 → 32 → 64) before global average pooling and softmax classification — compact enough for real-time inference on mobile hardware.

Abstract

Sensor-based human activity recognition systems deployed in the real world face a fundamental challenge: models trained on pooled multi-user data fail to capture the idiosyncratic ways individuals perform everyday activities. While incremental learning offers a path toward personalization through continuous adaptation, and deep learning has proven superior to traditional approaches for user-independent recognition, these two paradigms have remained largely separate.

We bridge this gap with an approach that makes three key contributions:

  1. We combine deep neural networks (ResNet and a lightweight Simplified CNN) with incremental learning to create personalized HAR models that continuously adapt to individual users from a stronger starting point than traditional methods.
  2. We introduce a data augmentation strategy based on multiple overlapping window segmentation with 0.2-second time shifts, addressing the limited labeled data problem inherent in personalization scenarios.
  3. We design three personalization strategies — non-supervised, semi-supervised (90% confidence threshold), and fully supervised — enabling a flexible trade-off between annotation effort and recognition accuracy.
  4. We demonstrate through experiments on three public datasets that deep learning models adapt faster to new users than the Learn++ ensemble baseline, requiring fewer labeled samples to reach equivalent performance.

Evaluated on the Anguita, Shoaib, and Siirtola benchmarks, our S-CNN achieves 97.34% accuracy with semi-supervised personalization and 99.94% with supervised labels — while starting from a user-independent baseline that already exceeds the best personalized accuracy of the traditional Learn++ method.

Method

Our approach begins by training a user-independent model on pooled data from multiple users using either a ResNet or a lightweight Simplified CNN (S-CNN). The S-CNN consists of four convolutional blocks with 1×5 filters, batch normalization, and ReLU activations, progressively increasing feature maps from 16 to 64 channels, followed by global average pooling and softmax classification — deliberately kept compact for mobile deployment.

Personalization via incremental learning. To adapt the pre-trained model to a new user, we apply fine-tuning on incoming user data with three labeling strategies: (1) non-supervised, where the model's own predictions serve as pseudo-labels; (2) semi-supervised, where only predictions exceeding a 90% confidence threshold are retained; and (3) supervised, where ground-truth labels are provided. Transfer learning with light fine-tuning enables rapid adaptation without catastrophic forgetting.

Data augmentation for scarce user data. To address the limited labeled samples available during personalization, we propose a novel augmentation technique based on multiple overlapping window segmentation with 0.2-second time shifts. This effectively multiplies the available training data without introducing synthetic artifacts, preserving the natural characteristics of each user's movement patterns.

Data augmentation via overlapping window segmentation

Figure 2. Data augmentation through overlapping window segmentation. The same accelerometer signal is segmented with a 0.2-second time shift, generating additional training samples from limited user data without introducing synthetic artifacts.

Results

We evaluate on three public HAR benchmarks (Anguita, Shoaib, and Siirtola) and compare against Learn++, a state-of-the-art ensemble-based incremental learning method. Deep learning models substantially outperform the traditional baseline across all personalization strategies.

Average Accuracy Across Three Datasets

Method User-Independent Non-Supervised Semi-Supervised Supervised
Learn++ [baseline] 82.74% 83.64% 96.73% 98.74%
ResNet 94.14% 94.39% 96.86% 99.90%
S-CNN 94.09% 94.16% 97.34% 99.94%

Table 1. Average accuracy (%) across three datasets for each personalization strategy. Bold indicates best result per column. Both deep learning models start from a dramatically higher user-independent baseline.

Key findings:

  • Deep models start from a +11.4 pp higher user-independent accuracy (94.14% vs. 82.74%), providing a fundamentally stronger foundation for personalization.
  • S-CNN reaches 97.34% accuracy with semi-supervised personalization — outperforming Learn++ while requiring no manual labeling effort.
  • With supervised personalization, S-CNN achieves 99.94% accuracy, approaching perfect activity recognition.
  • Deep models require significantly fewer labeled samples to reach equivalent accuracy on the Shoaib dataset, demonstrating faster adaptation to new users.
Learn++ vs S-CNN personalization comparison across three datasets

Figure 3. Personalization curves comparing Learn++ (dashed) and S-CNN (solid) across three datasets. S-CNN converges faster and starts from a substantially higher baseline, requiring fewer labeled samples to reach peak accuracy.

BibTeX

@inproceedings{amrani2020personalized,
  title     = {Personalized Models in Human Activity Recognition using Deep Learning},
  author    = {Amrani, Hamza and Micucci, Daniela and Napoletano, Paolo},
  booktitle = {International Conference on Pattern Recognition (ICPR)},
  year      = {2020},
  organization = {IEEE}
}