P2LHAP:Wearable sensor-based human activity recognition, segmentation and forecast through Patch-to-Label Seq2Seq Transformer

Sensor-based human activity recognition primarily relies on IMU sensors in wearable devices like smartwatches, smart insoles, and exercise bands. Traditional approaches, such as fixed-size sliding windows and dense labeling, have demonstrated strong performance [16]. Recent studies have employed diverse techniques. [17] distinguished static and dynamic activities using statistical techniques, then classified specific activities with random forests (RF) and CNNs. In[18], NAS was used to investigate effective deep learning (DL) architectures for smartphone inference. [19] introduced a lightweight CNN-BiGRU model, leveraging inertial sensor data from smartwatches and smartphones, achieving satisfactory results. However, the inherent uncertainty in activity duration, these methods will lead to multi-class window problems, where multiple activities may coexist within a window, introducing noise and negatively impacting model recognition performance. [20] employed an LSTM-based method to identify fine-grained patterns using high-level features from sequential motion data, [21] introduced a deformable convolutional network for activity recognition from complex sensory data, and [22] presented a GRU-INC model that initializes attention-based GRUs to effectively leverage spatiotemporal information in time series. Additionally, [23] proposed a multi-branch CNN-BiLSTM network that extracts features from raw sensor data with minimal preprocessing. [24] further developed Conditional-UNet, which models conditional dependencies between dense labels for coherent HAR. Current advancements in activity segmentation and recognition, such as [1, 7], leverage dense labeling methods. However, these methods rely solely on individual sensor samples as neural network inputs, and will obtain different representations depending on the task. Our proposed approach employing a patch block, situated between the scales of a large sliding window and a single sample point, for data segmentation. We then utilize a Seq2Seq architecture to unify activity segmentation and recognition tasks, obtaining a unified activity label sequence representation to perform different tasks while mitigating the challenges associated with multi-class windows and over segmentation.

Sensor-based human activity recognition systems have gained popularity with the widespread use of wearable devices in daily life. However, there remains a significant gap in research focusing on sensor-based activity forecasting. Recent studies have introduced attention models to predict multivariate motion signals from IMU sensors, thereby preserving pattern and feature information across multiple channels. Predictive models for these time series data can be utilized for human activity forecast by generating and classifying future activity signals. In [25], a combination of a neural network and Fourier transform was employed to convert the sensor activity data into an image format for predicting future signals. In [26] introduced the SynSigGAN model, which utilizes bidirectional grid long short-term memory as the generator network of the GAN model, and a convolutional neural network as the discriminator network to generate future signals. Furthermore, recent work [27] proposes a Transformer-based GAN that can successfully generate real synthetic time series data sequences of arbitrary length, similar to real data sequences. The GAN model’s generator and discriminator are built using a pure Transformer encoder architecture, which outperforms traditional GAN structures based on RNN or CNN models. Despite these advancements, existing methods predominantly rely on training models to forecast and emulate future sensor activity signals, subsequently employing classification algorithms to derive activity categories from the forecasted signal sequences. Any inaccuracies in the forecasted sensor signals inevitably impact the final classification outcome. In contrast, our proposed P2LHAP architecture bypasses this intermediate step by directly forecasting future activity category token sequences, thereby enhancing forecast accuracy. An additional study [11] presented the PatchTST time series forecast model, which is based on channel independence and patching, and demonstrated promising results. While there has been progress in forecasting multivariate sensor time series data, research on multi-category forecast of human activities based on multi-sensor signal channels remains limited.

Sensor-based multi-task activity recognition presents a significant challenge in the field. [28] employs recurrent neural networks to segment and identify activities and cycles using inertial sensor data. [8] proposes an efficient multi-task learning framework that utilizes commercial smart insoles. This framework addresses three health management-related problems by converting smart insole sensor data into recursive graphs and employing an improved MobileNetV2 as the backbone network. Experimental results demonstrate that this multi-task learning framework outperforms single-task models in tasks such as activity classification, speed estimation, and weight estimation. [29] introduces a multi-task model based on LSTM for predicting each activity and estimating activity intensity using wearable sensor signals. [7] combines the segmentation and classification tasks by passing the sensor signal through the MS-TCN module to predict sample-level labels and activity boundaries. This joint segmentation and classification approach calculates sample-level loss and boundary consistency loss to achieve improved performance. [6] proposes a novel multi-task framework called MTHARS, which leverages the dynamic characteristics of activity length to segment activities of varying durations through multi-scale window splicing. This approach effectively combines activity segmentation and recognition, leading to improved performance in both tasks. However, prior studies employed distinct backbone networks for various tasks or failed to achieve unified task representations, our proposed architecture integrates these tasks within a single model for simultaneous learning and representation. By doing so, we obtain a unified label sequence representation.

The Patch-to-Label Seq2Seq framework we propose is shown in Fig. 2. We define the $M$-dimensional input wearable sensor sequence as $\mathbf{X}_{1:L}=[\mathbf{x}_{1},\cdots,\mathbf{x}_{L}],$ $\mathbf{x}_{t}\in\mathbb{R}^{1\times M}$, where $\mathbf{x}_{t}$ represents the sensor signals collected at timestamp t. The activity label for each sample is denoted as $y_{1:L}=[y_{1},\cdots,y_{L}],$ $y_{t}\in R^{\textit{C}}$, where C represents the number of activity categories, t is the time step, L is the length of the wearable sensor sequence, and the length of the patch block is defined as P. In this study, we do not employ a fixed-length sliding window to segment the sensor data. The number of patches is denoted as L/P. Our approach utilizes a Transformer encoder and decoder as the underlying framework. To simultaneously process the sensor patch of each channel in parallel within the Transformer backbone, it is crucial to duplicate the Transformer weights multiple times and employ the patching operator to create a sensor sequence sample of a size $B\times M\times L$. The patch-level sequence extends to $M\times P\times N$. Reshaping the dimension sequence into a 3D tensor of size $(B\cdot M)\times P\times N$, the batch of samples serves as the input for the encoder. The purpose of the encoder is to map multiple sequences of single-channel sensor data patches into a latent representation. Following this, the decoder is used to derive the representation of the activity sequence, denoted as $\mathbf{Z}^{{}^{\prime}(i)}$. The ultimate output signifies the active label result obtained through the classification head, the active label sequence and the forecast of future activity label sequence of length $T_{pre}$, denoted as $\hat{y}_{pre}=\left\{\hat{y}_{1},\cdots,\hat{y}_{\textit{L}+T_{pre}}\right\}$.

we employ a Flatten layer with a linear head and a Softmax layer to generate the final human activity label sequence $\hat{y}$. The representation of $\hat{y}$ is summarized as follows:

During the training phase, we utilize patch-level label predictions and true labels to construct a patch-level classification loss, aiming to effectively optimize the model parameters. This loss is defined as follows:

Among them, $\hat{y}$ and $y$ represent the patch-level predicted label and the real label, respectively. Given the imbalance of human activity classes under unconstrained conditions, we employ the Softmax CrossEntropy loss function $l_{p}(\cdot)$ and take into account the number of effective samples for classification. The calculation formula is as follows:

Where $L$ signifies the length of input sequence, and C represents the activity class number of dataset. In sensor-based HAR tasks, we leverage this loss function and take into account the number of effective samples to amplify the weight assigned to minority classes. This approach aims to yield fairer and more robust results for each individual class.

Given that the classification loss treats each patch independently, it may result in unwarranted over-segmentation errors. To achieve smoother transitions between patches and minimize over-segmentation errors, we employ the truncated mean square error as our chosen smoothing loss function. The formulation of the smoothing loss is as follows:

$C$ represents the activity category, $T_{c}$ denotes the number of samples belonging to the c-th category, and $\widetilde{y}_{t,c}$ signifies the probability of the activity category c being represented by the patch at time t and $\tau$ is the smoothing threshold($\tau=2$).

For the activity label sequence $\hat{y}$ output by the model, in order to reduce the impact of over-segmentation on subsequent activity merging, we set each activity label as the center, set a smoothing window with a size of smooth_size, and set the starting position to the Half of the activity label sequence within the window. Within the window, we count and sort the activities of each category, and select the activity category with the most surrounding distribution as the current activity label according to the sorting result. After updating the activity label sequence, the performance of activity recognition and segmentation. Finally, it is enhanced by linking Patches of the same activity class to determine the start and end position of each activity in the data flow. We set the boundary set of each activity $B=\{(s_{i},e_{i})\}_{i=1}^{N}$, where $s_{i}$ represents the start subscript of the i-th activity, $e_{i}$ represents the end subscript of the i-th activity, and N is the number of activities in the activity label sequence. The activity set corresponding to each boundary is represented by $\hat{a}=\{a_{i}\}_{i=1}^{N}$, where $a_{i}$ is the activity corresponding to the i-th boundary. The Process of Algorithm is described in the Algorithm 1.

Within the activity forecast module, the continuous input Patch blocks retain one Patch block at the end. This involves filling P repeated values of the last $\mathbf{x}_{i}\in\mathbb{R}$ value at the end of the original sequence. The resulting sequence of generated Patches, $\mathbf{x}_{p}^{(i)}$, is then represented as $\mathbf{z}^{{}^{\prime}(i)}\in\mathbb{R}^{D\times N}$ through the Transformer backbone. By utilizing a linear layer and a fully connected layer with Softmax as the activation function, we obtain the predicted activity sequence, $\hat{y}_{pre}={\hat{y}_{L+1},\cdots,\hat{y}_{L+T_{P}}}\in\mathbb{R}^{1\times T_{P}}$, using the representation of the last patch. It is important to note that $T_{P}$, the length of the future segment, is determined by the number of Patch blocks within the time $T_{pre}$, specifically $T_{p}=\frac{T_{pre}}{P}$.

In the training stage, based on the predicted activity label sequence $\hat{y}_{pre}$ and the ground-truth activity sequence $y_{gt}$, all patch-level prediction losses can be expressed as:

Finally, the previously mentioned loss functions from Equations (6), (8), and (11) are incorporated into the equation, resulting in the final loss function:

We have introduced the main structure of P2LHAP. Algorithm 2 summarizes the training process of P2LHAP.

Our method was implemented using PyTorch and trained on an RTX 3090 GPU. The batch size was set to 64, and we utilized the Adam optimizer with a learning rate of 0.0001. The fully connected dropout rate is set at 20% (0.2), head dropout is fixed at 5% (0.05), the default patch length is 10, with a step size of 10. The model employs 8 heads by default, consists of 6 encoder-decoder layers, and undergoes training for 100 epochs.

For classification, segmentation, and forecast tasks, we employed the entire continuous dataset as input and divided it into training, validation, and test sets with a ratio of 0.7, 0.1, and 0.2 respectively. To evaluate the classification and segmentation performance of the model, we compared the forecast output with the real label. In the forecast task, the forecast patch length was set to 8, and performance assessment was based on a comparison between the forecast output patch label and the real label.

The choice of several parameters values for the models is shown in the table I:

1) Evaluation metrics: For the evaluation of the obtained Patch-level label sequences, we use the following metrics:

1. Activity classification

Accuracy: Accuracy is the overall measure of correctness across all classes and is calculated as follows:

Weighted F1 score: The weighted F1 score is calculated based on the proportion of Patch samples:

Where $prec_{c}=\frac{TP_{c}}{TP_{c}+FP_{c}}$ and $recall_{c}=\frac{TP_{c}}{TP_{c}+FN_{c}}$ represent precision and recall, respectively. The weight $w_{c}$ is calculated as the ratio of $n_{c}$ to $N$, where $n_{c}$ is the number of Patch samples per class, and $N$ is the total number of Patch samples.

2. Activity segmentation

Jaccard Index: The proposed metric assesses the degree of overlap between actual activity segments and predicted activity segments, specifically focusing on the sensitivity towards over-segmentation errors. For the purpose of representation, $G_{i}$ and $P_{i}$ are introduced as respective variables denoting the actual and predicted activity ranges. In this context, the subscript i signifies the ith segment within class C. The calculation of this metric can be determined utilizing the prescribed formula:

3. Activity forecast

Mean Square Error(MSE): MSE serves as a prevalent metric to evaluate the performance of forecast models. It computes the average of the squared differences between predicted values and actual values, thereby providing a quantification of the degree of deviation between the predicted and actual values. A lower MSE signifies a more precise alignment between the model’s predictions and the actual values, indicating superior performance. The formula for calculating MSE can be represented as follows:

To evaluate our proposed method, we adopt three challenging public HAR datasets: WISDM, PAMAP2, UNIMIB SHAR. A brief introduction to the dataset follows.

WISDM Dataset[30]: The data were obtained by 29 participants using phones with triaxial acceleration sensors placed in their trouser pockets with a sampling frequency of 20 Hz. Walking, strolling, walking up stairs, walking down stairs, standing motionless, and standing up were among the six daily activities undertaken by each participant. Fill missing values in a dataset using linear interpolation.

PAMAP2 Dataset[31]: This dataset comprises data collected from nine participants wearing IMUs on their chest, hands, and ankles with a sampling frequency of 100 Hz. These IMUs gather information on acceleration, angular velocity, and magnetic sensors. Each participant completed 12 mandatory activities, including lying down, standing, and going up and down stairs, as well as 6 optional activities, such as watching TV, driving a car, and playing ball.

UNIMIB SHAR Dataset[32]: The dataset was collected from the University of Milano-Bicocca. Thirty volunteers, aged 18 to 60, had their Samsung phones equipped with Bosch BMA 220 sensors with a sampling frequency of 50 Hz. The study recorded 11,771 activities by placing phones in the participants’ left and right pockets. The dataset comprises 17 categories of activities, divided into two groups: 9 activities of daily living (ADL) and 8 falling behaviors. Each activity was repeated 3 to 6 times.

In this section, we compare our method with other competing methods that have been evaluated on three publicly available datasets for wearable-based Human Activity Recognition. To conduct a thorough performance analysis, we use non-overlapping patches of size 10 to compare the overall recognition performance.

We compared the recognition performance of our proposed method with other similar algorithms as mentioned earlier. The results of various competing methods are summarized in Table II. Our method demonstrated improvements in recognition accuracy across three publicly available HAR datasets. Specifically, on the Unimib and PAMAP2 datasets, our method achieved the highest recognition performance, obtaining weighted average F1 scores of 82.04% and 98.92% respectively. Remarkably, our method surpassed the performance of the leading competing method by 3.26% in terms of weighted average F1 score on the Unimib dataset.

In Fig. 3, we present the confusion matrix of our method for activity classification on the PAMAP2 dataset using patch sizes of 200 and 10, respectively. The confusion matrix provides insight into the accuracy of our method for various activity categories. It is evident that, despite the different patch sizes, there is minimal variation in overall accuracy. However, when the patch size is small, over-segmentation occurs, necessitating active segmentation through dividing the patch blocks. This approach enhances the recognition performance for most activity categories.

To demonstrate our proposed method’s segmentation performance, this section presents a visualization of patch-level activity predictions in Fig. 4, with each color representing a different activity category. We compare sequence fragments obtained through model segmentation with selected sequence fragments from the PAMAP2 and WISDM datasets. In practical application scenarios, excessive segmentation can adversely affect prediction accuracy. Nonetheless, as depicted in the figure, our method successfully reduces most over-segmentation errors, leading to more precise and consistent patch-level predicted segments within each active segment.

Table III displays the segmentation performance of our proposed method in comparison with competing algorithms. Our method excels in both segmentation and recognition tasks, consistently outperforming the comparison algorithms. Notably, our Jaccard Index scores on the Pamap2 dataset surpassed those of A&D and JSR by 9.82% and 3.47% respectively, where (w/o) represents the segmentation Jaccard Index scores obtained without using the smoothing strategy. This justifies our employment of a smoothing algorithm, which effectively mitigates the boundary irregularities resulting from excessive segmentation. The algorithm serves to enhance clarity by refining the boundaries and ensures clearer and more distinct segmentation for each individual activity.

Sensor-recorded human activity data forms a continuous data stream, encompassing diverse actions like standing, sitting, walking, and transitions such as stand-to-sit and sit-to-lying. These activities are classified into two categories: basic and transition, based on their duration. Basic activities, characterized by longer periods, can be either dynamic (e.g., walking) or static (e.g., sitting), whereas transitional activities involve brief actions (seconds), like posture shifts (e.g., sit-to-stand). Despite the prevalence of research on basic activities, transition events are frequently overlooked due to their lower frequency and shorter duration compared to basic activities. When considering the short-lived activities that intervene between consecutive basic activities, two distinct scenarios may arise:

• •

  When two consecutive basic activities differ, the activity segments in between are classified as transition activities.

• •

  If adjacent basic activities are identical, the segments in between are designated as the disturbance process of the activity. This occurs due to external influences or human factors disrupting the sensor data, causing irregular fluctuations over time. Consequently, the activities preceding and following the disturbance are considered of the same type.

Given the brief nature of transition activities, our framework efficiently divides the original sensor data into multiple patches, ensuring that the information of these short events can be effectively captured and learned. As demonstrated in Fig. 4(c), the short duration of the Jogging activity does not hinder our framework’s ability to accurately segment it, showcasing its effectiveness.

In this section, we assess the performance of the activity forecast module using our model to forecast the duration of activity categories in the next 8 patch blocks. Fig. 5 displays the confusion matrix for the future activity forecast in the pamap2 dataset. Fig. 5 (a),(b),(c) present the prediction confusion matrices of Conv2LSTM[44], Seq2Seq-LSTM[45], and Seq2Seq-LSTM-PE-MA[46] predictors for five activity categories in the dataset, while Fig. (d) illustrates the predictions of our proposed model. The confusion matrix results for all campaigns indicate an average accuracy of 96.59%. Additionally, we compare the predicted outputs of future sensor signals with the ground truth signals by evaluating their mean squared error (MSE). To enable the model to predict future sensor signals, we exclude the last classification header and utilize the first half of the input sensor sequence for model input, reserving the second half to match the size of the predicted output data. Table IV presents the performance comparison of our model with five forecasters for sensor sequence signal prediction.

In this section, we conduct two ablation experiments to investigate the influence of varying patch block sizes on model performance and the efficacy of patching operations in generating label sequences.

Impact of Patch Size: various sizes and quantities of patches can yield different outcomes. Hence, we employ a scaling approach with multiple patches to analyze their performance. We provide the comprehensive results in Table V. We observed a decrease in accuracy when the Patch size is reduced significantly. This decline can be attributed to the issue of excessive segmentation. However, when the Patch size is set to 10, it consistently exhibits exceptional recognition performance across all three datasets.

The impact of Patching modules: we examined the effects on the results by not patching the sensor data stream, but instead using the Transformer backbone directly. The F1 score obtained on the benchmark dataset is presented in Table VI. It was observed that when the data was not patched during processing, the accuracy of the experimental results deteriorated. Hence, applying data patching enhanced the performance of activity recognition.

The impact of the number of multiple heads and the number of encoder-decoder layers:this study investigates the influence of model components on classification outcomes by varying the number of multi-heads and encoder-decoder layers. Table VII presents the activity recognition accuracy of our proposed framework on the PAMAP2 dataset, showcasing the performance changes with different configurations. The results indicate that an increase in both multi-heads count and model layer depth facilitates better learning of activity sequence representations, leading to enhanced recognition accuracy. The optimal configuration is achieved when the model has 6 layers and 8 multi-heads, achieving the highest accuracy.

To thoroughly assess the practicality of our proposed methodology, we will examine its computational complexity and parameter count, aiming to determine its feasibility for real-time deployment on mobile or embedded devices. The computational analysis was conducted on a Linux platform equipped with an RTX 3090 GPU and an Intel Core i9-13900K. The experiments employed a batch size of 32 for both training and testing, with 3 encoder and decoder layers, and 4 multi-heads. Utilizing the PAMAP2 dataset, we set the input sequence length to 512, patch size to 24, and step size to 12.

The computational complexity comparison, as presented in the table VIII, reveals that our method exhibits superior parameter efficiency and lower inference time compared to the benchmarked techniques.