Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Recognition

Automated Human Activity Recognition (HAR) has a pivotal role in mobile health, physical activity monitoring, and rehabilitation. Body-worn sensor based HAR can broadly be defined as classification of human physical activity based on signals from multitude of wearable sensors worn at different body locations. Human physical activities include activities of daily living as well as complex activities comprised of multiple simpler micro-activities. Increasing usage of smart handheld devices with multi-modal sensors has paved the way to deploy HAR system in applications of elderly activity monitoring, physiotherapy exercise evaluation, and smart home solutions.

HAR techniques rely on spatial information and temporal dynamics of physical activity captured from heterogeneous sensors placed at different human body locations. Activities involve different dominant body parts and thus hierarchical fusion of sensor signals from different sensor placements is able to capture salient information required for the task. Most human activities can be viewed as a session comprising a number of short time windows containing low level actions. Hierarchical fusion of temporal information is able to take this phenomenon into account. Further, HAR systems in real world scenarios are likely to encounter samples different from training classes. An optimal framework should be able to distinguish them from known classes with explainable visualizations.

Though initial works relied on domain knowledge and heuristic based statistical feature representation [19] for activity recognition, recent end-to-end deep learning models utilize convolutional [8] and recurrent [16] architectures for learning representations. Attention mechanism, described as weighted average of feature representations, is adopted to model HAR task like other sequence modeling problem such as NLP in recent works [15, 29, 12, 31, 13]. However, such approaches do not utilize hierarchical modelling of spatio-temporal information. Moreover, these approaches follow conventional training method under closed set assumption where unknown samples are forced to be recognised as one of the prior known classes.

Considering the aforementioned requirement towards hierarchical fusion of spatio-temporal features, the methodological approach taken in this paper incorporates hierarchical aggregation of sensor signals from different placements across time to construct representation for a specific window. Feature representations from different windows within the same session are aggregated to yield representation for that session. In the case of predicting label for specific window, we utilize session information guided window feature representation instead. The proposed approach models HAR under open set assumption where test samples are identified as seen or unseen and labeled as one of the known classes from training set simultaneously. Such capabilities are highly desirable in the scenario of a subject performing an activity in a completely unexpected way e.g. doing rehabilitation exercise incorrectly due to physical limitations or malfunctioning of sensor devices. In this regard, we have designed autoencoder architecture along with hierarchical encoder to model the distribution of the known activity classes where unseen activities are supposed to yield higher reconstruction loss. Explainable feature attention maps are obtained from hierarchical self-attention layers to demonstrate dominant sensor placement and temporal importance within session to classify specific physical activity.

We conduct extensive experiments on five publicly available benchmark activity recognition datasets: PAMAP2, Opportunity, USC-HAD, Daphnet and Skoda. Our model outperforms prior methods in several datasets. Furthermore, we evaluate the robustness of the model to noise and subject-specific variability through leave-one-subject-out validation experiments. Moreover, we evaluate open-set recognition performance and generate feature attention maps to demonstrate the activity distinguishing characteristics in the learned representation. In brief, the key contributions of our work are listed below:

1. 1.

   Proposed hierarchical self attention encoder models spatial and temporal information of raw sensor signals in learned representations which are used for closed-set classification as well as detection of unseen activity class with decoder part of the autoencoder network in open-set problem definition

2. 2.

   Interpretable visualization of feature attention maps indicate fusion of causal and coherent features for activity recognition

3. 3.

   Our extensive experiments achieve superior performance in several benchmark datasets and demonstrate robustness to subject-specific variability in sensor readings

Wearable Sensor Based HAR The earlier research works on HAR mostly relied on hand-crafted statistical or distribution-based [11, 20] features that have been designed based on domain expertise [9]. However, recent works for wearable-based HAR have mostly focused on end to end deep learning systems for modeling effective feature representation. In that regard, various forms of convolutional, recurrent and hybrid architectures such as [8, 7, 16] were proposed and demonstrated varying levels of success in recognition of the activities under consideration. In recent years, the incorporation of attention mechanism with deep learning-based architectures [15, 29, 31, 12] have demonstrated significant performance improvement. However, most of these works [4, 18] do not rely on hierarchical modeling of spatio-temporal information from wearable sensors.

Self Attention Architecture Recently, self-attention [27] based models have emerged as a popular alternative to recurrent networks for various NLP tasks and has also been proposed for HAR [13]. Using self-attention in a hierarchical manner has been proposed for various tasks such as classifying text documents [6], generating recommendations [10] etc. in order to break up the task into relevant hierarchical parts. However, no such work exists for wearable sensor data to the best of our knowledge though such hierarchy allows for intuitive representation of complex human activities.

Interpretability and Open-Set Recognition The data from wearable sensor devices are usually high dimensional involving different body placements over some time duration. Furthermore, most of the deep-learning based models used for classification of such data offer little to zero interpretability towards the predicted outcome. Some progress has been made in this regard for video based action recognition task [14, 21]. Although attention-based models for wearable HAR offer more interpretability using the attention-scores, it is still scarce in HAR community. With regards to HAR systems, it is often useful to be able to identify previously unseen activities. Class conditioned [17] or variational [26, 2] autoencoder for unseen sample detection has been proposed for image data. Although unseen activity recognition for skeleton data [25] & smart-home environment [1, 5] have been proposed, autoencoder architectures for unseen sample detection in wearable based HAR are very scarce.

Datasets: We use five publicly available benchmark datasets - PAMAP2 [22], Opportunity [23], USC-HAD [30], Daphnet [3] and Skoda [24] for our experiments. A summary of the datasets used is presented in Table 1.

Construction of Activity Window & Session: Activity sessions are constructed using overlapping sliding-window across the temporal axis. Each activity session comprises a number of non-overlapping short activity segments which we refer to as windows.

Open-set Experiment: We randomly hold out the data for a fraction of the classes (22% and 27% of the classes in case of Oppotunity & Skoda, 33.33% for rest) as part of the open-set and include the benchmark test data as defined in Table 1 to construct the test set for evaluation. We train the model with the remaining data and report the cross validation results.

Training and Hyperparameters: We implement the proposed model in Tensorflow and train on eight Tesla K80 GPUs. We use Adam optimizer with learning rate set to $10^{-3}$ with momentum $\beta_{1}=0.9$ and $\beta_{2}=0.999$ and weight decay $\epsilon=10^{-7}$. The number of identical blocks and head for multi-headed self-attention is set to $N=2$ and $n=4$ respectively for our experimental setup. The dropout applied to placement specific encoder block and the size of the representation vector learned from each session is configured to $0.2$ and $d_{model}=64$ respectively.

Evaluation Metric: For the evaluation of activity recognition performance, we use the macro average F1-score defined in (11) as metric where $|C|$ and $i=1,...,C$ in (11) indicate number of classes and the set of classes respectively.

The aim of this work was to design self-attention based model with hierarchical fusion of spatial and temporal features. Our extensive experiments confirmed that hierarchical aggregation leads to better modelling of spatio-temporal dependency in multimodal time-series sensor signal. These findings have significant implications for the understanding of how to construct feature representation that leverages better separability for classification and unseen class detection. The findings reported here lays the groundwork of future research into natural language description generation from multimodal sensor signals. However, publicly available benchmark HAR dataset is limited by the lack of complex activity annotations. Notwithstanding these limitations, the hierarchical self-attention model demonstrates interpretable activity recognition as well as robust feature representation.