Challenges in Video-Based Infant Action Recognition:
A Critical Examination of the State of the Art

We present a new dataset called InfActPrimitive as a benchmark to evaluate infant action recognition models. Videos in InfActPrimitive are provided from two sources. (1) Videos submitted by recruited participants: We collected infant videos using a baby monitor from their home and in an unscripted manner. The experiment was approved by the Committee on the Use of Humans as Experimental Subjects of Northeastern university (IRB number:22-11-32). Participants provided informed written consent before the experiment and were compensated for their time. (2) Videos gathered from public video-sharing platforms. This portion of video clips in our dataset has been adapted from [9], which was acquired by performing searches for public videos on the YouTube platform. InfActPrimitive contains $814$ infant action videos of five basic motor primitives representing specific postures such as sitting, standing, prone, supine, and all four. The start and end time of every motor primitive is meticulously annotated in this dataset. The InfActPrimitive, with its motor primitives defined by the Alberta Infant Motor Scale (AIMS) as significant milestones, is ideal for developing and testing models for infant action recognition, milestone tracking, and detection of complex actions. Figure 1 shows the screenshots from various videos within the InfActPrimitive dataset, illustrating the diversity of pose, posture, and action among the samples. The diverse range of infant ages and a wide variety of movements and postures within the InfActPrimitive dataset pose significant challenges for action recognition tasks. The right side of the panel in Figure 1 shows the statistical analysis of InfActPrimitive for each sources of data separately.

Infant specific prepossessing, skeleton data prediction, and action recognition are the key components of our pipeline, as shown in Figure 2. To achieve this, input frames are processed through the pipeline’s components, enabling infant-specific skeleton data generation and alignment as input to the different state-of-the-art action recognition models.

Preprocessing– Input video $V$ is represented as sequence of $T$ frames, $V=\left(f^{1},\ldots,f^{t},\ldots,f^{T}\right)$. We customized the YOLOv7 [25] to locate the bounding box around the infants at every frame as a region of interest. We then extracted either a 2D or 3D infant skeleton pose prediction $x^{t}\in\mathbb{R}^{J\times D}$, where $J=17$ is the number of skeleton joints (corresponding to the shoulders, elbows, wrists, hips, knees, and ankles), and $D\in\left\{2,3\right\}$ is spatial dimension of the coordinates. The underlying pose estimators—the fine-tuned domain-adapted infant pose (FiDIP) model [7] for 2D and the heuristic weakly supervised 3D human pose estimation infant (HW-HuP-Infant) model [17] for 3D were specifically adapted for the infant domain.

Infant-adult skeleton alignment– One of the major challenges in the domain of skeleton-based action recognition lies in the significant variability of skeleton layouts across different datasets and scenarios. The diversity in joint definitions, proportions, scaling, and pose configurations across these layouts introduces complexity that directly impacts the efficacy of action recognition algorithms and makes transferring knowledge between two different datasets inefficient. The challenge of reconciling these layout differences and enabling robust recognition of actions regardless of skeletal variations is a critical concern in our studies.

As shown in Figure 3, NTU RGB+D indicates the location of 25 joints in a 3D space. The layout of the infants 3D skeletons in the InfActPrimitive on the other hand, is based on the Human3.6M skeleton structure, which supports a total of 17 joints. To match the number of keypoints and align the skeleton data in these two datasets, We only select a subset of joins of NTU RGB+D skeleton that are common with the Human3.6M layout. We also reordered these joints, so the structures became as similar as possible. For the 2D skeletons, layouts of both NTU RGB+D and InfActPrimitive are based on the COCO structure.

Action recognition– After preprocessing, we fed the extracted sequence of body keypoints from the input video into various state-of-the-art skeleton-based action recognition models leveraging different aspects of infant-specific pose representations. We categorize these skeleton-based models into three groups: CNN-based, graph-based, and RNN-based models to fully exploit the information encoded in the pose data and perform a comprehensive comparative analysis of the results.

• •

  Recurrent neural network structures capture the long-term temporal correlation of spatial features in the skeleton. We applied the part-aware LSTM (P-LSTM)[22] to segment body joints into five part groups and used independent streams of LSTMs to handle each part. At each timeframe $t$, the input $x^{t}$ is broken into $\left(x^{t}_{1},\dots,x^{t}_{P}\right)$ parts, corresponding to $P$ parts of the body. These inputs are fed into $P$ streams of LSTM modules, where each LSTM has its own individual input, forget, and modulation gates. However, the output gate of these streams will be concatenated and will be shared among the body parts and their corresponding LSTM streams.

• •

  Graph convolutional networks (GCNs) represent skeletal data as a graph structure, with joints as nodes and connections as edges. To capture temporal relationships, we applied ST-GCN, which considers inter-frame connections between the same joints in consecutive frames. Furthermore, we employed InfoGCN [2], which integrates a spatial attention mechanism to understand context-dependent joint topology, enhancing the existing skeleton structure. InfoGCN utilizes an encoder with graph convolutions and attention mechanisms to infer class-specific characteristics. $\mu_{c}$ and diagonal covariance matrix of a multivariate Gaussian distribution $\sigma_{c}$. With an auxiliary independent random noise $\epsilon\sim N(0,I)$, $Z$ is sampled as $Z=\mu_{c}+\Sigma_{c}\epsilon$. The decoder block of the model, composed of a single linear layer and a softmax function, converts the latent vector $Z$ to the categorical distribution.

• •

  3D convolutional networks are mainly employed in RGB-based action recognition tasks to capture both spatial and temporal features across consecutive frames. To utilize the capabilities of a CNN-based framework, We first convert keypoints in each frame into heatmaps. These heatmaps were generated by creating Gaussian maps centered at each joint within the frame. Subsequently, we applied the PoseC3D [4] method, which involved stacking these heatmaps along the temporal dimension, enabling 3D-CNNs to effectively handle skeleton-based action detection. Lastly, the representations extracted from each input sequence using the 3D convolutional layer were fed into a classifier. This classifier consists of a single linear layer followed by a softmax function, ultimately yielding the final class distribution.

NTU RGB+D[22] is a large-scale action recognition dataset with both RGB frames and 3D skeletons. This dataset contains 56,000 samples across 60 action classes. Video samples have been captured by three Microsoft Kinect V2 camera sensors concurrently. 3D skeletal data contains the 3D locations of 25 major body joints at each frame. HRNet is used to estimate the 2D pose, which results in the coordination of 17 joints in the 2D space. Given that each video in this dataset features a minimum of two subjects, our approach involves evaluating the models within a cross-subject setting. In this particular setup, the models are trained using samples drawn from a designated subset of actors, while the subsequent evaluation is carried out on samples featuring actors who were not part of the training process. We have employed a train-test split paradigm that mirrors the methodology outlined in [22]. Specifically, we partition the initial cohort of 40 subjects into distinct training and testing groups, with each group composed of 20 subjects. In the context of this evaluative exercise, both the training and testing sets encompass a substantial number of samples, totaling 40, 320, and 16,560, respectively. It is noteworthy to mention that the training subjects for this particular evaluation bear the following identification numbers: 1, 2, 4, 5, 8, 9, 13, 14, 15, 16, 17, 18, 19, 25, 27, 28, 31, 34, 35, and 38. The remaining subjects have been thoughtfully reserved for the purpose of conducting rigorous testing.

InfActPrimitive, as detailed in subsection 3.1, combines video clips from two primary sources: data collected from the YouTube platform and data acquired through our independent data collection efforts. To evaluate our pipeline’s performance on this dataset, the training set comprises all videos collected from YouTube, totaling 116 (sitting), 79 (standing), 62 (supine), 74 (prone), and 69 (all-fours) actions. Similarly, the test set consists exclusively of videos from our independently collected data, including 171 clips for sitting, 58 clips for standing, 62 clips for supine, 185 clips for prone, and 92 clips for all fours. This partitioning strategy enables us to assess the pipeline’s ability to generalize across previously unobserved data and diverse sources, ensuring a comprehensive representation of various actions in both the training and test sets. This approach enhances the robustness of our evaluation by encompassing a wide range of settings and conditions found in YouTube videos and our collected data.

In this section, we detail the series of experiments conducted using our infant action recognition pipeline. We will also provide a comparative analysis, examining the outcomes in relation to the adult skeleton data.

Baseline experiment– In our baseline experiment, we trained various action recognition models, as detailed in subsection 3.2, separately on both the NTU RGB+D and InfActPrimitive datasets from scratch. With the exception of PoseC3D, all these models established baseline performance levels for both 2D and 3D-based action recognition tasks across both adult and infant domains. This baseline performance provides a starting point against which the performance of future experiments, such as fine-tuning or incorporating domain-specific knowledge, can be compared. We set the hyperparameter for ST-GCN, InfoGCN, deepLSTM and PoseC3D models a exactly as they were specified in [29],[2], and [4]. In Table 1, the first pair of columns illustrate the experimental findings with 2D skeleton sequences from both the NTU RGB+D and InfActPrimitive datasets, respectively. Simultaneously, the fourth and fifth columns present the results in the context of 3D data. As demonstrated, PoseC3D consistently outperforms other models in both adult and infant action recognition domains. Nevertheless, a significant performance gap persists between infant and adult action recognition, which can be attributed to disparities in sample size and class distribution. The adult model benefits from a more abundant dataset, enabling it to effectively capture the spatiotemporal nuances of various actions, a characteristic that the InfActPrimitive dataset lacks.

Figure 4 displays the confusion matrices for PoseC3D, InfoGCN, and ST-GCN methods. As illustrated, the sequences associated with the ”Sitting” action class exhibit superior separability compared to other classes. However, it is evident that the InfoGCN model miserably fails in the infant action recognition

Transfer learning experiment– To utilize the knowledge embedded in the adult action recognition, we initialized the model weights using the learned parameters obtained from prior training on the NTU RGB+D dataset. To address the substantial class disparities between the two datasets, we excluded the classifier weights, and for this experiment, initialized them randomly.

Given the significant disparity in the number of classes between the two datasets and the substantial impact of training set size on model performance, we chose to delve deeper into the implications of this experimental parameter. Notably, limited data availability posed challenges to achieving high accuracy in models trained on InfActPrimitive. To determine whether this issue extended beyond the domain of infant action recognition, we made modifications to the training subset of NTU RGB+D. Specifically, we curated a subset comprising only five action classes, namely, ’sit down,’ ’stand up,’ ’falling down,’ ’jump on,’ and ’drop,’ which closely matched those in InfActPrimitive. We then restricted the number of samples per class in this subset to align with the size of the InfActPrimitive training subset. The validation samples for these selected classes remained unchanged.

As shown in Figure 5, the latent variables demonstrate a significantly greater degree of separability within the adult domain compared to the infant domain. This finding highlights the potential limitations of models pretrained on infants in capturing the underlying patterns specific to the infant domain. The disparity can be attributed to the substantial differences between the adult and infant domains, emphasizing the necessity for domain-specific model adaptations or training approaches.

Intra-class data diversity experiment– In our final experiment, we investigate the impact of intra-class diversity on action recognition model performance. We hypothesize that the absence of structural coherence and the inherent variations among samples from the same class can significantly reduce validation accuracy. While traditional action recognition datasets like NTU RGB+D are known for rigid action instructions and minimal intra-class variation, our InfActPrimitive dataset, derived from in-the-wild videos, exhibits a higher level of variability in performed actions. To test this hypothesis, we conducted cross-validation training, dividing our training dataset into five subsets and training on four while validating on the fifth. The original validation set of InfActPrimitive was used for testing. Given the superior results achieved with the PoseC3D model using 2D skeleton data, we considered this model as an infant action recognition model. Our findings, presented in Table 2, shed light on the influence of intra-class diversity on action recognition model performance.

As shown in Table 2, although each experiment yields high training accuracy, there are substantial variations in validation and testing accuracies across experiments. These outcomes reveal discrepancies in the training datasets, leading to inconsistent learning, and underscore distinctions between videos collected from diverse sources.