Weakly Supervised Online Action Detection for Infant General Movements

Human pose keypoints named skeleton is usually denoted as $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where nodes set $\mathcal{V}=\{v_{i}|1\leq i\leq N\}$ represents joints, and edges set $\mathcal{E}=\{e_{i}|1\leq i\leq E\}$ represents connectivity between joints. Formally, we use adjacency matrix $\mathbf{A}\in\mathbb{R}^{N\times N}$ to denote the edges set where $A_{i,j}=1$ if there is an edge between $v_{i}$ and $v_{j}$, otherwise 0. Assume that we are given a skeleton sequence $\mathcal{X}=\{\mathbf{x}_{t,n}\in\mathbb{R}^{C}|t,n\in\mathbb{Z},1\leq t\leq T,1\leq n\leq N\}$, where $T$ is the sequence length, and $C$ denotes the feature vector dimension for one joint. Then the feature of this sequence can be written as $\mathbf{X}\in\mathbb{R}^{T\times N\times C}$.

As detailed in section 1, FMs have complex movement patterns. To well modeling the local spatio-temporal information, we need to capture the relation between different joints in one frame and the variations of joints over time. Multi-scale graph convolution is often used to fuse long-range relations in one graph [14]. MS-G3D [15] utilized cross-spacetime skip connections to construct a spatio-temporal subgraph and proposed a disentangled multi-scale graph convolution to model human action dynamics. We extend the feature extractor in MS-G3D with vertices fusing to capture the local spatio-temporal information of FMs.

In detail, we first split the pose sequence $\mathcal{X}$ into clip-level set $\{C_{1},C2,\cdots,C_{L}\}$ with a sliding window of temporal size $\tau$ and stride size $s$, where $C_{1}\cup C_{2}\cup\cdots\cup C_{L}=\mathcal{X}$. Then a spatio-temporal graph $(\mathcal{V}_{\tau},\mathcal{E}_{\tau})$ is constructed by tiling $\mathbf{A}$ into a block adjacency matrix $\mathbf{A}_{\tau}\in\mathbb{R}^{\tau N\times\tau N}$. A multi-scale graph convolution is applied to each clip feature tensor $\mathbf{X}^{in}_{(\tau)}\in\mathbb{R}^{1\times\tau N\times C_{in}}$ as

where $\mathbf{X}^{out}_{(\tau)}\in\mathbb{R}^{1\times\tau N\times C_{out}}$ is output feature, $\sigma$ is activation function, $M\in\mathbf{R}$ is the number of scales to aggregate. $\tilde{\mathbf{D}}_{(\tau,m)}$ is the corresponding diagonal degree matrix of disentangled $\tilde{\mathbf{A}}_{(\tau,m)}$. $\mathbf{\Theta}_{(m)}\in\mathbb{R}^{C_{in}\times C_{out}}$ denotes the learnable parameter matrix.

For each clip, $\mathbf{X}^{out}_{(\tau)}$ will be collapsed into one skeleton by a 3D convolution operator with kernel size $(1,\tau,1)$ to get a feature tensor $\mathbf{X}^{out}\in\mathbb{R}^{1\times N\times C_{out}}$. Then each joint in $\mathbf{X}^{out}$ has got rich information from their neighborhood. To extract more complex local spatio-temporal fused feature, we will aggregate joints information as $\mathbf{f}_{i}=\mbox{agg}(\mathbf{X}_{i}^{out})$, where subscript $i$ denotes the index of clip. We use 2D convolution as aggregator followed with ReLU in this paper. Then we get a feature sequences as $[\mathbf{f}_{1},\mathbf{f}_{2},\cdots,\mathbf{f}_{L}]$. All the clips share the same parameter in this module.

For inference of weakly supervised online action detection, only accumulated history information is available. We introduce a branch to generate pseudo labels with future information and long-range information during training. Since infants’ FMs are continual and hard to distinguish from other mixed movements [5], combination future information with long time receptive field is helpful for better classification and detection.

Take the features $\mathbf{F}^{0}=[\mathbf{f}_{1},\mathbf{f}_{2},\cdots,\mathbf{f}_{L}]\in\mathbb{R}^{L\times C_{out}}$ extracted from LFEM as input, we use the 1D temporal convolutions followed by ReLU to aggregate long-range temporal information from neighbour clips features, i.e. $\mathbf{F}^{i+1}=\mbox{Relu}(\mbox{Conv1D}(\mathbf{F}^{i}))$. The output feature of the last 1D convolution layer is $\mathbf{F}_{out}$. Then for each clip, fully connected layers are used to get the action scores $\mathbf{S}=[\mathbf{s}_{1},\mathbf{s}_{2},\cdots,\mathbf{s}_{L}]$, where $\mathbf{s}_{i}\in\mathbb{R}^{n_{c}}$ is the score of $n_{c}$ actions of clip $i$.

Multiple Instance Learning Loss(MILL) is widely used to get accurate actions scores [19, 7]. In this paper, we consider the entire video clip-level sequence set $\{C_{1},C2,\cdots,C_{L}\}$ as a bag of instances, where each instance denotes one clip. To use the video-level labels, we compute video-level scores with Top-K strategy for each action class. That is $s^{c}=\frac{1}{K}\sum_{j\in\mathcal{M}}s^{c}_{j}$, where $\mathcal{M}$ is the Top-K indices set of clips over class $c$, and $K$ is got by $\max(1,\left\lfloor\frac{L}{\kappa}\right\rfloor)$ with hyper parameter $\kappa$. Then, to obtain the action class probabilities $\mathcal{P}$, sigmoid or softmax (multi-class dataset) is applied to the video-level scores. The MILL is computed as

where $y_{c}$ and $p_{c}$ is the ground truth label and prediction probability of class $c$. Supervised by $\mathcal{L}_{MIL}^{p}$, the network can learn clip-level scores which will be used to generate clip-level pseudo labels with future information by a two-stage threshold strategy [7]. First, action class will be discarded if video-level score is less than a threshold $\theta_{class}$. Then, for the remaining action classes, a second threshold, $\theta_{score}$, is applied on clip-level action scores $\mathbf{S}$ to get pseudo labels $L_{P}$. After that, the video-level ground truth labels are used to filter out wrong pseudo labels.

As shown above, CPGB can utilize future information during training. However, future information is not available during inference. LSTM is used to accumulate the historical temporal information in this branch.

Given one clip-level feature $\mathbf{f}_{i}$, previous hidden state $\mathbf{h}_{i-1}$ and cell state $\mathbf{c}_{i-1}$ as input, updated states is got by $\mathbf{h}_{i},\mathbf{c}_{i}=\mbox{LSTM}(\mathbf{h}_{i-1},\mathbf{c}_{i-1},\mathbf{f}_{i})$. For $i\mbox{th}$ clip, online action score is computed as $\mathbf{a}_{i}=\operatorname{softmax}\left(\mathbf{W}_{a}^{\top}\mathbf{h}_{i}\right)$, where $\mathbf{a}_{i}\in\mathbb{R}^{n_{c}+1}$ is the action scores including a background class. Then, cross entropy loss is applied to $\mathbf{a}_{i}$ over all clips with pseudo labels $L_{P}$ to get frame loss, i.e.

To further utilize the ground truth video level information, another MILL is used in this branch. As shown in Figure 1, we use the same top-K strategy as CPGB to get video-level scores for each action class. Here, $\mathcal{L}_{MIL}^{o}$ is used to denote MILL in this branch.

In the training stage, two branches and the local feature extraction module are jointly optimized by $\mathcal{L}_{MIL}^{p}+\mathcal{L}_{FML}+\mathcal{L}_{MIL}^{o}$. During inference, the future information is not available for online action detection tasks, so we only use the feature extraction module and online action modeling branch.

Video-level classification performance. For our model, one video is classified as F+ if the video level score got by Top-K strategy is greater than $0.5$ in the seen video. For STAM [18], MS-G3D [15], Zhu et.al[27], the classification threshold is set to 0.5. Since action detection tasks also include classification, we also report the video-level performance of WOAD [7] and W-TALC [19]. Following previous appearance-based methods [7, 25], the image features and optical features are extracted by I3D [2] pre-trained on Kinetics [12], a huge dataset contains 306k videos. For a fair comparison, none-skeleton based methods also use the first 6000 frames. Both the clip window size and window stride are 20. Other experiments settings are the same as original papers.

Results are shown in the left part of Table 1. For skeleton-based methods, we report the 5-fold cross-validation results. For other methods, since the results perform much worse than skeleton-based methods and calculating image and optical flow features requires a lot of computing power, we only report results of the fifth fold with frame-level annotations. Though only accumulated history information is used, video-level results of our model outperform previous works by a lot margin, demonstrating the superiority of WO-GMA. Moreover, for both WOAD[7] and W-TALC [19], the model with the image input feature performs worse than the optical flow input feature. This result indicates that motion information is more important than appearance information in this task.

Online action detection performance. To the best of our knowledge, we are the first to develop weakly supervised online action detection method for GMA. 60 F+ samples in the fifth dataset split-fold were annotated by experts with frame-level labels, and 56 valid samples are used as ground truth to report detection results. The right part of Table 1 shows that our model achieves the best detection performance. Compared with the image features, the detection results using the optical flow features are better, which further demonstrates the importance of motion information. Since the skeleton contains only motion information, this result also proves the necessity of using the skeleton as input. The performance gap between left part and right part of Table 1 shows that detection is much more difficult than classification in this task. There are two main possible reasons. First, only video-level supervision information may not enough. Second, compared with everyday life actions like shaking hands, the boundary of FMs is even harder to determine for annotators accurately. Figure 3 demonstrates that our model can get video-level performance as good as fully observed when only the first 20% video frames are observed. This result shows a significant benefit of our model: the assessment time of automated GMA for real-world applications can be greatly shortened. The visualisation detection results in the top subplot of Figure 3 shows the acceptable detection performance.

Since the complex movements pattern of FMs, we argue that both local spatio-temporal information and long-range information are critical in this task. This part will analyze the effect each component in the fifth dataset split-fold with frame-level annotations.

To study the effect of CPGB which can combine future information, we remove this branch(w/o pseudo). As shown in Table 2, both the classification and detection performance will drop compared with WO-GMA, demonstrating the necessity to generate pseudo clip labels for training. We replace the LFEM with only clip skeletons vertices features concatenation to get results without complex local features(w/o local). As shown in the second row of Table 2, the video level accuracy will drop by $2\%$, and the detection results will drop when IoU is higher. Furthermore, to analyze the effect of long-range information, we remove the 1D convolutions used to capture long-range information in CPGB(w/o long-range). Results in Table 2 imply the importance of long-range information.

To better illustrate the influence of different modules, we plot three more curves of the same infant in Fig.3 and report the number of detection instances in Table 2. These results show that the detection action instances are fragmented without long-range information, which is unsuitable for detecting continuous FMs. Without local feature extraction, the detection score is less confident than WO-GMA. Without pseudo, the model may ignore the gap between intermittent FMs, which the long-range information in CPGB will make up. Moreover, generating pseudo labels without long-range information will bring the noise. The detection results further show the difficulty of detection task mentioned in previous subsection. Compared with appearance-based methods, our skeleton-based methods achieve better performance.