Part-level Action Parsing via a Pose-guided Coarse-to-Fine Framework

Figure 2 shows the overall structure of the Pose-guided Coarse-to-Fine (PCF) framework for the PAP task. It includes three stages, i.e., instance and part detection, video-level action recognition, and part action parsing. In the first stage, as shown in the upper part of Figure 2, we adopt YOLOF [21] as the backbone of the person detector and part detector to locate each person and body parts. To overcome the ambiguity of body parts, we insert the pose estimator and the positional embedding module between the person detector and part detector to improve the accuracy of the part location. The second stage is shown in the lower part of Figure 2. Based on the short-term persistence of human actions, we exploit segment-level action prediction to approximate the frame-level action state to balance the accuracy and computation cost. In particular, we divide the original video into multiple segments that last three seconds or so. Then tag each segment with six segment-level pseudo action labels based on the original frame-level part action labels which significantly reduce the computation cost for frame-level action parsing. After that, we train models for segment-level action and video-level action respectively. In the final stage, we integrate the output of all previous stages to get the final output for the PAP task. Next, we will introduce each stage of our framework in detail.

To our knowledge, body part detection is an unprecedented task in traditional object detection tasks. Different from general object detection, body part detection needs to overcome the ambiguity of body parts and model the prior of human body structure. For example, normally people only have one left foot and one right foot, but the local features of the left one and the right one are often very similar. Fortunately, this structural prior is very common in human pose estimation and has been widely explored. To maximize the usage of this structural prior, we propose a pose-guided part detection method, which is shown in the top half of Figure 2.

In detail, the person detector $F_{person}(\cdot)$ first extracts the bounding box of a person $X_{p}$ from the input frame $X_{f}$ by

Then the pose estimator $F_{pose}(\cdot)$ takes $X_{p}$ as the input and outputs the keypoints $K_{p}=\{(x_{i},y_{i})\}^{N}_{i=1}$ of the person, which is formulated by

where $(x_{i},y_{i})$ is the coordinates of keypoint $i$, $N$ is the number of keypoints. After that, the keypoints $K_{p}$ are integrated by the positional embedding module $G$ with dots of different colors and radius on the original $X_{p}$ to generate an augmented person image $X^{\prime}_{p}$. This process can be formulated by

By this means, we can increase the appearance difference between different body parts and facilitate the learning of body parts detector $F_{body}$. Finally, the part detector $F_{body}$ is implemented to localize each body part $Y_{part}$ by

In addition, we also fine-tune the person detection box with the results of the pose estimator. In a nutshell, the pose estimator has the ability to predict the possible human keypoints outside the person box, and we fine-tune the detected person box until all possible human keypoints are included.

Fine-grained frame-level part state parsing requires more computation and hardware cost than coarse video-level action recognition. However, we find this frame-level action parsing problem can be transformed into a simpler segment-level action recognition task due to the overwhelming “Long Tail Effect” caused by the short-term persistence of actions in video segments. For example, in the segment of “hurling sport” that lasts about three seconds, we just need to predict “None” for the heads in every frame and easily achieve 97.7% frame-level part state accuracy. To take advantage of the significant “Long Tail Effect”, as shown in the bottom half of Figure 2, we tag each video segment with six part-level pseudo labels based on the original frame-level action state label. The fine-grained part-level label consists of three parts: coarse video-level action, body part, and the most frequent frame-level action of the body part. As the example shown in Figure 2, “(hurling_sport) head: none” means the video-level action of this video is “hurling_sport”, and the most frequently frame-level action of the “head” in this segment is “none”. Through this transformation, we can directly apply individual action recognition networks without sharing parameters, such as CSN [19]. By this means, we can predict each fine-grained segment-level label and coarse video-level label respectively without any other models related to the computation-intensive frame-level action prediction.

Dataset. The experiments are performed on the Kinetics-TPS dataset [20] that provides 7.9 M annotations of 10 body parts, 7.9 M part state (i.e., how a body part moves) of 74 body actions, and 0.5 M interactive objects of 75 objects in the video frames of 24 human action classes. Kinetics-TPS contains 3,809 training videos (4.96 GB in size) and 932 test videos (1.26 GB in size). It’s worth noting the source videos of Kinetics-TPS come from Kinetics 700. Hence, all the Kinetics-pretrained models are forbidden in the PAP task.

Evaluation Metrics. We adopt the official evaluation metric, i.e., ROC score, of the Kinetics-TPS dataset [20]. ROC scores are calculated based on the Part State Correctness (PSC) and the action recognition conditioned on PSC. The PSC calculates the accuracy of the whole human detection results and body part action parsing in each frame. The action recognition conditioned on PSC draws the ROC curve and calculates the ROC score according to the top-1 video-level action recognition accuracy and PSC accuracy. Please refer to [23] for more details.

Details of Detector and Pose Estimator. For person detector and part detector on keyframes, we adopted the YOLOF [21], which is an anchor-free model with a ResNet-101 [7] backbone. The model is pre-trained on the COCO dataset [24] and then fine-tuned on Kinetics-TPS. The final models obtain 93.8 AP@50 in the person category and 79.7 AP@50 in the 10 body part categories on the Kinetics-TPS validation set. For the pose estimator, we directly adopted the HRNet-w48 [25] pre-trained on COCO [24] to extract the keypoints of each person without any fine-tuning.

Details of Action Parsing and Action Recognition Network. We use the CSN networks [19] as the backbone in our action recognition and action parsing framework. We use the ip-CSN-152 implementation pre-trained on the IG-65M [26] dataset with input sampling $T\times\tau=32\times 2$. In particular, we freeze the Batch Normalization (BN) layers in the backbone during fine-tuning on Kinetics-TPS.

Details of Training. The detection model and action recognition models are trained separately. Each model is trained in an end-to-end manner. In detail, we train the YOLOF detector using SGD with a mini-batch size of six on four V100 GPUs and train it for 24 epochs with a base learning rate of 0.01, which is decreased by a factor of 10 at epoch 16 and 22. We perform linear warm-up [27] during the first 1800 iterations. For the CSN model, we train it using SGD with a mini-batch size of four on four V100 GPUs for 58 epochs with a base learning rate of 8e-5, which is decreased by a factor of 10 at epoch 32 and 48. We perform linear warm-up [27] during the first 16 iterations. By default, we use weight decay of 1e-4 and Nesterov momentum of 0.9 for all models.

Details of Inference. Following the official guideline [23], we extract the top-10 results from the person detector and the top-1 results of each body part from the part detector during testing. For the action recognition task and action state parsing task, we set the number of sampling clips as seven for each video segment at test time and scale the shorter side of input frames to 256 pixels.

To demonstrate the effectiveness of our method, We compare our PCF framework with the official baseline method. Meanwhile, to illustrate the fairness of comparison, we replace our ip-CSN-152 backbone with the TSN-Res50 used by the official baseline.

We present our results on Kinetics-TPS in Table I. The ”Input” in the second column refers to the video input form, and the calculation of optical flow is based on the tvl1 algorithm [28]. The “video acc” in the fourth column refers to the top-1 video-level action recognition accuracy, while the “ROC score” in the fifth column refers to the final ROC score of the methods.

From the results, we can first find that directly applying our PCF framework with TSN-Res50 backbone and RGB input form, our performance achieves a significant enhancement of +19.44 ROC score. Out of our expectation simply changing the input mode from RGB to optical flow gives a total boost of +5.1 ROC score improvement. This may indicate that the body part action encoded by optical flow carries more effective information than RGB input when using 2D-CNN based network in the PAP task. Furthermore, with the strong CNN-based model ip-CSN-152 pretrained on IG-65M, our PCF framework achieves the 60.89% ROC score on the Kinetics-TPS dataset.

Effect of Adding Pose Estimator. We investigate the effect of the pose estimator on detection mAP. For person detector and part detector, we train the lightweight CNN-based model YOLOF with human location and body parts location respectively. As shown in Table II, adding the pose estimator brings consistent AP and AP@50 increases for these two models. More specifically, equipped with the pose estimator, our $YOLOF_{part}$ model achieves a significant enhancement of $+26.6$ AP@50 on the Kinetics-TPS dataset.

Frame-level Prediction v.s. Segment-level Prediction In this subsection, we quantitatively compare the action prediction accuracy and the computation cost between the frame-level action parsing and the segment-level action parsing in Table III. The action prediction accuracy and the “TFLOPs” are calculated with the ip-CSN-152 model on the Kinetics-TPS dataset. From the results, we can see that the computation “TFLOPs” decreases greatly with the increase of segment duration, while the loss of the accuracy is just 0.4% or less. Especially when the duration of the segment is less than three seconds, the accuracy of frame-level prediction and segment-level prediction is almost the same (decrease less than 0.1%), while the computation decreases about 68.16%.