Temporal Fusion Network for Temporal Action Localization:
Submission to ActivityNet Challenge 2020 (Task E)

To improve the performance of video classification, we try to encode more temporal and spatial information. The Slowfast [6] achieves excellent performance on video classification by decoupling spatial and temporal information in the temporal-spatial space. Specifically, we choose Slowfast101 as our backbone, and each input clip has 32 frames by 15fps. We first pre-train our backbone on Kinetics-600 [3] and then fine-tune it on the HACS dataset [13]. The output of the network is a prediction of 200 categories. Because there is no background class, we only sample frames in segments that involve with action in training. Finally, we add Batch Nuclear-norm [5] to the loss function to improve the generalization capabilities.

Features. The fine-tuned model has a better representation of the HACS dataset. So we use the fine-tuned classification model’s backbone to extract features for all videos. To encode more temporal information, we apply a dense sample strategy. Specifically, we sample 32 frames with a 0.5s stride, and resize the short side of the clip to 256 pixels, and input the backbone to extract discriminative features. Then we take the features of stage “Res5” in Slowfast and use global average pooling to get a 2034-dimensional vector as the feature expression for each clip. Considering that every 32 frames represents a video length of about 2.13s, the overlaps between different clips will cause some frames to be repeatedly sampled. However, same frames in different clips will have a different temporal context, so dense sampling can provide a more fine-grain feature representation, which can improve the quality of the generated proposals effectively.

Boundary-Matching Network. In order to generate high-quality temporal proposals, Lin et.al [9] proposed the Boundary-Matching Network(BMN). BMN utilizes Temporal Evaluation Module (TEM) to generate temporal proposals. Each proposal generated by TEM can map to BM confidence map which is generated by Proposal Evaluation Module (PEM), and provide confidence for proposals. BMN generates proposals with precise temporal boundaries as well as reliable confidence scores simultaneously. For a video with a duration of $t$ seconds, we can use Slowfast101 to extract features with a temporal length of $T=round(2t)$. In training and inference process of BMN, we resize the feature temporal length of each video to $D$. In our experiments we set $D$ to 200. This is because the quality of the proposals is relatively high at a temporal length of 200 in the comparison of different temporal lengths. LSTM can encode sequences very well, and in fact video also belongs to sequence information. We use a bidirectional LSTM to capture forward temporal information and backward temporal information of the video. Our experimental results show that adding bi-directional LSTM to the base module of BMN can also improve performance.

Refine Network. BMN can generate a good deal of proposals. In the HACS dataset, most videos have long duration, while the temporal length of the above-mentioned features can be only set as a small number, e.g., 200 in our method, limited by the amount of calculation and memory. Obviously, downsampling procedure will lose some temporal information. To solve this problem, we propose to further refine the proposals generated by BMN, which we call Refine Network. We first perform soft-nms [1] on the proposals, and sort them according to the corresponding scores. By this mean, a large number of redundant proposals can be removed, but only some good quality proposals would be fed for further retaining. Then we input these proposals into the Refine Network. The Refine Module employs RoI Align Pooling [7] technology to extract features for each proposal from the enhanced original video features. By applying the features, we can further predict the corresponding IoU and temporal offsets of the proposals. Then we repeat the Refine Modules several times to refine the proposals iteratively. In this way, a cascade [2] architecture is formed. In our method, we embed three Refine Modules in the refine network. As in  [2], we set IoU thresholds for assigning positive labels to $0.5$, $0.6$ and $0.7$, respectively.

Refine Module. The refine module aims to further refine the candidate proposals and predict the corresponding IoUs. Firstly, we predict score for each point on video feature whether there is an action, which we call Action Score, Similar to Actionness in  [10]. Note that it is a supervised procedure to learn Action Score. By adopting the supervision information, the learned features can possess discriminative power. Secondly, in order to predict IoU as accurately as possible, we should encode long-term temporal information. It is obvious that local features can not know the actual duration of the action (especially when proposals intercept part of the action). Therefore, we need to extract features which have a global perception of videos. In order to achieve the purpose, before performing RoI Align Pooling [7], we treat the video features as sequences and use Transformer [11] in temporal dimension to obtain a global receptive field. Finally, we also use bidirectional LSTM network [8] and Nonlocal [12] to enhance local features in the RoI features, which allows the network to fully integrate local temporal information.

We choose Slowfast101 as the backbone of our classification model. In order to make the classification model more feature-rich, we train a variety of classification models based on this backbone. For example, different learning rates, adding Batch Nuclear-norm [5] in training, and adding Transformer [11] to video features. While improving the results of a single model as much as possible, we also need the complementarity of features between different models. So even if the result of the single model is not the best, it can still improve the final classification results when performing ensemble procedure. Finally, we fuse these classification results as our final classification results. When there are $N$ classification models for ensemble, we set an adaptive weight for each model. These $N$ parameters are multiplied by the prediction results of each model before Softmax, and they will automatically converge to the value that makes the classification performance the best. Some of our experimental results are shown in the Table 1.

The purpose of BMN [9] is to initially obtain a large number of proposals for the fine network. We also train many different models with different parameters or embedded modules to increase the complementarity between different results for ensemble scheme. For the ensemble of multiple BMN models, we first take the two $D$-dimensional vectors output by the TEM in each model, which are the action starting and ending probabilities, respectively. Then we take the two $D*D$ Boundary-Matching confidence maps output by PEM. We perform weighted summation on the four maps between all models to obtain the results of model ensemble. When multiple BMN models have different $D$, we resize the four maps to the same scale by linear interpolation. The experimental results are shown in Table 3.

All Refine Networks in our experiments apply 3 Refine Module cascades. During the inference, because each Refine Module will change the position of input proposals, so we only take the prediction result of the last layer of Refine Module as our final proposals. We use the batch of proposals obtained after the BMN model ensemble as the input of $N$ different Refine Networks, and we can get $N$ batches of different adjusted proposals. Finally, we use weighted summation to fuse these proposals as our final result. Our final detection result achieves $40.55\%$ on the validation set and $40.53\%$ on the test set in terms of mAP. The experimental results are shown in Table 2.