Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation

Assume that an arbitrary untrimmed video $V=\{v_{t}\}_{t=1}^{T}$ contains a collection of $N$ action instances $\Psi=\{\psi_{n}=(t_{s,n},t_{e,n}\}_{n=1}^{N}$, where $\psi_{n}$ refers to the $n$-th action instance and $(t_{s,n},t_{e,n})$ correspond to its annotated start time, end time. The aim of temporal action proposal generation task is to predict a set of action proposals $\Phi=\{\phi_{w}=(t_{s,w},t_{e,w},p_{w}\}_{w=1}^{W}$ as close to the ground truth annotations as possible based on the content of $V$. Here $p_{w}$ is the $w$-th proposal confidence.

We propose a Pyramid Region-based Slot Attention Network (PRSA-Net) to generate temporal action proposals precisely. The pipeline of our PRSA-Net is illustrated in Fig. 2. PRSA-Net consists of three major components: feature extractor, PRSlot modules and dual branch proposal generation head. The original video is first fed into the feature extractor to produce the snippet-level features $X$ of size $L\times C_{\texttt{input}}$, where $L$ is the snippet length and $C_{\texttt{input}}$ is the feature dimension. Then, PRSlot modules are employed to enhance action-matter contextual representations. After this, a dual branch generation head maps the output embedding from the PRSlot modules to final boundaries and confidence scores respectively. The Boundary classifier is used to detect the boundaries, while the proposal regressor is utilized to evaluate the proposal-level confidence. We provide detailed descriptions of PRSA-Net below.

Given an untrimmed video $V$, we utilize the Kinetics [21] pre-trained I3D [6] to extract video features. We consider RGB and optical flow features jointly as they can capture different motion aspects. Following the implementation of previous methods [26, 45], each frame feature is flattened into a $C$-dimensional feature vector and then we group every consecutive $\sigma$ frames into $L=\lceil T/\sigma\rceil$ snippets. In this work, these generated video snippets are the minimal units for further feature modeling. For convenience, we denote the extracted snippet features as $X\in\mathbb{R}^{L\times C}$, where $C$ is the feature dimension and $L$ is the total number of video snippets. In the end, a 1d convolution is applied to transform the channels into $C_{\texttt{input}}$. For fair comparisons with [26, 25, 45], we also conduct experiments based on the Kinetics pre-trained TSN [42] to extract video features.

The original slot attention module is firstly proposed for updating the object-centric representations (slots) by similarity-based attention. Slots produced by slot attention bind to any object in the input embedding and are updated via a learned recurrent function for object-centric learning. Inspired by this, we propose a tailor-modified Pyramid Region-based Slot Attention (PRSlot) module for capturing action-matter representations by a novel attention mechanism.

Next, slot embedding provided by PRSlot modules serves as the latent action-aware representations for the proposal estimation.
Boundary Classifier. A lightweight 1d convolution layer is introduced to transform the input channels into 2 for (start, end) detection and a non-linear $\mathsf{sigmoid}$ function is followed to form the start/end probabilities $\mathcal{P}^{s}$/$\mathcal{P}^{e}$ separately.
Proposal Confidence Regressor. Following the conventional proposal regression method [25], we use pre-defined dense anchors to generate densely distributed proposals shaped as $D\times L$ and then apply 1DAlignLayer [45] to extract the proposal-level features for corresponding proposal anchors shaped as $D\times L\times C_{\texttt{out}}$. Finally, 2 FC layers are used to predict the proposal-level completeness maps $M^{com}$ and classification map $M^{cls}$.

Label Assignment. We first generate temporal boundary label $G_{s}$ and $G_{e}$ followed by BSN [26]. Next, we generate the dense proposal label map $G^{c}\in\mathbb{R}^{D\times L}$. As described in [25], for a proposal $G^{c}_{i,j}$ with start frame $i$ and end frame $i+j$, we calculate the intersection over union (IoU) with all ground-truth and denote the maximum IoU as the value of $G^{c}_{i,j}$.
Training. We define the following loss function to train our proposed model

Here the boundary classification loss $\mathcal{L}_{b}$ is a weighted binary logistic regression loss used to determine the starting and ending scores $\mathcal{P}^{s}\in\{\mathcal{P}_{l}^{s}\}_{l=1}^{L}$ or $\mathcal{P}^{e}\in\{\mathcal{P}_{l}^{e}\}_{l=1}^{L}$. $\mathcal{L}_{norm}$ is the regularization term for network weights, we set $\lambda=0.0002$. We also construct the proposal confidence losses $\mathcal{L}_{p}$ which is the combination of cross-entropy loss and mean square error loss to optimize dense proposals confidence scores $M^{com}$ and $M^{cls}$. It can be formulated as,

where $\mathcal{L}_{cls}$ is binary logistic regression loss and $\mathcal{L}_{com}$ is the MSE loss, usually we set the balance term $\lambda_{c}=10$.
Inference. Based on the outputs of the boundaries classifier, we select the valid starting snippets from $\{\mathcal{P}^{s}_{l}\}_{l=1}^{L}$ by two conditions: (1) $\mathcal{P}^{s}_{l-1}<\mathcal{P}^{s}_{l};\mathcal{P}^{s}_{l}>\mathcal{P}^{s}_{l+1}$; (2) $\mathcal{P}^{s}_{l}>0.5\times\max_{n=1}^{L}\{\mathcal{P}_{n}^{s}\}$. We apply the same rule for recording ending snippets. Then, we combine these valid starting and ending snippets and obtain the candidate proposals denoted as $\Phi=\{\phi_{w}=(t_{s,w},t_{e,w},p_{w},M_{w}\}_{w=1}^{W}\}$, where $\mathcal{P}_{w}=\mathcal{P}^{s}_{t_{s,w}}\cdot\mathcal{P}^{e}_{t_{s,w}}$ and proposal-level confidence $M_{w}=M^{cls}_{t_{s,w},t_{e,w}}\cdot M^{com}_{t_{s,w},t_{e,w}}$. Finally, we fuse the proposal confidence scores and output the predicted proposals $\Phi=\{\phi_{w}=(t_{s,w},t_{e,w},S_{w}\}_{w=1}^{W}\}$, here $S_{w}=p_{w}\times M_{w}$.
Post-Processing. The non-maximum suppression (NMS) algorithm [30] is adopted to suppress redundant proposals with high overlaps. Also, we use the Soft-NMS [2] algorithm to suppress the proposals and report the performance for a fair comparison with previous methods.

THUMOS14 [18]. THUMOS14 dataset contains respectively 200 videos in the validation for training and 213 videos in the test set for inference. It has a total number of 20 classes, and each video has around 15 action instances on average.
ActivityNet-1.3 [4]. ActivityNet-1.3 dataset contains 19994 untrimmed videos labeled in a wider range of 200 action categories with a lower 1.5 action instances per video on average. These videos are split into the training, validation, and test set by the ratio of 2:1:1. We evaluate on the validation set of ActivityNet-1.3.
Evaluation metric. To assess the performances of action proposal generation, we report the average recall (AR) under different intersections over union thresholds (tIoUs) with various average number of proposals (AN) for each video. Following the conventions, we adopt the tIoUs of $\{0.5:0.05:1.0\}$ on THUMOS14 and $\{0.5:0.05:0.95\}$ on ActivityNet-1.3. To evaluate the quality of our generated proposals, we also evaluate the performances of action detection using the mean average precision (mAP) at different tIoUs. Following the official evaluation API, the tIoUs of $\{0.3,0.4,0.5,0.6,0.7\}$ are used for THUMOS14, and $\{0.5,0.75,0.95\}$ are used for ActivityNet-1.3.

Following the conventional setting, we extracted the 2048-dimensional video features by two-stream I3D [6] pre-trained on Kinetics [21] on THUMOS14. Besides, for a fair comparison, we also conduct experiments based on the two-stream network TSN [42] backbone, where ResNet network [15] and BN-Inception network [19] are applied as the spatial and temporal network respectively. We also set $C_{\texttt{input}}=256$. On ActivityNet-1.3, the video features were extracted by the pre-trained model provided in [44, 4]. In data preparation, we set the snippet interval $\sigma$ to 4 on THUMOS14 and 16 on ActivityNet-1.3. Then we cropped each video feature sequence with the overlapped windows of stride of $100$ and length $L=250$ on THUMOS14. While on ActivityNet-1.3, we set the temporal length to $L=100$ by linear interpolation. Also, we enumerate all possible anchors where the max durations is $D=64$ on THUMOS14 and $D=100$ on ActivityNet-1.3. For the PRSlot module, we set the embed dimension $C_{\texttt{embed}}=256$, $C_{\texttt{out}}=256$, and the local region scale $S=\{4,8\}$. In post-processing, we set the NMS threshold $\theta$ to 0.65 and 0.45 on THUMOS14 and ActivityNet-1.3 respectively. During the training, it was optimized on one NVIDIA TESLA V100 GPU with batchsize 16 on ActivityNet1.3 and 8 on THUMOS14. We adopted the Adam optimizer [22] for network optimization. For THUMOS14, the models were tuned for 10 epochs with the learning rate set to $2\times 10^{-4}$. For ActivityNet-1.3, we trained our models by setting the learning rate to $10^{-3}$ in the first 7 epochs and decaying it to $10^{-4}$ in the last 3 epochs.

We conduct extensive experiments on THUMOUS14 using the I3D [42] backbone and NMS for post-processing to investigate the effectiveness and proper settings of the different components of our proposed models.

In Fig. 4, we visualize some action detection results generated by our PRSA-Net on THUMOS14. We can find that our model can successfully detect the action instances in these examples with precise action boundaries. For the untrimmed video with multiple action instances, the foreground actions and background scenes can be well distinguished.

When evaluating the quality of our proposals, we follow the conventional two-stage action detection works [45, 25, 26]. Therefore, we classify the candidate proposals using external classifiers. We use the video classifier in UntrimmedNet [41] to assign the video-level action classes on THUMOS14, while on ActivityNet1.3, we adopt the video-level classification results from [44] to assign the action labels to detect the action class. Furthermore, we also introduce the proposal-level classifier P-GCN [49] to predict action labels for every candidate proposals. We evaluate the final action detection performances and make comparisons with the state-of-the-art. The results on THUMOS14 are summarized in Table 6. Compared with the other state-of-the-art methods, our approach achieves significant improvements under all tIoU settings. Especially, the mAP at the typical IoU = 0.5 was boosted from 45.4% to 55.0%, reaching a considerable improvement ratio of 21.1%. Also, when proposal-level classifier P-GCN is applied to classify our generated proposals following the same implementations in [45, 38], the performance can be boosted rapidly and achieve 58.7% [email protected]. Table 7 shows our action detection results on the validation set of ActivityNet-1.3 compared with previous works. Again, our method outperforms most other methods by a large margin in almost all cases, including the average mAPs over different tIoUs. These experiments demonstrate that the proposals generated by our PRSA-Net are able to boost the action detection performance better.