Augmented Transformer with Adaptive Graph for
Temporal Action Proposal Generation

We apply transformers to improve the semantic representation of snippet-level features by capturing long-range dependencies. A vanilla transformer block is composed of a self-attention layer and position-wise feed-forward layers, performing multi-head attention to compute a weighted sum of input snippet features $F^{g}\in\mathbb{R}^{T\times C}$, resulting an augmented feature $\tilde{F}^{g}$ with global context. In particular, for each attention head, the attention map is computed by matching the transformed input features (a.k.a. the queries) $Q=f(F^{g})$ to another transformation of the input features (a.k.a. the keys) $K=g(F^{g})$, with $f$ and $g$ being learnable linear transformation. Namely,

where $A\in\mathbb{R}^{T\times T}$ is the generated attention map, and $d$ is the dimension for $Q$ and $K$. The above attention map computation can find the relationship between features $Q$ and $K$ of global snippets and then aggregate their information into another linear transformation of input features $V=h(F^{g})$ (a.k.a. the values). The multi-head attention is followed by an FFN layer, which contains two linear layers with ReLU activation and a residual connection after each layer. We also include layer normalization and dropout to facilitate training.

Directly employing the vanilla transformer presents two issues here. First, the main task of TAPG is the boundary regression, and the corresponding loss functions supervise proposal-level signal and cannot guide the transformer to effectively learn long-term semantic relations for each snippet. Second, according to Eq. 1, the computation of any element $A_{mn}$ in the attention map matrix $A$ depends on only the features of snippet m and n, namely, $F_{m}^{g}$ and $F_{n}^{g}$, which indicates that the attention map generation does not consider any temporal context, especially in limit transformer layers. To solve the above issues, we add a snippet actionness loss and a front block to the vanilla transformer and it is dubbed augmented transformer.

The snippet actionness loss explicitly guides the transformer to learn effective long-range dependencies at snippet level. An actionness predictor is equipped upon the FFN, and it is used to predict the probability of an action instance existing in the input snippet, by minimizing the following binary classification loss $L_{a}$ over action/background categories:

where $\alpha^{+}={T}/\sum(g_{i}^{a})$, $\alpha^{-}={T}/{\sum(1-g_{i}^{a})}$, and $p^{a}_{i}$ and $g_{i}^{a}$ are the actionness predicted probability and action/background label of snippet $i$.

This loss function supervises the network to obtain snippet-wise classification only depending on each snippet feature from the output of transformer, which is critical to achieve noisy action instances localization. For those noisy snippets that have action labels, the network shall build correct relations among the noisy snippets and others to make correct decisions. Thus, this loss function helps the transformer learn how to selectively capture long-range dependencies and understand the complicated noisy action instances.

The front block is a light-weight network consisting of three parts to expand the temporal receptive field. We first apply a gated linear unit [6], followed by a parallel 1 $\times$ 1 convolutional layer and 3 $\times$ 1 average pooling layer with stride 1 to expand the receptive field, and the small 3 $\times$ 1 average pooling layer can also smooth the snippet-level feature to filter out the tiny noisy frames. The last part is a convolutional layer with a large kernel size, for example, 7 $\times$ 1. To avoid over-fitting from large size kernels, we adopt the depth-wise separable convolution. We apply residual connection for each part and layer normalization behind each part. The structure of front block is depicted in the top-right corner in Fig. 2.

To capture the local context, we design a new graph convolution layer to construct the local branch. We build a video graph $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$, where $\mathcal{V}=\{v_{i}\}_{i=1}^{T}$ and $\mathcal{E}$ represent the node and edge sets, respectively. Each node represents a snippet and each edge shows the dependency between two snippets. For local context modeling, edges between two nodes are constructed according to their temporal distance. The edge set is defined as:

where $T$ is the number of snippets, and $\delta$ is defined as the maximum connection distance. Our graph convolution operation $\mathcal{F}$ is defined as:

where $W\in{\mathbb{R}^{C_{\mathrm{out}}\times C_{\mathrm{in}}}}$ includes trainable weights, $M$ is a fixed $T\times T$ binary mask matrix to limit the connection range according to $\delta$, and $\odot$ is the dot product. Instead of using a pre-defined adjacency matrix, the matrix in Eq. 4 is divided into two parts: $A_{a}$ and $A_{d}$ and treated differently.

The first part ($A_{a}$) is completely parameterized and optimized in the training phase. The elements in $A_{a}$ can be arbitrary values without any constraints, which means that the edges of the graph are completely learned based on the training data and the positions of nodes in the video.

The second part ($A_{d}$) is a data-dependent graph which adaptively learns a unique graph for each video. Our GCN is used to capture local temporal context, therefore we pay more attention to the difference of features. To determine whether there is a connection between two nodes and how strong the connection is, the difference of two nodes $m$ and $n$ is computed as:

where $\sigma$ is an activation function, ${\theta}^{T}$ is the trainable parameter vector to reduce the dimension of $f_{m}-f_{n}$ to $1$. The detailed visual description is shown in Figure 2.

Early fusion and late fusion are widely used in two-stream feature fusion. In our method, we use the early fusion, i.e., the global and local features are fused by concatenation before the output module.

Due to our global-local combination mechanism, the two-branch features have precise and discriminative action/background representation. Here, we input them into two same convolutional networks consisting of two temporal convolutional layers and a sigmoid activation function to generate the probability of the start $p^{s}$ and end $p^{e}$ for each snippet, respectively.

Besides the prediction of boundaries, the network is trained to predict the completeness of proposals to boost the final results. Our completeness regression network follows  [16] and also introduces the Boundary-Matching mechanism to generate completeness scores for densely distributed proposals. For each proposal, we sample $32$ features from the corresponding two-branch features. Sampled features are then input into the completeness prediction network to generate two completeness score maps $M^{cc}\in\mathbb{R}^{T\times D}$ for completeness classification, and $M^{cr}\in\mathbb{R}^{T\times D}$ for completeness regression, where $D$ is the maximum proposal duration and is set depending on the dataset.

In order to make full use of various predicted scores for each proposal $\phi_{n}$, we fuse its boundary probabilities and completeness scores by multiplication. The confidence score $p^{f}$ can be defined as:

Hence, the proposals set can be re-written as $\psi_{p}=\{\phi_{n}=(t_{s},t_{e},p^{f})\}_{n=1}^{N}$.

Because of our dense proposal generation, it is inevitable to output numerous redundant proposals which highly overlap with each other. Thus, based on the completeness score for proposals, we apply the Soft-NMS algorithm to remove redundant proposals. Candidate proposal set $\psi_{p}$ turns to be $\psi_{p}^{\prime}=\{\phi_{n}=(t_{s},t_{e},p^{f\prime})\}_{n=1}^{N^{\prime}}$, where $p^{f\prime}$ and $N^{\prime}$ is the final completeness score and the number of final proposals, respectively.

The evaluation is performed on two popular TAPG benchmark datasets, THUMOS14 [12] and ActivityNet1.3 [3]. We use the subset of THUMOS14 that provides frame-wise action annotations. The model is trained with 200 untrimmed videos from its validation set and evaluated using 213 untrimmed videos from the test set. This dataset is challenging due to the large variations of the frequency and duration of action instances across videos. ActivityNet1.3 is a large-scale dataset covering 200 complicated human activity classes with 19,994 untrimmed videos, which are divided into training, validation and test set in the ratio 2:1:1. The model is trained on the training set and evaluated with the validation set.

For feature encoding, following previous works [16, 1], we adopt two-stream network [28] pre-trained on training set of Activity1.3. The frame interval is set to $5$ and $16$ on THUMOS14 and Activity1.3, respectively. For ActivityNet1.3, we resize video feature sequences by linear interpolation to 100, i.e. $T=100$, and set the maximum duration length as $100$. For THUMOS14, we slide the window on video feature sequence with $50\%$ overlap and $T=128$ and set the maximum duration length as $64$, which can cover $98\%$ action instances.

Adam [13] optimizer is adopted and the learning rate is set to $10^{-3}$ and decayed by a factor of 0.1 after every 10 epochs. To preserve the positional information in the transformer, we adopt sine positional encoding and add it to queries and keys. There are 8 heads in the multi-head attention module is $8$. $\delta=2$ in local branch. In Eq. 8, the coefficients $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are all set to $1$. Early fusion is adopted unless otherwise specified.

Our model is compared with state-of-the-art TAPG methods on THUMOS14 and ActivityNet1.3. Table 1 shows the comparison on THUMOS14. To ensure a fair comparison, both C3D and two-stream features used by previous methods are adopted in our experiments. Following previous methods [17, 16, 15, 1], we apply NMS and soft-NMS as post-processing methods to evaluate our method, respectively. For both C3D and two-stream features, our results outperform other state-of-the-art methods by a large margin. Additionally, NMS achieves better average recall performance than soft-NMS under small proposal numbers.

The results on ActivityNet1.3 are summarized in Table 2. Our method again surpasses all of the leading methods.

These experiments demonstrate the effectiveness of our model by adequately exploiting the global and local context.

Fig. 4 visualizes some representative results on ActivityNet1.3 and THUMOS14 featuring different challenging cases. The proposals with the highest $k$ scores are visualized in each video, where $k$ is the number of ground truth. In the top video, background frames and action frames have the similar scene and there are many irrelevant frames which are nearly the same as background frames in the action. However, our top-1 proposal still successfully aligns the position of ground truth action instance. Our network is trained to regard the noisy snippets in action instance as action. We concern that whether the network will overfit on the noisy frames and many background snippets are mistakenly treated as action instance. In the middle video, a cheering segment exists between the true action and background so it needs to be classified as background, which our model predicts correctly. Note that this is different from the case in Fig. 1, where the cheering segment is embedded in an action instance. The correct prediction made by our model shows that it correctly understands the semantic information and does not overfit on the noisy frames. The bottom video has multiple ground truth action instances, while our top-5 proposals perfectly cover them in an accurate way, suggesting the high quality of our generated proposals. More examples and comparison with the previous method are shown in the supplement material.

The augmented transformer and adaptive GCN are the core components in our method. We evaluate several variants of this model by ablating its augmented transformer and GCN in Table 3. Base represents the version without augmented transformer and adaptive GCN. We can see that any component can individually improve the performance. The existence of snippet actionness loss and front block can boost the effect of vanilla transformer, which demonstrates that they can assist the transformer to learn long-range dependencies. For local context capture, the result shows that adaptive graph is beneficial for this task and removing any adjacency matrix will degrade the performance. With the global and local fusion, the full model reaches the best result.

We explore other fusion strategies. Specifically, in the early fusion, besides our early fusion with concatenation, we also evaluate the global and local features are fused by summation; in the late fusion, each path is individually fed into independent prediction networks of the output module and concatenate the dual-branch features till entering classifiers to aggregates predictions of boundaries and completeness scores. We show the results in Table 3. The Early fusion with concatenation outperforms other fusion strategies.

Stacking multi-layer transformers normally brings better performance, which is demonstrated in other tasks, for instance, NLP [25, 7] and object detection [4]. We stack different numbers of transformers and show their results in Table 4. Different from other tasks, more transformers in our model do not improve the performance. We believe the reason is that the scale of datasets in this task is too small to train a deeper transformer network. Although there are about 20k videos in ActivityNet1.3, the density of action instances is low and each video includes only 1.5 action instances on average, which is far less than the datasets in other tasks.

For the local context capture, we use a GCN with a hybrid adaptive adjacency matrix. Further, we experiment with different structures to replace our GCN, including (i) a general GCN with predefined adjacency matrix, (ii) a self-attention style GCN where the adjacency matrix $A$ is calculated by $A=softmax(X^{T}W_{\theta}^{T}W_{\phi}X)$ where $X$ is the input feature sequence and $W_{\theta}$ and $W_{\phi}$ are trainable weights, and (iii) a convolutional layer. Table 5 compares all these variants, with our choice outperforming other two variations.