Action Unit Memory Network for Weakly Supervised Temporal Action Localization

Assume we have $N$ untrimmed training videos $\{v_{i}\}_{i=1}^{N}$. Each video $v_{i}$ has its ground-truth label ${\bf y}_{i}\in\mathbb{R}^{C}\$, where $C$ is the number of action categories. ${\bf y}_{i}(j)=1$ if the action category $j$ is present in the video and ${\bf y}_{i}(j)=0$ otherwise¹¹1If there are multiple action categories in one video, ${\bf y}_{i}$ is normalized with the $\ell_{1}$ normalization.. During testing, the goal of the temporal action localization is to generate a set of action proposals $\{(c,s,e,q)\}$ for each video, where $c$ and $q$ denote the predicted category and the confidence score, $s$ and $e$ represent the start and the end time respectively. In this paper, we follow previous works [29, 28, 34] to extract features for both RGB and optical flow streams. Given an untrimmed video $v_{i}$, we first divide it into non-overlapping 16-frame segments and apply the I3D pretrained on the Kinetics dataset to extract features for each segment. Then we get two segment-wise features ${\bf X}^{RGB}_{i}\in\mathbb{R}^{{l_{i}}\times{D}}$ and ${\bf X}^{FLOW}_{i}\in\mathbb{R}^{{l_{i}}\times{D}}$, where $l_{i}$ denotes the number of segments in video $v_{i}$ and $D$ is the dimension of features. Because the RGB and FLOW streams are trained independently, we use ${\bf X}_{i}$ to represent them in the rest of this paper for simplicity. Since the extracted features from I3D are learned for the action recognition task originally, it is desired to add a task-adaption layer to refine the extracted features. In specific, we adopt a temporal convolutional layer with the ReLU activation as

$${\bf X}^{e}_{i}=\mathrm{ReLU}(W^{emb}*{\bf X}_{i}+b^{emb}),$$

(1)

where the $*$ represents the convolution operation, $W^{emb}$ and $b^{emb}$ are the weights and bias of temporal filters, ${\bf X}^{e}_{i}\in\mathbb{R}^{{l_{i}}\times{F}}$ is the learned embedding feature, and $F$ is the dimension of learned features.

The overall architecture of our action unit memory network is shown in Figure 2. It consists of three parts including feature extraction, memory bank construction, memory bank for classification and updating. The details are introduced as follows.

Memory Bank Construction. The action unit memory bank stores multiple templates ${\bf M}\in\mathbb{R}^{{K}\times{F}}$, and each template represents an action unit, where $K$ and $F$ are the number and dimension of templates respectively. We adopt two encoders named ${\bf Enc}_{K}$ and ${\bf Enc}_{V}$ to embed the templates into pairs of keys and values respectively. The ${\bf Enc}_{K}$ is designed to reduce the dimension of the templates for efficient reading from the memory, which is implemented as a fully connected layer (FC). And the ${\bf Enc}_{V}$ is designed to encode each template into a template-specific classifier, which is a MLP network consisted of two FC layers with a bottleneck structure among them to reduce parameters. In this way, the keys store appearance and motion related information for the templates and can be used for template matching during memory reading, and the values store templates specific classifiers and can be used for segment classification. Formally, we formulate the encodings as follows:

$${\bf K}^{M}={\bf Enc}_{K}({\bf M}),$$

(2)

$${\bf V}^{M}={\bf Enc}_{V}({\bf M}),$$

(3)

where ${\bf K}^{M}\in\mathbb{R}^{{K}\times{F/m}}$ and ${\bf V}^{M}\in\mathbb{R}^{{K}\times{CF}}$ are keys and values, $M$ denotes the memory, and $m$ is a hyper-parameter to control memory reading efficiency. Given the memory bank and an input video, we introduce how to perform video classification and memory updating next.

Memory Bank for Classification. For video classification, we use an encoder ${\bf Enc}_{Q}$ which is implemented as a FC layer to encode video feature ${\bf X}^{e}_{i}$ into a set of queries ${\bf Q}_{i}\in\mathbb{R}^{{l_{i}}\times{F/m}}$, and then feed the segment features and queries into a self-attention module and a cross attention module to generate classification results. In the self-attention module, we first calculate the similarity scores among video segments with queries and then use these scores to refine the segment features by aggregating context information, which can be formulated as

$${\bf X}^{s}_{i}=(\mathrm{softmax}(\frac{{\bf Q}_{i}{\bf Q}^{T}_{i}}{\sqrt{F/m}})+{\bf I}){\bf X}^{e}_{i},$$

(4)

where ${\bf I}$ is the identity matrix used to preserve the original information, and ${\bf X}^{s}_{i}$ keeps the same dimension with ${\bf X}^{e}_{i}$. Via this message passing between segments, we can extract global context information and get more discriminative features for both classification and localization.

In the cross-attention module, we read from the memory and get a set of segment-wise classifiers. To achieve this goal, we first calculate the similarity scores ${\bf S}_{i}\in\mathbb{R}^{{l_{i}}\times{K}}$ between video segments and memory templates with the scaled dot-product

$${\bf S}_{i}=\mathrm{sigmoid}(\frac{{\bf Q}_{i}({\bf K}^{M})^{T}}{\sqrt{F/m}}).$$

(5)

Based on the similarity scores, we can obtain a set of segment-wise classifiers by using the similarity scores to aggregate memory values as

$${\bf V}^{O}_{i}={\bf S}_{i}{\bf V}^{M},$$

(6)

where ${\bf V}^{O}_{i}\in\mathbb{R}^{{l_{i}}\times{CF}}$. Later, to perform classification, we reshape ${\bf V}^{O}_{i}$ into a set of segment classifiers ${\bf W}^{cls}_{i}=\{{\bf W}^{cls}_{i}(t)\in\mathbb{R}^{{F}\times{C}}\}_{t=1}^{l_{i}}$, which is adaptive to the appearance or motion variations of each segment.

With the refined features of the self-attention module and the segment-wise classifiers of the cross-attention module, we can obtain the segment-level classification results by applying each classifier on the corresponding segment. Since we only have video-level ground-truth supervision, we need to aggregate these segment-level classification results into a video-level prediction. In specific, since the second dimension of the similarity matrix ${\bf S}_{i}$ denotes the similarity between a segment and a template, we can apply the max-pooling operation along the second dimension of the ${\bf S}_{i}$ to get the foreground attention weight ${\bf a}_{i}\in\mathbb{R}^{l_{i}}$ as

$${\bf a}_{i}=\mathrm{MaxPool}({\bf S}_{i}),$$

(7)

The video-level classification result $\hat{\bf y}_{i}$ is then obtained as an attention weighted pooling

$$\hat{\bf y}_{i}=\mathrm{softmax}(\frac{1}{l_{i}}\sum_{t=1}^{l_{i}}{\bf a}_{i}(t)({\bf X}^{s}_{i}(t){\bf W}^{cls}_{i}(t))),$$

(8)

where $\hat{\bf y}_{i}\in\mathbb{R}^{C}$. Then the classification loss is defined as the cross-entropy loss between the prediction and the video label ${\bf y}_{i}$

$$\mathcal{L}_{cls}=-\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{C}{\bf y}_{i}(j)\log\hat{\bf y}_{i}(j).$$

(9)

Memory Bank Updating. For the memory updating, we find that the above classification loss alone is not enough to learn a satisfying memory bank. Thus we design three mechanisms (diversity, homogeneity and sparsity) to guide the updating of the memory bank. The diversity mechanism encourages that each template in the memory bank is different from other templates, the homogeneity mechanism encourages that each template in the memory bank is meaningful, and the sparsity mechanism encourages that the templates in the memory bank can suppress background segments. In specific, in the diversity mechanism, we design a diversity loss to ensure the uniqueness of each template in the memory as

$$\mathcal{L}_{d}=\left\|{\bf M}{\bf M}^{T}-{\bf I}\right\|_{F},$$

(10)

where ${\bf I}$ is the identity matrix and $\left\|\cdot\right\|_{F}$ is the Frobenius norm of a matrix. While the diversity loss encourages the templates in the memory bank to be unique, it does not guarantee that each template in the memory bank is useful. For example, a template may not represent an action unit and have low similarities with all video segments during training. To deal with this issue, we design a homogeneity loss to encourage a uniform distribution for the occurring probability of templates in the homogeneity mechanism. In specific, we first pool the similarity matrix ${\bf S}_{i}$ over time by a sum operation and then use a softmax function to obtain the occurring probability of each template as

$${\bf p}^{O}_{i}=\mathrm{softmax}(\sum_{t=1}^{l_{i}}{\bf S}_{i}(t)),$$

(11)

where ${\bf p}^{O}_{i}\in\mathbb{R}^{K}$. Then the homogeneity loss can be formulated as

$$\mathcal{L}_{h}=\left\|\frac{1}{B}\sum_{i=1}^{B}{\bf p}^{O}_{i}\right\|_{2},$$

(12)

where $B$ is the mini-batch size. And based on the observation that an action usually occupies a small portion of a untrimmed video, we design a sparsity loss in the sparsity mechanism to relieve the interference of background segments. And the sparsity loss is designed as

$$\mathcal{L}_{s}=\frac{1}{B}\sum_{i=1}^{B}\left\|{\bf a}_{i}\right\|_{1},$$

(13)

which encourages background segments to have low similarities with all the templates.

Training. For training the whole network, we compose the classification loss and three auxiliary losses as

$$\mathcal{L}=\mathcal{L}_{cls}+\alpha\mathcal{L}_{d}+\beta\mathcal{L}_{h}+\gamma\mathcal{L}_{s}.$$

(14)

where $\alpha$, $\beta$ and $\gamma$ are hyper-parameters to balance the contribution of each loss function. To summarize, we maintain a set of templates in representation of various action units and update them in the video-level classification task. To better guide the learning of templates, a diversity loss and a homogeneity loss are devised to keep the variety and effectiveness of each template, and a sparsity loss is introduced to relieve background interference.

Inference. After modeling action units by the AUMN, we can localize actions by examining whether a segment belongs to a kind of action unit. In specific, we take a two-step approach to perform localization. First, we threshold on video-level prediction scores $\hat{\bf y}_{i}$ and discard categories which have confidence scores below a threshold $\eta_{cls}$. Thereafter, for each of the remaining action categories, we apply the threshold $\eta_{act}$ on the foreground attention weight to generate action proposals. To assign a confidence for each proposal, we compute the class activation sequence (CAS) ${\bf C}_{i}\in\mathbb{R}^{{l_{i}}\times{C}}$ first, where

$${\bf C}_{i}(t,;)={\bf X}^{s}_{i}(t){\bf W}^{cls}_{i}(t),$$

(15)

then ${\bf C}_{i}$ is passed through a softmax function along the category dimension to get class scores at each time location, denoted as ${\bf\bar{C}}_{i}$. And the confidence score $q$ in the proposal $\{(c,s,e,q)\}$ is computed as

$$q=\sum_{t=s}^{e}\frac{\theta{\bf a}^{R}_{i}(t){\bf\bar{C}}^{R}_{i}(t,c)+(1-\theta){\bf a}^{F}_{i}(t){\bf\bar{C}}^{F}_{i}(t,c)}{s-e+1},$$

(16)

where the superscripts $R$ and $F$ denote $RGB$ or $FLOW$ streams respectively, $\theta$ is a scalar denoting the relative importance between the two modalities and is set to 0.3 in this work. To remove proposals with a high overlap, the class-wise Non-Maximal Suppression (NMS) is used.

Datasets. The proposed AUMN is evaluated on two benchmark datasets including THUMOS14 [13] and ActivityNet [3]. THUMOS14 dataset contains 200 validation videos and 213 testing videos annotated with temporal action boundaries belonging to 20 categories. This dataset is particularly challenging as it consists of very long videos with multiple action instances of small duration. Following previous works [39, 29, 31, 23, 28, 16, 52], we use the 200 validation videos for training and the 213 testing videos for evaluation. ActivityNet dataset includes ActivityNet1.2 and ActivityNet1.3. ActivityNet1.3 consists of 10024 training videos, 4926 validation videos and 5044 testing videos belonging to 200 action categories. And ActivityNet1.2 is a subset of ActivityNet1.3, which covers 100 action categories with 4819 training, 2383 validation and 2480 testing videos. ActivityNet only contains 1.5 instances per video on average and most videos only contain one action category with only 36% background averagely. Following previous works [39, 29, 31, 23, 28, 16, 52], we train our model on the training set and evaluate it on the validation set.

Evaluation Metrics. Following the standard evaluation protocol, we evaluate the TAL performance with the mean Average Precision (mAP) values under different intersection over union (IoU) thresholds.

Implementation Details. We use the two-stream I3D networks [4] pre-trained on Kinetics as our feature extractor. Note that for fair comparison, we do not finetune the I3D network. We apply the TV-L1 algorithm to extract optical flow from RGB data. Then we divide both streams into non-overlapping 16 frames segments as the input to the I3D network, the dimension $D$ of the output feature for each segment is 1024. We train separate AUMNs for RGB and FLOW streams and collect the generated proposals from both networks during inference. In AUMN, the embedding layer is composed of a temporal convolutional layer with 1024 input channels and 512 output channels. The number of templates $K$ is $7$ if not mentioned specifically. In Eq. (14), the loss function weights $\alpha$ = 0.01, $\beta$ = 0.02, and $\gamma$ is set to 0.05 and 0.03 for the RGB stream and the FLOW stream, respectively. During inference, the threshold $\eta_{cls}$ is 0.1 and $\eta_{act}$ is the mean value of the corresponding foreground attention ${\bf a}_{i}$ for video $v_{i}$. And we use the class-wise NMS with a threshold 0.3 to remove highly overlapped proposals. Our model is trained using Adam optimizer [14] with the learning rate $10^{-4}$ and batch size 32.

Experiments on THUMOS14. Table 1 summarizes the performance comparison between the proposed AUMN and state-of-the-art TAL methods on the THUMOS14 test set. Weakly⁺ denotes methods that adopt additional supervision during training, e.g., the number of action instances in a video, and AVG indicates the average mAP for IoU thresholds 0.1:0.1:0.5. From the results, we can see that the proposed AUMN outperforms all the previous weakly supervised models and achieves a new state-of-the-art performance (33.3% mAP at IoU0.5). And an absolute gain of 5.1% is achieved in terms of the average mAP when compared to the best previous method (TSCN [52]). It is worth noting that EM-MIL [25] gets a higher mAP at IoU thresholds 0.6 and 0.7 than ours. However, we get 7% improvement than EM-MIL at average mAP. Besides, EM-MIL adopts a pseudo label scheme to relieve background interference while we adopt a simple sparsity prior. We believe the performance of our approach can be promoted further when equipped with a more effective background suppression techniques. Compared to the weakly⁺ methods, our method outperforms 3C-Net [28] at all IoU thresholds and achieves 5.1% improvement over STAR [43] in average mAP. When compared with fully supervised methods, we note that the performance of AUMN drops faster than fully supervised methods as the IoU threshold increases. However, we can also get a comparable result at low IoU thresholds, e.g., AUMN outperforms TAL-Net at IoU thresholds 0.1, 0.2 and 0.3.

Experiments on ActivityNet. On the ActivityNet dataset, we follow the standard evaluation protocol [3] by reporting the average mAP scores at different thresholds (0.5:0.05:0.95). The performance comparisons on the ActivityNet1.2 and ActivityNet1.3 are shown in Table 2 and Table 3, respectively. The results are consistent with those on the THUMOS14 dataset, and our AUMN outperforms all previous weakly supervised models in average mAP on both ActivityNet1.2 and ActivityNet1.3, with 25.5% and 23.5% average mAP respectively. It is worth noting THUMOS14 dataset and ActivityNet dataset have different characteristics. For the THUMOS14 dataset, the most important thing is the background suppression. While for the ActivityNet dataset, the most important thing is the localization completeness. At high IoU thresholds, EM-MIL has better performance on the THUMOS14 dataset while worse performance on the ActivityNet dataset. This is because EM-MIL mainly considers background suppression while ignores the localization completeness. Different from existing methods, our AUMN takes both background suppression and localization completeness into consideration, and can achieve favorable performance on both datasets.

In this section, we conduct a series of ablation studies on the THUMOS14 dataset to evaluate the influence of each design.

Influence of Each Loss Function. As introduced in Section 3.2, we design three auxiliary losses (diversity loss $\mathcal{L}_{d}$, homogeneity loss $\mathcal{L}_{h}$ and sparsity loss $\mathcal{L}_{s}$) to guide the memory updating. To explore the influence of each loss function, we conduct experiments with different loss combinations, and the results are shown in Table 4. From the results, we have the following observations: (1) The sparsity loss $\mathcal{L}_{s}$ can bring a significant performance improvement at all IoU thresholds. Because there is no frame-level label as the supervision of the foreground attention, $\mathcal{L}_{s}$ can serve as a prior to guide the action unit templates to focus on the action related segments. (2) The diversity loss $\mathcal{L}_{d}$ is designed to encourage the action unit templates in the memory bank to be different from each other. Without the diversity loss, we can only rely on the random initialization to achieve our goal. From the results, we can see that the diversity loss can bring a 1.0% performance gain in average mAP, which indicates that the diversity loss is necessary. (3) The homogeneity loss $\mathcal{L}_{h}$ is designed to guarantee that each learned action unit template is useful. Without the $\mathcal{L}_{h}$, some templates in the memory bank may be useless, which may decrease the representative ability of the memory bank. When equipped with this loss, we observe a 0.6% performance gain in average mAP. (4) These three losses can promote each other. For example, the $\mathcal{L}_{d}$ can keep the difference between templates, but cannot ensure each learned template is useful. On the other hand, although $\mathcal{L}_{d}$ can ensure no template is redundant, it may lead to learning a set of identical memory templates. By combining them together, the mAP is increased by 3% at IoU = 0.5, which is much more significant than applying them independently.

Influence of the Self-attention Module. As introduced in Section 3.2, the self-attention module is designed to incorporate context information so as to encourage a smoother temporal classification score, which is important for complete action localization. From the results in Table 4, the self-attention module can consistently improve the performance at all IoU thresholds. And it is worth noting that the performance gain at IoU = 0.4, 0.5, 0.6 is more significant than that at IoU = 0.1, 0.2, 0.3. To further verify the self-attention design, we show several visualization results in Figure 3. With the self-attention module, some less discriminative action segments can be assigned higher confidence scores, and it means that the self-attention module can indeed help to improve localization completeness.

Influence of the Template Number. To explore the influence of the template number, we conduct experiments on the THUMOS14 dataset and report the average mAP at IoU 0.1:0.1:0.5 of AUMN with different template numbers. The results are shown in Figure 4, the average mAP can be consistently improved as the template number $K$ grows from 1 to 7, which means 7 templates are sufficient to model all the action units on the THUMOS14 dataset.

To better understand our method, the qualitative results of our AUMN on three videos from the ActivityNet1.2 validation set are presented in Figure 5. The action instances from left to right are javelin-throw, long-jump and high-jump respectively. We visualize the similarities between video segments with different templates in third to fifth rows. And the 6th row is the foreground attention ${\bf a}$. We find that different templates attend to model different visual patterns. For example, the first template has a high similarity to the segments which contain the action unit running while the second is similar to jumping. The third template tends to focus on throwing, which is an important action unit in javelin-throw. Interestingly, some segments of throwing are a little similar to the first template, because the man still keeps running while throwing the javelin. It is worth noting that long-jump and high-jump both contain segments about jumping, to distinguish them from each other, the segment-wise classifiers defined in Eq (6) are desired. In summary, by finding action units in untrimmed videos via the templates from memory and utilizing the segment-wise classifiers, we can correctly recognize an action and obtain robust foreground attentions for complete action localization.