Entity-aware and Motion-aware Transformers for Language-driven Action Localization in Videos

We encode the input language query $S$ into entity query features $\textbf{F}_{q}^{e}$ and motion query features $\textbf{F}_{q}^{m}$. The words in $S$ are classified into three classes: entity, motion, and others, by using the part of speech tags¹¹1https://www.nltk.org/ of the words. The classification probabilities of the word $i$ are denoted by ${\textbf{p}}_{i}=[p_{i}^{e};p_{i}^{m};p_{i}^{o}]\in\{0,1\}^{3}$. For the word $i$, if its part of speech tag is related to entity (i.e., noun, adjective), then $\textbf{p}_{i}=[1,0,0]$; if its part of speech tag is related to motion (i.e.,verb, adverb), then $\textbf{p}_{i}=[0,1,0]$; otherwise, $\textbf{p}_{i}=[0,0,1]$.

The word features $\textbf{Q}=[\textbf{w}_{1},\textbf{w}_{2},\cdots,\textbf{w}_{N}]^{\top}\in\mathbb{R}^{N\times d_{w}}$ of $S$ are first initialized using the GloVe embedding  Pennington et al. (2014), where $\textbf{w}_{i}$ denotes the $i$-th word feature with dimension $d_{w}$ and $N$ denotes the word number in the language query. And then a Transformer block is used to learn the relationships between the words, given by

where $\textbf{F}_{q}=[\textbf{f}_{q,1},\textbf{f}_{q,2},\cdots,\textbf{f}_{q,N}]^{\top}\in\mathbb{R}^{N\times d}$ are the learned linguistic query features; $FC_{1}(\cdot)$ is a fully connected layer that projects the word feature from dimension $d_{w}$ to $d$; $Transformer_{q}(\cdot)$ is a standard Transformer block, as shown in Figure 3(a), which consists of multi-head self-attention, residual connection, layer normalization and feed-forward network. Finally, the entity query features $\textbf{F}_{q}^{e}=[\textbf{f}_{q,1}^{e},\textbf{f}_{q,2}^{e},\cdots,\textbf{f}_{q,N}^{e}]^{\top}\in\mathbb{R}^{N\times d}$ and motion query features $\textbf{F}_{q}^{m}=[\textbf{f}_{q,1}^{m},\textbf{f}_{q,2}^{m},\cdots,\textbf{f}_{q,N}^{m}]^{\top}\in\mathbb{R}^{N\times d}$ of the language query are calculated by

As the objects in video are more easily accessible and provide rich indication information for actions, it is natural to narrow down the searching space from all the frames to the actual relevance using the entities in the language query. So we propose an entity-aware Transformer to coarsely select the video clips that are related to the input entity queries. As illustrated in Figure 2, the entity-aware Transformer first learns relationships between video frames via cross-frame attention to provide more contextual information, then fuses the entity query features into each video frame via cross-modal attention, next, attends complementary information across different frames via cross-frame attention, and finally predicts an action-relevant score for each frame to indicate whether the frame is action-relevant or not. According to the predicted action-relevant scores, we select action-relevant video clips where the desired action may happen.

Given the coarsely located video clips by the entity-aware Transformer, we propose a motion-aware Transformer to refine the action boundaries by capturing both fine-grained local and global motion changes. As shown in Figure 2, the motion-aware Transformer first learns contextual motion information by a novel LSTM Transformer, then attends the action-relevant parts by the action-relevant score of entity-aware Transformer and fuses motion query features into them via cross-modal attention, next, captures the fine-grained motion changes and global motion changes via the LSTM Transformer, and finally predicts the action boundaries.

Specifically, given an input sequence $\textbf{V}^{in}=\{\textbf{v}_{i}^{in}\}_{i=1}^{T}$, the multi-head self-attention module of our LSTM Transformer is given by $MSA(\textbf{f}_{Q},\textbf{f}_{K},\textbf{f}_{V})=[h_{1},h_{2},\cdots,h_{n}]$ where a single head is calculated as $h_{i}=SA_{i}(\textbf{f}_{Q},\textbf{f}_{K},\textbf{f}_{V})=softmax(\textbf{f}_{Q}\textbf{f}_{K}^{\top}/\sqrt{d})\textbf{f}_{V}$, where $d$ is the dimension of intermediate features and $\textbf{f}_{\nu}=LSTM_{\nu}^{S}(\textbf{V}^{in}),\nu\in[Q,K,V]$. The $LSTM^{S}$ is a multi-scale version of the LSTM, denoted as $LSTM^{S}(\textbf{V}^{in})=[L_{1}(\textbf{V}^{in});L_{2}(\textbf{V}^{in});\cdots;L_{S}(\textbf{V}^{in})]$, where $[\cdot]$ is concatenation. The $s$-th scale LSTM is calculated by

One-time running of $L_{s}$ can update input sequence every $s$ frames, and sliding $L_{s}$ in the input sequence one frame for $s$ times can update all input sequence.

For each video $V$ and its visual features $\textbf{F}_{v}=[\textbf{f}_{v,1},\textbf{f}_{v,2},\cdots,\textbf{f}_{v,T}]^{\top}\in\mathbb{R}^{T\times d_{v}}$, the LSTM Transformer is used to learn both fine-grained local and global motion changes:

where $\textbf{F}_{v}^{m}=[\textbf{f}_{v1}^{m},\textbf{f}_{v2}^{m},\cdots,\textbf{f}_{vN}^{m}]\in\mathbb{R}^{T\times d}$ are the updated motion features that pay more attention to the motion changes; $FC_{2}(\cdot)$ is the fully connected layer used in Equation (3) that projects the visual feature from dimension $d_{v}$ to $d$; $Transformer_{m}(\cdot)$ represents the LSTM Transformer.

Given the predicted probability distribution of action boundaries $\textbf{P}_{s}^{b}$ and $\textbf{P}_{e}^{b}$ , the training objective for action boundary prediction is formulated by

where $f_{XE}(\cdot)$ is a cross-entropy function, and $(\tau_{s},\tau_{e})$ are the ground-truth boundaries. Given the inner probability $\textbf{P}^{in}$, the training objective for action frame prediction is formulated by

where $f_{BXE}(\cdot)$ is a binary cross-entropy function and $\textbf{Y}^{in}=\{y^{in}_{i}\}_{i=1}^{T}\in\{0,1\}$, when $\tau_{s}\leq i\leq\tau_{e}$, $y^{in}_{i}=1$, otherwise $y^{in}_{i}=0$. The overall objective is given by

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters.

We evaluate our method on two datasets: Charades-STA Gao et al. (2017) and TACoS Regneri et al. (2013). The Charades-STA dataset is built on the Charades dataset Sigurdsson et al. (2016) and contains 16,128 annotations, including 12,408 for training and 3,720 for test. The TACoS dataset is built on the MPII Cooking Compositive dataset Rohrbach et al. (2012) and contains 18,818 annotations, including 10146 for training, 4589 for validation, and 4083 for test.

We use the metrics of $R@n;IoU=\mu$ and $mIoU$ for evaluation. $R@n;IoU=\mu$ denotes the percentage of test samples that have at least one result whose IoU with ground-truth is larger than $\mu$ in top-n predictions and $mIoU$ denotes the average IoU over all test samples. We set $n=1$ and $\mu\in[0.3,0.5,0.7]$.

Following the previous methods, 3D convolutional features (C3D for TACoS, and I3D for Charades-STA) are extracted to encode videos. We adopt Adam Kingma and Ba (2014) for optimization with an initial learning rate of 5e-4 and a linear decay schedule. The loss weights $\lambda_{1}$ and $\lambda_{2}$ in Equation (14) are set to 1 and 10, respectively. The number of Transformer Blocks is set to 1 and 3 for early and late Transformers in entity-aware and motion-aware Transformers. The feature dimension of all intermediate layers is set to 512, the head number of multi-head self-attention is set to 8, the layer number and scale number of long short-term memory are set to 1 and 3, respectively.

We compare our method with the latest state-of-the-art methods on the Charades-STA and TACoS datasets in Table 1 and Table 2, respectively. From the results, it is interesting to observe that our method achieves the best performance in terms of all evaluation metrics on both two datasets, clearly validating the superiority of the proposed entity-aware and motion-aware Transformers on improving the localization precision via a coarse-to-fine strategy.

We perform in-depth ablation studies to evaluate each component of our method on the Charades-STA dataset. The results are shown in Table 3.

We show several examples of action localization results on the Charades-STA dataset in Figure 4 by visualizing the corresponding action-relevant scores predicted by the entity-aware Transformer where bright colors indicate higher values. From the first three cases, we see that the action-relevant scores make it easier to accurately localize the action boundaries by paying more attention to the target action in a shrunken temporal. However, in the last case, the entities (“person & cup”) remain unchanged that the action-relevant scores of different frames are similar (the margin between maximum and minimum is less than 0.1), thus contributing less to the final boundary localization. Moreover, the entity word “drink” is wrongly classified to a motion word, so that the predicted boundaries wrongly fall in the boundaries of “drink”.