Exploring Motion and Appearance Information for Temporal Sentence Grounding

Video encoder. Unlike previous detection-based methods only using Faster R-CNN (Ren et al. 2015) pre-trained on image detection dataset to extract appearance-aware object features, we consider additionally extracting motion-aware object information to obtain action-oriented features for temporal modeling. Specifically, for appearance features, we first sample fixed $T$ frames from the original video averagely, and then obtain $K$ objects from each frame using Faster R-CNN that is built on a ResNet (He et al. 2016) backbone. Therefore, there are total $T\times K$ objects in a single video, and we can represent their appearance features as $\bm{V}^{a}_{local}=\{\bm{o}^{a}_{t,k},\bm{b}_{t,k}\}_{t=1,k=1}^{t=T,k=K}$, where $\bm{o}^{a}_{t,k}\in\mathbb{R}^{D},\bm{b}_{t,k}\in\mathbb{R}^{4}$ denotes the local appearance feature and bounding-box position of the $k$-th object in $t$-th frame. Since the global feature of the whole frame also contains the non-local information of its internal objects, we utilize another ResNet model with a linear layer to generate frame-wise appearance representation $\bm{V}_{global}^{a}\in\mathbb{R}^{T\times D}$. For motion features, we first extract the feature maps of each video clip from the last convolutional layer in C3D (Tran et al. 2015) network, and then apply RoIAlign (He et al. 2017) on such feature maps and use object bounding-box locations $\bm{b}_{t,k}$ to generate motion-aware object features $\bm{V}^{m}_{local}=\{\bm{o}^{m}_{t,k},\bm{b}_{t,k}\}_{t=1,k=1}^{t=T,k=K}$. To extract the clip-wise global features $\bm{V}_{global}^{m}\in\mathbb{R}^{T\times D}$, we directly apply average pooling and linear projection to the extracted feature maps of C3D.

Query encoder. We first utilize the Glove (Pennington, Socher, and Manning 2014) to embed each word into dense vector, and then employ multi-head self-attention (Vaswani et al. 2017) and Bi-GRU (Chung et al. 2014) to encode its sequential information. The final word-level features can be denoted as $\bm{Q}=\{\bm{q}_{n}\}_{n=1}^{N}\in\mathbb{R}^{N\times D}$, and the sentence-level feature $\bm{q}_{global}\in\mathbb{R}^{D}$ can be obtained by concatenating the last hidden unit outputs in Bi-GRU.

Cross-modal interaction. Learning correlations between visual features and query information is important for query-based video grounding, which helps to highlight the relevant object features corresponding to the query while weakening the irrelevant ones. Specifically, for the $k$-th object in the $t$-th frame in the motion branch, we interact its feature $\bm{f}^{m}_{t,k}$ with word-level query features $\{\bm{q}_{n}\}_{n=1}^{N}$ by:

where $\bm{W}_{1}^{m},\bm{W}_{2}^{m}$ are learnable matrices, $\bm{b}^{m}_{1}$ is the bias vector and the $\bm{\text{w}}^{\top}$ is the row vector as in (Zhang et al. 2019). The query-enhanced object features $\widehat{\bm{f}}^{m}_{t,k}$ can be obtained by:

where $\sigma$ is the sigmoid function, $\odot$ represents element-wise product, $\bm{W}_{3}^{m},\bm{b}_{2}^{m}$ are parameters. $\widehat{\bm{F}}^{m}=\{\widehat{\bm{f}}_{t,k}^{m}\}_{t=1,k=1}^{t=T,k=K}\in\mathbb{R}^{(T\times K)\times D}$, and the enhanced object features $\widehat{\bm{F}}^{a}$ of appearance branch can be obtained in the same way.

Spatio-temporal graph. Since the detected objects have both spatial interactivity and temporal continuity, as shown in Figure 2, we construct object graph to capture spatio-temporal relations in each branch, respectively. For motion branch, we define object-wise features $\widehat{\bm{F}}^{m}=\{\widehat{\bm{f}}_{t,k}^{m}\}_{t=1,k=1}^{t=T,k=K}$ including all objects in all frames as nodes and build a fully-connected motion graph. We adopt graph convolution network (GCN) (Kipf and Welling 2016) to learn the object-relation features via message propagation. In details, we first measure the pairwise affinity between object features by:

where $\bm{W}^{m}_{4},\bm{W}^{m}_{5}$ are learnable parameters. $\bm{A}^{m}\in\mathbb{R}^{(T\times K)\times(T\times K)}$ is obtained by calculating the affinity edge of each pair of objects. Two objects with strong semantic relationships will be highly correlated and have an edge with high affinity score in $\bm{A}^{m}$. Then, we apply single-layer GCN with residual connections to perform semantic reasoning:

where $\bm{W}^{m}_{6}$ is the weight matrix of the GCN layer, $\bm{W}^{m}_{7}$ is the weight matrix of residual structure. The output $\widetilde{\bm{F}}^{m}=\{\widetilde{\bm{f}}_{t,k}^{m}\}_{t=1,k=1}^{t=T,k=K}\in\mathbb{R}^{(T\times K)\times D}$ is the updated features for motion-aware objects. Feature $\widetilde{\bm{F}}^{a}$ for appearance-aware objects can be obtained in the same way.

Object feature fusion. After obtaining the object features, we aim to integrate object features within each frame to represent fine-grained frame-level information under the guidance of query information. To this end, we compute the cosine similarity between object feature $\widetilde{\bm{f}}^{m}_{t,k}$ and the sentence-level query feature $\bm{q}_{global}$ at frame $t$ as :

where $\bm{W}_{q}$ is the linear parameter, $c^{m}_{t,k}$ indicates the relational score between the visual object and the given query. Then, we integrate the object features within each frame $t$ to represent query-specific frame-level feature as:

The final query-specific frame-level features in both appearance and motion branches can be denoted as $\bm{H}^{a}=\{\bm{h}^{a}_{t}\}_{t=1}^{T},\bm{H}^{m}=\{\bm{h}^{m}_{t}\}_{t=1}^{T}\in\mathbb{R}^{T\times D}$.

Motion-guided appearance enhancement. Compared to appearance-aware feature $\bm{H}^{a}$, the motion-aware feature $\bm{H}^{m}$ captures the temporal contexts of objects. Considering the motion context contains implicit appearance information, we abstract the motion context and integrate it into the appearance features for generating motion-enhanced appearance features. Firstly, the motion feature $\bm{H}^{m}$ is adapted to appearance feature space through a frame-independent feed-forward network (FFN) which consists of three linear layers. Then, We add the adapted motion information to the appearance feature $\bm{H}^{a}$ in an element-wise way, leading to motion-enhanced appearance feature $\widehat{\bm{H}}^{a}$. This procedure can be formulated as:

In this soft and learnable way, contexts appeared in the motion space are aggregated into the appearance feature space.

Appearance-fused motion enhancement. Compared to motion-aware feature $\bm{H}^{m}$, the feature $\bm{H}^{a}$ represents more on the appearance details and spatial location clues of a certain object. We utilize the dot-product attention to attend appearance-aware object features into motion space for inferring motion contexts as:

where $\widehat{\bm{H}}^{m}$ is the appearance-enhanced motion feature.

Query-guided motion-appearance fusion. To decide which information are most discriminative among appearance and motion features $\widehat{\bm{H}}^{a},\widehat{\bm{H}}^{m}$ corresponding to the query, we integrate them under the guidance of sentence-level query features through an attention-based weighted summation as:

where $\widetilde{\bm{H}}=\{\widetilde{\bm{h}}_{t}\}_{t=1}^{T}\in\mathbb{R}^{T\times D}$ is the integrated frame-level features to be fed into the last grounding head.

Charades-STA. Charades-STA is a benchmark dataset for the video grounding task, which is built upon the Charades (Sigurdsson et al. 2016) dataset. It is collected for video action recognition and video captioning, and contains 6672 videos and involves 16128 video-query pairs. Following previous work (Gao et al. 2017), we utilize 12408 video-query pairs for training and 3720 pairs for testing.

TACoS. TACoS is collected by (Regneri et al. 2013) for video grounding and dense video captioning tasks. It consists of 127 videos on cooking activities with an average length of 4.79 minutes. In video grounding task, it contains 18818 video-query pairs. We follow the same split of the dataset as (Gao et al. 2017) for fair comparisons, which has 10146, 4589, and 4083 video-query pairs for training, validation, and testing respectively.

Evaluation metric. We adopt “R@n, IoU=m” proposed by (Hu et al. 2016) as the evaluation metric, which calculates the IoU between the top-n retrieved video segments and the ground truth. It means the percentage of IoU greater than m.

Compared methods. To demonstrate the effectiveness of our MARN, we compared it with several state-of-the-art methods: (1) Proposal-based: CTRL (Gao et al. 2017), QSPN (Xu et al. 2019), BPNet (Xiao et al. 2021), DRN (Zeng et al. 2020), CBLN (Liu et al. 2021b); (2) Proposal-free: CBP (Wang, Ma, and Jiang 2020), GDP (Chen et al. 2020), VSLNet (Zhang et al. 2020a), IVG-DCL (Nan et al. 2021); (3) Detection-based: MMRG (Zeng et al. 2021).

Comparison on Charades-STA. We compare our MARN with the state-of-the-art proposal-based and proposal-free methods on the Charades-STA dataset in Table 1, where we reach the highest results over all evaluation metrics. Particularly, our C3D+Object variant outperforms the best proposal-based method CBLN by 4.87% and 9.86% absolute improvement in terms of R@1, IoU=0.7 and R@5, IoU=0.7, respectively. Compared to the proposal-free method IVG-DCL, the C3D+Object model outperforms it by 14.23% and 10.21% in terms of R@1, IoU=0.5 and R@1, IoU=0.7, respectively. We also compare our model with the detection-based method MMRG in Table 2. To make a fair comparison, we follow the same data splits for training/testing. It shows that our C3D+Object model brings a further improvement of 4.38% and 5.57% in terms of R@1, IoU=0.5 and R@5, IoU=0.5. We further utilize the I3D to present a new I3D+Object variant, which performs better than C3D+Object since I3D can obtain stronger features.

Comparison on TACoS. Table 1 and 2 also report the grounding results on TACoS. Compared to CBLN, our C3D+Object model outperforms it by 7.35%, 8.09%, 4.01%, and 6.94% in terms of all metrics, respectively. Our model also outperforms IVG-DCL by a large margin. Compared to the detection-based method MMRG, our C3D+Object model brings the improvements of 3.20% and 3.99% in strict metrics of R@1, IoU=0.5 and R@5, IoU=0.5, respectively. The I3D+Object variant further achieves better results.

Main ablation. We first perform main ablation studies to demonstrate the effectiveness of each component. To construct the baseline model, we utilize a general ResNet50 based Faster-RCNN for appearance-aware object extraction and another ResNet50 for frame-level global feature extraction, and do not encode any motion contexts. Instead of building the appearance branch, we employ a co-attention (Lu et al. 2016) module to interact object-level cross-modal information and simply concatenate query-object features for semantic enhancement. We also utilize another co-attention module to capture object-relations, and a mean-pooling layer to fuse object features to represent frame-level features. We utilize the same grounding head in all ablation variants. The performances of each variants are shown in Table 3, and we can observe the following conclusions: (1) It is worth noticing that the baseline model achieves better performance than all existing methods in Table 1, demonstrating that the detection-based method is more effective in distinguishing the frames with high similarity. Our object-level features filter out redundant background information in frame-level features of previous works, thus leading to fine-grained activity understanding and more precise localization. (2) The proposed appearance branch (AB) can capture more fine-grained object relations, thus bringing a significant improvement. (3) The motion encoder (ME) and the corresponding motion branch (MB) can incorporate action-oriented contexts into appearance-based features for better understanding the activity. (4) Motion-appearance association module (MAA) further brings improvement, which proves the effectiveness of incorporating appearance and motion features for bi-directional enhancement.

Analysis on the video encoder. As shown in Table 4, we conduct the investigation on different video encoding. We find that the full model performances worse if we remove the global feature learning. It demonstrates that the frame-wise features help to better explore the non-local object information in the frame. Besides, it also presents the effectiveness of the position encoding in identifying temporal semantic and improving the accuracy of temporal grounding.

Analysis on the reasoning branches. We also conduct ablation study within the reasoning branches as shown in Table 5. For object-level cross-modal interaction, our attention based mechanism outperforms the mechanism of concatenating query-object features by 1.47% and 1.38%. For spatio-temporal graph network, replacing the graph model with simple co-attention model will reduce the performance, since the latter lacks temporal modeling. We can observe that our model achieves best result when the number of graph layer is set to 1, and more graph layers will result in over-smoothing problem (Li, Han, and Wu 2018). For frame-level object-feature fusion, attention based mechanism performs better than the mean-pooling.

Analysis on the associating module. As shown in Table 6, equipped with only motion-guided appearance enhancement, there is a 1.12% and 1.04% point gain compared to the model w/o MAA. This is because the motion context contains implicit appearance information. Beside, when adopting appearance-fused motion enhancement alone, we achieve an improvement of 1.29% and 1.41%, since the appearance-aware objects can also contribute to motion modeling. Applying both of them together, the performance boost is larger than using only one of them.