Relation-aware Video Reading Comprehension for
Temporal Language Grounding

The goal of this task is to answer where is the semantically corresponding video moment given a language query in an untrimmed video. Referring to the forms of MRC, we treat the video $\mathbf{V}$ as a text passage, the query sentence $\mathbf{Q}$ as a question description and provide a set of video moment candidates as a list of answer options $\mathbf{A}$. Based on the given triplet $(\mathbf{V,Q,A})$, temporal language grounding in videos is equivalent to cross-modal MRC, termed as video reading comprehension.

For each query-video pair, we have one natural language sentence and an associated ground-truth video moment with the start $g^{s}$ and end $g^{e}$ boundary. Each language sentence is represented by $\mathbf{Q}=\{q_{i}\}^{L}_{i=1}$, where $L$ is the number of tokens. The untrimmed video is represented as a sequential snippets $\mathbf{V}=\{v_{1},v_{2},\cdots,v_{n_{v}}\}\in\mathcal{R}^{n_{v}\times C}$ by a pretrained video understanding network, such as C3D Tran et al. (2015), I3D Carreira and Zisserman (2017),VGG Simonyan and Zisserman (2014) etc..

In temporal language grounding, the answer should be a consecutive subsequence (namely time span) of the video passage. For any video moment candidate $(i,j)$, it can be treated as a valid answer if it meets the condition of $0<i<j<n_{v}$. Hence, we follow the fixed-interval sampling strategy in previous work Zhang et al. (2020b) and construct a set of video moment candidates as the answer list $A=\{a_{1},\cdots,a_{N}\}$ with $N$ valid candidates. After these notations, we can recast the temporal language grounding task from the perspective of multi-choice reading comprehension as:

However, different from the traditional multi-choice reading comprehension, previous solutions in temporal language grounding also compare their performance in terms of top-$K$ most matching candidates for each query sentence. For fair comparison, it requires our approach to scores $K$ candidate moments $\{(p_{i},t^{s}_{i},t^{e}_{i})\}_{i=1}^{K}$, where $p_{i},t^{s}_{i},t^{e}_{i}$ represent the probability of selection, the start, end time of answer $a_{i}$, respectively. Without additional mention, the video moment and answer/choice are interchangeable in this paper.

As shown in Figure 2, we describe the details of each component in our framework as followings: Modality-wise Encoder. This module aims to separately encode the content of language query and video. Each branch aggregates the intra-modality context for each snippet and token.

$\cdot$ Video Encoding. We first apply a simple temporal 1D convolution operation to map the input feature sequence to a desired dimension, which is followed by an average pooling layer to reshape the sequence into a desired length $T$. To enrich the multi-hop interaction, we use a graph convolution block called GC-NeXt block Xu et al. (2020), which aggregates the context from both temporal and semantic neighbors of each snippet $v_{i}$ and has been proved effective in Temporal Action Localization task. Finally, We get the encoded visual feature as $\hat{\mathbf{V}}\in\mathcal{R}^{C\times T}$

$\cdot$ Language Encoding. Each word $q_{i}$ of $\mathbf{Q}$ is represented with the embedding vector from GloVe 6B 300d Pennington et al. (2014) to get $\mathbf{Q}\in\mathcal{R}^{300\times L}$. Then we sequentially feed the initialized embeddings into a three-layer Bi-LSTM network to capture semantic information and temporal context. We take the last layer’s hidden states as the language representation $\hat{\mathbf{Q}}\in\mathcal{R}^{C\times L}$ for cross-modality fusion with video representation $\hat{\mathbf{V}}$. In addition, the effect of different word embeddings, is also compared in  4.4.4.

The encoded visual and textual features can be formulated as follows:

To ensure the token-aware visual feature for each choice $a_{i}$, we adopt attention mechanism to learn the token-moment relations between each choice and the query. Firstly, we adopt a 1D convolution layer to project the visual and textual features to a common space and then calculate their semantic similarities, which depict the relationships $\mathbf{R}\in\mathcal{R}^{N\times L}$ between $N$ candidates and $L$ tokens. Secondly, we generate query-related feature for each candidate based on the relationships $\mathbf{R}$. Finally, we integrate these two features of candidates for token-aware visual representation.

where $T$ denotes the matrix transpose, $\odot$ and $\otimes$ are element-wise and matrix multiplications, respectively. We add the sentence-aware $\mathbf{F}_{1}$ and token-aware features $\mathbf{F}_{2}$ as the output of this module $\hat{\mathbf{F}}_{A}$.

Concretely, we take each answer candidate $a_{i}=(t^{s}_{i},t^{e}_{i})$ as a graph node, and there is a graph edge connecting two candidate choices $a_{i}$, $a_{j}$ if they share the same start/end time spot, e.g., $t^{s}_{i}=t^{s}_{j}$ or $t^{e}_{i}=t^{e}_{j}$. An example is shown in Figure 3 (b), where neighbors of the target moment choice (the red one) is denoted as dark green in a criss-cross shape. As shown in Figure 3 (c), our model is also able to achieve the information propagation between two unconnected moment choices. For other moments (dark green) that are not connected with the target moment (red) but have overlapped, their relations can be implicitly captured with two loops, namely two graph attention layers. We can guarantee the message passing between the dark green moment and cyan moments in the first loop. And then, in the second loop, we can construct relations between cyan moments and target moment, where the information from the dark green moment is finally propagated to the red moment.

Given the choice-query $\hat{\mathbf{F}}_{A}\in\mathbb{R}^{C\times N}$, there are $N$ nodes and approximately $2TN$ edges in the graph. A GAT layer inpsired by Huang et al. (2019) is applied on the graph: for each moment, we compute attention weights of its neighbours in a criss-cross path, and average the features with the weights. The output of the GAT layer can be formulated as:

where $\hat{A}$ is the adjacency matrix of the graph to determine the connections between two moment choices, defined by the predefined answer set $A$. Answer Ranker. Since we have captured the relationship between multi-choices by GCNs, we adopt this answer ranker to predict the ranking score of each answer candidate $a_{i}$ for selecting the best-matched one. This ranker takes the query-aware feature $\hat{\mathbf{F}}_{A}$ and relation-aware feature $\hat{\mathbf{F}}_{A}^{*}$ as input and concatenate them (denoted as $\|$) to aggregate more contextual information. After that, we employ a convolution layer to generate the probability $P_{A}$ of being selected for $a_{i}$ in the predefined answer set $A$. The output can be computed as:

where $\sigma$ represents the sigmoid activation function.

Following Zhang et al. (2020b), we first calculate the Intersection-over-Union (IoU) between the answer set $A$ and ground-truth annotation $(g^{s},g^{e})$ and then rescale them by two thresholds $\theta_{min}$ and $\theta_{max}$, which can be written as:

where $g_{i}$ and $\theta_{i}$ are the supervision label and corresponding IoU between $a_{i}$ and ground-truth respectively. Hence, the total training loss function of our RaNet is:

where $p_{i}$ is the output score in $P_{A}$ for the answer choice $a_{i}$ and $N$ is the number of multi-choices. In the inference stage, we rank all the answer options in $A$ according to the probability in $P_{A}$.

TACoS. It consists of 127 videos, which contain different activities that happened in the kitchen. We follow the convention in Gao et al. (2017), where the training, validation, and testing contain $10,146$, $4,589$, and $4,083$ query-video pairs.
Charade-STA. It is extended by Gao et al. (2017) with language descriptions leading to $12,408$ and $3,720$ query-video pairs for training and testing.
ActivityNet Captions. It is introduced into the temporal language grounding task recently. Following the setting in CMIN Lin et al. (2020), we use val_1 as validation set and val_2 as testing set, which have $37,417$, $17,505$, and $17,031$ query-video pairs for training, validation, and testing, respectively.

Our RaNet is compared with recent published state-of-the-art methods: VLSNet Zhang et al. (2020a), 2D-TAN Zhang et al. (2020b), DRN Zeng et al. (2020), CMIN Lin et al. (2020), DEBUG Lu et al. (2019), QSPN Xu et al. (2019), MAN Zhang et al. (2019a), ExCL Ghosh et al. (2019), CTRL Gao et al. (2017), etc.. The top-2 performance values are highlighted by bold and underline, respectively.