Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering

Given a video $V$ and linguistic question $q$, our goal is to learn a mapping function $F_{\theta}(.)$ that returns a correct answer $\bar{a}$ from an answer set $A$ as follows:

In this paper, a video $V$ is abstracted as a collection of object sequences tracked in space and time. Function $F_{\theta}$ is designed to have the form of a hierarchical object-oriented network that takes the object sequences, modulates them with the overall video context, dynamically infers object interactions as instructed by the question $q$ so that key information regarding $a$ arises from the mix. This object-oriented representation is presented next in Sec. 3.2, followed by Sec. 3.3 describing the key computation unit and Sec. 3.4 the resulting model.

With the videos represented as object sequences, we need to design a scalable reasoning framework that can work natively on the structures. Such a framework must be modular so it is flexible to different input formats and sizes. Toward this goal, we design a generic reasoning unit called Object-oriented Spatio-Temporal Reasoning (OSTR) that operates on this object-oriented structure and supports layering and parallelism.

Algorithmically, OSTR takes as input a query representation $q$, a context representation $c$, a set of $N$ object sequences $X=\{X_{n}\mid X_{n}\in\mathbb{R}^{T\times d}\}_{n=1}^{N}$ of equal length $T$, and individual identities $\left\{n\right\}$. In practice, $X$ can be a subsegment of the whole object sequences $O$, and $c$ is gathered from the frame features $g_{t}$ constructed in Sec. 3.2. The output of the OSTR is a set of object instances of the same identity.

Across the space-time domains, real-world objects have distinctive properties (appearance, position, etc.) and behaviors (motion, deformation, etc.) throughout their lives. Meanwhile, different objects living in the same period can interact with each other. The OSTR closely reflects this nature by containing the two main components: (1) Intra-object temporal attention and (2) Inter-object interaction (see Fig. 2).

Even though the partitioning of temporal and spatial interaction in OSTR brings the benefits of efficiency and modularity, such separated treatment can cause the loss of spatio-temporal information, especially with long sequences. This limitation prevents us from using OSTR directly on the full video object sequences. To allow temporal and spatial reasonings to cooperate along the way, we break down a long video into multiple short (overlapping) clips and impose an hierarchical structure on top. With such division, the two types of interactions can be combined and interleaved across clips and allow full spatio-temrporal reasoning.

Based on this motive, we design a novel hierarchical structure called Hierarchical Object-oriented Spatio-Temporal Reasoning (HOSTR) that follows the video multi-level structure and utilizes the OSTR units as building blocks. Our architecture shares the design philosophy of hierarchical reasoning structures with HCRN [14] as well as the other general neural building blocks such as ResNet and InceptionNet. Thanks to the genericity of OSTR, we can build a hierachy of arbitrary depth. For concreteness, we present here a two-layer HOSTR corresponding to the video structure: clip-level and video-level (see Fig. 3).

In particular, we first split all object sequences $O$ constructed in Sec. 3.2 into $K$ equal-sized chunks $C=\left\{C_{1},C_{2},...,C_{K}\right\}$ corresponding to the video clips, each of $T$ frames. As the result, each chunk includes the object subsequences $C_{k}=\left\{o_{n,t}\right\}_{n=1,t=t_{k}}^{N,t_{k}+T}$, where $o_{n,t}$ are the object features extracted in Eq. 2, and $t_{k}$ is the starting time of clip $k$.

Similarly, we divide the sequence of global frame features $\left\{g_{t}\right\}_{t=1}^{L}$ into $K$ parts corresponding to the video clips. The global context $c_{k}$ for clip $k$ is derived from each part by an identical operation with the temporal attention for objects in Eqs. 3,4: $c_{k}=\textrm{temporal\_attention}\left(\left\{g_{t}\right\}_{t=t_{k}}^{t_{k}+T}\right)$.

Clip-level OSTR units work on each of these subsequences $C_{k}$, context $c_{k}^{\textrm{clip}}$and query $q$, and generate the clip-level representation of the chunk $y_{k}^{\textrm{clip}}\in R^{N\times d}$:

Outputs of the $K$ clip-level OSTRs are $K$ different sets of objects $y_{k}^{\textrm{clip}}=\left\{y_{n,k}\right\}_{n=1}^{N}$ whose identities $n$ were maintained. Therefore, we can easily chain these objects of the same identity from different clips together to form the video-level sequence of objects $Y^{\textrm{clip}}=\left\{y_{n,k}^{\textrm{clip}}\right\}_{n=1,k=1}^{N,K}$.

At the video level, we have a single OSTR unit that takes in the object sequence $Y^{\textrm{clip}}$, query $q,$ and video-level context $c^{\textrm{vid}}$. The context $c^{\textrm{vid}}$ is again derived from the clip-level context $c_{k}^{\textrm{clip}}$ by temporal attention: $c^{vid}=\textrm{temporal\_attention}\left(\left\{c_{k}^{\textrm{clip}}\right\}_{k=1}^{K}\right)$ .

The video-level OSTR models the long-term relationships between input object sequences in the whole video:

The output of this unit is a set of $N$ vectors $Y^{\textrm{vid}}=\{y_{n}^{\textrm{vid}}\mid y_{n}^{\textrm{vid}}\in\mathbb{R}^{d}\}_{n=1}^{N}$. The set is further summarized using an attention mechanism using the query $q$ into the final representation vector $r$:

We follow the common settings for answer decoders (e.g., see [9]) which combine the final representation $r$ with the query $q$ using an MLP followed by a softmax to rank the possible answer choices. More details about the answer decoders per question types are available in the supplemental material. We use the cross-entropy as the loss function to training the model from end to end for all tasks except counting, where Mean Square Error is used.

We evaluate our proposed HOSTR on the three public video QA benchmarks, namely, TGIF-QA [9], MSVD-QA [25] and MSRVTT-QA [25]. More details are as follows.

MSVD-QA consists of 50,505 QA pairs annotated from 1,970 short video clips. The dataset covers five question types: What, Who, How, When, and Where, of which 61% of the QA pairs for training, 13% for validation and 26% for testing.

MSRVTT-QA contains 10K real videos (65% for training, 5% for validation, and 30% for testing) with more than 243K question-answer pairs. Similar to MSVD-QA, questions are of five types: What, Who, How, When, and Where.

TGIF-QA is one of the largest Video QA datasets with 72K animated GIFs and 120K question-answer pairs. Questions cover four tasks - Action, Event Transition, FrameQA, Count. We refer readers to the supplemental material for the dataset description and statistics.

Implementation: We use Faster R-CNN¹¹1https://github.com/airsplay/py-bottom-up-attention for frame-wise object detection. The number of object sequences per video for MSVD-QA , MSRVTT-QA is 40 and TGIF-QA is 50. We embed question words into 300-D vectors and initialize them with GloVe during training. Default settings are with 6 GCN layers for each OSTR unit. The feature dimension $d$ is set to be $512$ in all sub-networks.

We compare the performance of HOSTR against recent state-of-the-art (SOTA) methods on all three datasets. Prior results are taken from [14].

MSVD-QA and MSRVTT-QA: Table 1 shows detailed comparisons on MSVD-QA and MSRVTT-QA datasets. It is clear that our proposed method consistently outperforms all SOTA models. Specifically, we significantly improve performance on the MSVD-QA by 3.3 absolute points while the improvement on the MSRVTT-QA is more modest. As videos in the MSRVTT-QA are much longer (3 times longer than those in MSVD-QA) and contain more complicated interaction, it might require a larger number of input object sequences than what in our experiments (40 object sequences).

TGIF-QA: Table 2 presents the results on TGIF-QA dataset. As pointed out in [14], short-term motion features are helpful for action task while long-term motion features are crucial for event transition and count tasks. Hence, we provide two variants of our model: HOSTR (R) makes use of ResNet features as the context information for OSTR units; and HOSTR (R+F) makes use of the combination of ResNet features and motion features (extracted by ResNeXt, the same as in HCRN) as the context representation. HOSTR (R+F) shows exceptional performance on tasks related to motion. Note that we only use the context modulation at the video level to concentrate on the long-term motion. Even without the use of motion features, HOSTR (R) consistently shows more favorable performance than existing works.

The quantitative results prove the effectiveness of the object-oriented reasoning compared to the prior approaches of totally relying on frame-level features. Incorporating the motion features as context information also shows the flexibility of the design, suggesting that HOSTR can leverage a variety of input features and has the potentials to apply to other problems.

To provide more insight about our model, we examine the contributions of different design components to the model’s performance on the MSVD-QA dataset. We detail the results in Table 3. As shown, intra-object temporal attention seems to be more effective in summarizing the input object sequences to work with relational reasoning than BiLSTM. We hypothesize that it is due to the selective nature of the attention mechanism – it keeps only information relevant to the query.

As for the inter-object interaction, increasing the number of GCN layers up to 6 generally improves the performance. It gradually degrades when we stack more layers due to the gradient vanishing. The ablation study also points out the significance of the contextual representation as the performance steeply drops from 39.4 to 37.8 without them.

Last but not least, we conduct two experiments to demonstrate the significance of video hierarchical modeling. “1-level hierarchy” refers to when we replace all clip-level OSTRs with the global average pooling operation to summarize each object sequence into a vector while keeping the OSTR unit at the video level. “1.5-level hierarchy”, on the other hand, refers to when we use an average pooling operation at the video level while keeping the clip-level the same as in our HOSTR. Empirically, it shows that going deeper in hierarchy consistently improves performance on this dataset. The hierarchy may have greater effects in handling longer videos such as those in the MSRVTT-QA and TGIF-QA datasets.

To provide more analysis on the behavior of HOSTR in practice, we visualize the spatio-temporal graph formed during HOSTR operation on a sample in MSVD-QA dataset. In Fig. 4, the spatio-temporal graph of the two most important clips (judged by the temporal attention scores) are visualized in order of their appearance in the video’s timeline. Blue boxes indicate the six objects with highest edge weights (row summation of the adjacency matrix $A$ calculated in Eq.6). The red lines indicates the most prominent edges of the graph with intensity scaled to the edge strength.

In this example, HOSTR attended mostly on the objects related to the concepts relevant to answer the question (the girl, ball and dog). Furthermore, the relationships between the girl and her surrounding objects are the most important among the edges, and this intuitively agrees with how human might visually examine the scene given the question.

Each word in a question length $S$ is first embedded into a vector space of dimension. The vector sequence is the input of a BiLSTM to provide a sequence of forward-backward states. The hidden state pairs is gathered to form the contextual word representations $\{e_{s}\}_{s=1}^{S}\ e_{s}\in\mathbb{R}^{d}$, where $d$ is the length of the vector. The global representation of the question is then summarized using the two end states: $q_{g}=\left[\overleftarrow{e_{1}};\overrightarrow{e_{S}}\right],\ q_{g}\in\mathbb{R}^{d},$where $[\thinspace;]$ denotes vector concatenation operation.

In this subsection, we provide further details on how HOSTR predicts answers given the final representation $r$ and the query representation $q$. Depending on question types, we design slightly different answer decoders. As for open-ended (MSVD-QA, MSRVTT-QA and FrameQA task in TGIF-QA) and multiple-choice questions (Action and Event Transition task in TGIF-QA), we utilize a classifier of 2-fully connected layers with the softmax function to rank words in a predefined vocabulary set $A$ or rank answer choices:

We use the cross-entropy as the loss function to training the model from end to end in this case. While prior works [5, 14, 9] rely on hinge loss to train multiple-choice questions, we find that cross-entropy loss produces more favorable performance. For the count task in TGIF-QA dataset, we simply take the output of Eq. 12 and further feed it into a rounding operation for an integer output prediction. We use Mean Squared Error as the training loss for this task.

In this paper, each video is split into $K=10$ clips of $T=10$ consecutive frames. This is done simply based on the empirical results. For the HOSTR (R+F) variant, we divided each video into eight clips of $T=16$ consecutive frames to fit in the predefined configuration of the ResNeXt pretrained model²²2https://github.com/kenshohara/video-classification-3d-cnn-pytorch. To mitigate the loss of temporal information when split a video up, there are overlapping parts between two video clips next to each other.

We train our model using Adam optimizer [12] at an initial learning rate of $10^{-4}$ and weight decay of $10^{-5}$. The learning rate is reduced after every 10 epochs for all tasks except the count task in TGIF-QA in which we reduce the learning rate by half after every 5 epochs. We use a batch size of 64 for MSVD-QA and MSRVTT-QA while that of TGIF-QA is 32. To be compatible with related works [15, 18, 6, 14, 4], we use accuracy as evaluation metric for multi-class classification tasks and MSE for the count task in TGIF-QA dataset..

All experiments are conducted on a single GPU NVIDIA Tesla V100-SXM2-32GB installed on a sever of 40 physical processors and 256 GB of memory running on Ubuntu 18.0.4. It may require a GPU of at least 16GB in order to accommodate our implementation. We use early stopping when validation accuracy decreases after 10 consecutive epochs. Otherwise, all experiments are terminated after 25 epochs at most. Depending on dataset sizes, it may take 9 hours (MSVD-QA) to 30 hours (MSRVTT-QA) of training. As for inference, it takes up to 40 minutes to complete (on MSRVTT-QA). HOSTR is implemented in Python 3.6 with Pytorch 1.2.0.