Learning to Agree on Vision Attention for Visual Commonsense Reasoning

Conventional VQA mainly focuses on the recognition capability of models [1, 14]. Until recently, VCR has emerged to study commonsense understanding, putting forward higher requirements in AI systems’ cognitive reasoning abilities. To address this task, many approaches have been proposed, which can be roughly divided into the following two categories.

The initial methods endeavor to devise task-specific architectures for VCR. Some of them explore a variety of sophisticated reasoning structures (e.g., holistic attention mechanism) to construct interactions between the image and text. For instance, R2C [2] performs three inference steps – grounding, contextualization, and reasoning, to effectively solve the visual cognition problem. Inspired by the neuronal connectivity of the brain, CCN [8] designs a graph method to globally and dynamically integrate the local visual neuron connectivity; HGL [6] integrates the intra-graph and inter-graph to bridge the vision and language modalities. In addition, as some questions cannot be directly answered from only the image information, external commonsense knowledge is also exploited in the cross-modal reasoning process [15].

Another prevalent stream is to apply VL-Transformers to the downstream VCR for both Q$\rightarrow$A and QA$\rightarrow$R [12, 9]. For example, UNITER [11] designs four novel pre-training tasks with conditional masking to learn universal image-text representations for various downstream multi-modal tasks. ViLBERT [10] applies a dual stream fusion encoder to process visual and textual inputs in separate streams, followed by the modality interactions with co-attentional Transformer layers. MERLOT RESERVE [16] first learns task-agnostic representations through sound, language, and vision of videos [17], and then transfers these features to the downstream VCR task.

Although both categories of methods have achieved improved results, they all view the two processes in VCR as two independent VQA instances. As a result, the critical correlations between these two are ignored, resulting in weak visual reasoning. In this work, we propose an effective attention alignment framework to bridge these two processes directly.

The past few years have witnessed increasing growth in the research area of VQA. Among the existing methods, visual attention-based ones have demonstrated substantial advantages in feature learning. Traditional approaches focus more on the correlation learning between each image region and the whole question sentence [14, 18]. Thereafter, the co-attention mechanism jointly performs the question-guided attention over image regions and the image-guided attention over question words [19]. In addition to these top-down attention methods, BUTD [20, 21] combines both the top-down and bottom-up attention, which first detects the salient objects inside an image and then leverages the top-down attention technique to locate the most relevant regions according to the given question. VQA-HAT [22] and Attn-MFH [23] point out that the attention maps produced by VQA models are inconsistent with human cognition. They, therefore, attempt to design more explicitly visual reasoning methods and have achieved certain improvements.

Transformers have achieved great success in the field of Natural Language Processing (NLP) [24] and Computer Vision (CV) [25]. Because of their effectiveness, transformers also attract much attention in multi-modal studies. Based on how the vision and language branches are fused, current VL-Transformers can be roughly categorized into single-stream (e.g., UNIMO [26] and SOHO [27]) and dual-stream cross-modal Transformers (e.g., LXMERT [28] and ALBEF [29]). In general, VL-Transformers adopt a pretrain-then-finetune learning paradigm: these models are firstly pre-trained on large-scale multi-modal datasets (such as Conceptual Captions [30]) for learning universal cross-modal representations, and then fine-tuned on downstream tasks by transferring their rich representations from pre-training. In particular, the pretext tasks play an important role in pre-training, where masked language modeling, masked region prediction, and image-text matching are extensively studied. The fine-tuning step mirrors that of the BERT model [24], which includes a downstream task-specific input, output, and objective. VL Transformers mainly help the following three groups of downstream tasks: cross-modal matching, cross-modal reasoning, and vision language generation [31]. The first group focuses on learning cross-modal correspondences between vision and language, such as image text retrieval and visual referring expression. Reasoning ones require VL Transformers to perform language reasoning based on visual scenes, such as VQA. The last group aims to generate the targets of one modality given the other as input [32]. The desired visual or textual tokens are decoded in an auto-regressive generation manner.

Given an image $I$ and a question $Q$ about this image, the goal of visual commonsense reasoning is to both predict the right answer $A_{+}$, as well as the correct rationale $R_{+}$ (the right and false ones are denoted as $+$ and $-$, respectively). In general, VCR comprises two multiple-choice processes: question answering (Q$\rightarrow$A) and answer justification (QA$\rightarrow$R).

Q$\rightarrow$A aims to predict the correct answer $A_{+}$ from a set of answer choices $\mathcal{A}$ to the given question $Q$ upon the image $I$, which can be achieved by,

where $f$ denotes the Q$\rightarrow$A model. Specifically, both question $Q$ and candidate answer $A_{i}$ are expressed in terms of textual sentences, and the image $I$ consists of $n$ objects $\mathcal{O}=\{o_{i}\}_{i=1}^{N}$ detected by a Mask-RCNN model [33]. Previous methods train $f$ by minimizing the cross entropy loss⁴⁴4We employ the single instance loss rather than the batch-wise one for simplicity.,

where $y[A_{+}]$ denotes the index of the ground-truth answer.

QA$\rightarrow$R is different from Q$\rightarrow$A wherein the input is now composed of question $Q$ and its correct answer $A_{+}$ (a straightforward way is to concatenate these two together). The objective of QA$\rightarrow$R is to select the correct rationale $R_{+}$ from a rationale set $\mathcal{R}$ (see Figure 1),

where $g$ denotes the QA$\rightarrow$R model. The optimization function is similarly defined as follows,

where $y[R_{+}]$ represents the ground-truth rationale for the given image and question.

Note that both $f$ and $g$ share identical structures and are often trained separately [8, 34]. To bridge these two connected processes, we propose a visual attention alignment module in the next.

Inspired by human cognition, we argue that the visual evidence exploited in answering and justification should be the same. As these two processes share the same image (see Equation 1 and Equation 3), the fundamental visual attention thus offers a natural bridge for connecting them. In view of this, we propose to align the visual information based on the attention maps calculated by $f$ and $g$.

Visual attention is an integral component for current vision-language models [20, 35]. A typical VCR model often involves a visual attention module on the object set $\mathcal{O}$. In particular, the visual attention learns a set of attention score $\{c_{i}\}_{i=1}^{N}$ for each image object, where $\sum_{i=1}^{N}c_{i}=1$. A large $c_{i}$ usually represents that the $i$-th object has more influence for answering the current question. As discussed earlier, there are two sets of attention maps: $\mathcal{C}_{Q\mapsto A}$ – the attention weights from model $f$, and $\mathcal{C}_{QA\mapsto R}$ – the attention weights from model $g$. To achieve the goal that $f$ and $g$ employ the similar image regions, our idea is to learn another module that drives $\mathcal{C}_{Q\mapsto A}$ and $\mathcal{C}_{QA\mapsto R}$ closer. Specifically, we implement this by pulling the similarity of attention maps from the correct answer ${A_{+}}$ and the correct rationale ${R_{+}}$, while pushing away other negative pairs.

In this paper, we propose an attention alignment loss to make the Q$\rightarrow$A and QA$\rightarrow$R learn to agree on visual regions. We formalize our attention alignment loss below,

where $\mathcal{C}_{Q\mapsto A}^{A_{i}}$ represents the attention weights from model $f$ according to the $i$-th answer $A_{i}$; $\mathcal{C}_{QA\mapsto R}^{R_{i}}$ denotes the attention weights from model $g$ according to the $i$-th rationale $R_{i}$, and $s(\cdot,\cdot)$ is a similarity function. We detail how we implement $s(\cdot,\cdot)$ in the following:

• •

  One intuitive approach to measure the similarity between two sets of attention weights is the dot product. We refer to this method as Align-Dot,

  $$s(\mathbf{c}_{p},\mathbf{c}_{t})=\mathbf{c}_{p}^{T}\mathbf{c}_{t}.$$

• •

  Inspired by the learning to rank models in the field of information retrieval [36], we use the list-wise approach to align the ranking of attention weights and name this method Align-Rank. To this end, we first obtain the permutations of the two attention vectors prediction $\bm{\pi}_{p}$ and target $\bm{\pi}_{t}$. Thereafter, we employ the NDCG metric optimization [37, 38] in our experiments:

  	$\displaystyle s(\bm{\pi}_{p},\bm{\pi}_{t})$	$\displaystyle=NDCG(\bm{\pi}_{p},\bm{\pi}_{t})$		(6)
  		$\displaystyle=Z_{N}^{-1}\sum_{i=1}^{N}\frac{G(\pi_{pi})}{\log(1+\pi_{ti})},$

  where $G(\pi_{pi})$ denotes a gain function, e.g., $G(\pi_{pi})=2^{\pi_{pi}}-1$, $Z_{N}^{-1}$ acts as the maximum of $G(\pi_{pi})/\log(1+\pi_{ti})$, i.e., the value when the predicted attention permutation of $\bm{\pi}_{p}$ is the same as the target one $\bm{\pi}_{t}$. Nevertheless, directly optimizing NDCG with back-propagation is impossible due to its non-differentiable nature. To approach this problem, we then smooth the permutation function with [38],

  $$\hat{\pi}_{pi}=1+\sum_{j\neq i}\frac{\exp\{-\alpha(c_{pi}-c_{pj})\}}{1+\exp\{-\alpha(c_{pi}-c_{pj})\}},$$

  (7)

  where $\alpha$ is a hyper-parameter, and $c_{pi}$ denotes the attention value of the $i$-th object.

By combining the aforementioned loss functions, the final objective becomes,

where $\lambda$ is a trade-off hyper-parameter.

Our method is applicable to both vanilla visual attention models as well as the most recent VL-Transformers. In the following, we will show its implementation on these two typical models.

Before the prevalence of VL-Transformers in VCR, conventional methods all adopt the vanilla attention mechanism to focus on the most salient image regions based on the textual information. In view of this, we intend to explore whether our proposed visual attention alignment method works under such settings. Specifically, we take the TAB-VCR [7] as a typical baseline to implement our method. Note that the two processes, i.e., Q$\rightarrow$A and QA$\rightarrow$R share the same structure, we, therefore, use the Q$\rightarrow$A as an example as the QA$\rightarrow$R can be easily extrapolated.

VL-Transformers have been widely studied over the past few years [46, 47]. In this section, we show how our attention alignment method works under such self-attention settings.

We conducted extensive experiments on the VCR benchmark, a large-scale dataset alongside this task. The images are extracted from the movie clips in LSMDC [49] and MovieClips⁵⁵5youtube.com/user/movieclips., wherein the objects inside images are detected via the Mask-RCNN model [33]. We used the official dataset split, where the number of questions for training, validation, and testing are 212,923, 26,534, and 25,263, respectively. For each question, four answers are given with only one being correct, and there are also four rationale choices among which only one makes sense.

Regarding the evaluation metric, we used the popular classification accuracy for Q$\rightarrow$A, QA$\rightarrow$R, and Q$\rightarrow$AR⁶⁶6For Q$\rightarrow$AR, the prediction is right only when both the answer and rationale are selected correctly.. The ground-truth labels are available for the train and validation sets [2]. Therefore, we reported the performance of our best model on the testing set once and performed other experiments on the validation set.

The PyTorch toolkit [50] is leveraged to implement our models, and all the experiments were conducted on a single GeForce RTX 2080 Ti GPU. The specific details of each model are shown below.

Vanilla attention model. As for the input features, we employed the pre-trained object-level features extracted by TAB-VCR [7] as the visual features, and the pre-trained token embeddings from BERT as the textual features. All the trainable parameters are initialized with the default PyTorch settings. The training batch size is set to 96 and the alignment loss weight $\lambda$ is set to 1.0. The parameters are optimized with the Adam [51] optimizer with an initial learning rate $2\times 10^{-3}$.

VL-Transformers. In order to verify the generalizable capability of our attention alignment mechanism on VL-Transformers, we employ several classical VL-Transformer models [11, 12, 13] as the backbone. For a fair comparison, we strictly followed the original implementations.

We evaluated our method by comparing its performance with three kinds of methods: (1) advanced VQA models. (2) Traditional VCR baselines. (3) VL-Transformers in VCR. The results on both validation and testing sets are reported in Table I, and the key observations are as follows.

• •

  First, a significant performance gap exists between traditional strong VQA models and VCR methods, especially for QA$\rightarrow$R. This is because VCR demands higher-order reasoning capability, which differs from the simple recognition in VQA. In addition, predicting the right rationale is even more challenging.

• •

  Second, compared to the task-specific VCR models such as R2C [2] and CCN [8], VL-Transformers demonstrate advantages in all metrics, indicating the effectiveness of the universal cross-modal representation learned by pre-training.

• •

  Lastly, our method achieves the best performance over these state-of-the-art models. Especially, for both the vanilla attention and VL-Transformer VCR baselines, with our attention alignment mechanism, they can all achieve improved gains on both validation and test sets. For example, compared with TAB-VCR, an absolute improvement of 1.1$\%$, 1.2$\%$ and 1.6$\%$ can be observed on Q$\rightarrow$A, QA$\rightarrow$R and Q$\rightarrow$AR respectively.

To better illustrate the effectiveness of our model in detail, we conducted experiments on the essential parts of our method and reported the results on the validation set below.

Efficacy of the attention alignment. Figure 5 demonstrates the effect of our attention alignment mechanism for both the vanilla attention model and VL-Transformers. From this figure, we can observe that among all the variants, our attention alignment mechanism consistently improves the base model on all metrics by a large margin. Take the vanilla attention model as an example, our attention alignment mechanism boosts it by 1.7% (Q$\rightarrow$A), 1.7% (QA$\rightarrow$R), and 2.5% (Q$\rightarrow$AR). One main limitation of these baselines is that they neglect the visual consistency and the interactions between the answering and reasoning. In contrast, our visual attention alignment offers a bridge to connect these two processes and significantly enhances the baseline performance.

Figure 4 shows the convergence of answering and reasoning accuracy for baselines and our GIST model. It can be seen that the performance of baselines increases fairly or faster than our GIST model. Nevertheless, with more training steps, GIST outperforms all the baselines by a significant margin. One possible reason is that the baselines show certain disadvantages in these difficult instances as the model keeps training. In contrast, when aligning the visual attention between Q$\rightarrow$A and QA$\rightarrow$R, the model learns to leverage more accurate visual information to perform visual understanding, leading to more improvements.

Align-Dot v.s. Align-Rank. As discussed in Section III-B, we applied two candidate alignment loss functions to align the two attention maps from the two processes. Table II reports the results obtained from different alignment losses. One can see that both loss functions bring certain performance improvements over respective baselines. Specifically, for the vanilla attention model, the Align-Dot, which is more demanding, can achieve significantly better results. We suspect that one possible reason is that the carefully designed re-attention module can capture the attention precisely.

Re-Attention of the vanilla attention model. In order to investigate the effectiveness of our re-attention module in the vanilla attention model, we designed two variants and reported the results in Table III. After removing the token-wise attention module from our model, we can observe performance degradation on all three metrics. It validates that different textual tokens contribute distinctively to visual feature learning. We then replaced all the attention computation with the mean operation and showed the results in the last row of this table. One can see that the model performance drops sharply compared with the previous two.

In this section, we study the influence of the trade-off hyper-parameter $\lambda$ on the performance of our GIST model. The results on both the vanilla attention model and UNITER are shown in Figure 6. As we increase the loss weight $\lambda$, the model performance keeps being enhanced. This result demonstrates the effectiveness of our designed attention alignment mechanism. However, a too-large weight will lead to a deteriorating result. For instance, a loss weight larger than 0.4 for UNITER negatively hurts the model performance.

It is expected that our attention alignment operation should pull the attention maps from the two processes similarly so that the model prediction can be made according to consistent visual cues. To justify this, we performed some qualitative results from the following two angles.

Similarity of attention maps. We leveraged the Align-Dot to perform the attention alignment and estimated the attention similarity between Q$\rightarrow$A and QA$\rightarrow$R. The histogram of the similarity values is shown in Figure 7. As can be observed, the attention similarity from baseline models is mostly less than 0.2. In contrast, our GIST model yields more consistent visual reasoning results as the similarity is increased significantly.

Attention map visualization. To gain a deeper insight into our attention alignment mechanism, in this section, we also provide some qualitative examples in Figure 8 to compare the attention distribution of the baseline and our method. Regarding the first instance, the baseline puts more attention on the lamp regions while ignoring the critical person areas. However, though the answer is wrongly predicted, the rationale is unexpectedly selected correctly. This supports our argument that the answer prediction and rationale selection should depend on the same evidence otherwise may lead to unpredictable results. As to the second instance, without the attention alignment, the baseline focuses more on the chair1 object, which is less relevant to the given question. Our GIST model corrects this mistake and obtains the right selection for both Q$\rightarrow$A and QA$\rightarrow$R. The last one shows that the attention weights are distributed incorrectly for both processes. With the consistent alignment of our method, both Q$\rightarrow$A and QA$\rightarrow$R can be accurately predicted.