Natural Language Rationales with Full-Stack Visual Reasoning:
From Pixels to Semantic Frames to Commonsense Graphs

The GPT-2’s backbone architecture can be described as the decoder-only Transformer Vaswani et al. (2017) which is pretrained with the conventional language modeling (LM) likelihood objective.³³3Sometimes referred to as density estimation, or left-to-right or autoregressive LM Yang et al. (2019). This makes it more suitable for generation tasks compared to models trained with the masked LM objective (BERT; Devlin et al., 2019).⁴⁴4See Appendix §A.1 for other details of GPT-2.

We build on pretrained LMs because their capabilities make free-text rationalization of complex reasoning tasks conceivable. They strongly condition on the preceding tokens, produce coherent and contentful text See et al. (2019), and importantly, capture some commonsense and world knowledge Davison et al. (2019); Petroni et al. (2019).

To induce conditional text generation behavior, Radford et al. (2019) propose to add the context tokens (e.g., question and answer) before a special token for the generation start. But for visual-textual tasks, the rationale generation has to be conditioned not only on textual context, but also on an image.

We first outline types of visual information and associated external models that lead to the full-stack visual understanding. Specifications of these models and features that we use appear in Table 1.

Recognition-level understanding of an image begins with identifying the objects present within it. To this end, we use an object detector that predicts objects present in an image, their labels (e.g., “cup or “chair”), bounding boxes, and the boxes’ hidden representations (Fig. 2(a)).

The next step of recognition-level understanding is capturing relations between objects. A computer vision task that aims to describe such relations is situation recognition Yatskar et al. (2016). We use a model for grounded situation recognition (Fig.  2(b); Pratt et al., 2020) that predicts the most prominent activity in an image (e.g., “surfing”), roles of entities engaged in the activity (e.g., “agent” or “tool”), the roles’ bounding boxes, and the boxes’ hidden representations.

The object detector and situation recognizer focus on recognition-level understanding. But visual understanding also requires attributing mental states such as beliefs, intents, and desires to people participating in an image. In order to achieve this, we use the output of VisualComet (Fig. 2(c); Park et al., 2020), another GPT-2-based model that generates commonsense inferences, i.e. events before and after as well as people’s intents, given an image and a description of an event present in the image.

We now describe how we format outputs of the external models (§2.2; Table 1) to augment GPT-2’s input with visual information.

We explore two ways of extending the input. The first approach adds a vision model’s textual output (e.g., object labels such as “food” and “table”) before the textual context (e.g., question and answer). Since everything is textual, we can directly embed each token using the GPT-2’s embedding layer, i.e., by summing the corresponding token, segmentation, and position embeddings.⁵⁵5The segment embeddings are (to the best of our knowledge) first introduced in Devlin et al. (2019) to separate input elements from different sources in addition to the special separator tokens. We call this kind of input fusion uniform.

This is the simplest way to extend the input, but it is prone to propagation of errors from external vision models. Therefore, we explore using vision models’ embeddings for regions-of-interest (RoI) in the image that show relevant entities.⁶⁶6The entire image is also a region-of-interest. For each RoI, we sum its visual embedding (described later) with the three GPT-2’s embeddings (token, segment, position) for a special “unk” token and pass the result to the following GPT-2 blocks.⁷⁷7Visual embeddings and object labels do not have a natural sequential order among each other, so we assign position zero to them. After all RoI embeddings, each following token (question, answer, rationale, separator tokens) is embedded similarly, by summing the three GPT-2’s embeddings and a visual embedding of the entire image.

We train and evaluate our models with different fusion types and visual features separately to analyze where the improvements come from. We provide details of feature extraction in App.  §A.4.

We ask workers to judge whether a rationale supports a given answer or entailment label in the context of the image (visual plausibility). They could select a label from $\{$yes, weak yes, weak no, no$\}$. We later merge weak yes and weak no to yes and no, respectively. We then calculate the ratio of yes labels for each rationale and report the average ratio in a sample.¹³¹³13We follow the related work Camburu et al. (2018); Do et al. (2020); Narang et al. (2020) in using yes/no judgments. We introduced weak labels because they help evaluating cases with a slight deviation from a clear-cut judgment.

We compare the text-only GPT-2 with visual adaptations in Table 3. We observe that GPT-2’s visual plausibility benefits from some form of visual adaptation for all tasks. The improvement is most visible for vqa-e, followed by vcr, and then e-snli-ve (all). We suspect that the minor improvement for e-snli-ve is caused by the entailment-data artifact. Thus, we also report the visual plausibility for entailment and contradiction cases separately. The results for contradiction hypotheses follow the trend that is observed for vcr and vqa-e. In contrast, visual adaption does not help rationalization of entailed hypotheses. These findings, together with the fact that we have already discarded neutral hypotheses due to the high error rate, raise concern about the e-snli-ve dataset. Henceforth, we report entailment and contradiction separately, and focus on contradiction when discussing results. We illustrate rationales produced by Rationale${}^{\textsc{vt}}$ Transformer in Figure 4, and provide additional analyses in the Appendix §B.

We motivate different visual features with varying levels of visual understanding (§2.2). We reflect on our assumptions about them in light of the visual plausibility results in Table 3. We observe that VisualComet, designed to help attribute mental states, indeed results in the most plausible rationales for reasoning in vcr, which requires a high-order cognitive and commonsense understanding. We propose situation frames to understand relations between objects which in turn can result in better recognition-level understanding. Our results show that situation frames are the second best option for vcr and the best for vqa, which supports our hypothesis. The best option for e-snli-ve (contradiction) is hybrid fusion of objects, although uniform situation fusion is comparable. Moreover, VisualComet is less helpful for e-snli-ve compared to objects and situation frames. This suggests that visual-textual entailment in e-snli-ve is perhaps focused on recognition-level understanding more than it is anticipated.

One fusion type does not dominate across datasets (see an overview in Table 9 in the Appendix §B). We hypothesize that the source domain of the pretraining dataset of vision models as well as their precision can influence which type of fusion works better. A similar point was recently raised by Singh et al. (2020). Future work might consider carefully combining both fusion types and multiple visual features.

It has been shown that powerful pretrained LMs can reason about textual input well in the current benchmarks (e.g., Zellers et al., 2019b; Khashabi et al., 2020). In our case, that would be illustrated with a high plausibility of generated rationales in an evaluation setting where workers are instructed to ignore images (textual plausibility).

We report textual plausibility in Table 4. Text-only GPT-2 achieves high textual plausibility (relative to the human estimate) for all tasks (except the entailment part of e-snli-ve), demonstrating good reasoning capabilities of GPT-2, when the context image is ignored for plausibility assessment. This result also verifies our hypothesis that generating a textually plausible rationale is easier for models than producing a visually plausible rationale. For example, GPT-2 can likely produce many statements that contradict “the woman is texting” (see Figure 4), but producing a visually plausible rationale requires conquering another challenge: capturing what is present in the image.

If both textual and visual plausibility of the text-only GPT-2 were high, that would indicate there are some lexical cues in the datasets that allow models to ignore the context image. The decrease in plausibility performance once the image is shown (cf. Tables 3 and 4) confirms that the text-only baseline is not able to generate visually plausible rationales by fitting lexical cues.

We notice another interesting result: textual plausibility of visually adapted models is higher than textual plausibility of the text-only GPT-2. The following three insights together suggest why this could be the case: (i) the gap between textual plausibility of generated and human rationales shows that generating textual plausible rationales is not solved, (ii) visual models produce rationales that are more visually plausible than the text-only baseline, and (iii) visually plausible rationales are usually textually plausible (see examples in Figure 4).

The best performing models for vcr and e-snli-ve (contradiction) are still notably behind the estimated visual plausibility of human-written rationales (see Table 3). Moreover, plausibility of human rationales is similar when evaluated in the context of the image (visual plausibility) and without the image (text plausibility) because (i) data annotators produce visually plausible rationales since they have accurate visual understanding, and (ii) visually plausible rationales are usually textually plausible. These results show that generating visually plausible rationales for vcr and e-snli-ve is still challenging even for our best models.

In contrast, we seem to be closing the gap for vqa-e. In addition, due in part to the automatic extraction of rationales, the human rationales in vqa-e suffer from a notably lower estimate of plausibility.

We investigate further whether visual plausibility improvements come from better visual understanding. We ask workers to judge if the rationale mentions content unrelated to the image, i.e., anything that is not directly visible and is unlikely to be present in the scene in the image. They could select a label from $\{$yes, weak yes, weak no, no$\}$. We later merge weak yes and weak no to yes and no, respectively. We then calculate the ratio of no labels for each rationale. The final fidelity score is the average ratio in a sample.¹⁴¹⁴14We also study assessing fidelity from phrases that are extracted from a rationale (see Appendix B).

Figure 5 illustrates the relation between visual fidelity and plausibility. For each dataset (except the entailment part of e-snli-ve), we observe that visual plausibility is larger as visual fidelity increases. We verify this with Pearson’s $r$ and show moderate linear correlation in Table 5. This shows that models that generate more visually plausible rationales are less likely to mention content that is irrelevant to a given image.