Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanations

We first adopt a within-chunk semantic interactor to exploit the rich semantics contained in chunks. Each token has access only to the tokens in the same chunk group as itself. For example, as showed in Figure 2, “road” can only attend to “a road sign”. Specifically, for the $i$-th token $w_{i}$ in the $k$-th chunk, we calculate its attention score to the $j$-th token $w_{j}$ as follows,

where $\textbf{h}_{i}^{w}$ denotes the representation of $i$-th token, $d$ denotes the dimension of the h, $b_{k,s}$ and $b_{k,e}$ denote the start and end border of $k$-th chunk, respectively. $\textbf{W}_{1}^{q},\textbf{W}_{1}^{k}\in\mathbb{R}^{d\times d}$ are projection matrices. If $w_{i}$ and $w_{j}$ are not in the same chunk, the attention score between them is set to an infinitely negative value so that the resulting attention score after softmax becomes zero.

Then we apply softmax on $u_{i}$ to obtain the normalized scores $\alpha_{i}$, by which we gain the correlation between $w_{i}$ and other tokens. We aggregate the within-chunk semantics into each token,

where $\tilde{\textbf{h}}_{i}^{w}$ denotes the updated representation of $\textbf{h}_{i}^{w}$, $M$ is the number of tokens, $\textbf{W}_{1}^{v}\in\mathbb{R}^{d\times d}$ is the projection matrix.

The within-chunk semantic interactor explicitly integrates sentence constituents into $\textbf{H}^{w}$, which helps learn the local semantics. To model the relationship between image regions, previous works (Schuster et al., 2015; Liu et al., 2020; Ge et al., 2021) use scene graph parsers to encode regions into visual graphs. We adopt some advanced scene graph parsers approaches (Yang et al., 2018; Zhang et al., 2019) but find the dependencies between regions are ambiguous. These fuzzy relationships can cause error propagation, leading to sub-optimal performance. So we consider each image region as a separate vector, and each region can attend to other regions without limitation. In this way, we integrate the image information into each region representation.

After obtaining the chunk-level semantics via within-chunk semantic interactor, we utilize a cross-chunk semantic interactor to incorporate the global information into each token. We consider each token as the smallest unit and calculate the attention scores of $w_{i}$ to the concatenated text-image sequence as follows,

where $\textbf{h}_{j}$ denotes the representation of the concatenated sequence, $\textbf{W}_{2}^{q},\textbf{W}_{2}^{k}\in\mathbb{R}^{d\times d}$ are projection matrices.

Then we apply softmax on $u_{i}$ to obtain the normalized scores $\alpha_{i}$, and aggregate the cross-chunk semantics into each token,

where $\textbf{W}_{2}^{v}\in\mathbb{R}^{d\times d}$ is the projection matrix.

For image representations $\textbf{H}^{r}$, we consider each region as the smallest unit and update $\textbf{H}^{r}$ in a similar way. The cross-chunk semantic interactor helps learn the inter-chunk semantics, fusing the global information in a coarse level.

Cross-modal semantic interactor aims to conduct semantic vision-language fusion in a fine level. Unlike previous works (Park et al., 2018; Chen et al., 2020; Li et al., 2020) that consider each token separately and build vision-language alignments on token-level, we consider each text chunk as a component to build chunk-level alignments. More specifically, we use the average of the tokens in $k$-th chunk as the chunk representation $\textbf{v}_{k}$,

In this way, we aggregate the semantics within the same chunk. Thereafter, we calculate the relative attention of the $k$-th chunk to the $i$-th region,

where $\textbf{h}_{j}^{r}$ denotes the representation of $j$-th region, $\textbf{W}_{3}^{q},\textbf{W}_{3}^{k}\in\mathbb{R}^{d\times d}$ are projection matrices. $u_{kj}$ measures the correlation between $k$-th chunk and $i$-th region, by which we build chunk-level semantic alignments.

We apply softmax on $u_{k}$ to obtain the normalized scores $\alpha_{k}$, by which we capture the most salient regions related to the $k$-th chunk. We aggregate these regions to each token via $\alpha_{k}$,

where $\textbf{h}_{j}^{r}$ denotes the representation of the $j$-th region, $N$ is the number of regions, $\textbf{W}_{3}^{v}\in\mathbb{R}^{d\times d}$ is projection matrix.

Cross-modal semantic interactor incorporates each chunk and its semantically related regions, which helps remove the ambiguous vision-language alignments and obtain the high-order vision-semantic representations.

To improve the accuracy of semantic vision-language alignments, we pre-train CSI on the Flickr30k Entities dataset (Plummer et al., 2015). Flickr30k Entities dataset provides the alignments between noun phrases and image regions, where a phrase is aligned to only one region. Note that Flickr30k is also the source corpus of e-SNLI-VE (Kayser et al., 2021), so we split the Flickr30k Entities dataset along e-SNLI-VE to avoid data leakage. During pre-training, we assume that each token should attend to the most semantically relevant region. We sum the attention weights of the cross-modal semantic interactor layers of $w_{i}$ and apply softmax on it to get the normalized align score $s_{i}$. We utilize cross-entropy to enforce the alignment:

where $s_{ij}$ is the align score of $i$-th token to $j$-th region, $z_{ij}\in\{0,1\}$ is the label that indicates whether $i$-th token and $j$-th image origin should be aligned (i.e. 1) or not (i.e. 0), $M$ is the number of input tokens and $N$ is the number of image regions.

The optimization procedure of CALeC contains two stages. First, we train CSI and the relation inferrer for relation inference until the cross-entropy loss converges:

where $y^{g}_{i}\in\{0,1\}$ is the label of the $i$-th relation (e-SNLI-VE) or answer (VQA and VCR), $\tilde{y}_{i}$ is the probability of the $i$-th relation.

Then we freeze the their parameters and train LeCG for explanation generation. We minimize the negative log-likelihood of LeCG:

where $L$ denotes the length of the explanation, $e_{i}$ denotes the target token at time step $t$.

To enhance the constraints, we utilize a constrained beam sample during testing. Conventional beam sample (Holtzman et al., 2020) generates a sentence with the highest probability, which ignores the faithfulness of generated explanation. To alleviate this problem, we propose a constrained beam sample that scores each beam with the probability and the number of occurrences of the constraint words. In every step, we multiplied a constraint coefficient $\lambda$ to the candidate who generates a word that is in the lexical constraint set $\mathcal{S}$. By this way, the candidate who meets more constraints will have a higher score. We choose the candidate with the highest score as the output. The pseudo-code is in Algorithm 1.

Following the benchmark e-ViL (Kayser et al., 2021) for vision-language tasks with NLE, we evaluate our method on the VE-NLE dataset e-SNLI-VE (Kayser et al., 2021) and two VQA-NLE datasets VQA-X (Park et al., 2018) and VCR (Zellers et al., 2019). e-SNLI-VE is the current biggest VE-NLE dataset that combines SNLI-VE (Xie et al., 2019) and e-SNLI (Camburu et al., 2018). The training, validation, and test sets contain 401.7k/14.3k/14.7k image-text pairs, respectively. There are three relations of the input pair: entailment, contradiction and neutral. VQA-X is a subset of the VQA v2 dataset (Goyal et al., 2017), in which each sample contains an image, a question, an answer, and the corresponding explanation. The training, validation, and test sets contain 29.5k/1.5k/2k image-text pairs, respectively. VCR provides an image, a question and a list of annotated objects. For each question, a model needs to select one answer from four candidates. After that, it needs to select one explanation from four candidates. The test set for VCR is not publicly available. e-ViL (Kayser et al., 2021) reorganizes the dataset, and reformulates the explanation selection task as a generation task. The training, validation, and test sets contain 191.6k/21.3k/26.5k image-text pairs, respectively.

Following the e-ViL benchmark, we define three evaluation scores $S_{T}$, $S_{E}$, and $S_{O}$. $S_{T}$ represents the inference accuracy. $S_{E}$ represents the average explanation score of examples inferred correctly. This assumes that an explanation is wrong if it justifies an incorrect answer (Kayser et al., 2021). We adopt BLEU-4 (Papineni et al., 2002), ROUGE-L (Lin, 2004), METEOR (Banerjee and Lavie, 2005), CIDEr (Vedantam et al., 2015) and SPICE (Anderson et al., 2016) as the explanation scores. All scores are computed with the publicly available code¹¹1https://github.com/tylin/coco-caption. $S_{O}$ represents the overall performance, which is defined as $S_{O}=S_{T}\times S_{E}$.

Similar to the e-ViL benchmark, we compare our method with five strong baselines. Pointing and Justification (PJ-X) (Park et al., 2018) uses a simplified MCB model (Fukui et al., 2016) as the vision-language encoder and an LSTM-based language model as the decoder. Faithful Multimodal Explanations (FME) (Wu and Mooney, 2019) requires the answer and explanation to focus on the same image regions. It utilizes an improved Up-Down VQA model (Anderson et al., 2018) for answer inference, and an LSTM-based language model for explanation generation. Rationale-VT Transformer (RVT) (Marasović et al., 2020) utilizes different vision-language models to extract vision information and feeds the encoded representations with the question and ground-truth answer to the pre-trained GPT-2 (Radford et al., 2019). Note that RVT omits the question answering part, so we directly quoted the results from the e-ViL benchmark, which extends RVT with Bert (Devlin et al., 2018) to obtain the answer. e-UG (Kayser et al., 2021) combines the powerful pre-trained vision-language model UNITER (Chen et al., 2020) and GPT-2. NLX-GPT (Sammani et al., 2022) utilizes a large-scale pre-trained language model to generate the answer and explanation simultaneously.

We adopt $\text{Oscar}_{base}$ as the vision-language pre-trained model. We also utilize its parameters to initialize CSI. The number of layers of within-chunk, cross-chunk, and cross-modal semantic interactors are 3/6/3. We use a tagging model (Poth et al., 2021) pre-trained on Chunk-CoNLL2000 (Tjong Kim Sang and Buchholz, 2000) to get the text chunk borders. The number of attention layers of the relation inferrer is 3. We adopt $\text{GPT-2}_{base}$ (Radford et al., 2019) as the transformer-based language model in LeCG and randomly initialize the parameters of cross-attention sub-layers. We regard the input text and answer as prefix information and concatenate them before the explanation. For training, we use the Adam optimizer (Kingma and Ba, 2015) with the $10^{-5}$ initial learning rate and linear decay of the learning rate during CSI pre-training and CALeC training pipeline. To maintain the semantic alignment ability of CSI, the initial learning rate of CSI during the training pipeline is set to $10^{-6}$. The beam size and top-k of beam sample²²2https://huggingface.co/transformers/internal/generation_utils are set to 5 and 32. The constraint coefficient $\lambda$ is set to 0.86.

We compare our proposed method CALeC against five strong methods on three datasets. The automatic evaluation results are shown in Table 1. We can see that CALeC achieves the best performance, substantially surpassing all the baseline on $S_{O}$. By effectively performing chunk-aware semantic alignment and conducting inference over the fine-grained vision-language alignments, CALeC outperforms the strongest baseline model by $2.64$, $3.31$, and $3.23$ points on $S_{T}$ metric across the three datasets, respectively. Though the three datasets focus on different vision-language tasks, CALeC gains accuracy improvement all over them. It suggests that building accurate semantic alignment is a common yet crucial backbone for vision-language models. CALeC also surpasses the state-of-the-art model NLX-GPT on $S_{E}$ of the three datasets. This verifies that explicitly guiding the generator through lexical constraint can help improve the quality of generated explanations. We observe that $S_{E}$ of FME in VCR is slightly higher than CALeC. This may be attributed to the lower $S_{T}$ of FME, so FME only needs to count the more accessible samples when calculating $S_{E}$. Note that NLX-GPT does not provide its inference accuracy on the VCR dataset, so we calculate the scores based on their released output results³³3https://github.com/fawazsammani/nlxgpt. The $S_{T}$ of NLX-GPT is exceptionally low in VCR. This probably because that the answer of VCR is much longer than the other datasets, so it is harder for NLX-GPT to generate the correct answer.

We conduct ablation experiments to verify the effectiveness of CSI, relation inferrer, and LeCG in CALeC, which are presented in Table 2. We only list $S_{O}$ because it summarizes the performance on both $S_{T}$ and $S_{E}$. For a fair comparison, all the evaluated models have the same experimental settings and generate explanations through the beam sample algorithm. The second line verifies the effectiveness of the constrained beam sample. We can see that adding constraints to the conventional beam sample algorithm can help improve the quality of generated explanations. The third line shows the results when we drop LeCG and only retain the transformer-based generator. LeCG has a more significant influence than the constrained beam sample, indicating that directly guiding the generator with constraints can perform better than the post-hoc edit method. When we drop LeCG and the constrained beam sample simultaneously, the decrease in the overall score (1.69) is almost equal to the sum of separate reductions (1.79). This phenomenon shows that these two constrained approaches act at different but complementary points during generation and can jointly improve the quality of explanations. The constraint set is formed based on the relation inferrer and CSI, so they cannot be dropped solely. We drop the relation inferrer along with the constrained methods, in which we directly utilize the linear classifier on the concatenation of the two [CLS] outputs. The scores on the three datasets all decrease, indicating that the relation inferrer can better incorporate the fine-grained alignments of different level. We then drop CSI along with other components, in which the model degenerates into the vanilla transformer-based seq2seq model, i.e., Oscar-GPT. There is a 1.39 net decrease compared to just dropping the relation inferrer, which is higher than other components. This result shows that chunk-aware semantic alignment can greatly benefit vision-language tasks with NLE.

The automatic NLG metrics do not always reflect the faithfulness of the explanations because explanations can come in different forms and be very generic and data-biased. So we adopt human evaluation to evaluate the faithfulness of explanations. We conduct human evaluation on the test set of e-SNLI-VE, because we do not find the public result of the baseline models on other datasets. Following the e-ViL benchmark, we randomly select 100 test samples with correctly predicted answers. We ask annotators “Given the image and the hypothesis, does the explanation justify the answer?” with four choices: Yes, Weakly yes, Weakly no and No. To ensure the fairness of assessment, the explanations of each sample are shuffled. As shown in the last bar of Figure 3, CALeC gets about 77% positive scores (green region), and about 55% of them are strongly positive (dark green region), which far surpasses other models. The results indicate that our explanations can justify the answer better and reflect the inference process faithfully. We also conduct a human evaluation on CALeC w/o LeCG (the next-to-last bar). The proportion of Yes obviously decreases and the proportion of negative choices increases. This phenomenon verifies that adding constraints on explanation generation can guide the generator to focus on the input and generate explanations faithful to the inference process.

In Figure 4, we show an example with the inference result and explanations of each model on e-SNLI-VE. In this example, CALeC is the only model that infers the correct relation and generates a faithful explanation. In contrast, e-UG mistakes a house for a shop and generates an illogical explanation, and NLX-GPT predicts the wrong answer. In Figure 5, we show some qualitative results from our model on the three datasets. Based on the semantic alignments, the relation inferrer can accurately find the keywords (bold words in input text). LeCG can generate faithful explanations relevant to the inference process and input pair. We observe that although we only provide alignments for noun chunks during pre-training for CSI, it can learn alignments for other part-of-speech chunks (e.g. is giving) during fine-tuning, which may benefit from the cross-chunk semantic interactor.