Interpretable Visual Question Answering
Referring to Outside Knowledge

For each of $N$ training data instances, we denote it as $n$-th instance ($n\in 1,2,...,N$). Our method takes an image $I_{n}$, a question $Q_{n}$, and a ground truth sentence $G_{n}$. Then we generate an output sentence $W_{n}$ of length $M$, where $w_{m}$ refers to $m$-th word in the sentence, which contains both the answer and explanation.

We clip the image before extracting the feature as $N_{\rm grid}\times N_{\rm grid}$ ($N_{\rm grid}$ being the size of the grid) small pieces, which are represented as $I_{n}=\{p_{1},p_{2},...,p_{N_{\rm grid}\times N_{\rm grid}}\}$. Then we use a transformer-based vision encoder ${\rm E_{V}}$ to extract the feature of the input image. The image feature $\bm{f}_{n}^{I}$ can be calculated as follows:

$$\bm{f}_{n}^{I}={\rm E_{V}}(\{p_{1},p_{2},...,p_{N_{\rm grid}\times N_{\rm grid}}\}).$$

(1)

Through the encoder $\rm E_{V}$ with an attention structure, we can extract image features and capture the relationship between regions in $\bm{f}_{n}^{I}$, which is the crucial information to answer a question and generate an explanation.

A knowledge base containing various types of knowledge is needed to retrieve the outside knowledge of images. We set our sights on Wikipedia, which collects a large amount of structured data and has become a resource of enormous value with potential applications. We finally chose a recently proposed subset from Wikidata [14] following KAT [15] as our outside knowledge set $\mathcal{K}$ in this paper.

Since the image caption and the outside knowledge are natural language contents, we use a language encoder ${\rm E_{L}}(\cdot)$ to extract their features in the model. We first use a pre-trained image captioning model ${\rm G_{C}}(\cdot)$ to generate a caption set $C_{n}=\{c_{n}^{1},...,c_{n}^{L}\}$ containing $L$ captions from the $I_{n}$. The features of each caption in $C_{n}$ are extracted using ${\rm E_{L}}(\cdot)$, and then the obtained features are joined into a caption feature vector $\bm{f}_{n}^{C}$ by summing. Next, we use a retrieval model ${\rm R}(\cdot,\cdot)$ and the query $C_{n}$ to retrieve the knowledge item set $K_{n}=\{k_{n}^{1},...,k_{n}^{P}\}$ containing $P$ retrieved knowledge items from $\mathcal{K}$.

We use a query encoder ${\rm E_{Q}}(\cdot)$ and a passage encoder ${\rm E_{P}}(\cdot)$ to encode $C_{n}$ and $\mathcal{K}$, respectively, and then use the extracted features to get the retrieved outside knowledge $K_{n}$ by using the retrieval model ${\rm R}(\cdot,\cdot)$, which can be calculated as follows:

$$K_{n}={\rm R}({\rm E_{Q}}(C_{n}),{\rm E_{P}}(\mathcal{K})).$$

(2)

To introduce the outside knowledge into the final prediction process, we extract the outside knowledge feature vector $\bm{f}_{n}^{K}$ using the ${\rm E_{L}}(\cdot)$ in the same way as extracting $\bm{f}_{n}^{C}$.

In this way, we extract the feature vectors of the image caption and outside knowledge containing the internal and external information of the image, which will then be joined with the vision vector in the final prediction process.

We use a joint vector to expand the information because feature fusion is an essential step in multimodal tasks [16]. Because of the characteristics of our vision and language encoder, the image and language features are mapped to latent shared spaces. The concatenating operation allows us to use their features best and make our model contain much information. However, even if the extracted features are in a shared space, the simple concatenation between the different modalities will still lead to difficulties in model training. Thus, we use Multi-layer Perceptron (MLP) layers to better fuse the features with considering the combination effect and learning efficiency. First, three different MLPs, ${\rm g_{I}}$, ${\rm g_{C}}$ and ${\rm g_{K}}$, are used to process different reference information separately. Then we use the concatenation operation to generate the joint vector $\bm{f}_{n}^{J}$ as follows:

$$\bm{f}_{n}^{J}=<{\rm g_{C}}(\bm{f}_{n}^{C}),{\rm g_{K}}(\bm{f}_{n}^{K}),{\rm g_{I}}(\bm{f}_{n}^{I})>.$$

(3)

Through this feature fusion process, we can join the feature vectors of the different modalities to preserve information as much as possible.

The output of our model is a sentence $W_{n}$, which contains the answer to question $Q_{n}$ and the corresponding explanation. We use a decoder ${\rm D}$ to decode $\bm{f}_{n}^{J}$ to generate the context:$\{question\}+\{answer\}+because+\{explanation\}$ as follows:

$$W_{n}={\rm D}(\bm{f}_{n}^{J},Q_{n}),$$

(4)

where $Q_{n}$ is the input, and the decoder generates the answer and the explanation $W_{n}$. To train the model, we employ the cross-entropy loss and minimize the negative log-likelihood, which can be computed as

$$\mathcal{L}=-\sum_{m=1}^{N}{\rm log}\,{p_{\theta}}(w_{t}|\bm{w}_{<m}).$$

(5)

The probability mass function is represented by ${p_{\theta}}(\cdot)$, where $\theta$ is a parameter of the model distribution. The term $\bm{w}_{<m}$ refers to the words preceding $w_{t}$. By minimizing the loss $\mathcal{L}$, our model can produce accurate explanations that closely resemble the ground truth.

$\mathbf{Dataset}$. For our experiments, we utilized the VQA-X dataset [5], an extension of the VQA-v2 dataset [1], which included explanations for each answer. The dataset comprises 33k QA pairs from 28k images sourced from the COCO2014 dataset [17]. For the dataset division, we used all the images in the COCO2014 training set with 29k QA pairs as our training set. We divided the COCO2014 validation set into our validation set and test set according to the proportion of 3:4, which contains 1.5k QA pairs and 2k QA pairs, respectively.

Compared to traditional vision models with specific tasks such as image classification and image segmentation, for the vision encoder, we only rely on their primary network function to output simple grid features rather than their time-consuming top-down features. Thus, to better adapt to the Vision & Language task, we used the CLIP based on the structure of the vision transformer as the vision encoder. The CLIP makes the fusion of vision and language features easier by encoding them in the same hidden space. After experimentally verifying two of the most widely used pre-trained language models, BERT and CLIP-text, for the language encoder, we selected the CLIP-text model, which can better match the vision and language features. We used five different models to generate multiple captions containing various information, including GIT-large [18] fine-tuned on COCO, GIT-large [18] fine-tuned on TextCaps, BLIP [19], CoCa [20] and BLIP-2 [21]. In the outside knowledge retrieval, we used the same query and passage encoder as Dense Passage Retrieval [22] and the Faiss [23] to realize the retrieval process.

All the MLPs in our model have three layers and a 128 hidden size, and we set the caption number $L$ to 5 and the outside knowledge item number $P$ to 3. During the training, we resized all images to 224 $\times$ 224 pixels and added random flips. We trained our models for 30 epochs using a batch size of 32 and a learning rate of $2\times 10^{-5}$, which was decreased to $10^{-5}$.

$\mathbf{Comparison\ Methods}$. The comparison methods answer the question and generate natural language explanations based only on the input image. The NLX-GPT [10] and E-ViL [12] have separate answering process and explanation process, and RVT and QA-only [11] combine the two processes.

$\mathbf{Evaluation\ Metrics}$. We used the accuracy to evaluate the generated answers and used the following common language modeling evaluation metrics: BLEU-n (n being 1 to 4) [24], METEOR [25], ROUGE [26], SPICE [27] and CIDEr [28]. All the scores of the language metrics were computed by the publicly available code provided by Chen et al. [29].

As shown in Table 1, the obtained results prove that our model performs better than the comparison methods. Since introducing multiple references, we conducted ablation experiments to verify the validity of each reference information and the language encoder selection. Specifically, we experimented separately with models that only introduce outside knowledge (“Ours w/o C”) and caption (“Ours w/o OK”). Experimental results show that models that introduce captions or outside knowledge as additional references perform better than models that just refer to images. However, the model that uses captions and outside knowledge (“Ours”) performs best. These results prove that if the model has more information to refer to, it can generate answers and their corresponding explanations more correctly. The results of “Ours (via BERT)” also show that using the same language encoder CLIP-text model as the vision encoder can better align language and vision features and generate better results than the BERT.

We show the qualitative comparison between our model with the state-of-the-art method, NLX-GPT, in Fig. 3. We have bolded in black and purple some critical information in the caption and outside knowledge that is helpful for the results, respectively. We can see that this information is used for the generated answers and explanations. As shown in the example in the first row, with the same correct answer, we refer to more information. Thus, our approach yields a more detailed and reasonable explanation. Meanwhile, in the second row, our method correctly answers questions that the comparison model cannot answer because of the reference to the additional outside knowledge and image caption.