Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder

Visual Question Answering (VQA) is a good benchmark for context-specific reasoning and scene understanding that requires the combined skill of Computer Vision (CV) and Natural Language Processing (NLP). Given an image and a question in natural language, the task is to answer the question by understanding cues from both the question and the image. Tackling the VQA problem requires a variety of scene understanding capabilities such as object and activity recognition, enumerating objects, knowledge-based reasoning, fine-grained recognition, and common sense reasoning, etc. Thus, such a multi-domain problem yields a good measure of whether computers are reaching capabilities similar to humans.

With the success of deep-learning in CV and NLP, many datasets [3, 14, 21, 15, 19, 44], and models [2, 4, 6, 5, 22] have been proposed to solve VQA. Most of these models perform well in the existing benchmark datasets where the train and test sets have similar answer distribution. However, recent studies show that these models often rely on the statistical correlations between the question patterns & the most frequent answer in the train set, and shows poor generalized performance (i.e., poor performance on a test set with different answer distributions than the train set [1]). In other words, they are heavily biased on the language modality. E.g., most of the existing models overwhelmingly answer “tennis” by seeing the question pattern “what sport is.. ” or answer the question “what color of the banana?” as “yellow” even though it is a “green banana,” By definition and design, the VQA models need to merge the visual and textual modalities to answer the question. However, in practice, they often answer the question without considering the image modality, i.e., they tend to have less image grounding and rely more on the question. This undesirable behavior restricts the existing VQA models being applicable in practical scenarios.

One reason for this behavior is the strong language biases that exist in the available VQA datasets [3, 14]. Hence, the models trained on these datasets will often rely on such biases [1, 33, 7, 35, 41, 9]. E.g., in the most popular and large VQAv2 dataset [14], a majority of the “what sport is” question is linked to “tennis.” Hence, upon training, the model will blindly learn the superficial correlations between the “what sport is” language pattern from the question, and the most frequent answer “tennis.” Unfortunately, it is hard to avoid these biases from the train set when collecting real-world samples due to the annotation cost. Instead, we require methods that force the model to look at the image and question equally to predict the answer. Since the main source of bias is the over-dependency of the models on the language side, some approaches such as [33, 7, 9], tried to overcome this by reducing the contribution from the language side. On the other hand, some approaches [35, 1, 41] tried to improve the visual-grounding of the model in order to reduce the language-bias. All of these existing bias reduction methods cause a performance reduction in the standard benchmark VQA datasets.

A common pipeline in existing VQA models is to encode the image and question separately and then fuse them to predict the answer. Typically the image is encoded using a pre-trained CNN (e.g., ResNet [16], VGGNet [37], etc. ), and the question is encoded by sending the word-level features to an RNN (GRU [10] or LSTM [18]). In this approach, while encoding the question, it only considers the language modality, and such a question representation has not contained any distinguishable power based on the content in the image. In other words, the question representation is not grounded in the image. E.g., with this scheme, the question “what sport is this?” has the same encoded representation irrespective of whether the image contains “tennis,” “baseball,” or “skateboarding,” etc. Since the majority of such questions are linked to the answer “tennis” in the train set, the model will learn a strong correlation between the question pattern “what sport is” to the answer “tennis,” irrespective of the image. This leads to overfitting the question representation to the most frequent answer, and the model will ignore the image altogether.

In this paper, we propose a generic question encoder for VQA, the Visually-Grounded Question Encoder (VGQE), that reduces this problem. VGQE encodes the question by considering not only the linguistic information from the question but also the visual information from the image. A visual comparison of a generic VQA model with a traditional question encoder and with VGQE is shown in Fig 1. The VGQE explicitly forces the model to look at the image while encoding the question. For each question word, VGQE finds the important visual feature from the image and generates a visually-grounded word-embedding vector. This word embedding vector contains the language information from the question word and the corresponding visual information from the image. These visually-grounded word-embedding vectors are then passed to a sequence of RNN cells (inside VGQE) to encode the question. Since VGQE considers both the modalities equally, the encoded question representation will get sufficient distinguishing power based on the visual counterpart. For e.g., with VGQE, in the case of the “what sport is this” question, the model can easily distinguish the question for “baseball,” “tennis,” or “skateboarding,” etc. As a result, the question representation itself is heavily influenced by the image, and the learning of the correlation between specific language patterns and the most frequent answers in the train set can be reduced.

The VGQE is generic and easily adaptable to any existing VQA models, i.e., replace the existing traditional language-based question encoder in the model with VGQE. In this paper, we demonstrate the ability of VGQE to reduce the language bias on three recent best performing baseline VQA models, i.e., MUREL [6], UpDn [2] and BAN [22]. We did extensive experiments on the VQA-CPv2 dataset [1] and demonstrate the ability of VGQE to push the baseline models to achieve state-of-the-art results. The VQA-CPv2 dataset is specifically designed to test the VQA model’s capacity to overcome the language biases from the train set. Further, we show the effect of VGQE in the standard VQAv2 benchmark as well. Unlike existing bias reduction techniques [1, 33, 7, 9, 41], VGQE does not show any drop in the accuracy on the standard VQAv2 dataset; instead, it improves the performance.

A major role in the success of deep-learning models lies in the availability of large datasets. Most of the existing real-world datasets for various problems will have some form of biases, and it is hard to avoid this at the time of dataset collection, due to the annotation cost. Consequently, the models trained on these datasets often over-fit to the biases between the inputs and ground truth annotations from the train set. Researchers tried different procedures to mitigate this problem in various domains. For instance, some methods focused on the biases in captioning models [17], gender biases in multi-label object classification [45], biases in ConvNets and ImageNet [38], etc. Since our work focuses on reducing the language biases in VQA models, in the following, we discuss the related works lying on the same line.

Approaches in the dataset side: In VQA, various datasets have been proposed so far such as [3, 14, 21, 24]. The main source of bias in these datasets are from the language side, i.e., the question. There exist strong correlations between some of the word patterns in the question and the frequent answer words in the dataset. For e.g., in the large and popular VQAv1 dataset [3], there exist strong language priors such as “what color..” to “white,” “is there..” to “yes,” “how many..” to “2”, etc. To reduce the language biases in VQAv1, in [14], the authors introduced a more balanced VQAv2 dataset, by adding at least two similar images with different answers for each question. However, with this additional refinement also, there still exists a considerable amount of language biases that can be leveraged by the model during training. In [1, 27, 8], the authors empirically show that a question-only model (without using the image) trained on VQAv2 shows reasonable performance, indicating the strong language priors in the dataset. In [1], the authors show that since the train and test answer distributions of VQAv2 are still similar, and the models that solely memorize language biases in the train set show acceptable performance while testing. In this regard, they introduced a diagnostic dataset, the VQA-CP (VQA under Changing Priors), to measure the language bias learned by the VQA models. This dataset is constructed with vastly different answer distributions between the train and test splits. Hence the models that rely heavily on the language priors in the train set will show poor performance while testing. [1] empirically shows that most of the existing best performing VQA models on the VQAv2 dataset show a significant drop in the accuracy when tested on the new test split from VQA-CPv2. E.g., a model with $\approx 66\%$ accuracy in the standard VQAv2 shows only $\approx 40\%$ accuracy in the bias sensitive VQA-CPv2.

Approaches in the architecture side: Several approaches have been considered in the literature to remove such biases. In [1], the authors propose a specific VQA model built upon [42], the GVQA model, consists of specially designed architectural restrictions to prevent the model from relying on the question-answer correlations. This model is complicated in design and requires multi-stage training, which makes it challenging to adapt to existing VQA models. In [33, 7], the authors propose different regularization techniques that can be applied via the question encoder to reduce the bias. In [33], the authors added an adversary question-only branch to the model, and it is supposed to find the amount of bias that exists in the question. Then, a gradient negation of the question-only branch loss is applied to remove answer discriminative features from the question representation of the original model. They also propose a difference of entropy (DoE) loss captured between the output distributions of the VQA model and the additional question-only branch. In [7], the authors propose a similar approach as in [33], named RUBi, where instead of using the DoE and question-only loss, they mask (using the sigmoid operation) the output from the question-only branch and element-wise multiplying it with the output of the original model, to dynamically adapt the value of the classification loss to reduce the bias. However, both of these regularization approaches show a decrease in the performance on the standard VQAv2 benchmark, while reducing the bias. In [35], the authors propose a tuning method, called HINT, that is used to improve the visual-grounding of the existing model. Specifically, they tuned the model with manually annotated attention maps from the VQA-HAT dataset [11]. Similar to this, in [41], the authors propose a tuning approach called SCR, which uses additional manually annotated attention maps (from the VQA-HAT [11]) or textual explanations (from the VQA-X dataset [20]) to improve the visual-grounding of the model. However, both of these approaches [35, 41] require additional manually annotated data, and collecting the same is expensive. Also, these approaches reduce the performance on the standard VQAv2. Recently, in [9], the authors proposed an approach based on language-attention and is built upon the GVQA model [1]. The language attention module splits the question into three language phrases: question type, referring objects, and specific features of the referring object. These phrases further utilized to infer the answers from the image. They claim that splitting the question into different language phrases reduces the learning of the bias. However, this approach also shows a reduction in the performance on VQAv2.

The prior works are trying to overcome the bias either by reducing the contribution from the language side as in [33, 7, 9], or by improving the visual-grounding by tuning with additional manually annotated data as in [35, 41]. One common drawback of all of these methods is that they show a significant drop in the performance on the standard VQAv2 benchmark while reducing the bias. On the contrary, our approach improves the representation power of the question encoder to make visually-grounded question encodings to reduce the bias. The advantage of such an approach is that it is not only model-agnostic but also does not sacrifice the performance on the standard VQAv2 benchmark. Also, it does not require any additional manually annotated data or tuning.

Traditional question encoders and pitfall: The widely adopted question encoding scheme in VQA is passing the word-level features to a recurrent sequence model (LSTM [18] or GRU [10]) and taking the final state vector as the encoded question. Some early works use one-hot representations of question words and pass through an LSTM such as in  [3, 13, 43], or GRU as in [4]. The drawback of using the one-hot vector representations of question words is the manual creation of the word vocabulary. Later, the usage of pre-trained word embedding vectors such as GloVe [31], BERT [12], etc., becomes popular, since they do not require any word vocabulary and also rich in linguistic understanding. This question encoding scheme, i.e., pass the pre-trained word embeddings of the question words to some RNNs, is the currently popular approach in VQA [2, 22, 6, 5, 28, 30]. However, recently, with the success of Transformer based models in various NLP tasks [12], usage of Multi-modal Transformers (MMT) are becoming popular, such as ViLBERT [26], LXMERT [39], etc. These are co-attention models working on top of Transformers [40], where while encoding the question, the words are prioritized with the guidance of the visual information.

The above-mentioned question-encoding schemes only use the language modality, i.e., the word-embeddings of the question words. As a result, the encoded question contains only the linguistic information from the question, and it cannot distinguish the questions based on their visual counterpart. This situation forces the model to learn the unwanted correlations between the language patterns in the questions, and the most frequent answers in the train set without considering the image which leads to a language side biased model. On the contrary, our question encoder, the VGQE, considers both visual and language modalities equally and generate visually-grounded question representations. A VQA model with such a question representation can reduce the over-dependency on the language priors in the train set.

The VGQE is also related to approaches such as FiLM [32], in terms of the early usage of the complementary information. The FiLM uses the question context to influence the image encoder, whereas VGQE uses the visual information inside the question encoder.

We follow the widely-adopted RNN based question encoding scheme in VQA as the base. In VGQE, instead of the traditional RNN cell, a specially designed cell called the VGQE cell is used. A VGQE cell takes the word embedding of the current question word and finds its relevant visual counterpart feature from the image, then creates a visually-grounded question word embedding. This new word embedding is passed to an RNN cell to encode the sequence information.

Before going into the VGQE cell details, we explain the image and question feature representations used by the model. The image is represented by two sets of features $V=\{v_{i}\in\mathbb{R}^{d_{v}}\}_{i\in[1,k]}$ and $L=\{l_{i}\in\mathbb{R}^{d_{w}}\}_{i\in[1,k]}$ corresponding to the CNN features of objects and embeddings of the class labels respectively, where $k$ is the total number of objects. The question is represented as a sequence of words with corresponding word embedding vectors, the word embedding of the $t^{th}$ question word is being denoted by $q_{t}\in\mathbb{R}^{d_{w}}$.

Current VQA models rely heavily on the language priors that exists in the train set, without considering the image. Such models that fail to utilize both modalities equally would likely perform poorly in real-world scenarios. We propose VGQE, a novel question-encoder that utilizes both the modalities equally and generates visually-grounded question representations. Such question representations have sufficient distinguishing power based on the visual counterpart and help the models to reduce the learning of language biases from the train set. We did extensive experiments on the bias sensitive VQA-CPv2 dataset and achieved a new state-of-the-art. The VGQE is model agnostic and can easily incorporate into existing VQA models without the need for additional manually annotated data and training. We experimented with three best performing VQA models, and deliver consistent performance improvement in all of them. Further, unlike existing bias reduction techniques, VGQE does not sacrifice the original model’s performance on the standard VQAv2 benchmark.