Learning content and context with language bias for Visual Question Answering

In extra-based methods, depending on extra human supervision is the most straightforward solution to increase the image dependency. HINT [8] optimizes the alignment between human attention maps [15] and gradient-based network importance to reduce reliance on language bias. By human textual explanations [16], SCR [9] criticizes the sensitivity of incorrect answers to the influential object for the same purpose. In order to better guide models to get answers, some extra auxiliary tasks are added to the existing VQA models. To some extends, SCR and HINT also are seen as adding extra visual grounding task to guide VQA models. RankVQA [10] adds a image caption generator for reranking candidate answers from typical VQA model [7]. SSL-VQA [11] introduces Question-Image correlation estimation as auxiliary task to overcome language prior in a self-supervised learning framework. In addition, as causal inference has gradually attracted attention [4, 17],  [14] proposes a Counterfactual Samples Synthesizing training scheme to generate extra samples. Typically, extra-based methods can achieve better performance than non-extra methods, since they allow models to receive extra supervision, understand the task from an extra angle, or “see” extra samples.

Different from extra-based methods, non-extra-based methods does not introduce information outside VQA datasets. Currently, the mainstream methods to alleviate language bias are ensemble-based methods: they introduce a branch model and conduct joint training with the target VQA model by specific learning strategies. In adversarial training scheme, AReg [5] trains the VQA model and the question-only model which captures language bias by sharing the same question encoder. But this training scheme also affects the representations of questions and the stability of training [18]. Rubi [12] block the back propagation from the question-only model to the question encoder, and fuse the predicted answer distributions of the two branches to construct the loss function. Instead of the question-only model, LMH [13] introduces a pre-trained bias-only model to enhance the robustness of VQA model in a more targeted manner. In general, the current ensemble-based methods try to add an auxiliary branch to capture language bias so as to weaken its effect. However, this also impairs part of models’ ability in learning context prior. For achieving a balance, we propose CCB learning strategy.

The common formulation of VQA task is often considered as a multi-class classification problem. Specifically, given a VQA dataset $D$ consisting of N triplets $(v_{i},q_{i},a_{i})_{i\in[1,N]}$, where $v_{i}\in\mathcal{V}$, $q_{i}\in\mathcal{Q}$ and $a_{i}\in\mathcal{A}$ represent the image, the question and the answer for the $i_{th}$ instance, the VQA model aims to implement a function $\mathcal{F}:\mathcal{V}\times{\mathcal{Q}}\rightarrow\mathcal{R}^{\left\|\mathcal{A}\right\|}$ to produce a distribution over the candidate answer space $\mathcal{A}$. Generally, the entire process can be formulated as follow:

$\displaystyle\mathcal{F}(\mathcal{A}|v_{i},q_{i})=C(M(f_{q}(q_{i}),f_{v}(v_{i})))$

(1)

where the function $\mathcal{F}$ consists of an image encoder $f_{v}:\mathcal{V}\rightarrow\mathcal{R}^{d_{v}}$ to output visual features of dimension $d_{v}$ , a question encoder $f_{q}:\mathcal{Q}\rightarrow\mathcal{R}^{d_{q}}$ to output textual features of dimension $d_{q}$ , a multimodal fusion method $M:\mathcal{R}^{d_{v}}\times{\mathcal{R}^{d_{q}}}\rightarrow\mathcal{R}^{d_{m}}$ to output fusion features of dimension $d_{m}$ and a classifier $C:\mathcal{R}^{d_{m}}\rightarrow\mathcal{R}^{\left\|\mathcal{A}\right\|}$ to output the answer prediction $\mathcal{F}(\mathcal{A}|v_{i},q_{i})$ for the $i_{th}$ image and question pair.

Correspondingly, the classical learning strategy of VQA models is to conduct a multi-label loss $\mathcal{L}_{ml}$, minimizing the binary cross-entropy criterion over a dataset of size $N$.

	$\displaystyle\mathcal{L}_{ml}=-\frac{1}{N}\sum_{i}^{N}y_{i}\log(\sigma(\mathcal{F}(\mathcal{A}|v_{i},q_{i})))$
	$\displaystyle+(1-y_{i})\log(1-\sigma(\mathcal{F}(\mathcal{A}|v_{i},q_{i})))$		(2)

where $\sigma(\cdot)$ denotes the sigmoid function, and $y_{i}$ is the label of $a_{i}$, denoted as $y_{i}\in\{0,1\}^{\left\|\mathcal{A}\right\|}$.

Our method is motivated by an intuitive cognition: for both VQA models and humans, we need to get enough content and context from multimodal information to make a decision [17]. Figure 2 shows the overview of our approach, consisting of the Content branch $\mathcal{F}_{cn}$, the Context branch $\mathcal{F}_{cx}$ and a joint loss function $\mathcal{L}_{ccb}$ with language bias. We directly calculate the correlation between the question type $q_{type}$ and $\mathcal{A}$ to obtain language bias $b_{i}$ for the $i_{th}$ instance:

$\displaystyle b_{i}=P(\mathcal{A}|q_{type})$

(3)

where $q_{type}\in\{1,2,...,64\}$ and VQA datasets divide $\mathcal{Q}$ into 64 question types.

As common in the state-of-the-art, our base Up-Down [7] model encodes the image $v_{i}$ as visual features $f_{v}(v_{i})$ using the pretrained Faster RCNN network, encodes the question $q_{i}$ as question features $f_{q}(q_{i})$ by Glove embeddings and GRU. Ideally, we hope a VQA model captures content from visual features and context from textual features. However, due to the current shortcoming of visual-language representation capability and the semantic complementarity of multimodal information, we construct the two branches with the same inputs.

Content branch. We build the Content branch on top of a base VQA model Up-Down [7] and a typically ensemble method LMH [13]. Following the paradigm of VQA, $f_{v}(v_{i})$ and $f_{q}(q_{i})$ are first fed into the VQA model to capture the local key information and generate an answer distribution $\mathcal{F}(\mathcal{A}|v_{i},q_{i})$. Then we adopt ensemble method to reduce the effect of language bias and generate a new answer distribution as the output of our Content branch, which is denoted as:

$\displaystyle\mathcal{F}_{cn}(\mathcal{A}|v_{i},q_{i},b_{i})=E(\mathcal{F}(\mathcal{A}|v_{i},q_{i}),b_{i})$

(4)

Moreover, We try to further reduce the statistical prior caused by biased samples with reweighting method:

	$\displaystyle\mathcal{L}_{cn}=-\frac{1}{N}\sum_{i}^{N}(1-b_{i})^{r}[y_{i}\log(\sigma(\mathcal{F}_{cn}(\mathcal{A}|v_{i},q_{i},b_{i})))$
	$\displaystyle+(1-y_{i})\log(1-\sigma(\mathcal{F}_{cn}(\mathcal{A}|v_{i},q_{i},b_{i})))]$		(5)

where $r$ is a hyperparameter and we utilize $(1-b_{i})^{r}$ to adjust the contribution of different biased samples on the Content loss $\mathcal{L}_{cn}$.

Context branch. We build an additional Context branch, in which we try to use global information to produce a evenly predicted distribution $\mathcal{F}_{cx}(\mathcal{A}|v_{i},q_{i})$, helping the model learn a good context prior to filter out unnecessary answer candidates. Different from the Content branch that emphasizes on obtaining the most relevant information, the Context branch obtains all information that may affect $\mathcal{F}_{cx}(\mathcal{A}|v_{i},q_{i})$.

$\displaystyle\mathcal{F}_{cx}(\mathcal{A}|v_{i},q_{i})=C_{cx}(nn_{q}(f_{q}(q_{i}))\odot nn_{v}(f_{v}(v_{i})))$

(6)

where $\odot$ denotes element-wise product. Concretely, $\mathcal{F}_{cx}$ uses two neural networks $nn_{q}:\mathcal{R}^{d_{q}}\rightarrow\mathcal{R}^{d_{m}}$, $nn_{v}:\mathcal{R}^{d_{v}}\rightarrow\mathcal{R}^{d_{m}}$ to project multimodal features into a common space $\mathcal{R}^{d_{m}}$, then their element-wise product is fed into a classifier $C_{cx}:\mathcal{R}^{d_{m}}\rightarrow\mathcal{R}^{\left\|\mathcal{A}\right\|}$. To learn context priors with language bias, we convert $b_{i}$ into a binary vector $B(b_{i})$ as the label of computing $\mathcal{L}_{cx}$, denoted as follow:

	$\displaystyle\mathcal{L}_{cx}=-\frac{1}{N}\sum_{i}^{N}B(b_{i})\log(\sigma(\mathcal{F}_{cx}(\mathcal{A}|v_{i},q_{i})))$
	$\displaystyle+(1-B(b_{i}))\log(1-\sigma(\mathcal{F}_{cx}(\mathcal{A}|v_{i},q_{i})))$		(7)

where $B(\cdot)$ is the function to convert $b_{i}$ into the label, denoted as:

$\displaystyle B(b_{ij})=\left\{\begin{aligned} 1&&b_{ij}>0\\ 0&&others\end{aligned}\right.$

(8)

where $b_{ij}$ is the predicted probability of $a_{ij}$, $j\in{\{1,\left\|\mathcal{A}\right\|\}}$.

Predicting by Content and Context. To obtain the final prediction $\mathcal{F}_{p}(\mathcal{A}|v_{i},q_{i},b_{i})$ based on visual content and language context, we simply conduct an element-wise product between $\mathcal{F}_{cn}(\mathcal{A}|v_{i},q_{i},b_{i})$ and $\mathcal{F}_{cx}(\mathcal{A}|v_{i},q_{i})$.

$\displaystyle\mathcal{F}_{p}(\mathcal{A}|v_{i},q_{i},b_{i})=\mathcal{F}_{cn}(\mathcal{A}|v_{i},q_{i},b_{i})\odot\mathcal{F}_{cx}(\mathcal{A}|v_{i},q_{i})$

(9)

Follow the classical learning strategy, we conduct the Predict loss $\mathcal{L}_{P}$ to optimize $\mathcal{F}_{p}(\mathcal{A}|v_{i},q_{i},b_{i})$ as follow:

	$\displaystyle\mathcal{L}_{p}=-\frac{1}{N}\sum_{i}^{N}y_{i}\log(\sigma(\mathcal{F}_{p}(\mathcal{A}|v_{i},q_{i},b_{i})))$
	$\displaystyle+(1-y_{i})\log(1-\sigma(\mathcal{F}_{p}(\mathcal{A}|v_{i},q_{i},b_{i})))$		(10)

At last, we obtain our final loss $\mathcal{L}_{ccb}$ by summing $\mathcal{L}_{cn}$, $\mathcal{L}_{cx}$ and $\mathcal{L}_{p}$ together.

$\displaystyle\mathcal{L}_{ccb}=\mathcal{L}_{cn}+\mathcal{L}_{cx}+\mathcal{L}_{p}$

(11)

We evaluate our model on the VQA v2 dataset [2] and VQA-CP v2 dataset [3], following the standard VQA evaluation metric [1]. For fair comparisons, we use the same method as the Up-Down [7] model to extract multi-modal features and build on top of the publicly available model LMH [13] with the same settings.

We first compare CCB with other non-extra-based methods, which are proposed to reduce language bias. Table 1 shows that our approach significantly outperforms other methods over all question categories. Particularly, without using any extra data, we improve the performance of LMH from 52.05% to 57.99%, which is competitive even in extra-based methods. Then, compared with some extra-based methods, CCB achieves the state-of-the-art performance with CSS training scheme, which further proves the effectiveness of our method. For some simple “Yes/No” and direct “number” questions, our method achieves higher accuracy (89.12% and 51.04%), while answering these two types of questions is most susceptible to statistical prior. Finally, we evaluate CCB on the biased VQA v2 dataset. As can be seen from Table 1, there is an obvious gap in the performance of most methods between these two datasets. Notably, our method effectively reduces the performance gap of the model on the two datasets and improves the robustness of the model. The reason is that our learning strategy can decrease most statistical prior on VQA-CP v2, while retaining most context prior on VQA v2 so as to avoid a huge drop in performance.

Ablation studies on the VQA-CP v2 dataset are conducted to investigate factors that affect the performance of our approach. The results are listed in Table 2. Firstly we investigate the effectiveness of building Content and Context branches. To this end, we set “$r=0$, w/o”, in which the Content branch uses a multi-label loss $\mathcal{L}_{ml}$ and the Context branch calculates $\mathcal{L}_{cx}$ by setting an all-one constant vector as the label. The “$r=0$, w/o” performs better than the LMH model, which indicates that the two branches can jointly affect the generation of answers avoiding only depending on one-sided information. Then we set $r=1$ and add the context label to investigate the influence of using bias for reweighting in $\mathcal{L}_{cn}$ and labeling in $\mathcal{L}_{cx}$ respectively. The overall accuracy of 55.70% and 56.76% demonstrate that our loss functions are effective to reduce the effect of biased samples as well as retain their beneficial influence on learning prior knowledge. Finally, we investigate the influence of different hyper-parameter $r$, which are applied to adjust the importance of biased samples. The results show that the best performance can be achieved when $r=1$. For further analysis, a large $r$ may affect the model’s learning ability to biased samples, while a small $r$ may make the model get no complete rid of the excessive dependence on language bias.

We make qualitative evaluation of the effectiveness of our method by visualizing the top-3 important regions of three models and outputting the corresponding attention weights. In addition, we apply the softmax function to calculate the probability of top-3 candidate answers predicted by the models, which reflects the models’ confidence in the answer. As shown in Figure 3, for the question like “What color is the counter ?”, although both LMH and our model give the correct answer, our method makes the model pay more attention to “the counter” and ignore other regions that are not related to the question. This allows our model to give a more accurate color of “the counter”. Even for the question that requires more understanding of visual content like “How many feet do the women have on the ground ?”, our model can still give the correct answer. It can be seen that, the model correctly applies more weights to the corresponding regions in the image, including “the feet of the woman” on the ground and “other parts of the body” that are in contact with “the elephant”. In both cases, our method can avoid the model from being directly affected by the statistical prior and giving common answers (e.g., “white”, “2”). Moreover, our method can guide the model to understand the whole question and apply the correct attention weights , instead of just focusing on the n-gram “How many feet …”.