An Improved Attention for Visual Question Answering

Antol et al. [2] first introduced the task of visual question answering (VQA), by combining computer vision with natural language processing, to mimic human understanding about a particular visual environment. The model used a CNN for feature extraction and an LSTM for language processing. The features were combined using element-wise multiplication in service of classifying the answers.

Over the last few years, a large number of deep neural networks have been proposed to improve the performance on VQA. Moreover, attention-based approaches became widely used to solve various sequence learning tasks, including VQA. The goal of attention module is to identify the most relevant part of image or textual content. Yang et al. [33] introduced an attention network to support multi-step reasoning for the image question answering task. A combination of bottom-up and top-down attention mechanism was presented in  [1]. A set of salient image regions were proposed by bottom-up attention mechanism using Faster R-CNN [24]. On the other hand, task specific context was used to predict an attention distribution by top-down mechanism over the image regions. A model-agnostic framework is proposed by Shah et al.  [26] which relies on cycle consistency to learn VQA model. Their model not only answers the posed question, but also generates diverse and semantically similar variations of questions conditioned on the answer. They enforce network to match the predicted answer with the ground truth answer to the original question. Wu et al. [31] propose a differential networks (DN), a novel plug and play module where differences between pair-wise features are used to reduce noise and learn inter-dependency between features. To extract image and text feature, Faster R-CNN [24] and GRU [5] are used respectively. Both features are refined by a differential module and finally combined to predict the answers.

Recently, co-attention based approaches are becoming popular. The goal of co-attention model is to learn image and question attention simultaneously. Lu et al. [17] introduced a co-attention network that jointly reasons about image and question attention in a hierarchical fashion. Yu et al. [36] proposed an architecture to reduce irrelevant features by applying self attention for question embedding and question conditioned attention for image embedding. Multi-modal attention is proposed in [18, 25] to focus on images, questions or answers feature simultaneously. Recently, bilinear attention is proposed in [7, 12, 35] to locate more accurate objects. A multi-step dual attention for multimodal reasoning and matching is presented in [20]. One major limitation of these co-attention based approaches is lack of dense interactions between different modalities. To overcome this limitation, dense co-attention based methods are proposed in [34, 11]. But dense co-attention can generate irrelevant vector in scenarios where nothing is related to the query. To overcome the problem, motivated by [10], in this paper we combine Attention-on-Attention (AoA) module with Modular co-attention network to improve existing architecture. Our revised attention mechanism delivers significantly better performance in VQA.

To combine multi-modal features, sophisticated fusion technique is required. Depending on the type of fusion, existing VQA models can be divided into two categories: linear and bilinear [31]. Linear models use simple fusion approaches to combine image and question features. Simple element-wise summation and element-wise multiplication are used in [17, 33] and  [15, 20] respectively. On the other hand, bilinear model uses more fine-grained approache to fuse image and question features. Fukui et al. [7] used outer product to fuse multi-modal features. A low-rank projection followed by an element-wise multiplication is used by Kim et al. [12]. A Multi-modal Factorized Bilinear (MFB) pooling approach with co-attention learning is proposed in [35]. Wu et al. [31] proposed a Differential Networks (DN) based Fusion (DF) approach which first calculates differences between image and textual feature elements and then combines the differential representations to predict final answer.

In this paper, we propose an attention-based multi-modal fusion to combine image and question features by dynamically deciding how much weight to put on each modality; the weighted features are used to predict final answer.

Our SAoA unit (see Figure 2(a)) is an extension of multi-head self attention mechanism [34]. Multi-head attention consists of $N$ parallel heads where each head can be represented as a scaled dot product attention function as follows:

where attention function $\operatorname{f}(Q,K,V)$ operates on Q, K and V corresponds to query, key and value respectively. The output of this attention function is the weighted average vector $V^{\prime}$. To do so, first we calculate the similarity scores between Q and K; and normalize the scores with Softmax. The normalized scores are then used together with V to generate weighted average vector $V^{\prime}$. Here, $d$ is the dimension of $Q$ and $K$; both dimensions are the same.

The multi-head attention module always generates weighted vector, no matter whether it finds any relation between Q and K/V or not. So this approach can easily mislead or generate wrong answer for VQA. Therefore, following  [10], we incorporate another attention function over the multi-head attention module to measure the relation between attention results ($V^{\prime}$) and the query($Q$). The final AoA block will generate an information vector ($I$) and attention gate ($G$) through two separate linear transformations which can be represented as follows:

Here, ${\operatorname{W}}_{\operatorname{Q}}$, ${\operatorname{W}}_{\operatorname{V}^{\prime}}$, ${\operatorname{W}}_{\operatorname{G}}$, ${\operatorname{W}}_{\operatorname{G}^{\prime}}$ $\in$ ${\mathbb{R}}^{d\times d}$ and ${\operatorname{b}}_{\operatorname{I}}$, ${\operatorname{b}}_{\operatorname{G}}$ $\in$ ${\mathbb{R}}^{d}$. $d$ is the dimension of $Q$ and $V^{\prime}$ where $V^{\prime}$ = ${\operatorname{f}}_{\operatorname{att}}$ and $\sigma$ denotes sigmoid function. AoA block adds another attention via element-wise multiplication between both information vector and attention gate. Moreover, SAoA uses a point-wise feed-forward layer after the AoA block, considering only input features $X=[x_{1},x_{2},...,x_{m}]\in{\mathbb{R}}$.

Following  [34], we also propose another attention unit called guided attention on attention (GAoA) unit (see Figure 2(b)). Unlike SAoA unit, GAoA uses AoA block and a point-wise feed-forward layer along with two input features $X$ and $Y=[y_{1},y_{2},...,y_{n}]\in{\mathbb{R}}$ where $X$ is guided by $Y$. In both attention unit, feed forward layer takes the output feature of AoA block and apply two fully connected layers along with ReLU and dropout function (i.e. $\text{\tt FC}(4d)-\text{\tt ReLU}-\text{\tt dropout}(0.1)-\text{\tt FC}(d)$).

Modular Co-Attention on Attention (MCAoA) layer (see Figure 3) consists of two attention units discussed in Section 3.1. Here $X$ and $Y$ represents image and question feature respectively. MCAoA layer is designed to handle multimodal interactions. We use cascaded MCAoA layers, i.e., output from previous MCAoA is fed as input to the next MCAoA layer. For both input features, MCAoA layer first uses two separate SAoA units to caption intra-modal interactions for $X$ and $Y$ separately and then uses GAoA unit to capture inter-modal relationships where $Y$ guides $X$ feature.

In this section, we discuss our proposed modular co-attention on attention network (MCAoAN) (see Figure 4) which is motivated by  [34]. First we have to pre-process the inputs (i.e., image and query question) into appropriate feature representations. We use Faster R-CNN [24] with ResNet-101 as its backbone which is pretrained on Visual Genome dataset [13] to process input images. The intermediate feature of the detected object from Faster R-CNN is considered as visual feature representation. Following  [34], we also consider a threshold value to obtain dynamic number of detected objects, e.g., $x_{i}$ is corresponds to i-th object feature. The final image feature is represented by a feature matrix $X$.

The input query question is first tokenized and later trimmed to maximum 14 words. The pre-trained GloVe embedding [22] is used to transformed each word into a vector representation. This results a final representation of size $n\times 300$ for a sequence of words where $n\in[1,14]$ denotes the number of word in the sequence. The word embedding is fed to a one layer LSTM network [9] and generate final query feature matrix $Y$ by capturing the output features of all words.

Both input features are passed to the encoder-decoder module which contain cascaded MCAoA layers. Similar to  [34], encoder learns attention question features $Y_{L}$ by stacking $L$ number of SAoA units. On the other hand, decoder learns attended image features $X_{L}$ by stacking $L$ number of GAoA units by using query features $Y_{L}$.

The outputs (i.e image features $X_{L}=[x_{1},x_{2},...,x_{m}]\in{\mathbb{R}}^{m\times d}$ and question features $Y_{L}=[y_{1},y_{2},....,y_{n}]\in{\mathbb{R}}^{n\times d}$) from encoder-decoder contains attended information about image and query regions. Therefore, we need to apply an appropriate fusion mechanism to combine both feature representation. In this paper, we propose two kind of multi-modal fusion networks (see Figure 5) to aggregate features of both modality: (1) Multi-modal Attention Fusion and (2) Multi-modal Mutan Fusion. Following [34], we first use two layers of MLP (i.e. FC(d)- ReLU - Dropout(0.1) - FC(1)) for both $X_{L}$ and $Y_{L}$; and generate attended features $X^{\prime}$ and $Y^{\prime}$ as follows:

and

Now we have rich attended features from both modality and at the same time each modality should use to generate attention with one another for better prediction. Therefore, we have to decide, how much information should use from each modality. Following  [19], in multi-modal attention fusion, we apply concatenation on $\operatorname{X^{\prime}}$ and $\operatorname{Y^{\prime}}$ followed by a series of fully-connected layers (i.e., $\text{\tt FC}(1024)-\text{\tt Dropout}(0.2)-\text{\tt FC}(512)-\text{\tt Dropout}(0.2)-\text{\tt FC}(2)-\text{\tt Softmax}$) (see Figure 5 (a)). The output of these operations is a 2-dimensional feature vector that corresponds to the importance of two modality for answer prediction. After that, we generate weighted average of attended feature (i.e. $A$ and $B$) for each modality similar to eq. 4 and  5. $A$ and $B$ is combined with attended visual and textual features $X^{\prime}$ and $Y^{\prime}$. Finally, fused feature is fed to a LayerNorm to stabilize the training followed by a fully connected layer and sigmoid activation to generate predicted answer $Z$. We use binary cross-entropy loss (BCE) to train the network.

On the other hand, we also leverage a powerful fusion technique, MUTAN fusion [3], to integrate image and question features (see figure 5 (b)) in multi-modal mutan fusion. The network is similar to the above model but replacing the concatenation to MUTAN fusion with fully-connected layers (i.e., $\text{\tt Dropout}(0.2)-\text{\tt FC}(2)-\text{\tt Softmax}$).

To evaluate our method, in this paper we use VQA-v2 benchmark dataset [8] which consists of images from MS-COCO dataset [16] with human annotated question-answer pairs. There are 3 questions for each image and 10 answers per questions. The dataset has three parts: train set (80k images with 444k QA pairs), validation set (40k images with 214k QA pairs) and test set (80k images with 448k QA pairs). Moreover, test set is splited into two subsets: test-dev and test-standard where both are used for online evaluation performance. For measuring the overall accuracy, three types of answer are considered: Number, Yes/No and other.

To evaluate our method, we follow the experimental protocol proposed by [34]. The number of head in multi-head attention is 8. The latent dimension for both multi-head and AoA block is 512. Therefore, the dimension of each head is $512/8=64$. The size of the answer vocabulary is 3129.

To train the MCAoA network we use Adam solver with $\beta_{1}=0.9$ and $\beta_{2}=0.98$. We train our network up to 13 epoch with batch size 64 which takes around 24hrs to complete the training. The learning rate set to $min(2.5Te^{-5},1e^{-4})$ where $T$ represents current epoch. Learning rate starts to decay by $1/5$ every $2$ epochs when $10\leq T$.

We run a number of experiments to show the effectiveness of our proposed method and results of these experiments are presented in Table 1 and 2.

Moreover, we argue that a sophisticated way to aggregate language and visual features to support multi-modal reasoning is essential to further boost the performance. Table 2 also shows the comparison of different fusions with the MCAoA only where the former achieves better performance. More specifically, our proposed MCAoAN with both multi-modal fusion modules outperforms the baseline about $2\%$ accuracy on the whole validation set. This shows that the fusion module is important to combine vision and language representations. The proposed both fusion modules are suitable for VQA tasks. Among them multi-modal attention fusion performs the best. Beside that, Table 2 also shows that each individual component within our proposed method is important to increase the performance of VQA system.

We evaluate our model on VQA-v2 dataset and compare with other state-of-the-art methods. We re-run the PyTorch implementation provided by [34]¹¹1https://github.com/MILVLG/mcan-vqa and compare the results with our proposed method. Table 3 and 4 shows experimental results using test-dev and test-std respectively using online evaluation ²²2https://evalai.cloudcv.org/web/challenges/challenge-page/163/overview. Offline evaluation only supports on validation split (see table 2). Figure 6, shows some qualitative results using our method on validation set. From the experimental results, we can see that our proposed method outperforms other baseline methods on VQA. In Figure 7, we also visualize multi-modal fusion to compare how correctly MCAN [34] and our proposed multi-modal attention fusion can able to focus on image and question elements. The brighter bounding-box along with green color within the image and darker color in question represents higher attention score. We can see that our proposed method is able to focus more on correct answer. Beside that, Figure 8 shows typical failure cases using our method.