Attention Guided Semantic Relationship Parsing for Visual Question Answering

The traditional approach [12, 4, 5, 36, 42] for extracting question features for the VQA task is by sourcing pretrained semantic embedding vectors for each question words, concatenating and passing them through a recurrent neural network. The hidden state of the last recurrent block is extracted as the question feature. In contrast, we consider the question as a whole instead of separate word entities, thereby providing better contextual modelling. We use Bidirectional Encoder Representations from Transformers (BERT) [8] where we first tokenize each word of the question and then feed the tokenized question into a Transformer model pretrained for language modeling task. The question feature $\bm{q}\in\mathbb{R}^{m\times d_{q}}$ is extracted from the last hidden layer of the BERT model, where $m$ is the number of tokens identified in the question and $d_{q}$ denotes the feature dimension.

We represent the visual features of an input image as a set of bounding box coordinates and corresponding object-specific features. First, the object proposals are generated using a bottom-up [2] attention approach where a pretrained Faster-RCNN [32] model is employed the get the region proposals and extract visual features using a ResNet [15] backbone. Following [2], we use an adaptive threshold to select a range of region proposals $l\in[10,100]$ for each image. Further, to visually ground each region proposal we concatenate each region proposal with its bounding box coordinates. Thus, the visual feature representation $\bm{v}\in\mathbb{R}^{l\times(d_{v}+4)}$ of image $I$ consists of features of its object proposals $\{\bm{f}_{j}\in\mathbb{R}^{d_{v}}\}_{j=1}^{l}$ and corresponding bounding box coordinates $\{\bm{b}_{j}\in\mathbb{R}^{4}\}_{j=1}^{l}$, where $d_{v}$ is the object feature dimension.

The Semantic Relationship Parser (SRP) module is illustrated in Fig. 1. It has three major components. The first component is a region proposal network that operates in a similar manner as explained above in Sec. 3.1 for visual feature generation. Based on these object-wise features and box coordinates, a visual relationship detector is used in the second stage. The visual relationship detector generates subject, relationship and object¹¹1Here, the object refers to the grammatical component of a sentence. proposals from the region features which in-turn are used to generate a semantic relationship triplet set $\mathcal{T}=\{\bm{t}_{k}\}_{k=1}^{q}$. Each relationship triplet $\bm{t}_{k}$ consists of class labels predicted for subject, relationship, and object. In order to generate a triplet, a set of candidate subject, relationship and object visual features, denoted by $\bm{f}_{s}$, $\bm{f}_{r}$ and $\bm{f}_{o}$ respectively, are passed through the visual relationship detector. We follow the framework proposed by [44], where we assume a relationship exists only if a subject-object pair exists, not vice versa. Thus the relationship detector learns two mapping functions from visual feature space to semantic space, one for subject/object and the other one for relationship embedding.

The relationship feature embedding is generated by passing the concatenated version of three visual features $\bm{f}_{s}$, $\bm{f}_{r}$ and $\bm{f}_{o}$ through a two-layer Multi-layer Perceptron (MLP) network. The subject and object feature embeddings are generated in parallel by passing them through the same MLP network. On the other hand, class labels of subject, object and relationship are first converted to word vectors and then to semantic feature embeddings by passing them through a small MLP network. Three triplet losses are minimized [43] to match visual and semantic embedding for subject, object and relationship respectively. During inference, word vectors of all subject/object and relationship class labels are passed and a nearest neighbour search is performed to find the desired relationship labels. In practice, we perform the visual relationship detection on the input image as a pre-processing step where we train the visual relationship detector end-to-end on visual relationship dataset (i.e., Visual Genome [22], VRD [25]) and then run inference on the input images to generate relationship prediction, which consists of subject, object and relationship probability.

The third component of the SRP module is the semantic feature extractor which takes the filtered semantic relationship triplets $\mathcal{R}$, subject bounding box $\bm{b}_{s}$ and object bounding box $\bm{b}_{o}$ coordinates, and generates visually grounded semantic relationship features as follows,

$\displaystyle\bm{r}=\mathcal{F}_{\text{BERT}}(\mathcal{R},\bm{b}_{s},\bm{b}_{o}).$

(2)

Here, $\mathcal{F}_{\text{BERT}}$ denotes the BERT model (similar to the one described in Sec. 3.1) for extracting semantic features from the triplets. First, this function takes all entries from the set $\mathcal{R}$ and adds a period (‘.’) after each element. Each relation triplet $\bm{t}_{k}\in\mathcal{R}$ is now considered a complete sentence which is passed through the BERT model separately alongside its subject and object proposal coordinates. This generates corresponding semantic relationship features $\bm{r}\in\mathbb{R}^{n\times d_{r}}$ from image $I$, where $d_{r}$ is the dimension of the hidden feature of the BERT model.

Notably, the set $\mathcal{R}$ only contains refined relationship triplets obtained after a two stage thresholding and filtering process. At the first stage, we filter out the relationship predictions where the probability of the product of subject and object proposals are higher than a threshold $\alpha$. This ensures that we only select the high-confidence relationships between subject-object pairs. In the next stage, from the remaining relationship predictions, we select only those whose relation probability is higher than $\beta$. Intuitively, we set $\alpha$ at a higher value compared to $\beta$ to ensure that we first get the subject and object instances right, and ask the model to predict various relationships between them. The values of $\alpha$ and $\beta$ are set empirically with the objective to select at least three relationship triplet per image. We further filter out any duplicate relationships and end up with ‘$n$’ relationship predictions per image encoded in refined semantic relationship set $\mathcal{R}$.

The Mutual and Self Attention (MSA) module consists of two major components. The first component focuses on mutual attention where two separate multimodal fusion operations are performed to learn attention distribution over the input feature vectors. The second component applies self attention on the pair of input features and generates attention distribution over the input features themselves. We illustrate the MSA module in Fig. 2. For simplicity lets assume the input to the MSA module are a two feature embeddings $\bm{x}\in\mathbb{R}^{a\times d_{x}}$ and $\bm{y}\in\mathbb{R}^{b\times d_{y}}$, which will undergo mutual and self attention.

Mutual Attention: To capture the complex interaction between $\bm{x}$ and $\bm{y}$ we jointly embed these features by learning a multimodal embedding function. This is achieved by first concatenating the input features and then passing them through a 3 layer MLP network. The MLP model learns to capture the mutual interactions between the input feature vectors and produces a joint feature embedding. For input combinations $(\bm{y},\bm{x})$ and $(\bm{x},\bm{y})$, we have:

		$\displaystyle\bm{z}_{y,x}=\text{MLP}\,\,[\bm{y}\,\,\oplus\,\,\bm{x}]\in\mathbb{R}^{1\times b},\quad\textrm{and}$		(3)
		$\displaystyle\bm{z}_{x,y}=\text{MLP}\,\,[\bm{x}\,\,\oplus\,\,\bm{y}]\in\mathbb{R}^{1\times a},$		(4)

where $\oplus$ denotes the concatenation operation of the two vectors. $\bm{z}_{y,x}$ and $\bm{z}_{x,y}$ signifies the learned mutual attention distributions over the inputs $\bm{y}$ and $\bm{x}$ respectively. These attention distributions are used to take a weighted sum on the corresponding input feature vectors to generate mutually attended feature representation $\bm{y}_{m,yx}$ and $\bm{x}_{m,xy}$,

		$\displaystyle\bm{y}_{m,yx}=\sum_{i=1}^{b}(z^{i}_{y,x}\,\,\bm{y}^{i})\in\mathbb{R}^{d_{y}},\quad\textrm{and}$
		$\displaystyle\bm{x}_{m,xy}=\sum_{i=1}^{a}(z^{i}_{x,y}\,\,\bm{x}^{i})\in\mathbb{R}^{d_{x}},$		(5)

where, $\bm{x}^{i}$ and $\bm{y}^{i}$ denote the $i^{th}$ row from the feature embeddings $\bm{x}$ and $\bm{y}$ respectively.

Self Attention: For the self attention component, we follow the guided self attention module used in [41]. The input feature $\bm{x}$ is fed to a Transformer employing multi-head attention [38] to learn an attention distribution $\Psi_{x}$. The other input $\bm{y}$ undergoes a similar multi-head attention like $\bm{x}$, except the query input is replaced with $\Psi_{x}$, which allows the model to learn $\bm{x}$-guided self attention distribution over $\bm{y}$, denoted as $\Psi_{y,x}$. $\Psi_{x}$ and $\Psi_{y,x}$ are passed through separate fully-connected layers for dimensionality reduction and we get $\Psi_{x}\in\mathbb{R}^{a}$ and $\Psi_{y,x}\in\mathbb{R}^{b}$. These self attention maps are used to take a weighted sum over the corresponding feature representations and generate self attended features $\bm{y}_{s,yx}$ and $\bm{x}_{s,y}$,

		$\displaystyle\bm{y}_{s,yx}=\sum_{i=1}^{b}(\Psi^{i}_{y,x}\,\,\bm{y}^{i})\in\mathbb{R}^{d_{y}},\quad\textrm{and}$
		$\displaystyle\bm{x}_{s,y}=\sum_{i=1}^{a}(\Psi^{i}_{x}\,\,\bm{x}^{i})\in\mathbb{R}^{d_{x}}.$		(6)

In practice, we employ two MSA modules, where we feed $\bm{q},\,\bm{v}$ to the first one and $\bm{q},\,\bm{r}$ to the other. The intuition behind this is the first MSA module learns to identify which region of the image, and words of the question are important to answer the question. Similarly, the second MSA module tries to identify the salient relationship features and question parts for answering the question. For both MSA blocks, we pass question features as $\bm{x}$ which guides the attention learning process of the input. This is particularly important as the question sets the objective of the task, and the quality of the learned attention distribution depends more on the question than the other inputs. Thus the first MSA module outputs $\bm{v}_{m,vq},\bm{q}_{m,qv},\bm{v}_{s,vq},\bm{q}_{s,v}$ and the second one outputs $\bm{r}_{m,rq},\bm{q}_{m,qr},\bm{r}_{s,rq},\bm{q}_{s,r}$.

We perform multimodal attention fusion on the outputs of the MSA blocks. Each attended feature is projected to an intermediate space through fully connected layers followed by summation. As the attended features already capture rich feature description, we only use such a simple linear summation technique to capture their interaction before making the final answer prediction. The summed feature vector is then projected to the answer prediction space $d^{|\mathcal{A}|}$ through another fully connected layer where we minimize a cross-entropy loss to predict correct answer from the candidate answer set.

We perform experiments on two large-scale VQA datasets, namely VQAv2 [14] and GQA [18]. The VQAv2 has 200K images and 1.1M crowed-sourced questions. This is the biggest manually annotated VQA dataset. Further, GQA contains 11K images and 22M auto-generated questions, making it a more challenging evaluation setting.

The visual feature dimension is $d_{v}=2048$ for each object. To extract the semantic features from the question and relationship triplet, we use a pretrained bert-large-cased²²2https://huggingface.co/bert-large-cased model. Since a cased version is used, we do not convert the question or relationship triplets to lowercase. The extracted semantic feature dimensions for question and relationship are $d_{q}=d_{r}=1024$. Following the recommendation in [38, 41], the intermediate dimensions $d$ of the multi-head attention in transformer module is set to $512$ with $8$ heads and latent dimension of $64$. Adam optimizer [21] with $\beta_{1}=0.9$ and $\beta_{2}=0.98$ is used.

\sidecaptionvpos

tablec

In Tab. 1, we first establish the benefit of our proposed semantic relationship feature modeling. For a fair comparison with our proposed MSA model which uses bounding box coordinates of subjects and objects, we compare it with a version of MCAN [42]³³3official implementation of MCAN-large (frcn+bbox) model is at https://github.com/MILVLG/openvqa that also uses bounding box coordinates. To this end, we compare the VQA performance between ‘semantic’ and ‘visual’ relationship features to showcase the comparative advantage on the GQA validation dataset. To develop the baseline model with visual relationship features, we train the SRP module (Sec. 3.2) on the VRD dataset [25] with $100$ objects and $70$ predicate categories, and output visual feature of the subject and object relationship proposal alongwith relationship triplet. The visual feature of the subject and object proposals are concatenated and considered as visual relationship feature $\bm{r}^{vis}$, and the default semantic relationship features (denoted by $\bm{r}^{sem}$) are extracted from the relationship triplet. The models in Tab. 1 employ only the guided self-attention part of the MSA module for simplicity.

Blind models trained with visual relationship feature perform slightly better. In Tab. 1, we see that a VQA model trained with only visual relationship features (row 2) performs better than the model trained only with semantic relationship features (row 3). This is because when the visual feature $\bm{v}$ is not available, the $\bm{r}^{sem}+\bm{q}$ model is blind to the image and the answer prediction is based only on the relationship labels. On the other hand, the $\bm{r}^{vis}+\bm{q}$ model can see the image as a set of the visual feature of subject-object proposals, thus performs better than the completely blind model (row 3). However, in this extreme setting, the blind model perform reasonably well only relying on the semantic relationship labels.

Non-blind models trained with semantic relationship feature perform significantly better. When the visual feature is available, the VQA model with the complementary semantic relationship feature performs significantly better ($7.34\uparrow$) than its counterpart (rows 4, 5 in Tab. 1). This demonstrates the complementary effectiveness of the semantic relationship features, since both these settings are identical except for the nature of the relationship feature.

We simulate an oracle setting to further evaluate the effectiveness of using semantic relationships for VQA. We build this setting using scene-graph annotations available for GQA [18] train and validation sets. Each scene-graph entry consists of ground-truth subject, relationship and object label. We use a scene-graph parser which converts each scene-graph entry into a list of semantic relationship triplets similar to the output of visual relationship detector of Sec. 3.2, and denote the extracted semantic ground-truth relationship features as $\bm{r}^{oracle}$.

Both blind and non-blind oracle models significantly outperform the SOTA. The blind VQA model with a ground-truth relationship label $\bm{r}^{oracle}$ achieves an overall accuracy gain of $3.71\uparrow$ compared to the state-of-the-art MCAN [42] model which is a non-blind model (comparing rows 6 and 1 of Tab. 1). This is an interesting finding showing if good enough semantic relationship label are available, the VQA model could achieve better performance than SOTA without even looking at the image. Further, when visual feature of the image is available in the oracle setting, the model achieves $16.43(\sim 25\%)$ accuracy gain over [42].

‘Open-ended’ questions are answered better. Both oracle models report significant accuracy gain ($19.73\uparrow$ and $28.5\uparrow$ compared to MCAN) for the challenging ‘Open’ question category. These open-ended questions require diverse and broad reasoning ability to answer correctly. This is a significant finding as it sheds light upon effectiveness of using complementary semantic relationship features as an important line of research to break the bottleneck of VQA models mostly focusing on learning better visual representations.

\sidecaptionvpos

tablec

We perform extensive ablation on the VQAv2 test-dev and GQA test-dev datasets and report the results in Tab. 2. Our goal here is to identify which input and attention combination contributes to the overall performance of our model. This is a comprehensive setup as the VQA dataset and GQA dataset consist of natural crowed-sourced and auto-generated questions respectively. We use semantic relation features for all our experiments (i.e., $\bm{r}=\bm{r}^{sem}$).

Semantic relationship features provide accuracy boost when used in complement with visual features. The blind model which only uses parsed relationship features without any visual features (rows 1–3) performs worse compared to other models that explicitly use visual features. However, when used in complement with image and question features (rows 7–9), it helps models achieve better performance on both VQA and GQA datasets.

Guided self attention provides rich attention distribution over its inputs compared to mutual attention. For the three input combinations listed in Tab. 2, we ablate the MSA module by only activating mutual or self attention module. We can see that when only guided self attention module is activated, a better VQA accuracy is achieved in both the datasets. This is because the self attention module captures rich semantics over the the input features through its multi-head attention architecture. The mutual attention component works best in a VQA setting when the attention distribution is learned on the visual feature, which undergoes a second multimodal fusion with the question feature [12, 9, 4, 5]. By design, we want the mutual attention module to capture the multimodal interaction between the inputs and feed it to the attention fusion module (Sec. 3.4) for combining with other attention distributions. Thus a standalone setup for our mutual attention module performs sub-par to the guided self attention module.

MSA module with both mutual and self attention performs best. The full MSA model with both mutual and self attention modules achieves better performance compared to when a single block is activated (rows 3,6,7 in Tab. 2). The mutual attention module provides complementary information that helps in cases where self attention alone is not sufficient.

We report the performance of our single MSA model on benchmark VQAv2 Test-Standard dataset in Tab. 3. We show that by leveraging the semantic relationship features, our model is able to outperform other comparable state-of-the-art models, even without additional training on Visual Genome dataset [22]. Some recent models resort to ensembling with data augmentation techniques [42] or use pretrained model trained on different Vision-Language tasks and/or datasets [35, 24, 7] to achieve superior performance. However, such approach is tangential to the motivation of this paper and are not directly comparable to our proposed model. [43, 17] do not report their performance on the VQAv2 dataset and are thus not included here for comparison. We can see from Tab. 3, that our model achieves state-of-the-art accuracy on the benchmark VQAv2 dataset with comparable methods where the performance boost is achieved by incorporating semantic relationship features.

We provide some qualitative results in Fig. 3 of MSA model on VQAv2 dataset. We visualize the attention distribution over the region proposals and list two relationship triplet with highest attention for better visualization. For simplicity we do not visualize the mutual and self attention distribution over the question words. We can see that the self and mutual attention component provide complementary attention distribution over the input features. For example, in the second row, when asked ‘Are they wearing goggles?’ The visual self attention component focuses more on the sunglass of the person on the left. The mutual attention component looks at the person on the left and the right. Similarly, the self attention component gives more attention to a relationship triplet with sunglass and person, but further looks at person wear shirt relationship triplet for getting more semantic context. Such complementary relationship between various attention components helps VQA model to reason better over its input feature representations.