Multi-Clue Reasoning with Memory Augmentation for
Knowledge-based Visual Question Answering

For an image-question pair $\{\textbf{I},\textbf{Q}\}$, after the three-term phrase (i.e., Subject, Relation, Object) is predicted by the relation phrase detector, all the facts in the external KB with entities pointed by the subject or pointed to the object within $h$ hops are collected as candidate supporting facts. $h$ is set to 1 in our experiments. The relation predicted is leveraged to further filter out the facts with wrong relations. Then a set of candidate supporting facts $\{{\textbf{F}}_{i}\}_{i=1}^{N}$ is collected, where $N$ denotes the number of facts in the fact set, and each fact ${\textbf{F}}_{i}$ consists of a sequence of words.

Afterwards, similar to question embedding, each fact ${\textbf{F}}_{i}$ is transformed to a sequence of word embeddings ${\textbf{H}}^{f}_{i}$ based on GloVe’s vocabulary, and encoded using a BiLSTM to get its representation ${\textbf{f}}_{i}\in{\mathcal{R}}^{128}$.

To store the candidate supporting facts, a key-value structured memory network [25] is leveraged. The key memory is used in the addressing stage, while the value memory is used in the reading stage. The representation ${\textbf{f}}_{i}$ of each fact is passed through two separate linear layers to obtain its key embedding ${\textbf{M}}^{k}_{i}\in{\mathcal{R}}^{128}$ and value embedding ${\textbf{M}}^{v}_{i}\in{\mathcal{R}}^{128}$ respectively.

The visual-semantic embedding module has basically the same architecture as the relation phrase detector and the only difference is about the question embedding before top-down attention, where self-attention are applied over the sequence of word embeddings ${\textbf{H}}^{q}$ to get the semantic representation $\textbf{q}\in{\mathcal{R}}^{128}$.

Given visual-semantic representation h, we apply an attention over all the memory slots $\{{\textbf{M}}^{k}_{i},{\textbf{M}}^{v}_{i}\}_{i=1}^{N}$ to obtain the memory summary $\textbf{m}\in{\mathcal{R}}^{128}$ in light of the visual and semantic content of the image-question pair.

As we have obtained the memory summary m, we proceed to compute the attentions over all the question words and all the image features in light of the memory.

Given memory summary m, the sequence of word embeddings ${\textbf{H}}^{q}$ and the set of image features v, the memory-aware question embedding ${\textbf{q}}^{m}\in{\mathcal{R}}^{128}$ and the memory-aware image embedding ${\textbf{v}}^{m}\in{\mathcal{R}}^{2048}$ are derived as follows:

The visual-semantic representation ${\textbf{h}}^{m}$ in light of the memory is obtained by the fusion of the memory-aware question embedding ${\textbf{q}}^{m}$ and the memory-aware image embedding ${\textbf{v}}^{m}$.

Inspired by [6], another hop of the attention process is conducted over the memory before answer prediction. Attention mechanism of Section 3.3.3 is leveraged here, given memory aware visual-semantic representation ${\textbf{h}}^{m}$, to fetch the most relevant information ${\textbf{m}}^{h}\in{\mathcal{R}}^{128}$ from the memory to obtain the final visual-semantic representation $\hat{\textbf{h}}\in{\mathcal{R}}^{128}$. To be more specific, the fetched information ${\textbf{m}}^{h}$ is concatenated with ${\textbf{h}}^{m}$ and fed into a GRU to update the visual-semantic representation. To the end, a batch normalization (BN) layer is used.

Given the final visual-semantic representation $\hat{\textbf{h}}$, and the set of key embeddings ${\textbf{M}}^{k}\in{\mathcal{R}}^{128\times N}$, the key embedding ${\textbf{M}}_{i}^{k}$ of each candidate supporting fact is concatenated with the $\hat{\textbf{h}}$ to compute the probability of whether the fact is correct. The predicted supporting fact is

Once we have the candidate supporting facts retrieved from the external KB, all the learnable parameters of the proposed MCR-MemNN (besides the Relation Phrase Detector) are trained in an end-to-end manner by minimizing the following loss function over the training set.

For the training of the relation phrase detector, the model was trained with Adam optimizer [14] with an initial learning rate of $1\times 10^{-4}$ and weight decay of $1\times 10^{-6}$, and the batch size is set to 32.

For the training of the proposed MCR-MemNN (besides the relation phrase detector), the model was trained with Adam optimizer with an initial learning rate of $1\times 10^{-3}$ and weight decay of $1\times 10^{-6}$, and the batch size is set to 64. Top 40 predicted subjects and objects were adopted as clues to retrieve candidate supporting facts from the external KB. The memory size $N_{mem}$ was set to 96. Note that no matter the number of candidate supporting facts $N$ is larger than, equal to or smaller than $N_{mem}$, the ground-truth fact is preserved and the negative facts are randomly selected to fill up the remaining memory. Our code was implemented in PyTorch [28] and run with NVIDIA Tesla P100 GPUs.

For the FVQA dataset, CNN-RNN based approaches including LSTM-Question+Image+Pre-VQA [34] and Hie-Question+Image+Pre-VQA [34], semantic parsing based approaches including FVQA (top-3-QQmaping) [34] and FVQA (Ensemble) [34], learning-based approaches including Straight to the Facts (STTF) [27], Out of the Box (OB) [26], Reading Comprehension [17], and Multi-Layer Cross-Modal Knowledge Reasoning (Mucko) [42] are compared with the proposed MCR-MemNN.

For the Visual7W+ConceptNet dataset, learning based approaches including KDMN [16], Out of the Box (OB) are compared with the proposed MCR-MemNN. Note that both KDMN and MCR-MemNN adopted memory augmentation technique for storing the retrieved external knowledge.

As shown in Table 1, to evaluate the performance of the relation phrase detector, five different cases are considered, including Top-20, Top-30, Top-40, Top-60 and Top-100. For each case, the classification accuracies of subject (i.e., ‘Sub’), object (i.e, ‘Obj’) and both union (i.e., ‘Union’) are calculated. In addition, the answer recall is reported with the top-3 relation limitation (i.e., ‘w/ Relation’) or not (i.e., ‘w/o Relation’). Finally, the downstream question answering accuracy (‘QA Acc.’) is also calculated.

For more clarity, the ‘Sub’ in the case of Top-40 represents the fraction of test samples that the ground-truth subject is included in the top 40 predicted subjects, and these subjects are further adopted as the clues for KB retrieval. The Recall w/ Relation in the case of Top-40 represents the fraction of test samples that the correct answer is included in the candidate answer set corresponding to the supporting facts retrieved by the top-40 subject or object clues and filtered by the top 3 predicted relations. Note that the top-3 classification accuracy for relation prediction using the relation phrase detector is $93.20\%$.

Results in Table 1 show that both union accuracy and answer recall of the Top-40 case are higher than those of the Top-20 case, and improvements of $10.56\%$ and $6.09\%$ on downstream QA accuracies (with and without relation filtering) are caused. Even though much more relevant supporting facts are retrieved from the external KB in the Top-100 case, which delivers higher both union accuracy and answer recall, the downstream QA accuracies are lower than those of the Top-40 case. This observation clearly shows that excessive retrieved facts can lead to more redundant information, which is harmful to the reasoning process. We choose the top 40 subjects and objects as clues for KB retrieval as this gives the best downstream QA accuracies.

Tables 2 and 3 show the comparison of the proposed MCR-MemNN with the above-mentioned baselines on FVQA and Visual7W+ConceptNet, respectively, and the following observations can be made:

1) On FVQA dataset, MCR-MemNN outperforms almost all the baselines in terms of top-1 and top-3 accuracies. To be specific, MCR-MemNN outperforms the semantic parsing based approaches including FVQA (top-3-QQmaping) and FVQA (Ensemble), and achieves more than $12\%$ boost on top-1 accuracy and more than $15\%$ boost on top-3 accuracy. In addition, compared with the typical learning based approaches including STTF and OB, which wholly introduce the visual and semantic information without selection, MCR-MemNN gains an improvement by leveraging a two-way attention mechanism (i.e., Sections 3.3.3 and 3.3.4) to exploit the inter-relationships among image, question and KB and distill the most relevant information in each of the three modalities.

2) Even though a heterogeneous graph neural network with high complexity is employed in Mucko, the proposed MCR-MemNN can still deliver a comparable performance. Note that the full model of Mucko leveraged dense captions [13] as input for performance improvement. For fair comparison, the semantic graph for dense caption parsing is removed.

3) On Visual7W+ConceptNet dataset, MCR-MemNN consistently outperforms a series of models based on KDMN, which leverages a dynamic memory network to preserve the retrieved external knowledge. Since both MCR-MemNN and KDMN adopted the memory augmentation technique, this observation further evidences the effectiveness of modeling inter-relationships among image, question and KB for KB-VQA task. Note that Mucko does not provide the result without Semantic Graph.

To validate the superiority of the proposed MCR-MemNN on KB-VQA, several ablation experiments were conducted based on FVQA dataset, and we can make the following observations based on Table 4:

1) MCR-MemNN adopts both subject and object as clues for KB retrieval, which leads to better performance. Specifically, compared with Case-1, where only subject is adopted as clues for KB retrieval, there exists a jump on top-1 (i.e., $3.44\%$) and top-3 (i.e., $5.40\%$) accuracies when both of the subject and object are leveraged as clues in Case-3. A similar gain can be observed when Case-3 is compared with Case-2.

2) To validate the effectiveness of the relation filtering, the experiment is conducted in Case-4, which achieves $2.48\%$ improvement over Case-3 on top-1 accuracy. This observation clearly implies that the predicted relation can successfully remove the redundant supporting facts retrieved from the external KB. Similarly, compared with Case-5 without relation filtering, Case-6 brings up an improvement of $2.07\%$.

3) The two-way attention mechanism (i.e., Sections 3.3.3 and 3.3.4) can deliver an additional performance gain on KB-VQA task. For instance, compared with Case-4, where the inter-relationships are not exploited among image, question and KB, Case-6 brings up improvements of $3.40\%$ on top-1 accuracy. This indicates that the redundant information of the three modalities (i.e., image, question and KB) is removed during the reasoning process, and the most relevant information is collected by modeling inter-relationships among the three modalities.

Figure 4 shows several success and failure cases using MCR-MemNN. Two steps are indispensable to predict the correct answer: (1) Either the correct subject or object is included in the top-40 predicted subject set or object set. (2) The correct relation is predicted as the top-3 relations. The ground-truth fact will be retrieved as one of the candidate supporting facts, if both the two steps are successfully accomplished. For instance, all five samples in the first row have their corresponding ground-truth facts successfully predicted and the correct answers are delivered. Specifically, the first two samples have both of their subjects and objects correctly predicted. The 3rd sample have its subject correctly predicted while the last two have their objects correctly predicted. This clearly verifies the advantage of multiple clues reasoning, retrieval using two complementary clues (i.e., subject and object) makes it a lot easier to fetch the ground-truth fact and deliver the correct answer. Some other cases are presented in the second row. Generally, if a wrong fact is predicted (e.g., the 2nd, 3rd and 5th samples), the correct answer cannot be given. However, in some special cases, even if the ground-truth fact is not successfully predicted (e.g., the 1st sample), the correct answer can still be delivered. If the correct relation is not included in the top-3 (e.g., the 3rd sample), the correct answer cannot be given even if the subject or object is correctly predicted. The MCR-MemNN always fails if the question is intend to know the relationship between subject and object (e.g., 4th sample).