Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model

A general VLM is an AI model that can understand images and texts, along with their relationships, in various tasks. One of the earliest approaches to creating a general VLM was to combine separate vision and text encoders, aligning features from both encoders [18, 24]. Another approach utilized a pre-trained LLM, which can understand the meaning and context of texts. Tsimpoukelli et al. [20] proposed training and combining a vision encoder with a frozen LLM. Similarly, Alayrac et al. [2] proposed finetuning the connecting layers between an LLM and a vision encoder. More recently, Liu et al. [14] proposed training VLMs using the instructions of various tasks to make a general-purpose assistant. These general VLMs can be applied to various vision-language tasks, but they often lack expertise in specific domains. CoQAH addresses this issue by employing a specialized VQA model (e.g., chest X-ray VQA) that interacts with an LLM.

Creating VQA datasets manually is a time-consuming and expensive process. To overcome this challenge, some researchers have attempted to synthesize QA pairs using pre-defined templates [5, 7]. For instance, one study introduced a VQA dataset for chest X-rays by extracting abnormal findings from radiologist’s reports [5]. Similarly, another study created 3D-rendered images of various objects and generated complex template-based questions related to the visual features of those images [7]. Several other studies have used such datasets to train and evaluate the visual understanding capabilities of AI models [6, 17, 9, 5].

Several studies have used an LLM to solve VQA problems [22, 19, 4, 27, 23]. This is done by feeding the relevant information to the LLM, which then uses its high contextual understanding and reasoning ability to answer the question. For example, some researchers [22, 19, 4] have proposed feeding an image description from a captioning model to an LLM, while others have utilized VQA and captioning models together to give visual clues to an LLM [27]. Recently, some researchers have suggested using iterative interactions between an LLM and a VQA model to answer complex visual questions [23]. CoQAH uses a template-based VQA model that is specialized to questions in specific domains, which makes it different from other methods above.

The CoQAH method aims to conclude a reasonable answer for a given complex, human-written question (i.e., user question; see Figure 2a) by employing synthetic, template-based VQA data only. To achieve this, it utilizes interactions between two sub-models, a template-based VQA model ($VQA_{t}$; see Supplementary Table 1 and 2 for the performance of VQA models) and a large language model ($LLM$).

Suppose a user question and an image from an human-written VQA dataset ($q_{h}\in Q_{h},i_{h}\in I_{h};Q_{h}$ for human-written questions and $I_{h}$ for images) are given; first, the $LLM$ is prompted with a task instruction ($\rho$; see Section 3.2 for detail) that directs the model to generate a template-based question that conforms to the question templates used to train the VQA model ($q_{t}^{1}=LLM(\rho);$ $q_{t}^{1}\in Q_{t}$ where $Q_{t}$ includes all possible template-based questions; see Section 3.2). Next, $VQA_{t}$ answers the question ($a_{t}^{1}=VQA_{t}(q_{t}^{1},i_{h})$). After that, all the previous dialogue, including the task instruction, template-based questions, and answers from $VQA_{t}$ $(\rho+q_{t}^{1}+a_{t}^{1})$, is again fed into $LLM$, generating a next template-based question ($q_{t}^{2}=LLM(\rho,q_{t}^{1},a_{t}^{1})$). This QA process is repeated until $LLM$ has sufficient information to reach the final answer for the user question or it reaches the maximum number of questions to be queried (twenty questions for CLEVR-Human and five questions for VQA-RAD and SLAKE; see Section 5.4). Note that $LLM$ cannot access the image, whereas $VQA_{t}$ does.

The task instruction $\rho$ is a detailed description of how $LLM$ should respond to and interact with $VQA_{t}$ (Figure 2a). The instruction starts with Your task is to answer the following user question based on an image: followed by the user question. Then, we included a statement to declare that $LLM$ cannot see a given image (However, you cannot view the image, while I can.). After that, we asked $LLM$ to give a question that can only be generated based on a template format (You may ask me questions using the following format; see Section 3.3 for details). In the last part, we added some sentences to ask $LLM$ to query the next question step-by-step after having the response from $VQA_{t}$ (Ask me the next question after I have responded to you.) and conclude the answer for the user question whenever it is ready (Once you are able to answer the user question, stop asking further questions and provide me with the answer.).

Figure 2b describes how we instructed $LLM$ to generate a template-based question for the CLEVR dataset [7]. We first let $LLM$ know the structure of the question, which is composed of several different entities ($\langle question\rangle=\langle type\rangle+\langle object\rangle+\langle relation\rangle+\langle object\rangle$ in Figure 2b). Subsequently, we asked $LLM$ to choose one of the options available for each entity (e.g., small or large or $\langle Empty\rangle$ for $\langle Size\rangle$ entity). The template format for MIMIC-Diff-VQA is shown in Supplementary Figure 1. The detailed description of the task instructions for both CLEVR and MIMIC-Diff-VQA is displayed in Supplementary Figure 2 and 3.

Even if $LLM$ successfully generated a template-based question, in some cases, we observed that there were still some questions that had logical errors (see Figure 3): questions that ask the property of an object that does not exist in the image (existence violation in Figure 3b) and ask the property of multiple objects with the same condition (uniqueness violation in Figure 3c). For these types of questions, $VQA_{t}$ provided incorrect answers unexpectedly.

To address this problem, we introduced an existence and uniqueness handler (EUH), which continually checks the violation of both conditions on a template-based question generated from $LLM$. As described in Algorithm 1, we first extracted the object entity ($\langle object\rangle+\langle relation\rangle+\langle object\rangle$) from the question ($\langle type\rangle+\langle object\rangle+\langle relation\rangle+\langle object\rangle$). For example, the object entity ("small object" + "right of" + "shiny red object") was extracted from the question (What color is the small object right of the shiny red object?). Then, we checked the existence of the object entity by querying to $VQA_{t}$ (e.g., Is there a small object right of the shiny red object?). When the existence of the object was confirmed, we continued to check the uniqueness by asking $VQA_{t}$ to count the number of objects (e.g., How many small objects right of the shiny red object are there?). Using EUH, we prevented $LLM$ from directly providing a wrong answer without considering logical contradiction.

For datasets, we collected two types of images (and question-answer pairs), including 3D-rendered images and chest X-ray images, to show the effectiveness of CoQAH in different image types. We first trained VQA models using template-based datasets (CLEVR [7] for 3D-rendered and MIMIC-Diff-VQA [5] for chest X-ray) and externally validated the model’s performance on the human-written datasets (CLEVR-Human [8] for 3D-rendered; VQA-RAD [10] and SLAKE [13] for chest X-ray).

We measured exact-match accuracy for the human-written questions that select answers from predefined candidates (all questions of CLEVR-Human and closed-form questions of VQA-RAD and SLAKE). For the open-form questions of VQA-RAD and SLAKE, we calculated LAVE_GPT-4 [15] that uses an LLM (GPT-4 in this paper) for the evaluation of answers.

Table 1 compares accuracy between the general VLMs, template-based, and finetuned VQA models. Overall, the general VLMs reported lower accuracy than the template-based VQA models, except for the latest GPT-4-vision (e.g., 60.1% for GPT-4-vision vs. 59.9% for MDETR). CoQAH achieved the highest accuracy with large gaps compared to all the general VLMs and template-based VQA models (e.g., 74.3% accuracy for CoQAH vs. 59.9% for MDETR). There are still some performance gaps between CoQAH and the finetuned models (e.g., 74.3% for CoQAH vs. 81.7% for finetuned MDTER). However, it is worth noting that obtaining a large number of human-written QA pairs and finetuning a model is very difficult for most VQA tasks in general, and our framework substantially improves VQA performance using only synthetic QA pairs that are much easier to collect.

Table 2 summarizes the performance of the medical foundation and template-based VQA models for VQA-RAD and SLAKE. Among all the models, CoQAH reported the highest accuracy in the closed-form questions (e.g., VQA-RAD: 67.5% for CoQAH vs. 59.5% for OFA-MIMIC, SLAKE: 73.9% for CoQAH vs. 69.4% for OFA-MIMIC), and also the highest LAVE_GPT-4 in the open-form questions (e.g., VQA-RAD: 0.302 for CoQAH vs. 0.274 for MedVInT-TD, SLAKE: 0.425 for CoQAH vs. 0.396 for MedVInT-TD). There were huge gaps between the LAVE_GPT-4 of the OFA-MIMIC and CoQAH, meaning that the chain of QA process in CoQAH indeed improved the correctness of the answers significantly.

Since the CoQAH method utilizes an LLM to understand the context and finally answer the user question, we can ask the rationale of the final answer to the LLM (i.e., interpreting the AI model’s behavior), simply querying ‘why?’. For a few complex questions, we asked the LLM to explain the reasons to check whether it had logically driven their conclusions.

As shown in the examples of Figure 4, CoQAH successfully delineated the rationale behind the answers based on the information collected through the dialogues. For example, for the question of whether there are two cyan balls of the same size (Figure 4a), CoQAH reasonably explained the reason (one is large and one is small) for the answer (No) after checking the size of the cyan balls (Is there a large cyan ball? and Is there a small cyan ball?). More examples can be found in Supplementary Figure 8 (for 3D-rendered images) and 9 (for chest X-rays).

The maximum number of questions to be queried is important parameter since it can directly affect the quality of the answers from CoQAH. Therefore, we measured the accuracy of CoQAH on the CLEVR-Human validation dataset (5% used), changing that number from 0 to 30. As shown in Table 3, The maximum accuracy was achieved when the maximum number of questions was 20, and even if we increased the number to 30, the accuracy was the same.

We conducted an ablation study by excluding several components in CoQAH and measured the accuracy based on the CLEVR-Human validation data (5% used). The results of the ablation study are summarized in Table 4. When we ablated EUH in CoQAH, the accuracy on CLEVR-Human was slightly decreased (75.8% for CoQAH vs. 74.7% without EUH). When ablating the step-by-step QA process in CoQAH by enforcing the LLM to ask all the questions simultaneously (ask me up to 20 questions all at once), the accuracy was degraded mainly (75.8% for CoQAH vs. 63.1% without the process), indicating the importance of this process to derive the answers correctly. Finally, when we ablated the format description in the task instruction by letting LLM generate any form of questions during the chain of QA process, the accuracy was dramatically degraded (75.8% for CoQAH vs. 35.8% without the format description). This means that the questions from the LLM must conform to the template format so the VQA model can understand those questions. Additionally, to investigate the effect of few-shot prompting on CoQAH, we compared the performance of CoQAH in zero-shot, one-shot, and two-shot settings. Dialogues between the VQA model and the LLM were provided as few-shot exemplars to the LLM. The results are shown in Table  5. The performance increased when few-shot exemplars were given, but increasing the number of exemplars from one to two did not further improve performance.

We performed an error analysis by investigating the dialogues where CoQAH failed to answer correctly (5% CLEVR-Human validation used) and categorized them according to the reasons (e.g., incorrect answers from VQA model). The results of the error analysis are shown in Table 6.

The most significant portion of errors (48%) occurred when the LLM failed to query key questions or made a hasty conclusion after asking some irrelevant questions (e.g., Supplementary Figure 8e). It means that the LLM still has shown a limited reasoning ability during the QA process for some complex questions. The second largest portion (25%) was due to incorrect responses from the VQA model during the QA process (e.g., Supplementary Figure 8d). Some errors (11%) dedicated to the questions where the LLM was impossible to answer by querying template-based questions only (e.g., Supplementary Figure 8f). Other errors (15%) included failures of following instructions, data format (e.g., The answer should be in {}), and so on.