BloomVQA: Assessing Hierarchical Multi-modal Comprehension

First established by Bloom et al. (1956) and further revised by Anderson et al. (2001), Bloom’s Taxonomy describes a framework categorizing learning objectives based on levels of complexity of underlying cognitive processes. There are six major categories, each represented by a set of skills and actions describing the corresponding cognitive processes. Specifically, direct recall of concrete facts and knowledge is considered as the basis (Level 1) of objectives requiring higher-order reasoning and abstraction, from understanding and being able to explain (Level 2) to creating and generating original works (Level 6). While Bloom’s Taxonomy has been widely adopted in K-12 and college education as the guidance for designing teaching instructions and assessments Thompson et al. (2008); Shuhidan et al. (2009), limited effort has been made to transfer the taxonomy to the machine learning domain as the guidance for analyzing, assessing and improving learning. Zhang et al. (2021) propose a learning-based model for automatic classification of education questions based on Bloom’s Taxonomy. Sahu et al. (2021) use Bloom’s Taxonomy to form proximal clarifying contexts and study their impact on language models. In this work, we propose a novel VQA dataset to support hierarchical model assessment guided by Bloom’s Taxonomy. Specifically, we use the action verbs associated with different Bloom’s levels to create templates for data collection. While it is noted in recent studies that using the action verbs alone may not be sufficient in fully describing the underlying learning process Larsen et al. (2022), the verbs provide us a concrete operational definition of the taxonomy as the guidance for understanding a coarse trend of model performance over data categorization.

Motivated by the multi-modal nature of human comprehension and communication Alikhani et al. (2023), many recent datasets seek to challenge VQA models by tasks requiring reasoning in different domains Zakari et al. (2022); Garg et al. (2022); Kafle et al. (2018, 2020). Several works introduce tasks requiring understanding based on multiple images. MovieQA Tapaswi et al. (2016) dataset contains large scale multiple-choice samples collected for story comprehension based on video clips. RecipeQA Yagcioglu et al. (2018) is constructed based on recipes with procedural text and image instructions to evaluate visual understanding over events with temporal relations in the form of clozing and ordering tasks. SlideVQA Tanaka et al. (2023) proposes data and tasks requiring multi-hop reasoning over slide decks with multiple pages. GQA Hudson and Manning (2019) leverages Visual Genome annotations Krishna et al. (2017) to introduce tasks requiring reasoning with respect to entity relations in scene graphs.

In comparison to existing data where categorization is either based on the prefix of the question or intuitive summary of the tasks, we provide principled and theoretically-grounded categorization based on Bloom’s Taxonomy. Furthermore, we generalize the concept of scene graphs and propose the Story Graph which integrates entity-level relations, temporal relations and relations defined by the cognitive processes from Bloom’s Taxonomy, enabling hierarchical data augmentation and comprehensive model evaluation with novel metrics.

We propose a novel dataset for systematic assessment of multi-modal comprehension. We collect picture stories designed for educating young children from two Creative Commons CC BY 4.0 resources StoryWeaver ; Book Dash which provide collections emphasizing an Indian and an African cultural background respectively. We manually select 20 stories with relatively consistent artistic style, length (around 10-20 pages) and plot complexity so that tasks reflecting different levels of human comprehension and requiring different types of cognitive skills can be meaningfully collected based on the content of the stories. We filter out the stories with distorted artistic styles as they can pose additional challenges to both human annotators and learning models.

We provide a web-based UI (details included in the Appendix) where an annotator is asked to read through a picture story and then provide a set of 6 multiple-choice samples in English, each corresponding to one level in Bloom’s Taxonomy. In Bloom’s Taxonomy, each level of comprehension is associated with a set of action verbs describing underlying cognitive processes and skills (e.g., "identify" for Level 1). We use the action verbs to construct a set of more than 70 question templates as our operational definition of Bloom’s Taxonomy. For example, "How would you compare the <attribute> of <character> at the beginning and the end of the story?" is provided as a template based on the Level 4 action verb "compare". More examples are included in the Appendix. For each sample, we collect one template-based question and 4 free-form answers including 1 correct answer and 3 incorrect answers from the same annotator.

We collect data samples via Amazon Mechanical Turk (AMT) and manually review all inputs to select data with proper reflection of the story and the underlying Bloom’s level. For each story, we collect 10 sets of reviewed inputs from different annotators. To further reduce the bias in the dataset, we enforce annotators to provide answer choices with consistent lengths for the same question, with a length difference fewer than 5 words.

Detailed statistics about the data are shown in Table 1. In contrast to VQA data considering simple answers, we collect data with long free-form answers especially for high-level comprehension tasks. Overall we collect a core set of 1200 multiple-choice samples, based on which we perform systematic data augmentation as described in Sec 4. We demonstrate consistency analysis with a set of 12k augmented samples.

We perform human baseline on proposed dataset (1200 core VQA samples) with a small group of adult reviewers. The human baseline reports an average accuracy of $89\%$ with $2\%$ standard deviation. This preliminary study provides insights on the current gap between human and machine capabilities (GPT-4V reports an average accuracy of $75.3\%$ as shown in Table 2) on the proposed dataset.

Grounding VQA data to an established taxonomy provides not only theory-based categorization, but also guidance on systematic probing and generation of the data. Based on Bloom’s Taxonomy, we propose a hierarchical graph representation for picture story data. Referred to as the Story Graph, this novel representation naturally extends the concept of scene graphs Krishna et al. (2017). While scene graphs focus on representing low-level information such as geometric relations between local entities based on individual images, in a Story Graph (Figure 1), we represent different events in a single story as the nodes and represent event-level relations as the edges. In addition to low-level relations (e.g., temporal relations), we include a wide range of higher-level relations (e.g., logical relations, causal relations) corresponding to different levels of cognitive skills specified in Bloom’s Taxonomy (e.g., making comparison, making inference). As shown in Fig. 1, each of our templates correspond to one type of the edges in the Story Graph describing a particular cognitive skill. The type of edge is the same for all instantiations of the template. For example, the template "<Character><Action>, what would you do to change this action? <Answer>" always creates a Level 3 edge between two events, each encoding Level 1 details such as the character involved and the action taken. In this way, rich and diverse knowledge about the story requiring different levels of comprehension is organized in a hierarchical manner. The benefits of the proposed Story Graph are multi-fold. By traversing through the underlying graph, we can achieve systematic augmentation of VQA data following the taxonomy encoded in the graph. For example, given a task about a base event, we can construct an augmented question by incorporating knowledge about each connected event as the context. This augmentation not only expands the dataset combinatorially, but also introduces novel metrics for assessing the consistency of learning models. As context information introduced in the augmentation is labeled with Bloom’s levels, consistency analysis on how context knowledge requiring different levels of comprehension would affect base tasks can be performed to characterize model reliability.

We first evaluate the models with our core dataset of 1200 multiple-choice samples under two settings. In the first setting, we perform the "Hasty Student" experiment Tapaswi et al. (2016) by predicting multiple-choice answers solely based on the question text. This baseline measures underlying biases, as high accuracy indicates that the model is making up answers effectively without reading the actual picture stories. For experiments with three VLP models, we extract text embeddings of the question $q$ and each answer candidate $a_{i}$ from the set of answer choices $\{a_{i}\}_{i=1}^{N=4}$ using the text encoder and examine cosine similarity between the question embedding and each candidate answer embedding. We select the answer corresponding to the largest QA similarity as our prediction.

In the second setting, we perform the "Searching Student" experiment Tapaswi et al. (2016) by predicting the multiple-choice answer based on the cross-modal similarity defined as

where $g$ denotes the cosine similarity between normalized embeddings extracted from the text input (question $q$ and answer candidate $a_{i}$) and the image frames in the picture story where $\{v_{j}\}_{j=1}^{K}$ denotes a set of image frames from a story with $K$ pages and $v_{j}$ denotes the $j^{th}$ frame. We adopt max pooling for aggregating the similarity scores over image frames based on empirical performance of baselines. Other aggregation methods such as average pooling are tested with only minor performance differences noted. We select $a_{i^{\star}}$ with $i^{\star}=\arg\max_{i}l(\{v_{j}\}_{j=1}^{K},q,a_{i})$ as the model prediction.

Categorizing tasks based on Bloom’s Taxonomy enables evaluation of the consistency of model performance over tasks at different levels. As for humans, being consistent on relevant tasks is an important sign of comprehension on given subjects. In this work, we examine an intuitive hypothesis: models which succeed at more challenging tasks demonstrating comprehensive cognitive skills should be less likely to fail at easier ones requiring only basic skills. Let $X_{m}$ and $X_{n}$ denote a pair of VQA tasks from the same story and the same annotator at Bloom’s levels $m$ and $n$ respectively. We compute the likelihood of having a model solving $X_{m}$ correctly given that the model is correct on $X_{n}$ as

where $|S|$ denote the total number of annotation sets. $\hat{a}$ and $\bar{a}$ denote the predicted answer and correct answer for a task respectively. If $P_{m,n}$ with $m<n$ is higher than the unconditional accuracy at level $m$, it suggests that the model performance complies with the intuitive hypothesis. In other words, for a model demonstrating consistent comprehension patterns resembling human behavior, its probability of solving easier tasks from a lower Bloom’s level $m$ is expected to be high when the model is correct on more comprehensive tasks from a higher Bloom’s level $n$. Note that the formulation of consistency as conditional performance can be further expanded based on different hypotheses to examine various forward and backward consistency patterns Chen et al. (2023).

Based on the data augmentation strategy described in Sec. 3, we propose another strategy for probing model consistency by comparing its performance on visual comprehension tasks with and without background knowledge introduced. We consider the relevancy of background knowledge to a given task to be associated with the node connection in an underlying Story Graph constructed following Bloom’s Taxonomy. For a consistent model, its likelihood of success on a task should only be improved when relevant background information is provided and should not be affected by irrelevant information. Correspondingly, limited information from a lower-level task is less likely to affect the model performance on a higher-level task requiring comprehensive understanding of the story. While background information from a higher-level task involving advanced cognitive skills and reasoning is more likely to be useful for inferring the answer of a lower-level task. For example, considering a Level 3 base task ("If you are Nina’s father, what would you do to change her mood?") which requires distilling information from the story for problem solving, background information from a Level 1 context task ("What’s Nina’s father’s hair color? Black.") is less likely to be helpful. Therefore, a consistent model would have a similar performance on the augmented task ("If you are Nina’s father, who has black hair, what would you do to change her mood?") given the same set of candidate answers. Meanwhile, the background information from a Level 4 context task ("Why is Nina sad? Because she misses friends.") is expected to facilitate comprehension, leading to improved performance on the augmented task ("Nina is sad because she misses friends. If you are Nina’s father, what would you do to change her mood?").

Given a context task $X_{n}$ and a base task $X_{m}$ drawn from the core dataset at level $n$ and $m$ respectively, we construct an augmented task $X_{m|n}$ by prepending the question $q_{n}$ and the correct answer $\bar{a}_{n}$ of the context task to the base question $q_{m}$. We keep answer choices of the base task $a_{m}$. For experiments with three VLP models, we extend the Searching Student formulation for the augmented task by defining the cross-modal matching score between each candidate answer choice $a_{m,i}$ and the picture story with frames $\{v_{j}\}_{j=1}^{K}$ as

We select the answer corresponding to the highest score as the prediction for the augmented task. While combinatorial augmentation can be achieved, in this work, we examine a set of 12k augmented samples considering Level 1 data of the same story as the context task. Let $Y_{base}$ denote the model performance as a binary score for each base task without augmentation. Let $Y_{aug}$ denote the model performance as the average accuracy on the set of augmented tasks constructed by incorporating low-level contexts. We compute consistency defined as the average precision between two sets of scores Sahu et al. (2022): $AP(Y_{base},Y_{aug})$. This metric quantifies the predictability of model performance when probing the knowledge along the underlying hierarchical graph.

We perform zero-shot prompting experiments using the gpt-4-1106-preview model and the gpt-4-vision-preview model. In comparison to experiments using VLP models, we ask GPT-4V model to make a prediction based on all images in each story at once. We perform experiments on the core dataset with 1200 samples and a set of 1200 augmented data constructed by considering a random Level 1 context task from the same story for each base sample from the core dataset. We perform computation via OpenAI API. The total computation corresponds to approximately 8M tokens. We use the following prompts for multiple-choice evaluation. For each task, we randomly order the candidate answers and parse the output of LLM to compare the index generated following “My chosen answer is” to the ground-truth index of the correct answer. For GPT-4 experiments, we use the prompt:

Choose the best answer based on the question. End your response with ‘My chosen answer is’ followed by your chosen answer.
<Question>
<Candidate Answers>

For GPT-4V experiments, we use the prompt:

<Images>
Choose the best answer based on the story in the images. End your response with ‘My chosen answer is’ followed by your chosen answer.
<Question>
<Candidate Answers>

With prompts specified above, we receive indeterminate answers for 5% and 2% of the samples in QA-only and VQA experiments respectively. We report the average performance on the rest of the samples in Table 2-4.

In Table 2, we present graded evaluation on BloomVQA dataset with three recent VLP models Radford et al. (2021); Li et al. (2022b, 2023b) and a latest multi-modal LLM OpenAI (2023). We first compare the model performance on the core set of 1200 samples considering text inputs only ("QA") and multi-modal inputs ("VQA"). It is shown that all three VLP models have consistently low performance over different Bloom’s levels when making prediction based on QA similarity solely. In fact, the average performance is close to the random guess which demonstrates that the proposed dataset serves as a proper evaluation set for multi-modal data. In comparison to the QA baseline, all three VLP models demonstrate improved performance when exploiting multi-modal similarity for VQA prediction for most Bloom’s levels. We have similar observation for all three VLP models such that they achieve higher accuracy for multi-modal comprehension on lower-level tasks (level 1-2) in comparison to higher-level tasks (level 3-6). For comparison between three VLP models, we observe that the recent improved model (BLIP2) achieves the most gain in low-level (Level 1) tasks while similar performance is achieved by different models for higher-level tasks. This observation supports our hypothesis that VQA tasks corresponding to high-level human comprehension are not adequately addressed by current techniques, as existing datasets and models have been focusing on tasks reflecting memorization of low-level knowledge.

In Table 3, we consider the 1200 core samples and report model performance on tasks of a specific Bloom’s level (m) conditional on correct prediction on the task of the same story from the same annotator at a different Bloom’s level (n). With this set of results, we seek to compare the model performance with human comprehension patterns. We start with the hypothesis that comprehension at a lower level is less likely to fail when comprehension at a higher level is already achieved, as many lower-level comprehension skills are readily required in solving higher-level tasks. We observe a pattern in model performance aligned with this hypothesis in some scenarios. For example, as shown in Table 3(a), Level 2 accuracy given success of higher-level (Level 3-6) tasks are generally higher than Level 2 accuracy given success of lower-level (Level 1) tasks. However, this observation does not hold across different Bloom’s levels and different models. For example, as shown in Table 3(c), Level 5 accuracy conditioned on success of higher-level (Level 6) tasks is lower than Level 5 accuracy conditional on success of lower-level tasks (Level 1-4). The misalignment between model consistency patterns and human comprehension patterns surfaces the lack of grounding in model responses.

In Table 4, we further compare the performance of different models evaluated on VQA data samples with and without augmentation. For core data samples at each Bloom’s level, we augment the data by incorporating Level 1 knowledge about the same story as the context for the original question ("aug"). We further quantify the consistency of model performance between data with and without augmentation using Average Precision ("AP") scores as described in Section 4. Overall we observe that all three VLP models have a similar consistency level averaged over 6 different Bloom’s categories. The high consistency scores indicate that model performance can be perceived with confidence as low-level distractions have small effect on the model performance. Considering consistency measured with base tasks at different Bloom’s levels, we observe that Level 3 and Level 4 base tasks tend to be affected at a relatively greater scale by additional context from Level 1. This observation serves an example of model-specific comprehension patterns disclosed by proposed consistency analysis.

As shown in Table 2, GPT-4V shows strong performance across different Bloom’s levels. However, the comparison between text-only experiments using GPT-4 ("QA") and multi-modal experiments using full GPT-4V ("VQA") indicates that the high accuracy of the model may stem from biases and hallucinations. The model’s gain from incorporating visual inputs decreases on tasks with increasing Bloom’s levels, suggesting a tendency of the model to take shortcuts especially for tasks requiring higher-level comprehension. Specifically at Level 6, the model performance degrades when visual data is introduced. In comparison to VLP models, GPT-4V handles visual inputs at story level. This may contribute to the difference in model performance. We choose picture stories designed for young children as they have simple plots suitable for demonstrating various comprehension tasks without introducing unnecessary complexity. However, with such data there can potentially exist common-sense biases especially for tasks at higher levels which require more abstract comprehension, as the stories are often based on common sense. Therefore the higher performance of GPT-4V in comparison to other baselines can also be a sign that the model is better at leveraging common sense. In this work, instead of comparing models with respect to absolute accuracy, we focus on examine whether the model prediction is grounded to the visual data and the consistency patterns of human comprehension.

In Table 3(d), we show GPT-4V performance conditional on successful prediction on tasks about the same story at a different Bloom’s level. We observe that the model consistency pattern deviates from the hypothesis made based on human comprehension patterns in multiple cases. For example, Level 5 accuracy conditional on the success on higher-level (Level 6) tasks is lower than Level 5 accuracy conditional on the success on lower-level tasks (Level 1-4). Furthermore, as shown in Table 4, in comparison to the VLP models, GPT-4V has a lower average precision score which corresponds to lower consistency on data with and without augmentation by introducing irrelevant information. These observations suggest that, although GPT-4V is demonstrating generally improved VQA accuracy in comparison to earlier models, it may still fall short in demonstrating consistent comprehension following human intuitions.