Visual Reasoning Evaluation of Grok, Deepseek’s Janus, Gemini, Qwen, Mistral, and ChatGPT

Early benchmarks, such as VQAv2 [14], GQA [18], and AI2D [19], laid the groundwork for evaluating multimodal models through single-image visual question answering (VQA). However, these datasets primarily assess perception and image-text alignment rather than deeper reasoning across multiple images.

To address this, recent benchmarks have expanded to multi-image contexts. NLVR2 [32] introduced logical comparisons between paired images, while MMMU [42] and MathVista [23] incorporated mathematical and diagrammatic reasoning using multiple visual inputs. MINT [36] and BLINK [12] further extended these evaluations by integrating temporal and spatial reasoning. Despite these improvements, these benchmarks do not explicitly evaluate reasoning consistency, answer stability, or a model’s ability to reject unanswerable questions, leaving a gap in assessing robustness in real-world scenarios.

Recent benchmarks such as SEED-Bench [20] and ChartQA [26] have introduced confidence-based evaluation, but their focus remains limited to generative comprehension rather than multi-image, multi-choice answer stability. Meanwhile, MMLU-Math [16] and ReClor [41] analyze answer order biases in textual reasoning, yet they do not extend this approach to multimodal contexts.

MUIRBench [34] represents a significant advancement in multi-image visual reasoning by incorporating an unanswerable question framework to evaluate rejection-based reasoning, testing whether models can recognize when no correct answer exists. However, MUIRBench does not incorporate reordered answer variations, making it difficult to determine whether models rely on positional heuristics rather than genuine comprehension. Without testing reasoning consistency across reordering, models may appear competent in fixed-choice settings while failing to generalize across structurally altered answer formats. Addressing these limitations is essential for evaluating multimodal models’ robustness and ensuring they do not exploit shortcuts in answer positioning rather than engaging in true visual reasoning.

One of the overlooked weaknesses in current multimodal benchmarks is the reliance of models on heuristic patterns or randomness rather than true comprehension. This is particularly evident in the positional bias of multiple-choice answers, where models may favor specific answer choices based on their order rather than reasoning through the question. While text-based evaluations such as ReClor [41] and MMLU-Math [16] incorporate reordered answers to test robustness, no major benchmark systematically applies this technique in multi-image reasoning.

To provide a more comprehensive assessment, our benchmark builds on MUIRBench by incorporating four key criteria: (i) Multi-image reasoning, (ii) unanswerable question recognition, (iii) reordered answer variations, and (iv) entropy-based reasoning consistency. In particular, we introduce entropy as a novel metric to measure consistency in responses across reordered versions of the same question. By quantifying variability in answer distributions, entropy identifies models that exhibit instability in reasoning, revealing reliance on positional heuristics or superficial patterns rather than genuine content understanding. Moreover, our benchmark evaluates rejection accuracy, determining whether models correctly abstain from answering when no valid option exists.

By integrating multi-image reasoning, reordered answer variations, entropy-based reasoning consistency, and rejection-based evaluation, this benchmark contributes a novel framework for diagnosing model weaknesses and guiding future advancements in multimodal LLMs and visual reasoning benchmarks. This dual approach of reordered answers and entropy sets a new standard for reasoning consistency evaluation, ensuring more robust, reliable, and generalizable AI systems.

To evaluate the multi-image reasoning capabilities of multimodal LLMs, we curated a subset of 120 questions and 376 images from the MUIRBench [34] dataset, ensuring a balanced assessment across diverse reasoning tasks. Unlike traditional benchmarks that focus on single-image tasks, this dataset challenges models to process and integrate multiple visual inputs, making it a more rigorous evaluation framework for contextual, spatial, and logical reasoning.

The dataset maintains diversity in image types, as illustrated in Figure 1, spanning real-world photographs, medical imagery, scientific diagrams, and satellite views. By integrating both curated and synthetic data, the benchmark mitigates data contamination risks, ensuring models are tested on truly novel inputs.

Additionally, the dataset includes a range of multi-image relations, depicted in Figure 1, challenging models to process various dependencies such as temporal sequences, complementary perspectives, and multi-view representations. This structured approach ensures that models are tested beyond simple pattern recognition, requiring advanced spatial, logical, and comparative reasoning.

The selection of questions from MUIRBench was guided by the need for a balanced and diverse evaluation while maintaining computational feasibility. Given that MUIRBench consists of 2,600 multiple-choice questions across 12 multi-image reasoning tasks, we curated a representative subset of 120 questions and 376 images, averaging 3.13 images per question. This selection ensures an efficient assessment of multimodal LLMs while focusing on key reasoning abilities.

We prioritized coverage of core multi-image tasks, selecting 8 diverse tasks that span both low-level perception (e.g., difference spotting) and high-level reasoning (e.g., image-text matching) as seen in Fig 1. Tasks with high significance in multimodal evaluation, such as Image-Text Matching and Difference Spotting, were given higher representation, each comprising 28 questions. Additionally, we ensured diversity in multi-image relationships, including temporal sequences, complementary perspectives, and multi-view representations, to evaluate models’ ability to integrate spatial, logical, and comparative reasoning across multiple images.

Our selection also maintained MUIRBench’s answerable-unanswerable pairing strategy by including 40 unanswerable questions. This prevents models from exploiting shortcut biases and lucky guesses and encourages a deeper understanding of visual content.

Additionally, we incorporated alternate versions of each question by reordering the answer choices. This ensures that models do not rely on positional biases or heuristic patterns but genuinely understand and reason the multi-image context. By shuffling the answer choices, we can verify whether a model is consistently selecting the correct response rather than guessing based on positional tendencies.

This question refinement strategy improves model robustness by ensuring they rely on reasoning rather than heuristics. Our study offers a rigorous and efficient way to test how well next-generation vision-language models can reason about multiple images.

In our evaluation, we utilized a diverse set of multimodal language models, as seen in Table I, each designed with unique capabilities to process and reason over visual inputs. Below is a brief overview of each model:

To ensure a standardized and fair assessment of all multimodal LLMs, we established a strict evaluation protocol that handles varying model constraints, answer formats, and consistency measures.

As illustrated in Tables II and III, and Figures 5 and 6, ChatGPT-o1 was of the highest performance, achieving an accuracy of 0.825, followed by Gemini 2.0 Flash Experimental (0.717) and ChatGPT-4o (0.691). The lowest-performing models were Janus 1B and Janus 7B, with accuracy scores of 0.383 and 0.433, respectively. Grok 3, despite its large parameter count of 2.7 trillion, showed disappointing results with an accuracy of 0.580, underperforming in tasks requiring complex reasoning and consistency.

Additionally, QVQ-72B-Preview demonstrated high performance in geographic understanding, but struggled with image-text matching and difference spotting, which highlights a domain-specific strength but overall inconsistency. Qwen2.5-VL-72B-Instruct displayed moderate accuracy (0.633) but encountered challenges in cartoon understanding, likely due to content restrictions preventing the processing of specific images.

The difference in accuracy between the highest- and lowest-performing models is 0.442, highlighting a substantial performance gap across evaluated models. ChatGPT-o1 leads by a margin of 13.3 percentage points over the second-best model, Gemini 2.0 Flash Experimental.

Additionally, as seen in Figure 6, ChatGPT-o1 maintains the most balanced high performance across tasks, exhibiting consistently strong accuracy without significant weaknesses in any category. In contrast, models like Pixtral 12B and Grok 3 display more uneven performance, excelling in certain tasks while struggling in others. Pixtral 12B demonstrates relatively strong results in tasks such as cartoon understanding and diagram interpretation but underperforms in difference spotting and image-text matching.

As seen in Table IV and Fig 7, QVQ-72B-Preview recorded the highest rejection accuracy at 0.850, followed by ChatGPT-o1 at 0.700. Janus 7B (0.150) and Janus 1B (0.050) had the lowest rejection accuracies, showing a strong bias toward selecting an option.

For Abstention Rate, QVQ-72B-Preview had the highest rate at 0.425, followed by Grok 3 (0.375). Both exceed the threshold of 0.33, which corresponds to the proportion of questions where the correct answer was “None of the provided options.” This indicates a tendency to over-reject. In contrast, Janus 1B had the lowest abstention rate at 0.042, showing a strong reluctance to abstain even when uncertain. ChatGPT-o1 maintained a moderate abstention rate of 0.267, staying below the 0.33 threshold and demonstrating balanced uncertainty management.

ChatGPT-o1 achieved both high total accuracy (0.825) and balanced rejection accuracy (0.700) with an abstention rate of 0.267, showing well-calibrated uncertainty handling. QVQ-72B-Preview, despite high rejection accuracy (0.850), had an abstention rate (0.425) above the 0.33 threshold, suggesting overrejection. Grok 3 also showed an abstention rate of 0.375, indicating a tendency to overreject despite its high parameter count.

Janus 1B displayed the most extreme behavior with the lowest rejection accuracy (0.050) and abstention rate (0.042), reflecting an overconfident approach. ChatGPT-4o and Gemini 2.0 Flash Experimental both showed moderate abstention rates below 0.33, maintaining a balanced decision strategy.

To assess reasoning consistency across reordered answer variations, we compute entropy scores for each model. As shown in Table V, models with higher entropy scores, such as Janus 7B (0.8392), Janus 1B (0.787), and Pixtral 12B (0.557), exhibited greater response variability, suggesting fluctuations in answer selection across question variants.

In contrast, ChatGPT-o1 (0.1352), ChatGPT-4o (0.216), and Grok 3 (0.256) achieved the lowest entropy values, indicating the most consistent reasoning patterns. These results suggest that ChatGPT and $\mathbb{X}$AI models maintain more stable reasoning, whereas other models are more influenced by positional biases or random variations in answer selection.

A common misconception in the research community is larger models perform better due to increased capacity for processing and learning complex relationships. While this trend holds in many cases, our results suggest that size alone does not guarantee superior performance.

In this benchmark, the smallest models, Janus 7B and Janus 1B, significantly underperformed against their larger counterparts. Given their lower parameter count, this is expected, yet their frequent positional biases, and inconsistent reasoning indicate that their training or optimization was insufficient to compensate for their size limitations.

However, Grok 3, despite having the largest parameter count of 2.7 trillion, showcased low performance relative to its size, underperforming in tasks requiring complex reasoning and consistency. Its moderate rejection accuracy (0.525) indicate inconsistent reasoning stability, suggesting that sheer model size does not necessarily lead to more robust performance or consistent decision-making.

These findings underscore that while larger models generally have greater capacity, effective optimization and fine-tuning are critical for maximizing performance and consistency.

While open-source models offer transparency and adaptability, this benchmark demonstrates that proprietary models continue to hold a significant advantage in complex multimodal reasoning tasks. ChatGPT-o1 and Gemini 2.0 Flash outperformed all open-source models across most reasoning tasks, underscoring the benefits of high-quality fine-tuning and diverse pretraining data. In contrast, open-source models like Janus 1B and Janus 7B exhibited significant reasoning inconsistencies (i.e., hallucination) and biases, suggesting that limited access to high-quality data remains a major constraint for non-proprietary models. However, Pixtral 12B emerged as a surprising exception, performing well in specific tasks such as Cartoon Understanding, demonstrating that with targeted fine-tuning and optimization, open-source models can still be competitive in certain domains.

One of the most unexpected findings was the high rate of unanswered or incorrectly answered questions from Qwen2.5-VL-72B-Instruct and QVQ-72B-Preview due to potential content restrictions. This issue was particularly evident in Cartoon Understanding, where the model struggled with half of the test cases despite the images being non-explicit or hazardous.

This suggests that Qwen’s content filtering mechanisms are overly aggressive, preventing it from engaging with conventional content. This restrictive nature severely limits Qwen’s applicability in real-world multimodal reasoning, particularly in cases where understanding humor, memes, or non-literal content is necessary. A more balanced content moderation approach could improve its usability in diverse applications without compromising ethical safeguards.

Grok 3, currently in its Beta version, was positioned as a powerful contender to ChatGPT-o1, boasting an impressive 2.7 trillion parameters, the largest in this benchmark. Despite its scale and claims of advanced reasoning, Grok 3 fell short of expectations, achieving underwhelming results taking its size into consideration.

Additionally, Grok 3 exhibited an unusually high abstention rate, indicating a tendency to reject answers more frequently than average, even when correct options were available. This suggests an overly conservative approach to uncertainty, which undermined its decision-making effectiveness.

While it showed moderate success in specific tasks, Grok 3 failed to outperform ChatGPT-o1 or leverage its scale for superior reasoning stability. Overall, Grok 3, although it is still in its beta version, demonstrates that model size alone does not equate to improved accuracy or reasoning consistency.

The release of DeepSeek’s Janus models was widely anticipated, particularly following the success of DeepSeek’s R1 model, which made significant strides in competing with ChatGPT-o1 despite utilizing far fewer resources [15]. This led to growing expectations that DeepSeek could emerge as a serious competitor to OpenAI across multiple AI domains. However, this benchmark demonstrates that such expectations do not hold up in visual reasoning tasks.

Janus 7B and Janus 1B, the smallest models in this benchmark, also rank among the weakest performers, struggling significantly in numerous tasks, where they exhibit severe positional biases and incosistent reasoning. The accuracy evaluation shows that Janus models fail to generalize well across answer variations, reinforcing that their reasoning ability is not robust to minor changes in answer presentation.

While some expected DeepSeek Janus to be a strong open-source alternative to ChatGPT models, its current iteration suggests that DeepSeek’s approach does not yet scale effectively to multimodal reasoning. This indicates that DeepSeek’s advancements in language modeling with R1 do not yet extend effectively to multimodal reasoning, leaving the Janus models far behind proprietary competitors like ChatGPT-o1 in this domain.

While there has been significant development in multimodal models across different organizations, ChatGPT models continue to demonstrate an overwhelming advantage in visual reasoning tasks. ChatGPT-4o and ChatGPT-o1 outperformed every other model in almost all reasoning tasks, particularly excelling in Diagram Understanding, Image-Text Matching, and Visual Retrieval. The accuracy table further highlights that these models maintained stable performance across different variations of the same questions, indicating their robustness in reasoning and comprehension.

These models consistently provided the most stable and accurate responses, exhibiting the lowest rates of hallucination and positional bias. While some open-source models, such as Pixtral 12B, showed promising results in specific areas, none were able to compete across the board with OpenAI’s ChatGPT models. This result reinforces the idea that proprietary models still maintain a significant edge in multimodal visual reasoning, likely due to access to better training datasets, fine-tuning techniques, and alignment strategies that remain unavailable to the public. Future developments in open-source multimodal models will need to focus on improving robustness and reducing bias to close the gap with proprietary alternatives.

One of the key contributions of this study was the inclusion of reordered answer variants and the application of entropy to detect positional biases and randomness in model responses. This combined approach exposed significant differences in reasoning stability across models, highlighting the extent to which certain models rely on positional heuristics and arbitrary answers rather than genuine comprehension.

By using entropy to quantify variability in answer distributions across reordered variants, we effectively measured reasoning consistency and stability. Models with high entropy scores, such as Janus 7B (0.8392) and Janus 1B (0.787), exhibited significant inconsistencies across reordered variants, suggesting reliance on positional heuristics or randomness rather than content-based reasoning. Their fluctuations indicate that answer selection was influenced by surface-level patterns rather than a stable understanding of the question, particularly in tasks requiring multi-image integration.

In contrast, ChatGPT-o1 (0.1352) and ChatGPT-4o (0.216) demonstrated the lowest entropy scores, indicating strong reasoning stability and resistance to positional biases. These models consistently selected the same answer across reordered variations, suggesting they engage in more content-driven decision-making. While low entropy does not inherently guarantee higher accuracy, most of the top-performing models exhibited lower entropy values, showing a strong correlation between reasoning stability and overall model performance.

These findings highlight the limitations of traditional VQA evaluations, which measure correctness but not reasoning stability. By combining reordered answer variants with entropy as a consistency metric, this study provides a more nuanced and robust assessment of reasoning stability. This dual approach distinguishes models that genuinely comprehend content from those that rely on answer positioning and randomness. Future multimodal benchmarks should incorporate both reordered answers and entropy-based consistency metrics to assess reasoning robustness more effectively and mitigate heuristic-driven decision-making.

The majority of models demonstrated a consistent tendency to avoid selecting “None of the choices provided,” even when it was the correct answer. This pattern was most pronounced in Janus 7B and Janus 1B, which exhibited extremely low rejection accuracy and abstention rates. Their strong bias towards selecting an option, regardless of correctness, suggests an overcommitment to answers, likely influenced by pretraining datasets that emphasize choosing the best available choice.

In contrast, QVQ-72B-Preview and Grok 3 displayed the highest abstention rates, exceeding the 0.33 threshold for the proportion of questions where “None of the provided options” was correct. This indicates a more conservative approach to decision-making, with a tendency to over-reject. QVQ-72B-Preview consistently identified unanswerable questions, while Grok 3 exhibited more variability, reflecting inconsistent uncertainty calibration.

ChatGPT-o1 and ChatGPT-4o maintained balanced rejection reasoning, selectively abstaining without excessive avoidance. Their abstention rates remained close to the 0.33 threshold, indicating well-calibrated uncertainty recognition and strategic decision-making.

These findings reveal distinct patterns in rejection behavior, highlighting that while some models consistently avoid selecting “None of the choices provided,” others exhibit over-rejection tendencies. The balanced approach observed in ChatGPT models underscores the importance of effective uncertainty calibration for strategic rejection-based reasoning.