Instructions for *ACL Proceedings

We utilize ChatGPT to generate ideas on how to manipulate natural images via Chain of Thoughts (CoT). Given an initially determined L-H category, we prompt ChatGPT to identify a corresponding low-level perception to support it. For instance, in Figure 2, considering the “Behaviour-Background Substitution” category, ChatGPT first generates an idea to change the woman’s behaviour from attending a party to engaging in an experiment. Under this guidance, the background of the manipulated image should be a laboratory environment. Specifically, we provide auxiliary information such as the description of the manipulated image, which is incorporated into the textual prompt for image generation in Step 2.

To ensure coherence between the generated idea and the subsequent visual editing, we fixate on a specific subject at this initial step utilizing the visual grounding ability of Shikra (chen2023shikra). Specifically, we employ Shikra to retrieve the coordinates of a selected subject ($C_{sub}$) and utilize it to query low-level features (e.g., “What is the man holding?") from the image in the subsequent steps.

We define three categories of manipulated image generation based on the image-editing type: Overall Background Substitution, Partial Component Substitution, and Direct Manipulation.

1. Partial Component Substitution. Partial Component Substitution refers to manipulating an image by substituting an object or a part of the main subject. The pipeline utilizes Shikra to extract the target object’s coordinates ($C_{obj}$), with $C_{sub}$ serving as a constraint. After masking $C_{obj}$ as a blank, we apply the Stable-Diffusion-Inpaint stacchio2023stableinpainting as a tool, using the edited image’s caption obtained from step one as the prompt to generate a manipulated image. A set of defined L-L categories, $\{B_{2},B_{3},B_{4},R_{2},I_{1},I_{2},I_{3},E_{1}\}$, can be executed in this process.

2. Overall Background Substitution. Overall Background Substitution represents generating a manipulated image by retaining solely the main subject while replacing the entire background. In these cases, a standard rectangle cannot exactly mask the subject, potentially remaining unexpected element and distorting the background generation. To address this limitation, we employ the Segment Anything Model to produce a set of detected object masks ($\mathbbm{M}=\{M_{1},M_{2},...,M_{n}\}$) in irregular shapes for a given image. We identify a mask with the greatest overlap with $C_{sub}$.

Here, $Overlap$ refers to a function that calculates the overlapping square between two regions. To enhance flexibility and increase the case difficulty, we randomly translate the location of $C_{sub}$, rescale the $C_{sub}$, and resize the entire mask. Finally, with the new mask and the manipulated image’s caption obtained from Step 1, we utilize Stable-Diffusion-Inpaint to generate a new image with a different background from the original natural image. This process can handle $\{B_{1},R_{1},S_{1}\}$.

3. Direct Alteration. Direct Alteration addresses situations where nothing can be substituted, yet some alteration is necessary, such as changing facial expressions. With the original natural image and the manipulation instruction obtained from Step 1, we directly utilize the image-editing model InstructPix2Pix brooks2023instructpix2pix to generate a manipulated image for ${E_{2},S_{2}}$. However, since this process cannot focus on specific subjects, we mainly apply it to images containing a single person or cases requiring overall manipulations.

We generate Yes/No questions, Single- and Cross-Image multiple-choice questions using ChatGPT based on the ideas generated at Step 1. Single-Image questions focus on the discrepancy between image pairs, while Cross-Image tasks focus on the differences between each image pair. To ensure the quality of generated questions, two of this paper’s authors manually verified all $3205$ questions. A question was retained only when both annotators accept it. Finally, $1872$ questions are retained within the MVP-Bench. While verifying Yes/No questions, we focused on: (1) the quality of manipulation and (2) the consistency between images and ground truths. For multiple-choice questions, we paid additional attention to cases where distractors were not discrepant with the ground truth. We manually adjusted these distractors and double-checked the cases to ensure both annotators accept it.

As shown in Table 1, both open- and closed-source models perform worse on high-level perception tasks than low-level ones, e.g., $55.22\%$, $52.17\%$, and $56.09\%$ compared to $68.52\%$, $66.67\%$, and $74.44\%$ of qAcc on MiniCPM-V-2, LLaVA-1.5-13B, and GPT-4o, respectively. The performance gaps demonstrate the challenges of high-level visual perception tasks. Specifically, we observe that closed-source models present a larger relative performance gap between high-level and low-level perceptions. For example, GPT-4o achieves an accuracy of $34.80\%$ (reduced by $52.98\%$ from $74.01\%$) on cross-image MCQ, compared to $18.06\%$ (reduced by $29.74\%$ from $25.99\%$) of LLaVA-1.5-13B. This indicates that the performance gains from closed models mainly come from their superior low-level perceptions, yet they still encounter challenges in high-level tasks. We further discuss the potential cause of this observation in Section 5.3.

Small models can outperform the larger ones in Table 1. Among open-source models, MiniCPM-V-2-3B and DeepSeek-VL-7B achieve the best performance on high-level and low-level tasks respectively. As MiniCPM-V-2 is aligned with fine-grained correctional human feedback, it shows excellent trustworthiness and reduced hallucination. This implies that LVLMs’ trustworthiness may benefit their high-level visual perception. DeepSeek-VL demonstrates a strong capability of perceiving specific details with additional visual encoders for processing low-level features, indicating these features are crucial to low-level visual perception. Besides, comparing LLaVA and InstructBLIP with different sizes reveals that increasing parameters from 7B to 13B does not notably enhance their visual perception at either level. Therefore, to enhance LVLMs’ single-image visual perception, focusing on their ability to provide trustworthy answers and capture low-level features is more effective than simply scaling up.

Table 1 shows that closed-source models significantly surpass open-source models on cross-image task, especially at low perception level. For instance, GPT-4V and GPT-4o achieve accuracies of $44.50\%$ and $74.01\%$ respectively at low level, significantly surpassing the accuracy of LLaVA-1.5-13B ($25.99\%$). Furthermore, this performance gap is larger than that observed in single-image task. Specifically, in cross-image task, GPT-4o outperforms LLaVA-1.5-13B by $92.69\%$ and $184.76\%$ at two levels separately, compared to just $7.5\%$ and $11.65\%$ in single-image tasks. The significant gap indicates open-source LVLMs’ insufficient contextual attention, due to a lack of cross-image data during training.

As shown in Table 2, both open- and closed-source models show inferior performance on manipulated images compared to natural images. For example, MiniCPM-V-2, LLaVA-1.5-13B, and GPT-4o achieve an $iAcc$ of $68.64\%$, $58.58\%$, and $76.92\%$ on natural images, while exhibiting lower $iAcc$ of $53.85\%$, $55.62\%$, and $48.52\%$ on manipulated images. We attribute this observation to the discrepancy between the visual perception of manipulated images and LVLMs’ training data. Besides, closed-source models demonstrate a larger performance gap across image pairs than open-source models. The $iAcc$ gap of GPT-4V and GPT-4o is $40.3\%$ and $28.4\%$ separately, while LLaVA-1.5-13B and MiniCPM-V-2 have gaps of only $2.96\%$ and $14.79\%$. One reason for this is the rigorous manner of GPT-4V and GPT-4o (in Section 5.3). Unlike humans who only focus on relevant elements, these models equally scrutinize all the details with their prior knowledge. Their tendency to provide critical answers impedes the visual perception of manipulated images.

GPT-4V and GPT-4o present conflicting results on different tasks. Although both tasks are based on the manipulated images, two models perform poor on Yes/No task with an $iAcc$ of $30.77\%$ and $48.52\%$, while outperforming all open-sourced models on the MCQ task. From Table 2, we can witness that the results of MCQ and $iAcc$ on natural images share the same trend, which suggests that closed-source models’ inferior performance on manipulated images is owing to the nature of Yes/No questions. As an open-ended generative task, these models tend to perform rigorously and safely, while the MCQ task is less influenced by their rigorous manner. This is also a motivation for us to design both tasks for single-image perception.

Although GPT-4V exhibits the highest level of security among current LVLMs, its rigorous manner may hinder the straightforward perception of common visual contents. Specifically, GPT-4V usually approves only what it can directly observe from the image. It tends to refuse to interpret uncertain cases, such as conducting high-level perception without explicit visual clues. For example, as shown in Figure 4 (a), although GPT-4V accurately identifies the woman’s attire as a doctor’s uniform at the low perception level, it declines to provide the correct high-level perception that the woman is a doctor, as it cannot be directly observed in the image. This problem has been mitigated in GPT-4o, as it gives a correct answer.

To explore whether we can motivate GPT-4V to integrate commonsense knowledge via tuning the prompt, we add an instruction as follows:

You are a helpful visio-linguistic AI assistant who answers questions in short words or phrases on visual commonsense in the images.

As shown in Table 3, we observe a significant performance improvement in high-level Yes/No tasks on both GPT-4V and GPT-4o, while the performance changes on open-source models such as DeepSeek-VL-7B and LLaVA-1.5-7B are negligible. This implies that commonsense knowledge is essential to perform reasonable high-level perceptions, and specific designs of prompting are important to elicit this commonsense reasoning ability from closed-source models.

Although LLaVA-1.5-13B and DeepSeek-VL-7B can outperform GPT-4o on straightforward content like background ($qAcc$ of $92.42\%$, $86.36\%$ compared to $81.82\%$)²²2Appendix 4 demonstrates models’ performance on different categories of visual perceptions., they demonstrate worse performance on the object association perception requiring to recognize details ($qAcc$ of $50.00\%$, $58.93\%$ compared to $66.07\%$) and gesture perception requiring commonsense knowledge ($qAcc$ of $36.84\%$, $31.58\%$ compared to $59.46\%$). For instance, in Figure 4, LLaVA-1.5-13B and DeepSeek-7B respectively fail to detect the gun held by the elder man (left) and the emotion of the man (right), while GPT-4V and GPT-4o successfully identify both.

One hard case in MVP-Bench requires LVLMs to comprehend an entire image based on an inconspicuous object. In Figure 3 (bottom left), all LVLMs prioritize the shopping mall setting while overlooking the gun held by the woman. We attribute this to the data homogeneity of the training images, i.e., most training data is constructed by real-world images where a shopping mall closely correlates to shopping activities, misguiding the models to ignore the presence of the gun.

GPT-4V and GPT-4o tend to interpret occasional or dramatic scenes as staged images, especially when the co-occurrence frequency of visual elements is low based on commonsense knowledge. For example, in Figure 4 (bottom right), the case depicts that the behind woman is using a gun to attack another woman, while GPT-4V and GPT-4o regard this as a staged security training exercise. This suggests the over-reliance on prior commonsense knowledge of GPT-4V and GPT-4o, potentially obstructing their generalizability to understand and interpret occasional scenes and their inherent semantic meanings.