Quantity Matters: Towards Assessing and Mitigating Number Hallucination in Large Vision-Language Models

MSCOCO (Lin et al., 2014) is a large-scale object detection, segmentation, and captioning dataset. All tests in this study are conducted on the MSCOCO2014 validation set.²²2Test set is not used because we cannot find annotation results on the official website. Leveraging the provided detection and segmentation labels, we can construct a counting VQA dataset. Symbolically, for any given image $i$ with labeled bounding boxes $\{(b_{1},c_{1}),...,(b_{m},c_{m})\}$, where $b_{i}$ represents a bounding box and $c_{i}$ represents its corresponding object, the total number $n_{o}$ of a certain object $o$ in image $i$ can be denoted as:

where

Following this approach, we can construct a dataset with ground truth where each data instance can be denoted as $(i,c,n_{c})$, with $i$ denoting the given image and $c$ denoting the specified object. However, based on our observation, the distribution of answers is not uniform. For instance, in the original dataset, there are about 68% questions with answer $n_{c}=1$. What’s more, the distribution of objects is also non-uniform. To address these biases, we propose double k-uniform sampling. The core of this algorithm is limiting the number of data with object $c$ and answer $n_{c}=n$ not exceeding $k$. We present double k-uniform sampling in algorithm 1. Adjusting the value of $k$ allows us to strike a balance between achieving a more uniform distribution and retaining a larger dataset. In the subsequent discussion, we opt for $k=50$.

It is worth mentioning that, when constructing dataset, we only retain data $(i,c,n_{c})$ with $n_{c}>0$. There are two main considerations for this choice. First of all, there are many objects that do not appear in a given image and retaining all these questions is too expensive. If we perform k-uniform sampling similarly, this may result in random selection, which is insufficient for reflecting the model’s performance according to (Li et al., 2023a). More importantly, we conduct an experiment on LLaVA with randomly selected $(i,c,n_{c})$ with $n_{c}=0$ and find out that LLaVA correctly answers 95% of these questions, which meets the its high POPE evaluation result (Liu et al., 2023a). Based on these considerations, we only retain data $(i,c,n_{c})$ with $n_{c}>0$ to focus on the number of an existing object.

Another consideration is that, one bounding box in MSCOCO annotation may not exactly correspond to one existing object. Luckily, we found one solution of utilizing the ‘is_crowd’ field of the annotation. When constructing original dataset $S$, if there exists one annotation $(b_{j},c_{j})$ in image $i$ with ‘is_crowd’ being True, the data point $(i,c_{j},n_{c_{j}})$ is not included in original dataset $S$. This approach ensures the calculation of objects in the original dataset $S$ is correct.

After performing $k$-uniform sampling, we manually sampled and checked some labels in the final $S^{{}^{\prime}}$ to ensure the quality of the final dataset. Finally we obtain a dataset with 20200 pieces of data. The distribution of labels are listed in Figure 1.

As can be seen from the Figure 1, though k-uniform sampling is performed, questions with answer $\geq 10$ is still too less. Therefore to avoid randomness, when calculating F-scores, questions with answers $\geq 10$ are considered one category.

In evaluating the counting task, our focus extends to two perspectives. Firstly, we aim for the model to provide correct answers to as many questions as possible. Secondly, we seek minimal errors.

To address the first perspective, we treat this problem as a multi-classification task and employ F-scores(macro-F1 and weighted-F1) to evaluate the model’s performance. Macro-F1 is more suitable for non-uniformly distributed labels like the ones in our test set because it gives equal weight to both large and small categories. However, since there is less data in some categories, the value of macro-F1 may be more easily affected by the results of these data, which introduces more randomness. To address this problem, weighted-F1 is introduced to offer a more comprehensive understanding of the performance of models. Weighted-F1 gives larger weights to larger categories, which reduces randomness. However, if the model perform much better on larger categories than on smaller ones, weighted-F1 may be cheated. Thus only by combining macro-F1 and weighted-F1 can we fully reveal the capability of models.

For the second perspective, we use mean absolute error (MAE) as an evaluation metric. The employment of both F-scores and MAE provides a more comprehensive and quantitative understanding of the severity of number hallucination. For instance, a model that never accurately counts objects, even with a small MAE, is considered ineffective. Similarly, a model that occasionally counts objects correctly but often makes significant errors should also be deemed problematic. Only when a model that exhibits both high F-scores and low MAE can it be considered free from number hallucinations.

We use greedy decoding to make sure no randomness is introduced to the responses. All following evaluations use the same setting. Since the answers to our questions are relatively short, we believe this setting does not harm performance.

In this context, inner inconsistency refers to model responding ‘yes’ to even contradictory questions. An example is presented in Table 4. Despite the evident contradiction between the two questions (different $n$ setting), the model answer ‘yes’ to both. In fact, by comparing prompt I and prompt II, model shows this inner inconsistency at a ratio of 41%, which is very high because a model that randomly answers ‘yes’ or ‘no’ will bear an inner inconsistency of just about 25%. This high ratio reveals a significant challenge that it becomes uncertain what the model’s actual understanding is towards the question. Specifically, if the model answers ‘yes’ to a ground truth, we cannot confidently assert that the model precisely knows the number of objects in the given image. Moreover, even if the model answers ‘no’ to a hallucinated expression, it remains unclear whether the model accurately comprehends the ground truth or not. This inner inconsistency suggests that the model struggles to count the number of objects accurately and exhibits confusion, thereby contributing to number hallucination.

In this context, outer inconsistency refers to models responds ‘no’ to prompt II (because prompt II sets $n$ equaling to its own answer to the direct asking question). If it says this answers is incorrect, it conflicts with itself, which is inconsistent. As can be seen from the ‘yes’ ratio of prompt II, although model generally tends to agree with its own answers (prompt II), there are still notable instances where the model disagree with its own answers. In fact, models have an average inconsistency rate of 16.5% with its own answers except InstructBLIP. For InstructBLIP, it answers ‘yes’ to almost every question, and interestingly it still has the lowest ‘yes’ ratio of prompt II among the three prompts. This outer inconsistency further reveals the model’s uncertainty about the counting problem and may contribute to number hallucination.

Inner inconsistency is similar to what occurs in binary classification questions. In this context, inner inconsistency refers to the model providing different answers to the same question. By comparing within the same prompt format, it is surprising to find that after $c_{1}$ and $c_{2}$ are flipped, the model’s response sometimes changes. One example can be found in Table 6.

Our quantitative result shows that models generally output inconsistent answers for prompt I at a ratio of 25%, which is non-negligible. Prompt II also experiences this inconsistency even more severely (an example can be found in Table 6). For prompt II, the model responds with inconsistent answers of around 80%, which is disasterous. What’s more, if one only see the correct result(like the answers marked in green in) Table 6, he will probably fail to recognize the hidden hallucination inside the model, leading to oversight of the severity of hallucination.

This inconsistency reveals two facts. Firstly, the order of input and the format of prompt have a notable influence on the model’s output, making it challenging to evaluate the model’s performance with multiple-choice questions. Secondly, this inconsistency also reflects the model’s uncertainty about this question, which is a potential cause of number hallucination.

Outer inconsistency in this context refers to such a phenomenon that the model’s response to comparison task conflicts with its response to primal task. For instance, if the model responds there being $1$ cat and $2$ dogs in primal task but it answers there being more cats than dogs in comparison task, we say the model suffers from outer inconsistency.

Based on the definition above, we conduct a quantitative analysis, and it is not surprising to find that models also suffer from outer inconsistency severely on this comparison task. In fact, models only output 32% consistent answers averagely, which means under most circumstances the model does not completely agree with its own answer. In fact, a model that randomly responds to these questions will still achieve an outer consistency of about 33%, which is even higher than current results. Although this comparison task is more complicated and requires more complex reasoning, this low consistency is still absolutely a notable problem and should not be completely attributed to the complexity of this task.

In summary, through the analysis of results from these two perspectives, we observe severe inner and outer inconsistency problems. This inconsistency problem is not limited to any certain task or specific prompt. Instead, inconsistency appears everywhere without any regular pattern. These issues undermine the reliability of evaluation methods based on indirect questions. And, since this inconsistency represents the model’s uncertainty and confusion towards this task, we believe inconsistency is a potential contributing factor to number hallucination.

We utilize the MSCOCO2014 (Lin et al., 2014) training set as the foundational dataset and construct $S$ in the same manner as described in Section 3.2. We refrain from performing k-uniform sampling on the training set to avoid bias and retain as much data as possible. Since we have completely no idea about the data distribution in real-life scenarios, avoiding introducing prior biases and retaining as much data is more reasonable. This provides us with 233k pieces of data. For Direct method, all these data are used by directly asking like Section 3.1.

For Consistency(I), all these data are kept with prompt I, which provides us with 233k pieces of data similarly. The selection $m$ follows the same setting as setting III mentioned in Section • ‣ 4.1. That is, 50% of training data sets $m=n_{c}$, while the rest of training data uniformly selects $m$ around $n$.

As for Consistency(II), 100k pieces of data are randomly selected with prompt II. This selection aims at limiting the total amount of data. By combining these two datasets, we get a dataset with 333k pieces of data, corresponding to Consistency(I+II). Note that all these consistency data are automatically constructed from the whole original MSCOCO2014 training set.

We continue the experiments on LLaVA, InstructBLIP and MiniGPT-v2.⁴⁴4LLaVA and LLaVA-L bear the same architecture, so we only finetune LLaVA for simplicity. For simplicity and efficiency, we freeze the vision encoder as well as the LLM parts and only finetune the alignment part of each LVLM. For LLaVA, the alignment part is a two-layer MLP. For MiniGPT-v2, the alignment part is a linear layer. Regarding InstructBLIP, though the whole alignment part is a Q-former and a linear layer after it, we perform a preliminary experiment and observe that only finetuning the linear layer after Q-former instead of both Q-former and the linear layer offers a better performance on the Direct training method, so we also only finetune the linear layer after Q-former in InstructBLIP. Details of our hyper-parameters can be found in Appendix B.

We train with different consistency formats to fully reveal the performance of consistency method. Consistency(I) outperforms Direct on some metrics, which has already shown the effectiveness of consistency method. However, Consistency(II) does not perform as well, which we believe is because comparison only is too difficult and far from our primal task, making it less effective when tested with primal task.

We observe an interesting phenomenon that Consistency(I) often offer better weighted-F1 and worse MAE as well as macro-F1, while Consistency(I+II) may bear a little bit worse weighted-F1 with much better MAE than Consistency(I). We believe this phenomenon is because of the distribution of our training set. Since we do not perform double k-uniform sampling on our training set as mentioned, there are much more data with small answers rather than large answers, making the model more confident about generating smaller answers after introducing consistency with binary classification questions. But this actually harms the overall performance because it makes larger mistakes when counting more objects. However, when comparison question is introduced, the model is forced to focus on the whole image instead of trying to bypass by generating smaller answers, which result in a small drop in weighted-F1 while an increase in macro-F1 and MAE. This observation also highlights the importance of using these three metrics instead of one particular metric. As can be seen, using three metrics offers a more comprehensive understanding of the model’s performance.

Further, after combining prompt I and II, Consistency(I+II) outperforms Direct on almost all metrics and also beats Consistency(I) and Consistency(II) on most metrics, which highlights the effectiveness of our consistency method and the importance of both perspectives. Although either perspective only does not offer good enough results, combining them leads to an excellent result. This observation also meets our assumption that adding more perspectives may further ease the inconsistency problems and resulting in better overall performance.

Our method offers several advantages. Firstly, it solely focuses on data construction and is independent of specific model architecture. This model-agnostic characteristic makes it extendable to almost any large vision-language model (LVLM). As can be seen from our results, this method improves performance across all tested LVLMs.

Secondly, this method operates without the need to finetune the large language model (LLM) part of the model. Some mitigating methods necessitate finetuning the entire alignment and LLM part of the LVLM, which can be expensive and less efficient. In contrast, our method works with relatively small trainable parts, making it a more cost-effective and efficient solution.

Thirdly, this method may be extended to other hallucination-mitigating tasks. From a methodological perspective, most hallucinations typically involve various perspectives, thus consistency can be analyzed, and similar training methods can be applied.

Table 8 shows a successful example of our consistency method. This is also a case that we use to illustrate inconsistency problem in 4.2, which further proves the effectiveness of introducing consistency into training. As can be seen, with Direct method the model fails to identify the correct number of bicycles, while after introducing Consistency(I+II) method the model responds the correct answer. We believe this improvement is led by our consistency method forces the model to think more carefully and comprehensively, and thus reduces uncertainty and confusion.