FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Models

In the first part of FakeBench, we evaluate the fake image detection ability of LMMs, i.e., whether they can answer simple queries concerning objective image authenticity in natural languages. Therefore, for each fake image $\mathtt{I_{0}}$ and real image $\mathtt{I_{1}}$, we collect one authenticity-related question ($\mathtt{Q}$) in the closed-ended form, which is called basic mode prompting. In particular, $\mathtt{Q}$s are in two different forms: ‘Yes-or-No’ and ‘What’ questions (see Fig. 4(a)), i.e., ‘Is this a fake/real image?’ and ‘What is the authenticity of this image?’, which simulate common ways of human inquiry endow flexible evaluation approaches. The number of questions in different formats is balanced. Accordingly, we provide one correct answer ($\mathtt{A}$) to each question, which is in one of the forms of “yes”, “no”, “real”, and “fake” to reflect the real ability of fake image detection. The 6,000 pieces of $\mathtt{(I_{0},{Q}_{i},A)}$ tuples constitute FakeClass, where $i=0$ denotes the basic mode.

For the second part of FakeBench, we evaluate the reasoning and interpreting ability of LMMs fake image by constructing FakeClue, which focuses on model’s decision-making process and supports authenticity judgments in human-understandable terms respectively [31]. Regarding the reasoning process, the models need to comprehend image contents and invoke internal knowledge to spot telltale clues to draw the final authenticity judgment, requiring a sophisticated multi-hop inference to analyze and correlate diverse evidence meticulously. Meanwhile, interpreting is the ability of models to articulate clear and understandable descriptions to support the given judgment, which facilitates greater transparency and trust in detection results. Henceforth, we provide two prompting modes in FakeClue, including fault-finding mode and inference mode. As depicted in Fig. 4(b), the fault-finding mode utilizes few-shot in-context prompting [82] by providing true image authenticity directly in the prompt, i.e., describing the visual forgery cues when queried by “This is a fake image, explain why.” On the other hand, the inference mode (Fig. 4(c)) employs zero-shot chain-of-thought (CoT) prompting [83] that claims for explicit reasoning processes to analyze and correlate diverse evidence for the final authenticity conclusion.

Collecting authenticity-related descriptions is arduous for both humans and machines since it requires visual insights and knowledge reserve. LMMs normally excel in extensive internal knowledge, while humans are good at logical reasoning. Therefore, we incorporate human-supervised LMM assistance under the guidance of the taxonomy for visual forgery proposed in Sec. III-C to combine the strengths of both, which can unburden human workload and assure data reliability simultaneously. Specifically, as shown in Fig. 5, we adopt the human-in-the-loop strategy [84] by initially prompting the proprietary GPT-4V [66] in the fault-finding mode with extra manual step-by-step demonstrations to produce primitive descriptions of fake clues for further refining. Then, we recruit human experts to verify and supplement the descriptions forthwith. Each expert is asked to judge the authenticity first and can only refine the telltale clues for the correctly recognized images because perception and understanding are no longer reliable if an image successively fools the human expert. In particular, human refinement is based on the following principles: 1) Adding absent forgery descriptions that are obvious to human perception. 2) Removing conflicting annotations against human perception. 3) Reducing non-robust descriptions that do not uniquely exist in fake images. In this way, the remarkable logical inference ability of humans, together with the decent low-level vision and abundant internal knowledge of LMMs, can be centralized to make up the golden clues for fake images.

As shown in Fig. 5, the golden clues in FakeClue, denoted by $\mathtt{C}$, serve as the demonstration of a coherent flow of sentences revealing the premises of the conclusion mimicking the human logical inference process on image authenticity. Each fake image is assigned at least four pieces of descriptions on forgery clues. The 6,000 $\mathtt{(I_{0},\{Q\}_{i},C)}$ tuples constitute FakeClue, where $i\in\{1,2\}$ denotes fault-finding mode and inference mode respectively. Note that FakeClue contains no explanations for genuine images because the reasoning for fake images is to encompass and extend the recognition of real images. More examples of FakeClue data are provided in the supplementary materials.

The FakeQA dataset tends to investigate the ability of LMMs to analyze fine-grained forgery aspects. The data is in the $\mathtt{Q\&A}$ format on 14 dimensions of telltale clues (see Sec. III-C), which analyzes image authenticity from fine-grained aspects. We utilize single-modal GPT to reorganize the golden clues into a unified $\mathtt{Q\&A}$ format, e.g., “$\mathtt{Q}$: How is the texture regarding the authenticity of this image? $\mathtt{A}$: The texture on the robot’s surface is too uniform and lacks the variations you would expect from different materials.”. These $\mathtt{Q\&A}$s maximally cover the spread of forgery aspects defined by the proposed taxonomy. In total, around 42k pieces of $\mathtt{(I_{0},\{Q\}_{i},C)}$ tuples constitute FakeQA where $i=3$, indicating the free mode.

We evaluate 14 up-to-date LMMs on the entire FakeBench, including ten open-source models ([24, 65, 85, 25, 87, 26, 88, 89, 90, 86]) and four closed-sourced ones (GPT-4V by OpenAI [66], GeminiPro by Google [67], Claude3 Sonnet and Claude3 Haiku by Antrophic [68]). The LMMs vary in the architecture and modality alignment strategy, detailed in Table IV. Besides, we further report the classification accuracy (ACC) of five unimodal models dedicated to fake image detection on FakeBench, including CNNSpot [18], Gram-Net [7], FreDect [4], PSM [6], LGrad [9], and UnivDF [22]. To ensure fairness, all the LMMs and specialized image defaking models are evaluated under the zero-shot setting without further tuning on any additional data in FakeBench. Since the specialized models require fundamental training for classification, they are all trained under the generalization setting presented in [18], i.e., 360K fake images generated by ProGAN [49] and 360K real images from LSUN [91].

Generally, LMM’s accuracy in responding to the questions in FakeBench is utilized to signify their corresponding capabilities. We design rule-based systematic evaluation pipelines for both open- and closed-ended questions, which are elaborated in detail below.

The detection performance of 14 LMMs on FakeClass are shown in Table V, where humans, LMMs, and specialized defaking models are compared. First, LMM’s detection accuracy varies significantly, where GPT-4V and mPLUG-Owl2 achieve the best two overall performance, while Otter, Visual-GLM, IDEFICS-Instruct, and Kosmos-2 exhibit below-random performance. Second, most models perform better on the genuine image collection than on the fake, except for InstructBLIP and Kosmos-2. However, InstructBLIP achieves the best accuracy on fake images and multiple generator subsets simultaneously, exceeding specialized models. Claude3 Haiku, Sonnet, and Otter exhibit extreme discrepancy between the real and fake subsets, with notably high accuracy on the real subset and significantly lower accuracy on the fake subset, indicating they tend to classify any images as real ones. Third, most LMMs are robust to question formations, except for Visual-GLM and IDEFICS-Instruct, which prefer the Yes-or-No questions more. Besides, the FakeBench is a great challenge for existing fake image detection models because of the great domain shifting concerning generators and image contents compared with previous datasets.

Based on the observations, we obtain several intriguing insights for LMM’s fake image detection ability: (a) The majority of the LMMs can outperform random guess on the overall accuracy and even exceed the specialized models (e.g., GPT-4V, InstructBLIP, mPLUG-Owl2 and GeminiPro). Considering LMMs incorporate no explicit training on deepfake detection, this implies a promising prospect for general-purpose large foundational models as more robust and generalizable fake image detectors with careful knowledge augmentation. (b) Compared with open-source LMMs, proprietary models exhibit slight superiority on detection. However, even the strongest GPT-4V is outdone by the vanilla InstructBLIP on the fake image set ($59.87\%<67.80\%$). This is an unexpected finding since commercial LMMs are widely acknowledged to have leading performances on various occasions. Fortunately, this indicates that model scale is not the only answer for the image defaking task, where open-source LMMs have promising potential to achieve promising performance when fine-tuned properly. (c) Humans far exceed models on image defaking. Current Large Multimodal Models (LMMs) and specialized deep models lag behind humans in accuracy and generalization. Only the GPT-4V slightly outperforms the average human intelligence ($78.03\%>74.51\%$). However, all tested models significantly underperform compared to human peak performance, with the bottom five LMMs and two specialized models falling below the human minimum. This underscores a systematic advantage for humans in image defaking. (d) Distinguishing the fake is a harder task for most LMMs than recognizing real ones. They excel with specific image generators but struggle with certain types, sometimes achieving zero accuracy. This deficiency in image defaking tasks stems from the universal oversight in training and fine-tuning LMMs, where the distinction between synthetic and genuine data is often neglected.

LMM’s performance on fake image interpreting (effect-to-cause) and reasoning (cause-to-effect) is exhibited in Table VI and VII respectively. These results provide an overall reflection of causal investigations of LMMs. We observe that GPT-4V and InstructBLIP achieve the top two performances among all models. Nonetheless, the majority of LMMs still struggle to reason and interpret. Notably, the LMMs tend to gather in lower score segments on the reasoning task, whereas the other 12 out of 14 models achieve the overall score in the $[0.357,0)$ range. Besides, all the models perform better in interpreting fake images than reasoning. In particular, LMMs tend to exhibit higher relevance scores on the interpretation task, indicating the potential to generate authenticity-relevant descriptions when correctly guided by the given image authenticity.

We draw several findings on the explainability-related abilities of LMMs from the observations: (a) Reasoning is far more difficult than interpreting for LMMs on fake images. Few-shot prompting marginally improves forgery description by providing authenticity. However, constructing a coherent reasoning chain from these forgery clues remains a substantial challenge for current LMMs. (b) Open-source models exhibit competitive performance compared to the proprietary ones. For instance, InstructBLIP slightly outperforms GPT-4V in reasoning ($0.464>0.460$). This minor edge highlights the potential of further optimized LMMs as image defaking explainers with adequate fine-tuning data. (c) LMMs generally exhibit notable shortcomings on fake image reasoning and interpreting. This is largely because they are trained predominantly on general-purpose visual-text data, with a notable lack of authenticity-focused data.

FakeQA probes LMMs on analyzing fine-grained generative forgery clues of fake images, and the performances of LMMs are listed in Table VIII. From the results of fine-grained forgery analyses, we observe that GPT-4V again outperforms others. Moreover, the propriety LMMs, including GPT-4V, Claude3 Sonnet, and Claude3 Haiku, all exhibit superiority over their open-source counterparts. Besides, unlike the challenging reasoning task, InstructBLIP, Qwen-VL, and Visual-GLM exhibit even worse performance on FakeQA. In practice, they usually offer empty responses when required to analyze the forgery of a certain aspect, which can be reflected by their notably low completeness, BLUE, and ROUGE-L scores.

According to the observations, we discover that: (a) Proprietary LMMs typically outperform open-source ones in analyzing specific generative forgery aspects, though their best performance is still suboptimal (highest score of 0.477 out of 1). This reflects a fundamental deficiency in how current LMMs perceive and understand authenticity. (b) For some LMMs, analyzing fine-grained forgery is more challenging than broad reasoning. This difficulty arises because current LMMs lack the fine-grained conceptual framework for in-depth authenticity analysis. (c) Overall, most LMMs still struggle in fine-grained forgery analysis, highlighting a systematic inadequacy concerning concrete forensics knowledge in current training data used for LMMs.

We witness performance discrepancies from Table V to VIII, highlighting the varied capabilities across defaking-related subtasks. The highest Spearman Rank Correlation (SRCC) among model rankings across four criteria is 0.54, noted between reasoning and fine-grained forgery analysis, suggesting these tasks share operational mechanisms. In contrast, the correlation between interpreting and reasoning is notably low at 0.046, underscoring these tasks’ vastly different cognitive demands. This discrepancy emphasizes the specialized strengths and limitations of LMMs in the tasks requiring nuanced understanding and logical deduction versus those demanding broad contextual interpreting.

According to Table IX, explicit reasoning before judgment significantly impacts LMM’s performance on detection. However, the influence varies considerably among different models, exhibiting positive or negative effects. Specifically, 7 out of 14 models exhibit improved accuracy on genuine images, while only three models show improvements on fake images. Only Claude3 Sonnet attains improvements across all subsets and measures with CoT, albeit subtly. IDEFICS-Instruct and Claude3 Sonnet can see overall accuracy improvements with CoT prompting, where the former experienced a decrease in fake images but a substantial increase in genuine images. Conversely, Otter shows a 40.48% increase in accuracy on the fake image set yet suffered a significant drop on genuine images. In stark contrast, the other 6 out of 14 models have adverse impacts to different extents, with decreases in real and fake images. In particular, Qwen-VL loses 63.56% in accuracy on fake images. Regarding precision and recall, eight models showed improvements in precision and five in recall. The inconsistency among the metrics is due to the irrelevant responses from LMMs in some cases.

Therefore, the influence of explicit reasoning is various and inconsistent and negatively affects LMM’s performance on image defaking in most cases. It sometimes induces models to become overly picky in detecting the fake and thus erroneously label images as real (e.g., Q-Instruct, InterLM-XC.2-vl, LLaVA-v1.5, Otter, Visual-GLM). Also, it can globally impair detection capabilities (e.g., GPT-4V, GeminiPro), or hinder LMMs from providing relevant responses (e.g., InstructBLIP, Kosmos-2, Qwen-VL). However, as shown by $Answer_{2,3,5}$ of Fig. 7, CoT prompting can enrich authenticity assessments by fostering a broader and deeper level of visual comprehension, where additional semantics, contexts, and even external knowledge contribute to authenticity inferences. This suggests that LMMs can integrate various informational aspects beyond visual perception, potentially enhancing image authenticity judgments if properly directed through fine-tuning. Refer to the supplementary for exemplar cases of positive and negative effects from explicit reasoning.

The ineffectiveness of CoT prompting in our study is unexpected, as explicit reasoning generally benefits various tasks across different domains, e.g., causal inference [83], commonsense reasoning [101] and image quality assessment [72]. The reasons for this anomaly appear to vary across models and can be categorized into three main aspects: (a) Limited understanding of generative cues. LMMs often demonstrate insufficient visual reasoning capabilities concerning authenticity, as evidenced by their struggle to generate valid descriptions related to authenticity, even when explicitly prompted by CoT. This issue is highlighted in $Answer_{2}$ of Fig. 7, where essential reasoning processes fail to be effectively conducted, leading to flawed descriptions and, consequently, erroneous judgments due to error propagation. (b) Defective projection from descriptions to authenticity. Some models, like GPT-4V, can identify aspects of forgery but fail to correlate these descriptions accurately with authenticity. For instance, as shown in $Answer_{5}$ of Fig. 7, GPT-4V identifies an imaging abnormality but ultimately makes an incorrect judgment, resulting in decreased detection performance. (c) Limited instruction-following ability. The intrinsic ability of some models to follow instructions is compromised by insufficient instruction tuning, which hampers their capacity to adhere accurately to input queries. (d) Reliance on detection shortcuts. LMMs could have shortcuts in fake image detection. Fake image detection for LMMs is fundamentally a classification problem under the open-world setting and thus might partly depend on human-invisible cues left by generators, which are commonly exploited by DNN-based models [13, 22].

In summary, CoT-guided explicit reasoning divides the initial problem into sub-problems to be solved sequentially. However, difficulties in resolving sub-problems can collectively hinder the overall solution. Consequently, the observed decline in the effectiveness of CoT prompting in our research further reflects LMMs’ defective reasoning capabilities in image authenticity assessment, encompassing both the recognition of generative forgery and the projection to authenticity. Moreover, LMMs might have alternative pathways, aside from reason-to-judge, for detecting fake images, which remains an open question in LLM research. Investigating whether their judgments are more based on invisible cues left by generators or more on visual inconsistencies at various semantic levels could further benefit their detection capability.