A Sanity Check for AI-generated Image Detection

In this paper, our goal is to train a computational model that can distinguish the AI-generated images from the ones captured by the camera, i.e., $y=\Phi_{\text{model}}(\mathbf{I};\Theta)\in\{0,1\}$, where $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$ denotes an input RGB image, $\Theta$ refers to the learnable parameters. For training and testing, we consider the following two settings:

Train-Test Setting-I. In the literature, existing works on detecting AI-generated images (Wang et al., 2020; Frank et al., 2020; Ojha et al., 2023; Wang et al., 2023; Zhong et al., 2023) have exclusively considered the scenario of training on images from single generative model, for example, ProGAN (Karras et al., 2017), or Stable Diffusion (Sta, 2022), and then evaluated on images from various generative models. That is,

$$\mathcal{G}_{\text{train}}=\mathcal{G}_{\text{GAN}}\vee\mathcal{G}_{\text{DM}},\mathcal{G}_{\text{test}}=\{\mathcal{G}_{\text{ProGAN}},\mathcal{G}_{\text{CycleGAN}},...,\mathcal{G}_{\text{SD}},\mathcal{G}_{\text{Midjourney}}\}.$$

(1)

Generally speaking, such problem formulation poses two critical issues: (i) evaluation benchmarks are simple, as these randomly sampled images from generative models, can be far from being photo-realistic, as shown in Figure 1; (ii) confining models to train exclusively on GAN-generated images imposes an unrealistic constraint, hindering the model’s ability to learn from the diverse properties exhibited by more advanced generative models.

Train-Test Setting-II. Herein, we propose an alternative problem formulation, where the models are allowed to train on images generated from a wide spectrum of generative models, and then tested on images that are genuinely challenging for human perception.

$$\mathcal{G}_{\text{train}}=\{\mathcal{G}_{\text{GAN}},\mathcal{G}_{\text{DM}}\},\mathcal{G}_{\text{test}}=\{\mathcal{D}_{\texttt{Chameleon}}\}.$$

(2)

$\mathcal{D}_{\texttt{Chameleon}}$ refers to our proposed benchmark, as detailed below. We believe this setting resembles more practical scenario for future model development in this community.

The primary objective of the Chameleon dataset is to evaluate the generalization and robustness of existing AI-generated image detectors, for a sanity check on the progress of AI-generated image detection. In this section, we outline the progression of the proposed dataset in three critical phases: (i) dataset collection, (ii) dataset curation, (iii) dataset annotation, and (iv) dataset comparison. The statistical results of our dataset are illustrated in Table 1 and we compare our dataset with existing benchmarks in Fig. 1.

We leverage insights from the disparities in low-level patch statistics between AI-generated images and real-world scenes. Models like generative adversarial networks or diffusion models often yield images with certain artifacts, such as excessive smoothness or anti-aliasing effects. To capture such discrepancy, we adopt a Discrete Cosine Transform (DCT) score module to identify patches with the highest and lowest frequency. By focusing on these extreme patches, we aim to highlight the distinctive characteristics of AI-generated images, thus facilitating the discriminative power of our detection system.

Patch Selection via DCT Scoring. For an RGB image, we first divide this image into multiple patches with a fixed window size, $\mathbf{I}=\{x_{1},x_{2},\dots,x_{n}\}$, $x_{i}\in\mathbb{R}^{N\times N\times 3}$. In our case, the patch size is set to be $N=32$ pixels. We apply the discrete cosine transform to each of the image patches, obtaining the corresponding results in the frequency domain, $\mathcal{X}_{f}=\{x_{1}^{\text{dct}},x_{2}^{\text{dct}}\dots,x_{n}^{\text{dct}}\}$, $x_{i}^{\text{dct}}\in\mathbb{R}^{N\times N\times 3}$.

To acquire the highest and lowest image patches, we use the complexity of the frequency components as an indicator. From this, we design a simple yet effective scoring mechanism. Specifically, we design $K$ different band-pass filters:

$$F_{k,ij}=\begin{cases}1,&\text{if }\frac{2N}{K}\cdot k\leq i+j<\frac{2N}{K}\cdot(k+1)\\ 0,&\text{otherwise}\end{cases}$$

(3)

where $F_{k,ij}$ is the weight at the $(i,j)$ position of the $k$-th filter. Next, for $m$-th patch $x_{m}^{\text{dct}}\in\mathbb{R}^{N\times N\times 3}$, we apply the filters $F_{k,ij}\in\mathbb{R}^{N\times N\times 3}$ to multiply the logarithm of the absolute DCT coefficients $x_{m}^{\text{dct}}\in\mathbb{R}^{N\times N\times 3}$ and sum all the positions to obtain the grade of the patch $G^{m}$. We formulated it as

$$G^{m}=\sum_{k=0}^{K-1}2^{k}\times\sum_{c=0}^{2}\sum_{i=0}^{N-1}\sum_{j=0}^{N-1}F_{k,ij}\cdot\text{log}(|x_{m}^{\text{dct}}|+1)$$

(4)

where $c$ is the number of patch channels. In this way, we acquire the grades of all patches $G=\{G^{1},G^{2},...,G^{n}\}$. We then sort them to identify the highest and lowest frequency patches.

Through the scoring module, we can obtain the top $k$ patches $X_{\text{max}}=\{X_{\text{max}_{1}},X_{\text{max}_{2}},...,X_{\text{max}_{k}}\}$ with the highest frequency and the top $k$ patches $X_{\text{min}}=\{X_{\text{min}_{1}},X_{\text{min}_{2}},...,X_{\text{min}_{k}}\}$ with the lowest frequency, where $X_{\text{max}_{i}}\in\mathbb{R}^{N\times N\times 3}$, $X_{\text{min}_{i}}\in\mathbb{R}^{N\times N\times 3}$.

Patchwise Feature Encoder. Next, firstly, these patches are resized to a size of $256\times 256$. Secondly, they are input into the SRM (Fridrich & Kodovsky, 2012) to extract their noise pattern. Lastly, these features are input into two ResNet-50 (He et al., 2016) networks ($f_{1}(\cdot)$ and $f_{2}(\cdot)$) to obtain the final feature map $F_{\text{max}}=\{f_{1}(X_{\text{max}_{1}}),f_{1}(X_{\text{max}_{2}}),...,f_{1}(X_{\text{max}_{k}})\}$, $F_{\text{min}}=\{f_{2}(X_{\text{min}_{1}}),f_{2}(X_{\text{min}_{2}}),...,f_{2}(X_{\text{min}_{k}})\}$. We acquire the highest frequency embedding and lowest frequency embedding on the mean-pooled feature:

$$F_{\text{\text{max}}}=\text{Mean}(\text{AveragePool}(F_{\text{max}})),~{}~{}~{}~{}F_{\text{\text{min}}}=\text{Mean}(\text{AveragePool}(F_{\text{min}})).$$

(5)

To capture the rich semantic features within images, such as object co-occurrence and contextual relationships, we compute the visual embedding for input image with an off-the-shelf visual-language foundation model. Specifically, we adopt the ConvNeXt-based OpenCLIP model (Ilharco et al., 2021) to get the final feature map ($v\in\mathbb{R}^{h\times w\times c}$). To capture the global contexts, we append a linear projection layer followed by mean spatial pooling:

$$F_{\text{s}}=\text{avgpool}(g(v)).$$

(6)

To distinguish between AI-generated images and real images, we utilize a mixture-expert-model for the final discrimination. At low-level, we take the average of the highest frequency featured:

$$F_{\text{mean}}=\text{avgpool}(F_{\text{max}},F_{\text{min}}).$$

(7)

Then, we channel-wisely concatenate the representations between it and high-level embedding $F_{s}$. At last, the features are encoded into MLP to acquire the score,

$$y=f([\text{avgpool}(F_{\text{mean}};F_{\text{s}}]),$$

(8)

where $f(\cdot)$ denotes the MLP consisting of a linear layer, GELU (Hendrycks & Gimpel, 2016) and classifier, $[;]$ refers to the operation of channel-wise concatenation.

Detectors. We evaluate 9 off-the-shelf detectors including CNNSpot (Wang et al., 2020), FreDect (Frank et al., 2020), Fusing (Ju et al., 2022), LNP (Liu et al., 2022a), LGrad (Tan et al., 2023), UnivFD (Ojha et al., 2023), DIRE (Wang et al., 2023), PatchCraft (Zhong et al., 2023) and NPR (Tan et al., 2024) for comparison.

Datasets. To comprehensively evaluate the generalization ability of existing approaches, we conduct experiments across two kinds of settings: Setting-I and Setting-II, which are summarized in Sec. 3.1. For the Setting-I setting, we evaluate the detectors on two general and comprehensive benchmarks of AIGCDetectBenchmark ($\mathcal{B}_{1}$) (Zhong et al., 2023) and GenImage (Zhu et al., 2024) ($\mathcal{B}_{2}$). For the Setting-II setting, we evaluate the detectors on our Chameleon ($\mathcal{B}_{3}$) benchmark. More details can be found in Appendix.

Implementation Details. AIDE includes two key modules: Patchwise Feature Extraction (PFE) and Semantic Feature Embedding (SFE). For PFE channel, we first patchify each image into patches and the patch size is set to be $N=32$ pixels. Then these patches are sorted using our DCT Scouring module with $K=6$ different band-pass filters in the frequency domain. Subsequently, we select two highest-frequency and two lowest-frequency patches using the calculated DCT scores. These selected patches are then resized to $256\times 256$ and extracted their noise pattern using SRM (Fridrich & Kodovsky, 2012). For SFE channel, we use the pre-trained OpenCLIP (Ilharco et al., 2021) to extract semantic features. We adopt data augmentations including random JPEG compression (QF $\sim\text{Uniform}(30,100)$) and random Gaussian blur ($\sigma\sim\text{Uniform}(0.1,3.0)$) to improve the robustness of detectors. Each augmentation is conducted with 10% probability. During the training phase, we use AdamW optimizer with the learning rate of $1\times 10^{-4}$ in $\mathcal{B}_{1}$ and $5\times 10^{-4}$ in $\mathcal{B}_{2}$, respectively. The batch size is set to $32$ and the model is trained on 8 NVIDIA A100 GPUs for only 5 epochs. Our method trains very quickly, only 2 hours are sufficient.

Metrics. In accordance with existing AI-generated detection arpproaches (Wang et al., 2020; 2019; Zhou et al., 2018), we report both classification accuracy (Acc) and average precision (AP) in our experiments. All results are averaged over both real and AI-generated images unless otherwise specified. We primarily report Acc for evaluation and comparison in the main paper, and AP results are presented in the Appendix.

On Benchmark AIGCDetectBenchmark: The quantitative results in Table 3 present the classification accuracies of various methods and generators within $\mathcal{B}_{1}$. In this evaluation, all methods, except for DIRE-D, were exclusively trained on ProGAN-generated data.

AIDE demonstrates a significant advancement over the current state-of-the-art (SOTA) approach, PatchCraft, achieving an average accuracy increase of 3.5%. UnivFD utilizes CLIP semantic features for detecting AI-generated images, proving effective for GAN-generated images. However, it shows pronounced performance degradation with diffusion model (DM)-generated images. This suggests that as generation quality improves, diffusion models produce images with fewer semantic artifacts, as depicted in Fig. 1 (a). Our approach, which integrates semantic, low-frequency, and high-frequency information at the feature level, enhances detection performance, yielding a 5.2% increase for GAN-based images and a 1.7% increase for DM-based images compared to the SOTA method.

On Benchmark GenImage: In the experiments conducted on $\mathcal{B}_{2}$, all models were trained on SD v1.4 and evaluated across eight contemporary generators. Table 4 presents the results, illustrating our method’s superior performance over the current state-of-the-art, PatchCraft, with a 4.6% improvement in average accuracy. The architectural similarities between SD v1.5, Wukong, and SD v1.4, as noted by GenImage (Zhu et al., 2024), enable models to achieve near-perfect accuracy, approaching 100% on such datasets. Thus, evaluating generalization performance across other generators, such as Midjourney, ADM, and Glide, becomes essential. Our model demonstrates either the best or second-best performance in these cases, achieving an average accuracy of 86.88%.

On Benchmark Chameleon: As highlighted in Sec. 1, we contend that success on existing public benchmarks may not accurately reflect real-world scenarios or the advancement in AI-generated image detection, given that test sets are typically randomly sampled from generative models without ”Turing Test”. To address potential biases related to training setups—such as generator types and image categories—we evaluate the performance of existing detectors under diverse training conditions. Despite their high performance on existing benchmarks, as depicted in Fig. 3, the state-of-the-art detector, PatchCraft, experiences substantial performance declines. Additionally, Table 5 reveals significant performance decreases across all methods, with most struggling to surpass an average accuracy close to random guessing (about 50%), indicating a failure in these contexts.

While our method achieves state-of-the-art results on available datasets, its performance on Chameleon remains lacking. This underscores that our dataset, Chameleon, which challenges human perception, represents a critical issue requiring attention in this field.

In real-world scenarios, images often encounter unseen perturbations during transmission and interaction, complicating the detection of AI-generated images. Here, we assess the performance of various methods in handling potential perturbations, such as JJPEG compression (Quality Factor (QF) = 95, 90) and Gaussian blur ($\sigma$ = 1.0, 2.0). As illustrated in Table 6, all methods exhibit a decline in performance due to disruptions in the pixel distribution. These disruptions diminish the discriminative artifacts left by generative models, complicating the differentiation between real and AI-generated images. Consequently, the robustness of these detectors in identifying AI-generated images is significantly compromised. Despite these challenging conditions, our method consistently outperforms others, maintaining a relatively higher accuracy in detecting AI-generated images. This superior performance is attributed to our model’s ability to effectively capture and leverage multi-perspective features, semantics and noise, even when the pixel distribution is distorted.

Our method focuses on detecting AI-generated images with mixture of experts, namely patchwise feature extraction (PFE-H and PFE-L for high-frequency and low-frequency patches, respectively) and semantic feature extraction (SFE). These modules collectively contribute to comprehensively identifying AI-generated images from different perspectives. Herein, we conduct extensive experiments to investigate the roles of each module on $\mathcal{B}_{1}$.

Patchwise Feature Extraction. As shown in Table 7, removing either the high-frequency or the low-frequency patches results in obvious performance degradation in terms of accuracy. Without the high-frequency patches, the proposed method is unable to discern that the high-frequency regions of AI-generated images are smoother than those of real images, resulting in performance degradation. Similarly, without the low-frequency patches, the method cannot extract the underlying noise information, which is crucial for identifying AI-generated images with higher fidelity, leading to incorrect predictions.

Semantic Feature Extraction. As shown in Table 7, the performance degrades significantly (76.70% vs 92.77%) when we remove the semantic branch. Intuitively, if the branch for semantic information extraction is absent, our method struggles to effectively capture images with semantic artifacts, resulting in significant drops.

Visualization. To vividly demonstrate the effectiveness of our modules patchwise feature extraction (PFE) and semantic feature extraction (SFE), we conducted a visualization, as depicted in Fig. 4. In first row, the absence of semantic feature extraction results in many images with evident semantic errors going undetected. Similarly, the second row shows that, without patchwise feature extraction, numerous images lacking semantic errors still contain differing underlying information that remains unrecognized. Overall, our method, AIDE, achieves the best performance.