A Single Simple Patch is All You Need for AI-generated Image Detection

In recent years, image generative models have garnered considerable attention. These models are capable of generating images that are very similar to real ones. Generative Adversarial Network [15] (GAN) is one early generative model proposed for image generation. GAN consists of a generator and a discriminator, in which the generator accounts for creating images while the discriminator accounts for determining whether the images are real or fake. These two components are learnt through adversarial training. Afterwards, many researchers have proposed the variant models of GAN. For example, BigGAN [11] uses a hierarchical generator architecture that allows control over image details and diversity at different levels, which facilitates the generation of more detailed and diverse images. Recently, diffusion model was introduced for image generation. It starts with random noise, which is gradually transformed using a diffusion process. The diffusion-based text-to-image generation models [33, 28, 36] can be used to create images of high quality.

As the images produced by generative models become increasingly realistic, it has become more challenging for humans to distinguish between real and fake images. In earlier works, researchers typically employed learning-based approaches to address this problem, treating it as a binary classification task. Zhang  et al. [44] demonstrated that unique artifacts exist in GAN-generated images. These artifacts can be viewed periodically in the spectrum of images. Frank et al. [13] trained the detector in the frequency domain to detect them. Wang et al. [42] proposed that with appropriate training data and data augmentation, a classifier trained on one CNN generator can generalize well to other unseen generators. In [38], the author used a pretrained CNN model to convert images to gradients, which are used to present the artifacts in GANs.

However, after the emergence of diffusion models, these methods have shown poor generalization when applied to the images generated by diffusion models (DM). Ricker  et al. [35] found that DMs do not exhibit any obvious artifacts in the frequency domain and there is structural difference between images generated by DMs and GANs. Wang et al. [43] used the error between an input image and its reconstruction by a pre-trained diffusion model to identify fake images, but this method cannot generalize well to GAN-based models. Ojha  et al. [30] proposed UnivFD which utilizes the feature space of a frozen pretrained vision-language model followed by a linear classifier, but this model cannot be extended well with the emergence of new advanced generative models and the inference speed is much lower than other methods. Tan et al. [39] proposed to use the local interdependence among image pixels caused by upsampling operators to detect the artifacts. Zhong et al. [45] leverages two branches for rich and poor texture regions followed by carefully designed high-pass filters to classify the image. ChaiIn  et al. [5] also proposed a method in patch level as ours. They utilizes truncation at different layers of the neural network (e.g., ResNet [17], Xception [1]) to obtain model predictions based on the collective information from the whole set of patches. This method can help localize the manipulated regions. However, it was proved in [30] that this method does not perform well in detecting AI-generated images. It struggles to effectively distinguish even the images generated by the same generator. Unlike their work, our method uses the information from the original images. We just use one single simple patch followed by typical filters [14], which is capable of generalizing well across all the generators.

Given a test image, our task is to distinguish whether this image is real or not. The challenge lies in designing a detector that can generalize well across different generators. Inspired by [5, 45], we attempt to solve this problem in the patch level. We observe that using only a single simple patch from the original image is enough to identify the fake image. The overview of the proposed model is illustrated in Figure 1. In Section 3.2, we will first describe how to extract the simple patch and its noise pattern. In Section 3.3, we will introduce the enhancement module and perception module to address the issue of interference caused by low image quality.

In this section, we introduce our single simple patch (SSP) network. Given an image $\bm{X}$ with spatial size $W\times H$, SSP predicts the output based on one simple patch. We first randomly split the image into many small patches with patch size $M\times M$. We treat the patch with the smallest texture diversity as the simplest patch. The texture diversity is measured by pixel fluctuation degree proposed in [45], which can be calculated as follows,

	$\displaystyle\mathcal{L}_{div}=\sum_{i=1}^{M}\sum_{j=1}^{M-1}(\left|x_{i,j}\!-\!x_{i,j+1}\right|)\!+\!\sum_{i=1}^{M-1}\sum_{j=1}^{M}(\left|x_{i,j}\!-\!x_{i+1,j}\right|$		(1)
	$\displaystyle+\sum_{i=1}^{M-1}\sum_{j=1}^{M-1}(\left|x_{i,j}\!-\!x_{i+1,j+1}\right|\!+\!\sum_{i=1}^{M-1}\sum_{j=1}^{M-1}(\left|x_{i+1,j}\!-\!x_{i,j+1}\right|),$

Then, we resize the simplest patch with minimum $\mathcal{L}_{div}$ to the size of original image, and use standard SRM (Steganalysis Rich Model) [14] to extract its noise pattern, which records the high-frequency information by using $3$ high-pass filters. Finally, the noise pattern is delivered to ResNet50 [17] binary classifier to distinguish real images from fake images. The output of the classifier represents the probability for the input image being a real image. SSP network employs a standard binary cross-entropy loss:

$$\mathcal{L}_{cls}=\hat{y}\log y^{\prime}+(1-\hat{y})log(1-y^{\prime}),$$

(2)

where $\hat{y}$ denotes the ground-truth real/fake label and $y^{\prime}$ denotes the predicted label of SSP.

As mentioned in Section 1, generative models, such as diffusion models, focus more on generating subtle details in the regions with rich textures to make the generated images more realistic, while the simple patches in generated images are largely overlooked. These simple patches actually contain lots of hidden noise caused by camera capture which can be used as important fingerprints. With these insights, our SSP network directly focuses on the simple patches, reducing the influence of irrelevant and cluttered information from the original image. Additionally, focusing on the simplest patch also reduces the computational cost, leading to a simple yet effective model.

Our SSP network performs well in distinguishing between real and fake images in common situations. However, images may undergo a certain degree of degradation in the real world. In such scenarios, not only SSP but also other AI-generated detection methods would suffer from performance drop to some degree. Therefore, we design an enhancement module and a perception module to reduce the influence caused by interfering factors like blurring and compression.

Given a patch, both the perception module and the enhancement module work together to transform a lower-quality patch into a higher-quality one. Our enhancement module has commonly used U-Net [37] structure, which aims to enhance the patch quality if the input patch is blurry or compressed. Moreover, we design a perception module to provide guidance information (whether the input patch is blurry or compressed). The perception module is a lightweight three-class classifier (blurry, compressed, intact), which consists of a convolution layer, batch normalization layer, ReLU activation function, a pooling layer and a fully connected layer.

Based on the prediction of perception module, the enhancement module can perform the corresponding task, which helps alleviate the burden of enhancement module. Specifically, if the input patch is blurry or compressed, enhancement module performs the deblurring and decompression task respectively. If the patch is neither blurry nor compressed, enhancement module simply reconstructs the input patch. Corresponding to the abovementioned three tasks, we introduce three learnable task embeddings, which are combined based on the prediction of perception module and injected into the enhancement module as conditional information.

Formally, we use $F$ to denote the perception module, and use $\bm{w}^{\prime}=[w^{\prime}_{1},w^{\prime}_{2},w^{\prime}_{3}]$ to denote the predicted probability of input patch $\bm{x}$ being blurry, compressed, or intact, respectively.

$$\bm{w}^{\prime}=F(\bm{x}).\\$$

(3)

Then we get the normalized weights $\bar{\bm{w}}^{\prime}=[\bar{w}^{\prime}_{1},\bar{w}^{\prime}_{2},\bar{w}^{\prime}_{3}]$ by $\bar{\bm{w}}^{\prime}=\frac{\bm{w}^{\prime}}{||\bm{w}^{\prime}||_{1}}$. Based on the normalized weights, we combine three learnable task embeddings:

$$\bm{h}^{fus}=\bar{w}^{\prime}_{1}\cdot\bm{h}^{blu}+\bar{w}^{\prime}_{2}\cdot\bm{h}^{com}+\bar{w}^{\prime}_{3}\cdot\bm{h}^{rec},$$

(4)

in which $\bm{h}^{blu},\bm{h}^{com},\bm{h}^{rec}$ are the task embeddings for patch deblurring, decompression, and reconstruction, respectively.

The fused embedding $\bm{h}^{fus}$ serves as the conditional information of enhancement module, aiming to assist the enhancement module in removing the interference factors. Recall that the enhancement module has U-Net structure. The fused embedding is injected into the encoding and decoding blocks in the enhancement module using cross-attention. Specifically, the enhancement process can be written as follows,

$$\bm{x}^{\prime}=E(\bm{x},\bm{h}^{fus}),\\$$

(5)

in which $E$ denotes the enhancement module and $x^{\prime}$ denotes the output patch.

First, we conduct experiments on recently proposed GenImage dataset [48] containing some advanced generators. GenImage uses the real images in ImageNet [9] and their category labels to generate fake images. The fake images are generated by $8$ generators including Stable Diffusion V1.4 [36], Stable Diffusion V1.5 [36], GLIDE [28], VQDM [16], Wukong [3], BigGAN [11], ADM [10], and Midjourney [2]. Each generator is associated with a subset of real images and fake images, which are split into training subset and test subset. Therefore, GenImage consists of 8 training subsets and 8 test subsets, corresponding to 8 generators. In total, GenImage contains 1,331,167 real images and 1,350,000 fake images. Following [48], we train the classifier on the training subset of one generator and evaluate the classifier on the test subset of another generator, leading to $64=8\times 8$ settings. We also evaluate the degraded image classification task on this dataset.

To compare with previous works more thoroughly, we also conduct experiments on ForenSynths Dataset [42] which consists of images across 20 categories. The training set is composed of images generated by ProGAN generator. As for the testing set, we test our model on 7 GAN-based generators (ProGAN [21], CycleGAN [46], BigGAN [4], StyleGAN [22], GauGAN [31], StarGAN [7], STGAN [24]), 6 diffusion-based generators (DDPM [18], PNDM [23], IDDPM [29], Guided Diffusion Model [10], GLIDE [28], LDM [36]) and one autoregresive model (DALLE-E [34]) from UniversalFake Dataset [30].

Following existing works for AI-generated image detection, we adopt accuracy (ACC) and mean average precision (mAP) as evaluation metrics. The threshold for computing accuracy is set to 0.5.

All the images are first resized to 256$\times$256, and then we randomly crop 64 patches from each image. We choose the simplest patch as the input. Concerning the classifier of our SSP network, we use ResNet-50 [17] pretrained on ImageNet [9]. During the training stage, we apply Adam optimizer with learning rate $10^{-4}$. The batch size is set to 64 and we train our model for 30 epochs in total. As for the data augmentation, we use Gaussian blur and JPEG compression with the probability of 10%. The proposed method is implemented by PyTorch library. As for the hardware devices used for training, we use Intel(R) Xeon(R) Silver 4116 CPU, with 128GB memory and one NVIDIA GeForce RTX 3090 GPU.

We first conduct experiments on the recently proposed GenImage dataset which includes images generated by recent generators, so GenImage is more challenging than the ForenSynths dataset in Section 4.5. We compare our ESSP method with ResNet50 across different training and test subsets. The results are summarized in Table 2, which shows that our method significantly outperforms ResNet50 when the training and test subsets are from different generators.

In Table 3, for each test subset, we average the eight results obtained using eight training subsets. Our method surpasses all the baseline methods by a large margin. To the best of our knowledge, our ESSP is currently the only method that performs well in such challenging cross-generator settings, further highlighting the importance of noise fingerprints of simple patches.

Following [48], we also compare our ESSP with other methods when training subset is stable diffusion V1.4 [36]. The results are recorded in Table 1. We can observe that most methods have relatively low accuracy, which indicates that GenImage is a challenging dataset for AI-generated image dataset. Our ESSP surpasses the state-of-the-art PatchCraft [45] by 8.3%, achieving an accuracy of 90.6%. As for the method using large pretrained models, our ESSP achieves significant accuracy gain over DIRE [43] and UnivFD [30], with improvements of 19.9% and 17.3%, respectively. This also demonstrates the limited capability of pretrained models in classifying recently emerging generative models.

In this section, we compare our ESSP with previous works on ForenSynths Dataset. The experimental setting is consistent with previous works. To evaluate the generalization ability, the detector is trained on the images generated by ProGAN [21] and evaluated on the images generated by various generators from different categories. Table 5 (resp., Table 6) summarizes the accuracy (resp., mAP) of different methods on this dataset. The results show that our ESSP achieves competitive accuracy and mAP. It is worth noting that ESSP achieves an accuracy of 90.22%, surpassing the state-of-the-art baselines. Although the mAP of ESSP is slightly worse than UnivFD [30], our ESSP has lower computation cost and faster inference speed than UnivFD which uses pretrained CLIP-ViT.

To validate the effectiveness of individual components in our method, we conduct ablation studies on GenImage dataset. We train our model on stable diffusion V1.4 subset and test it on the others.

Following Section 4.6.2, we further explore the robustness of our method to blur and compression. We conduct experiments on GenImage dataset [48]. We train the model on stable diffusion V1.4 generator and test it on eight generators in GenImage dataset. The eight results are averaged to achieve the final result. All models are trained with the same strategy as in Section 4.6.2. For test images, we control the compression quality and blur degree.

We compare our model with the state-of-the-art methods: UnivFD [30], NPR [39], CNNSpot [42]. The results of different methods are plotted in Figure 3. It can be seen that as the image quality decreases, all models suffer from performance decline, which implies that the interfering factors substantially raise the difficulty of AI-generated image detection task. Nevertheless, our model still outperforms the latest models.

We also present the images that are processed by blurring or compression in Figure 3. When $\sigma$ is 2, the image quality is already very low. We believe that conducting forgery detection on excessively blurred images is not very meaningful. When JPEG compression quality is below 90, the accuracy of all methods is generally very low, so we set the limit of compression quality as 90 during experiments.

Although our ESSP method can generalize well across a wide range of generators, there is still room for improvement in detecting very low-quality images (e.g., compression quality is lower than 90). We would explore how to integrate simple patches with other information in the original image for better robustness.