UCF: Uncovering Common Features for Generalizable Deepfake Detection

Our encoder processes a pair of images $(\bm{x}_{0},\bm{x}_{1})$, where $\bm{x}_{0}$ represents the fake image and $\bm{x}_{1}$ represents the real image. The encoder $\bm{E}$ comprises a content encoder $\bm{E}_{c}$ and a fingerprint encoder $\bm{E}_{f}$, extracting the content and forgery features, respectively. We apply the encoder to each pair to obtain the corresponding fingerprint and content features as follows:

$$\bm{f}_{i}^{s},\ \bm{f}_{i}^{c},\ \bm{c}_{i}=\bm{E}(\bm{x}_{i}),$$

(1)

where $i\in\left\{0,1\right\}$ is the index of the image. Also, $\bm{f}_{0}^{s}$, $\bm{f}_{0}^{c}$, $\bm{c}_{0}$ and $\bm{f}_{1}^{s}$, $\bm{f}_{1}^{c}$, $\bm{c}_{1}$ are denoted by the specific fingerprint, common fingerprint, and content corresponding to each input image pair, respectively.

Our decoder (See Fig. 4) reconstructs an image by utilizing its content and fingerprint through a series of upsampling and convolutional layers. Unlike other disentanglement-based deepfake detection frameworks [54, 51, 27] that linearly add forgery and content features for recombination, our decoder applies Adaptive Instance Normalization (AdaIN) [19] for improved reconstruction and decoding, inspired by stylization techniques [20]. The AdaIN aligns the mean and variance of the content code to match those of the fingerprint. The formula is written as:

$$\operatorname{AdaIN}(\bm{c},\bm{f})=\sigma(\bm{f})\left(\frac{\bm{c}-\mu(\bm{c})}{\sigma(\bm{c})}\right)+\mu(\bm{f}),$$

(2)

where $\bm{c}$ and $\bm{f}$ are the content and the style vectors of the image pair, respectively. The functions $\mu(\cdot)$ and $\sigma(\cdot)$ compute the mean and variance of the input.

For classification loss, we propose two different losses of classification. First, we propose a binary classification loss $\mathcal{L}_{ce}^{c}$ computed by the cross-entropy for supervising the model to learn the common feature of different forgery methods:

$$\mathcal{L}_{ce}^{c}=\mathcal{L}_{ce}(\bm{H}_{c}(\bm{f}_{i}^{c}),\ \bm{y}_{i}),$$

(3)

where $\mathcal{L}_{ce}$ denotes the cross-entropy loss, $\bm{H}_{c}$ is the head for the common forgery feature, which is implemented by several MLP layers. $\bm{y}_{i}\in\left\{\text{fake},\text{real}\right\}$ is the binary classification label. In addition, $\mathcal{L}_{ce}^{s}$ is proposed to learn the method-specific patterns by guiding the model to identify which forgery method is applied to the fake image:

$$\mathcal{L}_{ce}^{s}=\mathcal{L}_{ce}(\bm{H}_{s}(\bm{f}_{i}^{s}),\ \bm{y}_{i}^{\prime}),$$

(4)

where $\bm{H}_{s}$ is the head for specific forgery feature and $\bm{y}_{i}^{\prime}\in\left\{\text{real},GAN_{1},GAN_{2},\cdots\right\}$ donates the label for identifying which instance belong to the image. Note that $\bm{H}_{c}$ and $\bm{H}_{s}$ share the same architecture but not share the parameters.

The multi-task classification loss enables the model to learn both method-specific textures and common features in different forgeries, enhancing the generalization ability of the model.

The objective of contrastive regularization loss is to optimize the similarity and dissimilarity measurements between images. The contrastive regulation loss is formulated mathematically as:

$$\mathcal{L}_{con}=\max\left(\left|\bm{x}_{A}-\bm{x}_{P}\right|_{2}-\left|\bm{x}_{A}-\bm{x}_{N}\right|_{2}+\alpha,\ 0\right),$$

(5)

where $\alpha$ serves as a margin hyper-parameter. This method minimizes the Euclidean distance between an anchor image ($\bm{x}_{A}$) and its similar counterpart ($\bm{x}_{P}$), while simultaneously maximizing the gap between the anchor image and its dissimilar counterpart ($\bm{x}_{N}$). For instance, if $\bm{x}_{A}$ denotes common features of a genuine image, $\bm{x}_{P}$ would represent common attributes of another real image, while $\bm{x}_{N}$ signifies the attributes of a manipulated image.

For the common features, we compute the loss between real and fake images to encourage the model to learn a generalizable representation that across different forgeries. For the specific features, we compute the loss between images of the same forgery to encourage the model to learn method-specific textures for each forgery.

In general, there are two types of reconstruction in our framework: self-reconstruction and cross-reconstruction. For the self-reconstruction, the decoder $\bm{D}$ is applied to the content and fingerprint encoded from the same image to reconstruct the corresponding image, and the formula is written as:

$$\mathcal{L}_{rec}^{s}=\left\|\bm{x}_{0}-\bm{D}(\bm{f}_{0},\bm{c}_{0})\right\|_{1}+\left\|\bm{x}_{1}-\bm{D}(\bm{f}_{1},\bm{c}_{1})\right\|_{1}.$$

(6)

For the cross-reconstruction, we consider the different combinations of fingerprint and content features encoded from the different images. Similarly, the formula of the cross-reconstruction loss is written as:

$$\mathcal{L}_{rec}^{c}=\left\|\bm{x}_{0}-\bm{D}(\bm{f}_{1},\bm{c}_{0})\right\|_{1}+\left\|\bm{x}_{1}-\bm{D}(\bm{f}_{0},\bm{c}_{1})\right\|_{1}.$$

(7)

Considering both the self-reconstruction and cross-reconstruction loss, the overall image reconstruction loss can be computed as follows:

$$\mathcal{L}_{rec}=\mathcal{L}_{rec}^{s}+\mathcal{L}_{rec}^{c}.$$

(8)

The image reconstruction loss ensures that the reconstructed image and the original image are consistent at the pixel level. In addition, the two reconstruction losses enhance the disentanglement of features. The self-reconstruction loss penalizes the reconstruction errors by leveraging the latent features of the input image, while the cross-reconstruction loss penalizes the reconstruction errors using the forgery feature and the swapped content features.

The final loss function of the training process is the weighted sum of the above loss functions.

$$\mathcal{L}=\mathcal{L}_{ce}^{c}+\lambda_{1}\mathcal{L}_{ce}^{s}+\lambda_{2}\mathcal{L}_{rec}+\lambda_{3}\mathcal{L}_{con},$$

(9)

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are hyper-parameters for balancing the overall loss.

To evaluate the generalization ability of the proposed framework, our experiments are conducted on four large-scale benchmark databases: FaceForensics++ (FF++) [37], DeepfakeDetection (DFD) [11], Deepfake Detection Challenge (DFDC) [12], and CelebDF [26]. FF++ [37] is a large-scale database comprising more than 1.8 million forged images from 1000 pristine videos. Forged images are generated by four face manipulation algorithms using the same set of pristine videos, i.e., DeepFakes (DF) [1], Face2Face (F2F) [45], FaceSwap (FS) [2], and NeuralTexture (NT) [44]. To evaluate the generalization ability of our framework, we follow prior research works [23, 7] and conduct experiments on three widely used face-manipulated datasets, i.e., DFDC [12], CelebDF [26], and DFD [11]. Note that there are three versions of FF++ in terms of compression level, i.e., raw, lightly compressed (HQ), and heavily compressed (LQ). Since realistic forgeries often have a limited quality, the HQ and LQ versions are used in experiments. Following previous works [23, 7], the HQ version of FF++ is adopted by default. If any deviation from this default, it will be explicitly stated.

We use a modified version of Xception [37] as the backbone network, with model parameters initialized by pre-training on ImageNet. Face extraction and alignment are performed using DLIB [39]. Following previous works [7], the aligned faces are resized to $256\times 256$ for both the training and testing. We use the Adam [21] for optimization with the learning rate of 0.0002, and the batch size is fixed as 32. In the overall loss function in Eq. (9), we set $\lambda_{1}$ to $\lambda_{3}$ as 0.1, 0.3, 0.05 empirically. The margin $\alpha$ in Eq. (5) is set to 3. We also apply some widely used data augmentations, i.e., image compression, horizontal flip, and random brightness contrast.

We report the Area Under Curve (AUC) metric to compare our proposed method with prior works, which is consistent with the evaluation approach adopted in many previous works [23, 36, 37, 30, 7]. The default evaluation metric employed is the AUC. We also report other metrics such as Accuracy (ACC), Average Precision (AP), and Equal Error Rate (EER) for a more comprehensive evaluation of our method. Please refer to our supplementary for more details.

To assess the generalization capacity of our framework, we reproduce ten competing methods under consistent conditions for a comprehensive comparison: Xception [37], Face X-ray [23], F3Net [36], SRM [30], SPSL [28], RECCE [6], CORE [35] , SLADD [7], and Liang et al. [27]. We use the provided codes of Xception, RECCE, SLADD, CORE, FWA, and SRM from the authors. We reimplement Face X-ray, F3Net, SPSL, and Liang et al. [27] rigorously following the companion paper’s instructions and train these models under the same settings.

We conduct this experiment by training the models on the FF++ [37] and then evaluate these models in DFD [11], DFDC [12], and CelebDF [26], respectively. This setting is challenging in generalization ability evaluation since the testing sets are collected from different sources and share much less similarity with the training set.

The results of the comparison between different methods are presented in Tab. 1, which shows the performance in terms of the AUC metric. It is evident that the proposed disentanglement framework and multi-task learning strategy lead to superior performance compared to other models in most cases, achieving the overall best results.

Liang et al. [27] proposes a disentanglement framework for content information removal, but their model is still prone to overfitting to method-specific patterns, leading to the limitation of the generalization. On the contrary, the disentanglement framework we proposed is designed to learn generalizable features across different forgeries by the multi-task learning strategy, thereby achieving improved generalization performance.

Face X-ray [23] and FWA [25] use blended artifacts in forgeries to achieve generalization. However, these two methods have limited generalization ability when the patterns in the training and testing datasets differ. This is because Face X-ray learns to identify the boundary patterns that are sensitive to the post-processing operations varying in different datasets. On the contrary, our proposed framework learns common representations that are not dependent on specific post-processing operations.

SRM [30], SPSL [28], and F3Net [36] utilize frequency components of images to distinguish between forgeries and pristine images. However, the experimental results show that their generalization performance is inferior to the proposed approach. This could be due to the fact that these frequency cues that are effective on the FF++ may not generalize to other datasets with different post-processing steps.

CORE [35], RECCE [6], SLADD [7] are recent detectors that focus on different detection algorithms: loss design, reconstruction learning, and adversarial training. However, these detectors could still be disrupted by unrelated factors such as race, gender, or identity because they operate in the entire feature space, which inevitably includes these unrelated aspects (similarly indicated in previous work [54]). From Tab. 1, our UCF (82.8%) largely outperforms SLADD (78.2%) in terms of the average AUC.

Finally, Xception [37] serves as a CNN baseline and does not incorporate any augmentation, disentanglement, feature engineering, or frequency information. Its performance drops dramatically in the case of unseen forgeries, highlighting the importance of incorporating these techniques in face forgery detection models.

For disentanglement-based detection frameworks, we identify one prior work, Liang et al. [27], that shares similarities with our approach. Their framework aims to remove content information and introduces two modules to enhance the independence of disentangled features. To ensure a fair comparison, we carefully implement their framework by following the settings of the original paper, as it is not available as an open-source resource. We train the baseline Xception, Liang et al. [27], and ours on the FF++ and evaluated them on FF++ (LQ), CelebDF, DFD, and DFDC. As reported in Tab. 3, we observe that Liang et al. [27] improve upon the baseline largely, demonstrating the essential of removing content information. Additionally, UCF outperforms Liang et al. [27] on all testing datasets, showing the efficacy of uncovering common features.

We further evaluate our method against other state-of-the-art models. The results, as shown in Tab. 2, demonstrate the effective generalization ability of our framework as it outperforms other methods, achieving the best performance in terms of the AUC metric on both CelebDF and DFDC. The results of some methods are directly cited from  [7, 53, 50]. Following a comprehensive evaluation against 27 state-of-the-art detectors (10 implemented in this study and 17 referenced), we demonstrate the robust generalization capability of our proposed framework. It is worth noting that both UCF and Zhuang et al. [60] aim to tackle the challenging issue of overfitting to method-specific artifacts. However, their technical methodologies are totally different (disentangle vs. adversarial learning). Actually, we offer a fresh and distinct solution to this problem. Moreover, UCF (82.4%) significantly outperforms Zhuang et al. [60] (72.8%) on CelebDF in terms of AUC.

To evaluate the impact of the proposed disentanglement framework and multi-task learning strategy on generalization ability, we conduct an ablation study on several datasets. Specifically, we train all models on FF++ and evaluate their performance on FF++ [37], DFD [11], DFDC [12], and CelebDF [26]. The results are reported in Tab. 4 using the AUC metric. The evaluated variants include the baseline Xception, Xception with the proposed disentanglement framework (Xception + D), the proposed disentanglement framework with the multi-task learning strategy (Xception + D + M), and the multi-task disentanglement framework with the contrastive regularization (Xception + D + M + C).

Regarding the ablation study, we observed the following. Firstly, the four variants achieve relatively similar results on FF++ (within-dataset evaluation). Secondly, implementing the basic disentanglement framework leads to a significant improvement in DFD, DFDC, and CelebDF (cross-dataset evaluation), indicating the generalization ability is improved largely when applying the proposed disentanglement framework for the content information removal. Thirdly, the multi-task disentanglement outperforms the basic disentanglement, indicating that the multi-task learning strategy is effective in improving the generalization ability of the model. Finally, combining the proposed multi-task disentanglement with the contrastive regularization loss achieves the best results in both within-dataset and cross-dataset evaluations, supporting the effectiveness of each module.

In contrast to other disentanglement-based detection frameworks [54, 51, 27] that use linear addition to combine fingerprint and content features for recombination, our proposed decoder utilizes AdaIN [19] to incorporate the fingerprint as a condition with the content for improved reconstruction and decoding. To evaluate the impact of the conditional decoder on the generalization ability, we conduct an ablation study on the proposed framework with and without the conditional decoder. Results in Tab. 5 demonstrate that our proposed conditional decoder can achieve improved performance on both within- and cross-datasets, highlighting the importance of using AdaIN layers for reconstruction and decoding.

To evaluate the effectiveness of the proposed multi-task learning strategy, binary classification results are compared based on the common, specific, and whole forgery features. Tab. 6 shows that the common features exhibit superior generalization performance compared to the specific features. The comparison of the common and whole forgery features reveals that the whole forgery features are not as effective as the common features, mainly due to the presence of specific features, which may lead to overfitting to method-specific textures.

In this section, we investigate the choice of backbone on the generalization ability of our proposed framework. We use Xception as the backbone in our previous experiments to align with other related works, but our multi-task framework is not limited to this choice. To evaluate the effectiveness of our framework with different backbones, we adopt a recent SOTA backbone ConvNeXt [29] and conduct an ablation study. The results of the ablation study, shown in Tab. 7, demonstrate that our proposed framework can largely improve the generalization performance of both Xception and ConvNeXt backbones. This suggests that our framework is effective and applicable to different backbone choices. Additionally, to further highlight the plug-and-play nature of our proposed framework, we extend its application to ResNet [17] and EfficientNet [43] backbones. More details can be accessed in our supplementary materials.

Within the framework we propose, the generation of reconstruction images occurs during the recombination phase of training. These images serve a pivotal role in ensuring the effective disentanglement of content and fingerprint features, as depicted in Fig. 5.

The content features within our framework are specifically designed to capture appearance, identity, gender, and other forgery-related features. In the visual examples of the reconstructed images (see Fig. 5), it is evident that the images sharing the same content code exhibit a marked similarity. This resemblance persists even when the fingerprint features, which represent unique identifiers separate from the content, are derived from other individuals. The observed similarity in the reconstructed images with identical content codes substantiates the efficacy of our framework in accurately isolating content features from other elements of the image. Contrary to the content features, the fingerprint features do not alter the content information of the original inputs. However, a close examination of the reconstructed images with the same content code but different fingerprint codes reveals subtle differences.