Exploring the Impact of Moire Pattern on Deepfake Detectors

Numerous studies have explored various methodologies for detecting synthetic media, leveraging a diverse array of features and techniques [13, 14, 15, 16, 17, 18]. Traditional approaches often relied on handcrafted features or heuristic-based methods to discern anomalies indicative of synthetic manipulation [19]. However, with the advent of deep learning, Convolutional Neural Networks (CNNs) have emerged as a dominant paradigm for deepfake detection [3, 20]. These models leverage the inherent spatial hierarchies within images and videos to learn discriminative features for distinguishing between genuine and manipulated content.

Despite the effectiveness of DNN-based approaches for various classification tasks [21], challenges persist in accurate classification, particularly in scenarios involving camera-captured footage. One such challenge arises from the presence of Moiré patterns, which can significantly degrade the quality of images and videos captured from digital screens. The intricate nature of these patterns poses a formidable obstacle to existing detection algorithms, often leading to misclassifications and reduced detection accuracy. However, it has not been studied in the context of deepfake videos. In this work, we aim to fill this gap.

We utilized two well-established deepfake datasets, CelebDF and Faceforensics++ (FF++), to ensure the diversity and representativeness of our data. CelebDF comprises videos featuring celebrities, while FF++ offers a broader range of synthetic and manipulated videos across various contexts. From the FF++ dataset, we selected the DeepFake, Face2Face, FaceSwap, and Neural Texture datasets.

We designed our experiments to compare the performance of deepfake detectors as follows:

The SOTA deepfake detectors we evaluated, including CoReD, CLRNet, Xception, BZNet, QAD, and ADD, typically exhibit accuracy rates in the high 90s when tested on benchmarking datasets like FF++ and CelebDF. However, we have observed a notable performance degradation when these detectors encounter deepfake videos incorporating Moiré patterns. In such cases, performance metrics plummet to as low as 47% for CoRed, with even the highest-performing detector, ADD, unable to surpass 68%. This issue raises significant concerns, as naturally occurring patterns introduced during real-world application have the potential to undermine the credibility of these detectors by yielding erroneous predictions.

1. 1.

   Finding #1. The performance of BZNet on the DeepFake (FF++) dataset demonstrates promise at 94%; however, it exhibits a near-random guess level (i.e., approximately 50%) on all other datasets, indicating potential overfitting to the DeepFake (FF++) dataset. Consequently, the overall performance of BZNet across the five datasets is 60%, as illustrated in the ‘Average’ section of Fig. 3.

2. 2.

   Finding #2. Like BZNet, ADD shows relatively high performance at 86%. However, unlike BZNet, ADD also performs well on the Face2Face (FF++) dataset, suggesting a higher degree of generalizability and positioning it as the best-performing detector among the five evaluated in our experiment, with 68% on average. Nonetheless, it’s important to note that this performance falls short of the typical high 90s range expected from these deepfake detectors. Therefore, the detectors need to be equipped with techniques to identify and mitigate Moiré patterns during training to achieve acceptable real-world performance levels (i.e., in the high 90s).

We present the AUROC scores of all evaluated detectors on the CelebDF dataset in Fig. 4. The AUROC scores align closely with the discussed results. Notably, CLRNet achieves the highest AUROC score of 0.64, which is notably low. This indicates limited discriminative power against camera-captured deepfakes with Moiré patterns, signaling the need for substantial improvements. The relatively low AUROC scores across the evaluated detectors underscore the challenges in effectively detecting deepfake videos, particularly those featuring Moiré patterns. Further research and refinement of detection techniques are imperative to enhance the robustness and efficacy of deepfake detection systems.

The PR curves illustrate the trade-off between precision and recall across varying threshold values for the deepfake detection model. As we can observe, the curves start at high precision (except for CoReD), indicating strong performance at higher threshold values where the model is conservative in making positive predictions. However, as the threshold decreases, allowing the model to be less conservative, recall increases while precision decreases. This behavior is characteristic of models where adjusting the threshold affects the precision-recall balance. Notably, the PR curves reveal dips below the no-skill line for the CoReD model, indicating varying performance across different threshold values. These fluctuations highlight the sensitivity of the model’s performance to threshold adjustments and underscore the importance of fine-tuning parameters to optimize detection performance. As depicted in Fig. 4, understanding the intricacies of precision-recall trade-offs is crucial for effectively evaluating and improving deepfake detection systems.

The results of our experiments demonstrated that Moiré patterns have a significant influence on the accuracy to which deepfake detectors present their results. As a consequence of this, there is an urgent requirement to develop innovative approaches and procedures for these detectors that are capable of effectively mitigating the impact of Moiré artifacts. Because of this, it is necessary to incorporate defenses against Moiré patterns into the training methodology of the detectors, particularly during the stage of data preprocessing and through methods such as data augmentation. These kinds of improvements are absolutely necessary in order to improve the effectiveness of the detectors across a wider range of situations that occur in the real world.

While preprocessing and data augmentation techniques have shown some promise in enhancing deepfake detectors against Moiré-patterned deepfakes through the synthetic introduction of such patterns into the training data, it is important to note that this approach has inherent limitations in terms of its ability to replicate all environmental factors that are present in the real world. A comprehensive deepfake dataset that authentically captures these patterns in real-world settings using physical devices such as computer screens and smartphone cameras is therefore required.

In this paper, we present a preliminary version of such a dataset developed on a smaller scale for experimental purposes. Our future endeavor, on the other hand, intends to significantly expand this dataset by incorporating well-established deepfake benchmark datasets and introducing a wide range of environmental conditions. These conditions will include variations in camera angles, lighting conditions, screen types, and camera models. We argue that such a dataset has the potential to be the best solution for strengthening deepfake detectors against patterns that are induced by physical environmental factors, which would result in an increase in the robustness of these detectors in real-world deployment scenarios.

In advancing deepfake detection systems, a critical consideration lies in evaluating their transferability across various domains, including those influenced by recent advancements in generative AI and diffusion-based deepfakes. While a detector may effectively identify deepfakes generated using specific techniques or originating from certain sources, its performance may degrade when confronted with unseen or adversarially crafted deepfakes, as evidenced by our findings. Therefore, assessing detectors’ robustness and generalization capabilities across diverse datasets and scenarios is essential, as emphasized in prior research [3]. Future endeavors should prioritize the development of evaluation methodologies that rigorously assess the transferability of deepfake detection models, thereby ensuring their reliability and suitability for real-world applications.

One of the most notable drawbacks of our paper is that it does not contain a more extensive selection of image and video detectors. These detectors are necessary for detecting deepfakes when Moiré patterns are present. Furthermore, our dataset was obtained by using a limited selection of smartphone devices, which may not accurately depict the wide variety of environmental variables that can cause Moiré to occur. In the future, research should incorporate a wider variety of image and video detectors in order to improve the accuracy of deepfake detection when it is confronted with Moiré patterns. It is absolutely necessary to collect datasets using a wide variety of smartphone devices in order to guarantee the reliability of the data acquired from a variety of monitor variations. Additionally, the incorporation of a variety of monitors that are capable of introducing distinct Moiré patterns when captured by smartphone cameras would provide a more comprehensive understanding of the impact that these monitors have on the detection of deepfakes. It is of crucial importance to develop methods that involve the reduction of Moiré patterns, as this will allow for the accurate detection of deepfake features. In order to determine the most effective strategies for evaluating deepfake detection methods, additional experimentation using these methods, in conjunction with a variety of detection techniques, will be invaluable value.