Deepfake Detection: A Comprehensive Survey from the Reliability Perspective

Deepfake is initially raised in the Reddit community when the open-source implementation was first published by the user ‘deepfakes’ simultaneously in 2017. Early research mainly focuses on subject-specific approaches, which can only swap facial identities that the models have seen during training. The most popular framework of the existing Deepfake synthesis studies (deepfakes, 2019; Perov et al., 2021) for the subject-specific identity swap is an autoencoder (Kingma and Welling, 2014). In a nutshell, the autoencoder contains a shared encoder that extracts identity-independent features from the source and target faces and two unique decoders each is in charge of reconstructing synthetic faces of a corresponding facial identity. Specifically, in the training phase, faces of the source identity are fed to the encoder for identity-independent feature extraction. The extracted context vectors are then passed through the decoder that corresponds to reconstructing the faces of the source identity. Similarly, the target face reconstruction is trained following the same workflow. When using a well-trained model to operate face-swapping, a target face after context vector extraction is fed to the decoder that reconstructs the source identity. Thenceforth, the decoder generates a look maintaining the facial expression and movement of the target face while having the identity of the desired source face. If the face-swapping model is trained for both directions, a target face may be face-swapped onto a source face following the same workflow.

Recent studies gradually focus on subject-agnostic methods to enable face-swapping for arbitrary identities with higher resolutions and have exploited Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) for better synthesis authenticity (Zhu et al., 2017; Ding et al., 2018; Natsume et al., 2018; Karras et al., 2018; Brock et al., 2019; Choi et al., 2018; Nirkin et al., 2019; Park et al., 2019; Gao et al., 2021; Zhang et al., 2021; Karras et al., 2021). In other words, they aim to consistently produce high-quality face-swapping results even on facial identities that are unseen during model training. GAN is a generator-discriminator architecture that is trained by having two components battle against each other to advance the output quality. In practice, the generator is periodically trained to fool the discriminator with synthetic faces. For instance, FaceShifter (Li et al., 2019) and SimSwap (Chen et al., 2020) each devise particular modules to preserve facial attributes that are hard to reconstruct and maintain fidelity for arbitrary facial identities. MegaFS (Zhu et al., 2021) and HifiFace (Wang et al., 2021a) accomplish face-swapping at high resolutions of 512 and 1,024 for arbitrary facial identities, respectively, relying on the reconstruction ability of GAN. In the last step, the generated fake face is usually blended back to the pristine target image with tuning techniques such as blurring and smoothing (Zhang et al., 2018) to reduce the visible Deepfake traces.

Benchmark datasets are vital in the development history of Deepfake detection models. Dolhansky et al. (Zi et al., 2020) raised the idea to break down previous datasets into three generations based on the two-generation categorization in the early work (Jiang et al., 2020). As listed in Table 1, UADFV (Yang et al., 2019), DeepfakeTIMIT (Korshunov and Marcel, 2018), and FaceForensics++ (FF++) (Rossler et al., 2019) are categorized to the first generation; Deepfake Detection Dataset (Gully, 2019), DeepFake Detection Challenge (DFDC) Preview (Dolhansky et al., 2019), and Celeb-DF (Li et al., 2020b) are in the second generation; DeeperForensics-1.0 (DF1.0) (Jiang et al., 2020) and DeepFake Detection Challenge (DFDC) (Dolhansky et al., 2020) are in the third generation. In summary, later generations contain general improvements over the previous ones in terms of dataset magnitude or synthetic method diversity.

Unlike the summary by Dolhansky et al. (Zi et al., 2020), the agreement from individuals appearing is not considered in this study, and we instead re-define the third generation such that the datasets are of better quality, broader diversity and magnitude, higher difficulty than the early generations, or challenging detailed discrepancies in early synthetic videos are resolved in the datasets. Specifically, besides DFDC with large manipulation diversity and dataset magnitude and DF1.0 with large magnitude and considerable difficulty by adding deliberate perturbations, we further classify the following datasets in the third generation, namely, FaceShifter (Li et al., 2019), WildDeepfake (Zi et al., 2020), and KoDF (Kwon et al., 2021).

FaceShifter (Li et al., 2019), although synthesized based on real videos of FF++, has specifically solved the so-called facial occlusion challenge that appears in previous datasets. In other words, the synthetic results are better handled even in difficult cases where parts of the face are blocked or obscured by objects such as accessories or other body parts such as hair tippings. WildDeepfake (WDF) (Zi et al., 2020) appears to be a special one in the third generation because videos are totally collected from the internet, which matches the real-life Deepfake circumstance the best. The most recent KoDF (Kwon et al., 2021) dataset is so far the largest Deepfake benchmark dataset that is publicly available with reasonable diversity and contains synthetic videos at high resolutions.

Deep learning models usually exhibit satisfactory performance on the same type of data that are seen in the training process but perform poorly on unseen data. In real life, Deepfake materials can be generated via various synthetic techniques (MarekKowalski, 2019; Lu, 2018; Chen et al., 2020; Gao et al., 2021; Zhu et al., 2021) as abundant on-the-shelf easily accessible face-swapping implementations are publicly available, and a reliable Deepfake detection model is expected to perform well on unseen data to imitate real-life Deepfake cases. Therefore, guaranteeing the transferability of the detection models for cross-dataset performance is necessary and frequently discussed.

With the fast development of deep learning techniques, various methods are devised using simple Convolutional Neural Network (CNN) based models. Zhou et al. (Zhou et al., 2017) fused a CNN stream with a support vector machine (SVM) (Hearst et al., 1998) stream to analyze face features with the assistance of local noise residuals. Afchar et al. (Afchar et al., 2018) studied the mesoscopic features of images with successive convolutional and pooling layers. Nguyen et al. (Nguyen et al., 2019a) designed a multi-task learning scheme to simultaneously perform detection and localization using CNNs. DFT-MF (Jafar et al., 2020) thinks that the mouth features can be important for detection and utilizes a convolutional model to detect Deepfake by verifying, analyzing, and isolating mouth and lip movements. Face X-ray (Li et al., 2020a) performs a medical x-ray on the candidate Deepfake faces by revealing whether the blending of two different source images can be decomposed.

Rather than the basic CNN architectures, well-designed and pre-trained CNN backbones are frequently exploited to improve model performance on Deepfake detection, especially for the cross-dataset performance on unseen data as more benchmark datasets have been released. An early approach (Amerini et al., 2019) proposes optical flow analysis with pre-trained VGG16 (Simonyan and Zisserman, 2015) and ResNet50 (He et al., 2016) CNN backbones and achieves preliminary in-dataset test performance on the FaceForensics++ (FF++) dataset (Rossler et al., 2019). Capsule (Nguyen et al., 2019b) employs capsule architectures (Sabour et al., 2017) with light VGG19-based network parameters but achieves similar detection performance to the traditional approaches leveraging CNNs. Li et al. (Li and Lyu, 2019) conducted a strengthened model, DSP-FWA, with the help of the spatial pyramid pooling (He et al., 2015) with ResNet50 as the backbone. This method is shown to be applicable to Deepfake materials at different resolution levels. FFD (Dang et al., 2020) leverages the popular attention mechanism by element-wise multiplication to study the feature maps and utilizes the XceptionNet CNN backbone, achieving marginally more promising performance than the work by Rossler et al. (Rossler et al., 2019). SSTNet (Wu et al., 2020) exploits Spatial, Steganalysis, and Temporal features using XceptionNet (Chollet, 2017) and exhibits reasonable intra-cross-dataset performance on FF++. Given the assumption that Deepfake only modifies the internal part of the face, Nirkin et al. (Nirkin et al., 2022) adopted two streams of XceptionNet (Chollet, 2017) for face and context (hair, ears, neck) identification and another XceptionNet to classify real and fake based on the learned discrepancies between the two. Later, Bonettini et al. (Bonettini et al., 2021) and Tariq et al. (Tariq et al., 2018) studied the ensemble of various pre-trained convolutional models including EfficientNetB4 (Tan and Le, 2019), XceptionNet (Chollet, 2017), DenseNet (Huang et al., 2017), VGGNet (Simonyan and Zisserman, 2015), ResNet (He et al., 2016), and NASNet (Zoph et al., 2018) backbones to detect Deepfake. Rossler et al. (Rossler et al., 2019) employed the pre-trained well-designed XceptionNet (Chollet, 2017) network and achieved state-of-the-art detection performance at the time on FF++. Besides, a special study introduced by Wang et al. (Wang et al., 2020) proves the transferability of the model trained on one CNN-generated dataset to the rest ten using pre-trained ResNet50 (He et al., 2016).

Meanwhile, frequency cues are also noticed and analyzed by researchers. While early image forgery detection work (Bai et al., 2020) focus on all high, medium, and low frequencies via Fourier transform, Frank et al. (Frank et al., 2020) were the first that raised the idea of finding frequency inconsistency between real and fake by employing low-frequency features to assist Deepfake detection using RGB information. Since low-frequency traces are mostly hidden by blurry facial features, later studies mainly analyze high-level features. F3-Net (Qian et al., 2020) exploits both low- and high-frequency features without RGB feature extraction operation. DFT (Durall et al., 2020), Two-Branch (Masi et al., 2020), SPSL (Liu et al., 2021a), and MPSM-RFAM (Chen et al., 2021) accomplish promising detection results by analyzing high-frequency spectrum along with the on-the-shelf RGB features. Li et al. (Li et al., 2021) extracted both middle- and high-frequency features and correlated frequency and RGB features for detection. By pointing out the drawbacks of using coarse-grained frequency information, Gu et al. (Gu et al., 2022) combined fine-grained frequency information with RGB features for better feature richness in a latest work. Moreover, Jeong et al. (Jeong et al., 2022) designed a new training scheme with frequency-level perturbation maps added, which further enhanced the generalization ability of the detection model regarding all GAN-based generators.

Since the CNN architecture and backbones lack generalization ability and mainly focus on local features, even XceptionNet is restricted in learning the global features for further performance improvements. Therefore, developed solutions have introduced convolutional spatial attention to enlarge the local feature area and learn the corresponding relations, and the detection AUC scores have been gradually raised above 70% on average upon unseen datasets accordingly. The SRM (Luo et al., 2021) approach makes two streams of network using XceptionNet as the CNN backbone and focuses on the high-frequency information and RGB frames with a spatial attention module in each stream. The MAT model (Zhao et al., 2021b) proposes the CNN backbone EfficientNetB4 and borrows the convolutional attention idea to study different local patches of the input image frame. Specifically, the artifacts in shallow features are zoomed in for fine-grained enhancement in the detection performance. With the success of the transformer architecture (Vaswani et al., 2017) in the natural language processing (NLP) domain, different versions of vision transformer (Dosovitskiy et al., 2021; Liu et al., 2021b; Peng et al., 2021; Touvron et al., 2021; Wang et al., 2021b) have derived reasonable performance in the computer vision domain due to its ability on global image feature learning. The architectures of vision transformers (ViT) have also been employed for Deepfake detection with promising results in recent work (Jeon et al., 2020; Heo et al., 2022; Wodajo and Atnafu, 2021; Wang et al., 2023; Cheng et al., 2023).

While reaching the bottleneck specially for the cross-dataset performance even resorting to the advanced powerful but potentially time-consuming neural networks, most recent work gradually focuses on strategies to enrich diversity in training data. Such attempts have improved the detection AUC scores even up to 80% on some unseen datasets. PCL (Zhao et al., 2021a) is introduced with an inconsistency image generator to add synthetic diversity and provide richly annotated training data. Sun et al. (Sun et al., 2022) proposed Dual Contrastive Learning (DCL) to study positive and negative paired data and enrich data views for Deepfake detection with better transferability. FInfer (Hu et al., 2022a) inferences future image frames within a video and is trained based on the representation-prediction loss. Shiohara and Yamasaki (Shiohara and Yamasaki, 2022) presented a novel synthetic training data called self-blended images (SBIs) by blending solely pristine images from the original training dataset, and classifiers are thus boosted to learn more generic representations. Chen et al. (Chen et al., 2022b) proposed the SLADD model to further synthesize the original forgery data for model training to enrich model transferability on unseen data using pairs of pristine images and randomly selected forgery references with forgery configurations including forgery region, blending type, and mix-up blending ratio. Cao et al. (Cao et al., 2022) introduced the RECCE model to emphasize the common compact representations of genuine faces based on reconstruction-classification learning on real faces. The reconstruction learning over real images enhances the learning representations to be aware of unknown forgery patterns. Liang et al. (Liang et al., 2022) proposed an easily embeddable disentanglement framework to remove content information while maintaining artifact information for training and Deepfake detection using reconstructed data with various combinations of content and artifact features of real and fake samples. The OST (Chen et al., 2022c) method improves the detection performance by preparing pseudo-training samples based on testing images to update the model weights before applying to the true samples.

Despite the promising ability of deep learning models, they suffer the weak interpretability problems due to the black-box characteristic (Loyola-González, 2019). In other words, it is hard to explain how and why a model comes up with a particular result. For the Deepfake detection task, while achieving promising detection performance statistically for in- and cross-dataset evaluations, the interpretability issue is still maintained to be fully resolved at the current stage. In other words, people tend to trust methods that are easily understandable via common sense rather than those with satisfactory accuracies but derived based on features that are hard to explain. Consequently, forensic evidence is critical to be probed to interpret and support the detection model performance by highlighting the reasons for classifying fake samples. In a nutshell, the interpretability challenge is to answer the following questions regarding a Deepfake suspect in order to be reliable:

• •

  Why is the content classified as fake?

• •

  Based on which part does the detection model determine the content as fake?

As confirmed by Baldassarre et al. (Baldassarre et al., 2022) using a series of quantitative metrics for evaluating the interpretability of the detection models, heatmaps are generally futile and unacceptable when being adopted to explain the detected artifacts (Xu et al., 2022). Moreover, heatmaps of the real faces are displayed in like manners, which are indistinguishable from that of the fake ones. Therefore, to answer the above questions, a group of noise-based research has been attempted to probe the Deepfake forensic traces for explanations upon the detection results. Early work mostly relies on the Photo Response Non-Uniformity (PRNU), a small factory-defects-generated noise pattern in the light-sensitive sensors of a digital camera (Lukas et al., 2006). PRNU has shown strong abilities in source anonymization (Picetti et al., 2020) and source device identification (Saito et al., 2017; Marra et al., 2017). Unfortunately, most of the PRNU-based Deepfake detection studies (Koopman et al., 2018; de Weever and Wilczek, 2020) have failed to show strong detection performance statistically. Therefore, the PRNU noise pattern can be a useful instrument for source device identification tasks, but it may not be a meaningful forensic noise tracing tool to satisfy the purpose of the Deepfake detection task with respect to the interpretability goal.

Later approaches (Amerini et al., 2019; Jafar et al., 2020; Li et al., 2020a; Chen et al., 2022a) step up to utilize CNNs for noise extraction and analyze the trace differences between real and fake. A milestone denoiser, DnCNN, proposed by Zhang et al. (Zhang et al., 2017) is able to perform blind Gaussian denoising with promising performance, and has been later extended to study the camera model fingerprint for various downstream tasks including forgery detection and localization for face-swapping images based on the underlying noises (Cozzolino and Verdoliva, 2020; Cozzolino Giovanni Poggi Luisa Verdoliva, 2019). Recently, studies (Guarnera et al., 2020b, a) aim to extract the manipulation traces for detection and interpretation. Guo et al. (Guo et al., 2021) proposed the AMTEN method to suppress image content and highlight manipulation traces as more filter iterations are applied. Wang et al. (Wang et al., 2022a; Wang and Chow, 2023) utilized the pre-trained denoisers for Deepfake forensic noise extraction and investigated the underlying Deepfake noise pattern consistency between face and background squares with the help of the siamese architecture (Bromley et al., 1993). Guo et al. (Guo et al., 2022) proposed a guided residual network to maintain the manipulation traces within the guided residual image and analyzed the difference between real and fake.

Besides studies (Qi et al., 2020; Ciftci et al., 2020) that rely on biological signals by analyzing minuscule periodic changes through the faces visualizing distinguishable sequential signal patterns as indicators, other studies mostly attempt to explore universal and representative artifacts directly from the visual concepts of the Deepfake materials. The PRRNet (Shang et al., 2021) approach studies region-wise relation and pixel-wise relation within the candidate image for detection and can roughly locate the manipulated region using pixel-wise values. Trinh et al. (Trinh et al., 2021) fed the dynamic prototype to the detection model and successfully visualized obvious artifact fluctuation via the prototype. Yang et al. (He et al., 2021) proposed to re-synthesis testing images by incorporating a series of visual tasks using GAN models, and they finally extracted visual cues to help perform detection. Obvious artifact differences can be observed by their Stage5 model on real and fake samples. Most recently, Dong et al. (Dong et al., 2022) introduced FST-matching to disentangle source-, target-, and artifact-relevant features from the input image and improved the detection performance by utilizing artifact-relevant features solely. The adopted features within the fake samples are visualized to explain the detection results.

As the transferability and interpretability challenges have been frequently undertaken and reasonable or even promising results have been achieved accordingly, there lacks a stage to be fulfilled in order for the detection approaches to be useful in real-life cases. The challenge of this stage can be summarized as robustness. To be specific, the quality of the candidate Deepfake material in real life is not as ideal as the benchmark datasets in experimental conditions most of the time. Consequently, they may experience different levels of compression operation in multiple scenarios due to objectively limited conditions (Marcon et al., 2021). Moreover, post-processing strategies and artificially added perturbations can cause further challenges and even disable the well-trained and well-performed detection models. Therefore, the robustness of the detection approaches is necessary to be significantly considered when facing various real-life application scenarios under restricted and special conditions.

Multiple studies have been conducted mainly regarding two types of real-life conditions, namely, the passive condition and active condition. The passive condition refers to scenarios with objective limitations such as video compression due to network flow settings. In an early study, Kumar, Vatsa, and Singh (Kumar et al., 2020) designed multiple streams of ResNet18 (He et al., 2016) to specifically deal with face reenactment manipulation at various compression levels. Hu et al. (Hu et al., 2022b) proposed a two-stream method that specifically analyzes the features of compressed videos that are widely spread on social networks. The LRNet (Sun et al., 2021) model is developed to stay robust when detecting highly compressed or noise-corrupted videos. Cao et al. (Cao et al., 2021) solved the difficulty of detecting against compressed videos by feeding compression-insensitive embedding feature spaces that utilize raw and compressed forgeries to the detection model. Wu et al. (Wu et al., 2022b, a) analyzed noise patterns generated via online social networks before feeding image data into the detection model for training. The method has won top ranking against existing approaches especially when facing forgeries after being transmitted through social networks. Le and Woo (Binh and Woo, 2022) employed attention distillation in the frequency perspective and have successfully raised the detection ability on highly compressed data using ResNet50. RealForensics (Haliassos et al., 2022) aims to solve Deepfake contents with real-life quality by training the detection model with an auxiliary dataset containing real talking faces before utilizing the benchmark datasets. The method has significantly advanced the detection performance against multiple objective scenarios and adversarial attacks such as Gaussian noise and video compression. Besides, a recent work (Wang et al., 2022b) has constructed a new dataset containing Deepfake contents under the near-infrared condition to prevent potential future Deepfake attacks in the corresponding scenarios.

On the other hand, the active condition summarizes deliberate adversarial attacks such as distortions and perturbations. Gandhi and Jain (Gandhi and Jain, 2020) explored Lipschitz regularization (Woods et al., 2019) and Deep Image Prior (DIP) (Lempitsky et al., 2018) to remove the artificial perturbations they generated and maintain model robustness. Yang et al. (Chuming et al., 2021) simulated commonly seen data corruption techniques on the benchmark datasets to increase data diversity in model training. Operations such as resolution down-scaling, bit-rate adjustment, and artifacting have effectively boosted the detection ability against in-the-wild Deepfake contents. Hooda et al. (Hooda et al., 2022) designed a Disjoint Deepfake Detection (D3) detector that improves adversarial robustness against artificial perturbations using an ensemble of models. The LTTD (Guan et al., 2022) framework is enforced to conquer the challenges brought by post-processing procedures of Deepfake generation like visual compression to models that rely on low-level image feature patterns. Lin et al. (Lin et al., 2022) proved the fact that having temporal information to participate in detection makes the detector less prone to black-box attacks.

Moreover, in the latest studies, researchers have frequently illustrated the necessity of robustness by fooling the well-trained detection models with stronger adversarial attacks (Hussain et al., 2022; Goodman et al., 2020). Jia et al. (Jia et al., 2022) even emphasized the robustness of the detection models against potential adversarial attacks by injecting frequency perturbations to fool the state-of-the-art approaches. Also, the robustness of Deepfake detectors is evaluated via the Fast Gradient Sign Method (FGSM) and the Carlini-Wagner L2-norm attack in several studies (Gandhi and Jain, 2020; Shahriyar and Wright, 2022). Moreover, Carlini and Farid (Carlini and Farid, 2020) employed white- and black-box attacks on a well-trained detector in five case studies. Obvious performance damping can be observed from the reported results. Faces with imperceptible visual variation even after perturbations and noises are added in have highlighted the importance of studying model robustness in the current Deepfake detection research domain.

In the ideal scenario, a reliable Deepfake detection model should retain promising transferability on unseen data with unknown synthetic techniques, pellucid explanation upon the falsification, and robust resistance against different real-life application conditions. As reviewed in Section 3, despite a method favorably satisfying all three challenges simultaneously is not yet accomplished, a metric that nominates the reliability of a method for real-life usages and court-case judgments is needed.

Regardless of the largely improved but still unsatisfied cross-dataset performance in the evolution of Deepfake detection, existing studies have only evaluated the model performance on each testing dataset to show the model detection abilities while the values for each evaluation metric (accuracy and AUC score) vary depending on different testing sets adopted in experiments. On the contrary, in real-life cases, people have no clue about fake content regarding its source dataset or the corresponding facial manipulation techniques, and the malicious attacker is unlikely to reveal such crucial information. Consequently, for a victim of Deepfake to defend his or her innocence or accuse the attacker (Kelion, 2018; Lee, 2018; Chinchilla, 2021; Miller, 2022), simply presenting a model detection decision and listing the numerical model performance on each benchmark dataset may not be convincing and reliable. Specifically, a unique statistical claim is necessary regarding the detection performance to discuss the model trustworthiness on any arbitrary candidate suspect instead of varying on each testing dataset when adopting the detection model as forensic evidence for criminal investigation and court-case judgments. To conclude, the following questions are to be solved:

• •

  Can a detection model assist or act as evidence in forensic investigations in court?

• •

  How reliable is the detection model when performing as forensic evidence in real-life scenarios?

• •

  How accurate is the detection model regarding an arbitrary falsification?

Unfortunately, to the best of our knowledge, no existing work has studied the model reliability or come up with a reliable claim for the model to play the role of forensic evidence. Therefore, in this study, inspired by the reliability study on antigen diagnostic tests (Administration, 2022; Committee, 2022; Government, 2022) for the recent COVID-19 pandemic (Osterman et al., 2021), we conduct a quantitative study by investigating the detection model reliability with a new evaluation metric with statistical techniques. Unlike the studies for antigen diagnostic tests that prefer to achieve a perfect specificity rather than sensitivity as the goal is to avoid missing any positive case (Organization, 2022), we wish to correctly identify both real and fake materials with no priority. In particular, we construct a population to imitate real-life Deepfake distribution and design a scientific random sampling scheme to analyze and compute the confidence intervals for the values of accuracy and AUC score metrics regarding the Deepfake detection models. As a result, numerical ranges indicating the reliable model performance at 90% and 95% confidence levels can be derived for both accuracy and AUC score.

In reality, a candidate Deepfake suspect can only have two possible categories, namely, real and fake. Admittedly, most public images and videos circulating on the internet are pristine without artificial changes. However, whenever a real-life Deepfake case is raised such that the authenticity of the candidate material needs to be justified, we could not consider all images and videos in the world as the target population (Tonucci, 2005) because most of the real ones are not likely to be disputed in the discussion of Deepfake. At the same time, the probability that the candidate material is fake does not necessarily equal to the proportion of fake ones regarding all images and videos in the world.

Despite the uncertainty of the real-life Deepfake population and distribution, in this study, we construct a sampling frame (Brown, 2010) with the accessible high-quality Deepfake benchmark datasets to imitate the target population of Deepfake in real life for detection model reliability analysis. Details of the participating datasets are introduced in Section 5.2.

We perform random sampling (Martino et al., 2018) from the constructed sampling frame with a sample size of $s$ for $t$ trials. Two sampling options are considered in this study: balanced and imbalanced. For a balanced sampling setting, we maintain the condition that the same amount of real and fake samples are randomly drawn. On the contrary, an imbalanced setting allows a completely random stochastic rule with respect to real and fake samples.

For an arbitrary Deepfake detection model $M$ after sufficient training, we randomly draw $s$ samples from the sampling frame following the sampling option. Then the $s$ samples are fed to model $M$ for authenticity prediction, deriving predicted labels and prediction scores. After that, the accuracy and AUC score metrics are computed accordingly based on the ground-truth authenticities of the sampled faces. Such a sampling process is repeated for $t$ trials and a total of $t$ accuracies and $t$ AUC scores are derived in the end. Take the $t$ accuracies as an example, we first compute the mean value $\bar{x}$ and standard deviation $\sigma$ following

and

where $x_{i}$ refers to the accuracy value of the $i$-th trial.

According to the central limit theorem (CLT) (Fischer, 2011), the distribution of sample means tends toward a normal distribution as the sample size gets larger. Therefore, the normal distribution confidence interval $CI$ can be calculated by

where parameter $z$ represents the z-score, an indicator of the confidence level following the instruction of the z-table [16]. When deducing statistical results for the $t$ AUC scores, the above workflow applies identically.

Since the target population is of unknown distribution, different values of the sample size $s$ are adopted in order to settle the confidence intervals at different confidence levels. Meanwhile, considering that an insufficient number of trials per sample size may cause bias when locating the sample mean, different values of trails for $t$ are chosen to eliminate the potential bias. Detailed steps of the model reliability study can be summarized as Algorithm 1.

While a video-level detector may rely on special data processing arrangement directly upon videos, for the frame-level detectors, since the detection results are evaluated based on all selected frames, to avoid potential biases toward particular videos, it is meaningful to keep the amount of extracted faces from each video equivalent during model training and testing. Therefore, we firstly obtain $c$ image frames using FFmpeg (FFmpeg, 2021) for each candidate video with an equal frame interval between every two adjacent extracted frames following

where $N$ refers to the number of frames that contain faces in the video and $P=\{p_{0},p_{1},...,p_{c-1}\}$ contains the sequentially ordered indices for which frames to be extracted from the video. In other words, image frames with no face detected are excluded from the sequential ordering and indexing. Besides, videos with fewer than $c$ frames containing detected faces are also omitted. The dlib library (King, 2021) is utilized for face detection and cropping where the face detector provides coordinates of the bounding box that locates the detected face. For the sequence of frames with frame indices $P=\{p_{0},p_{1},...,p_{c-1}\}$ from a video, we fix the size $l$ of a squared bounding box $b$ for all faces by

where $w_{i}$ and $h_{i}$ are the widths and heights of each bounding box. We then locate the center of each face $f_{i}$ with the help of the corresponding bounding box $b_{i}$ and place the fixed squared bounding box $b$ at the centers for face cropping.

Following the convention of the existing Deepfake detection work and considering the qualities of available benchmark datasets, we consider five datasets in experiments in this study, namely, FF++ (Rossler et al., 2019), FaceShifter (Li et al., 2019), DFDC (Dolhansky et al., 2020), Celeb-DF (Li et al., 2020b), and DF1.0 (Jiang et al., 2020). In detail, early datasets (Yang et al., 2019) are excluded due to low quantity and diversity. Meanwhile, although WDF (Zi et al., 2020) is similar to real-life Deepfake materials, the videos collected from the internet are manually labeled without knowing the ground-truth labels, which leads to credibility issues. KoDF (Kwon et al., 2021) is the largest Deepfake dataset up to date, but its huge magnitude requires unreasonably large storage ($\sim$4 TB) that we are unable to acquire and process⁵⁵5The 6 manipulation algorithms in KoDF are highly overlapped with the 8 manipulation algorithms in DFDC. This favorably suggests that the adoption of DFDC satisfies the demand for model diversity even without KoDF. . All involved benchmark datasets follow the pre-processing scheme for face extraction as discussed in Section 5.1, and special settings are mentioned in the following subsections when necessary.

Based on the Deepfake detection developing history and the three challenges of the current Deepfake detection domain, in the experiment, we selected several representative milestone baseline models and the most recent ones that have source code publicly available for reproduction. Specifically, Xception (Chollet, 2017), MAT (Zhao et al., 2021b), and RECCE (Cao et al., 2022) mainly attempt on the transferability challenge, Stage5 (Zhang et al., 2017) and FSTMatching (Dong et al., 2022) focus on the interpretability topic, and MetricLearning (Cao et al., 2021) and LRNet (Sun et al., 2021) are designed for the robustness issue. Models with publicly available trained weights are directly adopted for evaluation if the model is trained on FF++ or special arrangements other than the five benchmark datasets are necessary during training. The rest models are trained on FF++ in our experiment as discussed in Section 5 and all models converge commonly. The selected models are tested on all benchmark datasets. To guarantee complete fairness, we applied optimal parameter settings as reported in the corresponding published papers during training and testing.

During model testing, we recorded the video-level Deepfake detection performance. In particular, detection results of the cropped faces of each video are averaged to a unique output for detectors that are designed for detecting individual images. Methods that can directly generate a single output for each video are fed with raw videos via the corresponding processing scheme as provided by their published source codes. The well-trained models are firstly evaluated on the FF++ testing set for the in-dataset setting, in other words, tested on the same dataset they have seen during training. Then, to further validate the model performance on unseen datasets, the cross-dataset evaluation is conducted to test the models on DFDC, Celeb-DF, DF1.0, and FaceShifter. We set $N=10$ for training and $N=20$ for testing regarding Eq. (4) for frame extraction during data pre-processing when applicable.

We adopted accuracy (ACC) and AUC score at the video level as the evaluation metrics. In detail, the accuracy refers to the proportion of the correctly classified data items regarding all testing data, and the AUC score represents the area under the receiver operating characteristic (ROC) curve. In other words, the AUC score demonstrates the probability that a random positive sample scores higher than a random negative sample from the testing set, that is, the ability of the classifier to distinguish between real and fake faces. For testing sets that contain only fake samples, the AUC score is inapplicable and thus withdrawn.

Model performance for both in- and cross-dataset evaluations is listed in Table 2. It can be observed that all models of the transferability topic have derived reasonable detection performance on FF++ with accuracy values and AUC scores over 90% since their goal is to achieve better performance in cross-dataset experiments after maintaining promising performance on seen data. In particular, MAT (Zhao et al., 2021b) wins the comparison with the highest 97.40% accuracy and 99.67% AUC score. On the contrary, models that are designed specifically for interpretability or robustness purposes have exhibited relatively poor detection performance on FF++, and the potential causation is discussed together with their detection performance on other benchmark datasets in the following paragraphs.

As for cross-dataset evaluation, most models have suffered a performance damping since the testing data are unseen during training. In specific, no model has reached over 80% AUC scores on DFDC or Celeb-DF and some models even exhibit abnormal detection performance when validated on unseen fake testing sets solely (DF1.0 and FaceShifter). This may be caused by oblivious overfitting on real or fake data by the models. While models that are solving the transferability challenge have all exhibited normal and reasonable functionalities, hidden trouble can be discovered regarding the interpretability and robustness topics. Specifically, model weights of Stage5 (He et al., 2021) are adopted for testing because the model is trained with re-synthesized samples using exclusively GAN models under special settings, but this at the same time has led to unsatisfied results when detecting fake samples that are not synthesized using GAN architectures. FSTMatching (Dong et al., 2022) spends huge computing power on disentangling source and target artifacts for the explanation, which thus results in the failure against other models although similarly trained on FF++. MetricLearning (Cao et al., 2021) and LRNet (Sun et al., 2021) both are proposed and trained to deal with highly compressed Deepfake contents in special scenarios. Unfortunately, they suffer performance fluctuation when the compression condition varies without expectation. Furthermore, LRNet (Sun et al., 2021) executes detection based on facial landmarks solely, which is another main reason that leads to the unsatisfactory.

Besides, models are generally unstable on different testing sets. For instance, RECCE (Cao et al., 2022) wins the competition on Celeb-DF for both accuracy and AUC score, but its detection ability deteriorates immensely when facing DFDC. On the other hand, MAT (Zhao et al., 2021b) wins the battle on DFDC and achieves competitive performance on Celeb-DF, but an obvious overdependence on the real samples can be concluded from its poor accuracies on DF1.0 and FaceShfiter. Meanwhile, Xception has derived reasonable detection performance on each testing set even though not winning the comparison on any dataset. As a result, no model appears to be the overall winner according to Table 2 and it is hard to determine which model to use when facing an arbitrary candidate Deepfake suspect in real-life cases.

Moreover, in most cases, a well-trained model usually achieves a higher AUC score than the accuracy on each testing set. The reason is that the threshold to classify real and fake with softmax or sigmoid function applied is always fixed at 0.5 for the accuracy evaluation upon the output scores within the range of [0, 1] where 0 refers to real and 1 represents fake, while the actual threshold for the optimal model performance is usually located differently regarding 0.5. Therefore, despite a classifier with a threshold value set to 0.5 does not perform well, the model may still distinguish between real and fake with a relatively high AUC score. However, it is also worth noting that although a high AUC score may reveal the model’s ability to separate real and fake samples, the threshold may vary depending on different testing sets and different models. Hence, in order to stably determine real or fake, finding a fixed threshold to consistently satisfy the detection goal on arbitrary images and videos may help boost the overall model detection ability in the research domain.

The model reliability evaluation is conducted on a sampling frame with 5,368 videos composed of the testing set of each benchmark dataset, and the detailed data quantity is listed in Table 3. Following the workflow of Algorithm 1, we proposed reliability analyses on the well-trained Deepfake detection approaches. We computed the 90% and 95% confidence intervals in experiments. Specifically, two values of $t$, 500 and 3,000, are chosen to ensure a sufficient number of trials when locating the sample means. Various sample sizes $s\in\{\textrm{10, 100, 500, 1,000, 1,500, 2,000, 2,500}\}$ are selected to find the settled confidence intervals. Besides, both sampling options, balanced and imbalanced sampling, are executed in experiments. Detailed results are listed in Tables 4 to 7 following the Cartesian product settings of balancing options $O=\{\textrm{True, False}\}$, the number of trials $T=\{\textrm{500, 3,000}\}$, and evaluation metrics $E=\{\textrm{ACC, AUC}\}$.

It can be easily observed that for all models in the tables, the mean values $\bar{x}$ gradually settle as sample sizes become larger and the standard deviation (Std.) values $\sigma$ consistently decrease concurrently. Similarly, the 90% and 95% confidence intervals are progressively straitened and settled around the mean values. Moreover, statistical results with 3,000 sampling trials converge faster and are more stable than those with 500 trials as sample size increases for all experiments, and both trial numbers lead to similar final values once settled.

Regarding the balanced sampling setting, the results generally match the model detection performance in Table 2. On the other hand, since the constructed sampling frame is imbalanced for real and fake faces, an imbalanced sampling option may lead to more fake data than the real ones in the sample set. Consequently, for models that have shown relatively poor performance when evaluated on fake testing sets (DF1.0 and FaceShifter) solely, the mean values and confidence intervals of the accuracy are located at lower levels. Meanwhile, models that have achieved promising performance on merely the fake testing sets have led to higher accuracy values for mean and confidence intervals. The AUC score metric, as mentioned in Section 6.1, is impervious regarding the imbalanced dataset. Therefore, mean values and confidence intervals under balanced and imbalanced sampling options are generally identical.

With a closer look at the tables, the leading approaches, Xception (Chollet, 2017) and MAT (Zhao et al., 2021b), both have achieved mean accuracies above 68% and confidence intervals around 69% with respect to the balanced sampling option. All models regarding the interpretability and robustness topics have derived accuracies and confidence intervals around 50%. Stage5 (He et al., 2021) and MetricLearning (Cao et al., 2021) convey results with all values being 50% since the former recognizes all candidate samples as real and the latter classifies all as fake when checking the predicted labels accordingly. While Stage5 (He et al., 2021) may only be interpretable when facing GAN-based synthetic contents depending on its model design, MetricLearning (Cao et al., 2021) relies on a fixed threshold of 5 upon the output value without softmax or sigmoid activation. Since a perfect threshold value may vary depending on the testing dataset, the fixed threshold may be the main fact that causes the mistake. By looking at the AUC scores in Table 6 and Table 7, MetricLeaning (Cao et al., 2021) actually has displayed a reasonable ability in distinguishing real and fake with a mean value of 69.49% AUC score. As for FSTMatching (Dong et al., 2022) and LRNet (Sun et al., 2021), the experimental results are generally consistent with the ones in Table 2.

As for experiments under the imbalanced sampling setting, besides the foreseeably high and low performance by MetricLearning (Cao et al., 2021) and Stage5 (He et al., 2021) as discussed, Xception (Chollet, 2017) wins with the highest mean accuracy of 66.23% due to its stable performance in detecting both real and fake faces. Regarding the AUC scores, ignoring the imperceptible differences and taking a look at Table 6, 76.64% is derived by Xception because of its ability to separate real and fake samples at a certain threshold even though performing relatively unsatisfactory with the threshold value of 0.5 regarding the softmax output scores. Besides, MAT (Zhao et al., 2021b) is the only other model that has achieved above 70% AUC score. In the remaining approaches, RECCE has reached above 65% AUC scores, while the other two methods have performed relatively unsatisfactory in comparison.

It is also worth noting that despite the imbalanced sampling setting does not affect the final values of mean and confidence intervals, it may cause the AUC score incomputable for a tiny sample size. In particular, as shown in Table 7, there is a high possibility to randomly draw 10 samples that belong to the same category, which then leads to an incomputable AUC score since the sample set lacks data from the other category. Meanwhile, it is unlikely to randomly draw 100 or even more samples that are of the same category even though there is a possibility theoretically.

We obtained the available video clips of the four Deepfake cases from the internet and performed Deepfake detection using each of the well-trained models. Specifically, for video clips with sufficient numbers of image frames that contain faces, we randomly extracted 100 frames for face cropping when using frame-level detectors. As for the cheerleader case, as the sensitive contents are omitted, we were only able to acquire a total of 75 faces from the news clip. For frame-level detectors, the numbers of faces classified as real or fake by each model for each video are exhibited in Table 8, and except MetricLearning (Cao et al., 2021) that uses a fixed threshold, the softmax scores for each video are averaged to obtain a single score that indicates the model determined authenticity. In particular, except for MetricLearning, the fake scores lie in the range of [0, 1] where 0 refers to real and 1 refers to fake. For video-level detectors, the ultimate results are straightforwardly demonstrated in the table. As a result, given the fact that all four videos are known to be fake, about half of the selected models have made the correct classifications regarding both the number of fake faces and the average softmax score. Besides, we provided reliably quantified 95% confidence intervals regarding the models’ detection accuracy for reference when utilizing the results.

Despite achieving high statistical values regarding the accuracy and AUC score metrics in early experiments, Xception has failed to classify all four fake videos such that most faces are classified as real and all average softmax scores are below 0.43. Besides, the fake Emma Watson and Natalie Portman videos have also tricked the MAT model such that roughly two-thirds of the faces are classified as real and the final scores are below 0.4. Stage5 and FSTMatching, which proved to be relatively unsatisfactory in early experiments and discussions, have both failed to detect all four fake videos. MetricLearning and LRNet, although performing poorly on lab-controlled datasets, have shown robust detection ability especially because the videos circulated and downloaded from the internet have suffered incrementally heavy compressions. The RECCE model, as a result, turns out to be the winner with the highest fake scores and the number of correctly classified faces when facing real-life Deepfake suspects.

Considering real-life usages, accessible Deepfake detection models such as the ones being discussed in this survey can be gathered to an online platform and to provide real-time detection services. To go even further, similar to VirusTotal (Sistemas, 2004), a famous virus and malware intelligence platform that reports malicious threat intelligence, a fake video intelligence web portal can be established regarding the Deepfake detection research domain by endlessly integrating the detection models and real-life Deepfake intelligence beyond detection.