Deepfake Detection via Joint Unsupervised Reconstruction and Supervised Classification

Computer vision approaches are widely used for transferring facial expression and features by reconstructing 3D models for both source and target faces, and then exploring the correspondence among face geometry to warp between these two faces. Thies et al. [31] proposed a method to swap facial expression with an RGB-D camera. Face2Face [32] is a face reenactment system that uses only an RGB camera to manipulate facial expression in real-time. The extended work [14] even transfers more attributes other than expressions such as head position, eye blinking, and rotation from the source individual to a target individual in a video. FaceSwap [17] transfers the source individual’s features into the target face while preserving the expressions and head pose. Recently, deep learning-based techniques have attracted lots of attention in synthesizing or manipulating face images. Zhmoginov et al. [38] proposed a deep learning approach that utilizes the neural network to invert low-dimension face embedding for producing realistic face images. This technique is further extended to the mobile application called FaceApp [1], which can selectively edit facial attributes. Another branch of deep learning techniques is GAN-based methods. For example, ProGAN [13] achieves high-resolution face image synthesis by using progressively growing generator and detector. Although intended for age-oriented face synthesis, the Conditional Discriminator Pool proposed by Wang et al. [33] can be easily adapted for deepfake creation. Shao et al. [29] propose a facial expression transfer framework that specifically swaps the fine-grained expression of two unpaired images while well preserving other natural attributes. In [10], the authors introduce a stage-wise framework with an auto-encoder semi-learning manner. It predicts the image target boundary using pose and expression vector, where the boundary and input image are encoded into a latent space using two encoders. Then the input structure and texture are separated using the LightCNN network, and the concatenation of the boundary and input representation is decoded to produce the target synthesis. They considerably reduce the correlation bias in transferring poses and expressions for high-resolution input.

With the increasing magnitude of negative impacts on privacy and information security due to face forgery, researchers are racing to develop effective countermeasures. Some early works consider artifacts left by face manipulation methods as visual cues, e.g., unnatural eye blinking [21], abnormality of head pose [34], and biological signals from synthesized video [7]. Li et al. [22] proposed a detection method based on exploiting face warping artifacts from the manipulation pipeline. However, all these methods share a common drawback that these artifacts become invalid once the manipulation methods evolve. As the learning-based methods become popular, a recent study [27] proposes the deep learning-based framework that learns the features from the spatial domain and achieves outstanding performance on specific datasets. Zhou et al. [39] proposed a two-stream network to extract steganalysis features for tampered face detection. The work presented in [35] divides the original image into many patches where each patch is a particular facial region. The learned sub-feature maps of these patches are taken as input to the patch-based detector for classification. But their methods only consider a specific manipulation during training, thus, the model is not robust enough when tested on unseen face forgery datasets. Face X-ray [19] improves the generalization ability to detect fake faces without using fake images generated by existing manipulation methods. However, this method focuses on analyzing the artifacts left in the image by the post-processing step only: the blending step. While the artifacts due to the blending process are useful, other useful artifacts due to the processes at the prior stages in the manipulation pipeline are ignored. Discovering the intrinsic representation between samples from the same class is more useful for detecting general deepfake. By integrating Xception [6] with a triplet loss in [9], Feng et al. demonstrated the feasibility of their method in detecting deepfakes. In [11], Feng et al. applies the concept of unsupervised contrastive learning to manipulated face forensics by employing Xception as an encoder network. This method is divided into two training stages: 1) pre-training of the encoder using the original face data and its transformed version as a pair with a contrastive loss, followed by 2) the training of a simple classifier using the output from the encoder. It’s worth noting one merit of this method is that they train the encoder without using the ground truth. Although good performance can be observed in intra-data settings, there is still a significant performance gap when their method is applied in cross-data settings. Zhang et al. [36] introduced a self-supervised decoupling network(SDNN) to learn compression and authenticity features. The goal is to normalise the model using various compression rates in order to improve the classification performance of the authenticity classifier. However, the model might not be robust to unseen compression rates since the range of the compression ratios is variable and difficult to predict. In [18], a frequency-aware discriminative feature learning framework (FDFL) is proposed to minimize intra-class variations of authentic faces while increasing inter-class distance in the embedding space and compensate for the low efficacy of handcrafted features for forgery detection. The single-centre loss (SCL) is presented to drag real face features to the centre and push the manipulated features away. However, the model does not perform well on unseen datasets.

Most of the public face manipulation datasets are stored in the format of video. The face area only takes up a small proportion section in manipulated videos, which would raise difficulties for learning features and predicting probabilities. Since the purpose of deepfake is to swap the face or edit the attributes in the face region, locating the face region of each frame in the video and narrowing down the range of the bounding box to the face centre is necessary. In this paper, we use a popular face extractor Dlib [15] to locate the face region in each frame sampled from videos according to 68 facial landmarks and crop the image using a bounding box. To preserve as many tampered traces as possible, the faces are aligned at a centre position to incorporate the spatial information and eliminate the variance of head poses. Finally, the cropped-out faces will go through the transformation procedures (e.g., resize, normalization). The pre-processing allows our framework to work with different resolutions and postures.

Extracting representations effectively is critical for determining whether face images are real or fake. Different from conventional autoencoder, Convolutional Autoencoder (CAE) applies convolutional layers to learn representation by sharing the same weights among all locations in each input channel, which can preserve spatial locality [24]. As such, we employ a CAE to extract more distinguishable representations, which are beneficial in separating true and fake images in a high-dimensional space. To force the network to disentangle two different tasks, we feed the latent vector to a shallow linear network to handle the decision while the decoder restores the latent vector to reconstruct the original input by learning the local features of the input. With the combined supervised and unsupervised training, the distinguishable information associated with their classes (real or fake) is separated accordingly. The decoder upsamples the latent vector and reconstructs a realistic sample by learning the local spatial features of input data. Hence, the reconstruction task reinforces the shared encoder to extract robust features. The two-branch autoencoder design ensures the network shares useful information from both the reconstruction and classification tasks, yielding more robust forgery detection.

In this work, the CAE, denoted by g, consists of two sub-components, namely the encoder $\textit{g}_{e}$() and the decoder $\textit{g}_{d}$(). The face dataset, denoted as X, includes real and fake face images and their corresponding labels ${\{(\textit{x}_{i},\textit{y}_{i})\}}^{N}_{i=1}$, where $\textit{x}_{i}\in\textit{X}$, $N$ is the size, and $\textit{y}_{i}\in\{0,1\}$. Each training image $\textit{x}\in\mathbb{R}^{w\times h\times 3}$ is taken as input to the encoder for conversion into different low-dimensional feature maps that are projected to the latent vector space to produce the latent vector $\textit{v}\in\mathbb{R}^{\textit{z}}$, where z is the dimension of v.

where $\textit{W}_{e}$ denotes parameters for the encoder. Then, the latent vector z is sent to the decoder and a linear classifier C(), simultaneously as shown in Fig 1. The decoder decodes the latent vector v to reconstruct the input image $\hat{x}\in\mathbb{R}^{w\times h\times 3}$ while the linear classifier estimates the probabilities of v being real or fake.

where $\textit{W}_{d}$ and $\textit{W}_{c}$ are weights of the decoder and classifier, respectively. Note that we only adopt a shallow linear neural network for the classifier because the full-connected layers have a large number of trainable parameters, which are capable of fitting global spatial features rather than relying on the local features generated by convolutional layers. To this end, it is suitable to classify the distributed feature representation encoded in the latent space. Then the loss information is back-propagated to the encoder to support the extraction of effective representations.

During the training phase, we employ the jointly supervised and unsupervised fashion to optimize the two-branch autoencoder. We enforce the cross-entropy loss $\mathcal{L}_{cro}$ on the predicted authenticity label of the images and the Mean Square Error (MSE) loss $\mathcal{L}_{rec}$ on the images reconstructed from the autoencoder:

where $\beta_{1}$ and $\beta_{2}$ weigh the influence of each loss term. The network is end-to-end trainable with $\mathcal{L}$.

The cross-entropy loss is to measure the difference between the predicted label $\hat{\textit{y}}_{i}$ and the ground-truth label $\textit{y}_{i}$ corresponding to input $\textit{x}_{i}$. The classifier network is used to predict the probability of each image $\textit{x}_{i}$: $\sigma(\textit{x}_{i})=\frac{exp(\textit{x}_{i})}{\sum_{\textit{j=1}}^{N}exp(\textit{x}_{j})}$. Let $N$ denote the number of instances. This loss is defined as:

The reconstruction loss measures the Euclidean distance between the original image $\textit{x}_{i}$ and the reconstructed image $\hat{x}_{i}$, which is produced by $\textit{g}_{d}(\textit{g}_{e}(x_{i}))$:

We evaluate our proposed method on three different face manipulation datasets: UADFV [22], FaceForensics++ (FF++) [27], and Celeb-DF (version2) [23]. The UADFV is a small but commonly used dataset, which includes 49 pristine videos and 49 manipulated videos. Celeb-DF is a relatively large face forgery dataset with the forged images generated with advanced Deepfake algorithms to minimize the perceptibility of the artifacts, such as temporal flickering frames and color inconsistency. FF++ dataset contains 1,000 real videos collected from online archives and their corresponding forged versions generated with four different manipulation methods (DeepFake [2], Face2Face, FaceSwap [3], and NeuralTexture [30]). For simplicity, we use light compression (c23) [27] as our training data.

The number of images extracted from each dataset is set to 300 frames per video using the pre-processing introduced in Section 3.1. Table 1 shows the detailed volume of each dataset we used for training in our experiments.

We show some examples of pre-processed images from UADFV, Celeb-DF, and each category of the FF++ dataset in Fig. 2. The listed images show that they are at the similar visual quality level (e.g., brightness, image resolution). However, the fake images in UADFV are easier to be distinguished by human eyes because of the blending boundary discrepancies, unnatural facial expressions, and color inconsistency.

For image pre-processing, we use Dlib [15] as explained in Section 3.1 to detect face regions, crop the faces, and resize the images to $299$ $\times$ $299$. In this work, we adopt Xception [6] as the backbone of the shared encoder. The structure of the decoder and shallow linear classifier are pre-defined. Following each deconvolutional layer are a batch normalization [12] and a rectified linear unit (ReLU) [25] except the last deconvolutional layer. We resize the reconstructed images to 299 $\times$ 299 using interpolation with the bi-linear mode. For the classifier network, we apply ReLU after each fully connected layer excluding the last dense layer. The network is trained with the SGD optimizer with the learning rates of the convolutional autoencoder and the classification network set to 0.005 and 0.0004, respectively. We also add a scheduler on our optimizer with step size 5 and $80\%$ descending rate. The batch size is set to $4$ in our experiments. We empirically set $\beta_{1}=0.8$ and $\beta_{2}=0.2$.

In this section, we compare our proposed framework, JDFD with state-of-the-art face forgery detection methods. We provide intra-dataset and cross-dataset evaluations on UADFV, FF++ and Celeb-DF. AUC (area under receiver operating characteristic curve) is adopted as the metric to evaluate the performance.

From Table 2, we can observe the AUC values by different approaches. Regarding the evaluation result from the model trained on FF++, our method is $1.5\%$ weaker than Xception and $1.7\%$ weaker than Xception+Tri when tested on the FF++ dataset. For the cross-dataset setting, we observe a dramatic performance improvement of our method when tested on UADFV and Celeb-DF (e.g., outperforming Xception by $12.5\%$ on UADFV and Xception+Tri by $16.3\%$ on Celeb-DF). Because the decoder models the distribution of the latent vector corresponding to each input data, the reconstruction loss reinforces the learning of input data’s inherent features of the category it belongs to. To this end, our joint supervised and unsupervised learning enables the encoder to learn robust representations, allowing it to generalize better to unknown forged patterns. Celeb-DF is a challenging Deepfake dataset among these three in terms of quality, quantity, and diversity. For results from the model trained on UADFV, we reach the state-of-the-art performance in the intra-dataset setting. Also, our method outperforms the best-performing method of others by $4.9\%$ on Celeb-DF. However, FWA [22] achieves excellent AUC on the FF++ dataset which is $19\%$ higher than our method. This is similar to [9], which is because their method [22] focuses on specific face discrepancies due to the affine transformations employed by various deepfake generation methods used to generate some benchmarking datasets (e.g., UADFV, FF++). To enhance our method’s generalization power, we do not design it to detect specific manipulations but to learn both local batch patterns and global spatial information. Therefore, it might be slightly weaker than these methods in some specific cases. Given the fact that the deepfake generation algorithms evolve, countermeasures designed for detecting specific manipulations will soon become unfit for purposes in the near future. Celeb-DF is a more challenging dataset for deepfake detection. When trained on Celeb-DF, the result of our method is still competitive when compared to Xception+Tri [9]. With the performance on the cross-dataset setting, our method is 0.9% higher on FF++ and $0.5\%$ weaker on UADFV. As our method considers the learned local features into classification rather than just determining global features, the performance is better on FF++ (containing four different manipulation methods) demonstrating that our method improved the generalizability of the model. In summary, our method generally achieves better results on the cross-dataset evaluation setting (e.g., it is the best performer in 4 of the 6 tests).

The ROC curves of the corresponding AUCs of our method are also shown in Fig. 3. The diagonal line indicates the baseline result from random classification (50% in binary classification). Thus, the curve further away from the diagonal line from above indicates better performance.

In this section, we show two ablation studies: the effect of unsupervised learning, and the effect of data augmentation from other datasets.