Identifying Rhythmic Patterns for Face Forgery Detection and Categorization

Face forgery has received considerable attention over the past decade. In the earlier studies, graphics-based methods were used to implement this technology [4, 9]. Although graphics-based methods have been evolving, they have not yet become widespread due to their complexity and huge cost. With the publication of GANs by Goodfellow et al. [14] in 2014, it has made impressive progress and attracted a lot of research. Choi et al. [5] proposed StarGAN which performs multi-domain image conversion tasks in one dataset. To address this dilemma that GAN is hard and slow to train, Karras et al. [16] proposed ProGAN which gradually generates images from low quality to high quality. Overall, These GAN-based methods can easily use some random noise to create indistinguishable images, which makes face forgery easier and wildly available to the public.

Due to the incredible development of deep learning and GAN, face forgery technology has been progressing rapidly. To avoid its illegal use, researchers are exploring how to identify it. In earlier studies, researchers used hand-crafted features [24, 13] to detect. However, these methods don’t perform good generalization ability. In recent years, some methods with automatic learning ability based on deep learning have been proposed. Ding et al. [12] proposed a transfer learning model to detect swapped faces. Liu et al.  [21] proposed an architecture that can use textures in combination with the Gram Block module. Li et al. [17] assumed that the face is a blend of two different images, and transformed the task to detect its stitching boundary for better generalization. Although these methods achieve a high accuracy rate, they do not take advantage of deeper features and may fail in the face of new generations of forgery schemes.

With the proposal of remote photoplethysmography [31] (rPPG), a number of heart rate measurement methods based on it have been developed [22, 32, 23]. It is precisely because of the uniqueness of rPPG signal that researchers gradually apply it to the field of face anti-spoofing [18].

At the same time, face forgery detection methods based on rPPG have also been proposed. Ciftci et al. [7] analyzed the biological signals of fake and real video pairs, constructed a generalized classifier, and convert the signals into PPG maps to train a simple detection network. Qi et al. [25] proposed the motion-magnified spatial-temporal representation and dual-spatial-temporal attention network to adapt to changing faces and various fake types. However, these methods do not make full use of the PPG signal. To further exploit the advantages of the PPG signal, our method takes into account the noise in the PPG signals as well as the interaction and constraints of adjacent PPG signal.

Since the PPG signal does not simply rely on the texture information of the images, it cannot be represented directly using the original unprocessed images. Moreover, the PPG signal is very weak and susceptible to interference. To enable an accurate and comprehensive representation of the PPG signal, Inspired by [23], we generate a spatial-temporal map to represent the PPG signal, as shown in Figure 2.

It is observed that most existing face forgery methods share a common step: different original face images are transformed and then blended to generate forgery faces images. As mentioned in Section 2.3, each face video contains PPG signals corresponding to the unique heartbeat pattern in the human body. However, the loss of PPG signals is inevitable during the face forgery generation process. Nonetheless, these forgery face videos still exist the mixed PPG signals, which are the mixture of different residual PPG signals that may no longer correspond to human heart rate information but can reflect the unique rhythmic pattern of the face forgery method. Naturally, we propose a simple data blending to imitate the face forgery method for data augmentation.

Unlike forgery methods that process each image separately, since the spatial-temporal map is a combination of PPG signals from different sub-ROIs of the faces in the video clips. we take into account it and provide a new approach that is straightforward yet more effective.

Specifically, as shown in Figure 3, given two spatial-temporal maps $M_{1}$ and $M_{2}$, we generate a new spatial-temporal map $M_{new}$ by blending $M_{1}$ and $M_{2}$:

$$M_{new}=M_{1}\oplus M_{2}=\alpha\cdot M_{1}+\left(1-\alpha\right)\cdot M_{2},$$

(1)

where $\alpha$ denotes the blending coefficient, which is uniformly distributed between $[0,1]$. Moreover, since our detection is a classification task, the data blending can be divided into intra-source and inter-source blending, i.e., whether the two original spatial-temporal maps belong to the same source of face forgery or not. Correspondingly, the loss function of the framework will be different, as detailed in Section 3.4. Intuitively, intra-source blending expands the data diversity within the source, while inter-source blending can be considered as expanding the variety of sources even further.

In previous research of face forgery detection based on rPPG principle, the network architectures of some methods are too simple [7] to make comprehensive use of PPG signal, while some are too complex [25], and require considerable time. Moreover, since the input of the network is a spatial-temporal map rather than an ordinary two-dimensional image, the current mainstream network architecture may not be particularly suitable. Therefore, we design a framework containing a Spatial-Temporal Filtering Network (STFNet) for PPG signals filtering and a Spatial-Temporal Interaction Network (STINet) for constraint and interaction of PPG signals.

Overall, we summarize the loss function of our framework. The final loss function can be written as:

$$L=\alpha\cdot L_{ce}\left(t_{1}\right)+\left(1-\alpha\right)\cdot L_{ce}\left(t_{2}\right)+\beta\cdot L_{\rho},$$

(4)

where $t_{1}$ and $t_{2}$ denote the two classes of sources, respectively, $\beta$ is the weight for balancing the loss. And for intra-source data blending, $t_{1}$ and $t_{2}$ belong to the same source and the loss function can be simplified to:

$$L=L_{ce}\left(t\right)+\beta\cdot L_{\rho},$$

(5)

where $t$ denotes the class of source.

To verify that the presence and unique rhythmic pattern of the PPG signal in each forgery method, in other words, to verify that the PPG signal of each face forgery source does have its unique rhythmic pattern, rather than the PPG signals of all face forgery sources sharing the same pattern, we conduct pairwise detection experiments (real-fake and fake-fake) and face forgery categorization experiment (1 real and 5 fakes) in the six sub-datasets of FaceForensics++, as shown in Table 2 and Figure 5, respectively. The results show that we achieve high accuracy in both pairwise detection and face forgery categorization.

To demonstrate the superiority of our proposed method, we compare it with other methods. We select the mainstream network, i.e., VGG [27], ResNet [15], and some methods that have achieved good performance in FaceForensics++’s benchmark, i.e., Inception_V3  [28], Xception [6], and methods that are also based on the rPPG principle, i.e., Ciftciet al. [8], FakeCatcher [7], DeepRhythm [25].

It is worth noting that, except for the method based on the rPPG principle, the rest of the methods detect forgery videos by single images rather than video clips, so we adapt the training strategy for these methods to better compare. Specifically, we directly feed the face region detected by Dlib with appropriate scaling into the network. To speed up the training, we utilize every $64^{th}$ frame to train. This can be considered as a baseline for frame-based detection. For a fair and comprehensive comparison, in the face forgery detection and face forgery categorization experiments, we follow DeepRhythm and Ciftci et al. [8] to divide the dataset in the ratio of 8:1:1 and 7:3, respectively. All methods are experimented in six sub-datasets of FaceForensics++.

Overall, in Table 1, we can see that our method outperforms state-of-the-art methods in both face forgery detection and categorization, which fully demonstrates the effectiveness of our method. Baseline methods that do not utilize PPG signals can achieve high accuracy in some datasets. However, when facing NerualTextures-based rendering videos, these methods that only consider single-frame spatial information may fail and the accuracy will drop sharply. This proves that PPG signals have deeper information compared to 2D images, and PPG signals with spatial-temporal information can represent the more unique difference between real and fake videos. Compared with methods also based on the rPPG principle, our method achieves the same state-of-the-art accuracy as DeepRhythm in face forgery detection experiments (DeepRhythm only tests three sub-datasets). However, the model size of our method is much smaller than DeepRhythm (see Figure 6 (a)). Moreover, it can be seen that our method gets significant improvement in the accuracy of face forgery categorization compared with Ciftci et al. [8]. These comparisons demonstrate that our method performs a stronger ability to exploit the PPG signal.

To demonstrate the effectiveness of our method, based on face forgery categorization experiments, we perform several ablations to better understand the contributions of each component in face forgery categorization experiments, as shown in Table 3.

Effectiveness of PPG Signal. We feed face images and spatial-temporal maps containing PPG signals into the STFNet, respectively. the average detection accuracy of utilizing PPG signals is 1.79% higher than not. It is worth emphasizing that these results demonstrate that PPG signals from different face forgery sources have their unique rhythmic patterns, and the network can achieve high detection accuracy by learning their unique rhythmic patterns.

Effectiveness of STFNet. We utilize STFNet and VGG without classification layers as the backbone lay of the network for experiments, respectively. The results show that STFNet outperforms VGG in all aspects and improves the average accuracy by 1.15%, proving the effectiveness of the STFNet.

Effectiveness of Data Augmentation. We further evaluate the effectiveness of our intra-source and inter-source data blending. The results show that both data blending improve the accuracy. Moreover, inter-source blending is more effective than intra-source blending, with an improvement of 0.46%, and an improvement of 1.03% compared to no blending, which proves that our data augmentation can achieve amazing improvement by cleverly blends spatial-temporal maps without any additional computation and processing.

Effectiveness of STINet. Finally, we verify the effectiveness of STINet. We first add adjacency constraint and Bi-LSTM separately, with 0.12% reduction and 0.05% improvement in accuracy, respectively, while the accuracy increases by 0.11% after combining these two. After analysis, we speculate that although adding adjacency constraint alone has a certain constraint effect on the network, it does not play an optimization role because there is no information interaction between adjacent video clips. Therefore, combining the two can play a mutually reinforcing role.

To verify the generalization performance of our method, we conduct face forgery categorization experiments by adding the forgery sets of DFD, Celeb, and both of them, respectively. To save compute, only 300 videos are selected for each dataset including the six sub-datasets of FaceForensics++. The results are shown in Table 4, whether the two datasets are added separately or simultaneously, high accuracy is achieved, which proves that our method is also effective when other face forgery sources are expanded.

To test the performance and improvement of prediction aggregation with different lengths of video clips. We choose $T=\{32,64,128,256\}$ frames to conduct experiments.

The results are shown in Figure 6 (b), (c), and (d). We can see that the accuracy of detection of video clips with $T=\{64,128,256\}$ is very close, 95.28%, 95.22%, and 95.28%, respectively, while it decreases sharply at $T=\{32\}$, only 87.55%, which corresponds to the theoretical fact of rPPG principle: the length of video clips too short may not contain the complete heart rate cycle and thus fail to extract PPG signal accurately, while too long may include additional noise. This once again demonstrates that our method has an extremely strong theoretical basis. Moreover, prediction aggregation has a certain boost on the accuracy to different degrees.

To demonstrate the robustness of our proposed method for compressed videos, we conduct experiments on the HQ version (a light compression) and LQ version(a heavy compression) of FaceForensics++ dataset. As shown in Table 5, our method still achieves the average accuracy of 92.34% for face forgery categorization in HQ version, which proves the robustness of our method even in light compression video. In contrast, in LQ version, due to severe compression, the PPG signal usually suffers from noisy curve shapes and inaccurate peak locations due to information loss caused by intra-frame and inter-frame coding of the video compression process [32], which leads our method to obtain only an average accuracy of 55.37%.