IDRetracor: Towards Visual Forensics Against Malicious Face Swapping

For $\{\mathcal{F}_{1},\ldots,\mathcal{F}_{n}\}$ and $I^{s}$, there exist common mappings and features as well as unique ones. Consequently, employing one part of parameters to discern the common mappings and features, while adaptively assigning different parameters for the unique ones, can achieve the effect of fitting the current $\mathcal{F}_{i}$ through parameter recombination.

Inspired by [4], we introduce mapping-aware convolution that contains $n$ groups of convolution kernels, denoted by $C_{m}=\{\delta_{1},\ldots,\delta_{n}\}$. $C_{m}$ can learn the common contents and unique ones by different kernels. During the retracing process, the kernels will be adaptively recombined to different final kernels and they will be employed for different input images. The recombination is guided by a weight vector $\Omega=\{\omega_{1},\ldots,\omega_{n}\}$, which is produced by the mapping resolver. Subsequently, the final kernels for retracing can be written as $\delta_{final}=\sum_{i=1}^{n}\omega_{i}\delta_{i}$, where $i$ denotes the index of the convolution kernel in our mapping-aware convolution. Hence, through recombination, the generated $\delta_{final}$ can tackle both common and unique parts dynamically, thereby fitting to the corresponding $\mathcal{F}_{i}$ and produce $I^{r}\in\mathbb{T}(I^{t})$.

To resolve the potential mappings of the input images, we deploy a mapping resolver, denoted by $\mathcal{R}$. The mapping resolver aims to learn how to resolve different mappings and represent the resolved results through a weight vector $\Omega=\{\omega_{1},\ldots,\omega_{n}\}$, which can be utilized to guide the subsequent mapping-aware convolutions.

Given that different face swapping methods are considered different mappings, $\mathcal{R}$ could be interpreted as a mapping classifier to some extent. Consequently, the mapping resolver can utilize a standard classifier as its backbone architecture, for which we introduce ResNet18 [12]. Then, we employ the Cross-Entropy Loss, a widely adopted constraint in classification tasks, to regulate the resolver. Additionally, inspired by label smoothing [26], we introduce fusion regularization to promote the generation of non-binary weight, which brings three advantages to the IDRetracor: 1) Prevents the IDRetracor from regressing into an ensemble of different single-mapping models. 2) Encourages processing common content with sharing parameters while processing unique content with distinct parameters. 3) Provides the resolver with adaptability and generalization to some anomalous training image pairs, which inevitably occurs during crafting such a large dataset. Finally, the resolving loss to constrain the resolver can be written as:

where $c$ denotes the index of the current face swapping method and $\theta$ denotes the one-hot value after fusion regularization. Additionally, $\theta$ is set to $\frac{1}{n}$ when the input training sample is the source-source sample pair, which allows each group of convolutions to evenly learn the principles for processing real faces.

Firstly, given an input fake face $I^{f}$, the IDRetracor can resolve the mappings of $I^{f}$ and produce a weight vector via mapping resolver: $\Omega=\mathcal{R}(I^{f};\phi_{R})$, where $\mathcal{R}$ is parameterized by $\phi_{R}$. Then, a three-layer UNet [23] is employed as our reconstruction network for the original target face retracing. Notably, the UNet we deployed is a modified version, denoted by $\mathcal{N}$, where all vanilla convolutions are replaced by the proposed mapping-aware convolutions. During retracing, $\mathcal{N}$ is guided by $\Omega$ and recombines its kernels dynamically for the final retraced face $I^{r}=\mathcal{N}(I^{f},\Omega;\phi_{N})$, where $\mathcal{N}$ is parameterized by $\phi_{N}$. To constrain $\mathcal{N}$, we introduce a pixel-wise reconstruction loss $L_{rec}$, defined as follows:

where $||\cdot||_{2}$ denotes the Euclidean distance.

Moreover, since we aim to reconstruct the target face that has an identical ID to the original one, we implement $L_{id}$ to constrain the identity information of the retraced face. $L_{id}$ is calculated based on the cosine similarity:

where $v^{r}$ and $v^{t}$ denote the feature vectors of $I^{r}$ and $I^{t}$ extracted by Arcface recognition model [6]. Despite a complete pixel-wise identicalness (i.e., solely optimizing $L_{rec}$ to zero without introducing $L_{id}$) suggesting a complete identity alignment, introducing $L_{id}$ remains significant. This is due to the fact that $L_{rec}$ cannot be always effectively minimized while relaxing the pixel-wise constraint via $L_{id}$ may lead to superior alignment with high identity similarity.

Overall, the total loss function $L_{total}$ that is used to train the IDRetracor can be written as:

where $\alpha$ and $\gamma$ denote the trade-off parameters for $L_{id}$ and $L_{res}$, respectively.

The most widely adopted public datasets for face swapping are Celeb-DF-v2 [19] and FaceForensics++ [24]. However, these datasets do not align with the demands of our task due to two primary reasons: 1) They offer an inadequate number of target IDs that pair with one specific source ID. This inadequacy severely limits the generalization of a retracing network to arbitrary target IDs. 2) The face swapping methods employed to generate these datasets remain undisclosed, resulting in the dysfunction of $L_{res}$.

Consequently, we craft datasets for different source subjects based on VGGFace2 [2]. The dataset includes four face swapping methods reproduced following their official codes, that is, SimSwap [3], InfoSwap [9], HifiFace [30], and E4S [20]. For each face swapping method, we pair the specific source subject with various target IDs from VggFace2. Given that each ID includes multiple faces with distinct face attributes, we pair each different face and finally obtain 150000 training samples and 10000 testing samples for each source subject and face swapping method. The training set for each face swapping method contains 150,000 target-fake sample pairs and 4,636 target IDs, while the test set contains 10,000 target-fake sample pairs and 1,301 target IDs. Following Fig. 4, the target IDs and specific source faces for generating fake faces are completely unrepeated between the testing and training sets. We introduce 180 source-source sample pairs where 140 for training and 40 for testing. All faces are aligned and cropped to 224$\times$224. Examples of the fake faces in the dataset are presented in Fig. 6. Considering the inconsistent performances of every face swapping method, they are not always capable of generating optimal fake faces for training (as shown in row 5, column 4). Such suboptimal data, which can be perceived as noise, may obstruct the learning process of the network. By introducing label smoothing [26] into IDRetracor, we can to some extent mitigate this problem by preventing the model from overfitting to noisy data.

Although this dataset can be easily reproduced following the details we provided, we are still willing to release the dataset to the public subsequent to the publication of this paper. In this paper, the experimental results on the dataset of Bingbing Fan (Fan) will serve as the representative for comparisons and validations (See Fig. 5(a) for Fan).

Since we introduced the first applicable retracing method, there exists no previous baseline for comparison. To further investigate the retracing task, we trained two more models as described below. 1) VU-S: Vanilla UNet [23] trained with samples generated by a Single face swapping method. In our experiments, we manually deploy the model corresponding to the face swapping method that generates the input fake images. 2) VU-M: Vanilla UNet trained with samples generated by Multiple face swapping methods. UNet backbone is selected to experimentally verify the feasibility of the retracing task. It can be optionally replaced by other image-to-image translation backbones. All methods are trained in an end-to-end manner with 150 epochs and batch size 32. We adopt Adam [17] optimizer with $\beta_{1}=0$ and $\beta_{2}=0.99$ and the learning rate is set to 0.003. The hyper-parameters for IDRetracor are set as $\theta=0.9$, $n=4$, $\alpha=0.01$, and $\gamma=10$. All experiments are implemented in PyTorch on one NVIDIA Tesla A100 GPU.

The quantitative evaluations are performed in terms of two metrics: ID similarity and ID retrieval. For ID similarity, we employ two face recognition models (i.e., Arcface [6] and Cosface [28]) to extract the feature vectors of identity, denoted by ID Sim. (arc/cos). ID retrieval is employed to measure the model capability of retracing the potential perpetrators. To compute ID retrieval, we first randomly select 1000 target faces with different IDs for each face swapping method from the testing set and extract their feature vectors via the Arcface model. Then, we retrace the fake faces that are paired with the selected target faces. Finally, the ID retrieval is measured as the top-1 and top-5 (ID Ret. (t1/t5)) matching rates (%) of the retraced faces and their corresponding target faces.

As shown in Tab. III, we present the ID Sim. (arc/cos) and ID Ret. (t1/t5) between the retraced faces and the original target faces. Cross-FS denotes the ability to cross multiple face swapping methods. Notably, the proposed IDRetracor outperforms VU-M by over 20% and is even higher than VU-S, which is not Cross-FS. Such performances reveal the superiority of the proposed mapping-aware convolutions, that is, they can adapt the mappings of current input dynamically and retrace within the precise solution space of the original target face.

For a better illustration, we visualize the distribution of Arcface similarity of models that are Cross-FS on the testing set (see Fig. 8). The distributions demonstrate that the retraced faces can be used as reliable evidence for visual forensics and target face attribution.

As shown in Fig. 7, we present the retraced faces of VU-S, VU-M, and IDRetracor against four state-of-the-art face swapping methods [3, 9, 30, 20]. Considering that SimSwap modifies minor content of the target face during face swapping, the process of retracing becomes considerably easier. Consequently, all three models exhibit strong performance when retracing fake faces produced by SimSwap, while IDRetracor demonstrates a heightened capability of detail restoration. For HifiFace and InfoSwap, which erase more target ID information from the original target faces, IDRetracor can produce retraced faces that are more consistent with the original ID of the target faces. As for the hardest task E4S, the target feature residuals including skin color and facial expression are excessively removed from their original target faces. Despite the immense challenge of fully reconstructing details (e.g., makeup and wrinkles), IDRetracor can still produce retraced faces with the highest identity similarity with the original target faces.

To validate the effectiveness of intensity loss ($L_{id}$), we present the retraced results with different $\alpha$ in Fig. 9. Without $L_{id}$, the retraced results may contain some erroneous eye details (see the third row), while other facial features appear overly smooth (see the first and second rows). This could be due to the challenging conditions in the solution space, where solely constrained by $L_{rec}$ causes the model to lean towards creating a generic face, rather than a result more closely aligned with the original target face ID. In contrast, with the constraint of $L_{id}$, the model tends to supplement details that contribute to enhancing identity similarity. We also conduct additional experiments when $\alpha$ is large (i.e., $\alpha=0.1$) and solely using $L_{id}$. The results demonstrate that $L_{rec}$ primarily functions to preserve the contour and color of the retraced results, while $L_{id}$ acts as an auxiliary constraint to enhance the details.

Here, we analyze and validate the effectiveness of fusion regularization through experiments. As shown in Fig. 10, we measure the average Arcface similarities under different theta values on the testing set. When $\theta\in[0.0,0.4]$, the model degrades to a manner of mixed training like VU-M, where the model has a poor resolving ability for shared content and unique content. The worst performance occurs when $\theta=0.3$, as the generated weight vector is almost evenly distributed. When $\theta=1.0$, that is, when fusion regularization is not deployed, the model degrades to a framework that first identifies the face swapping method, and then processes the fake faces with the corresponding VU-S. Therefore, we set $\theta=0.9$ where the IDRetracor exhibits the best performance.

In Tab. II, we present the performances of IDRetracor on the testing set of seen target IDs (Seen-TI), seen source faces (Seen-SF), and unseen data. See Fig. 4 for the visualization and illustration of seen and unseen data. Notice that Seen-SF refers to the specific source faces that are applied to generate fake faces in the training set, and Unseen is the protocol we followed in all other experiments. The results show that there is minor variation in performance on seen and unseen data, which suggests that the abundant number of samples in the dataset has addressed the distinctions among Seen-TI, Seen-SF, and Unseen, thus achieving generalization.

In Fig. 11, we substantiate the efficacy of incorporating source-source sample pairs into our model. For testing, we exclusively use source faces that are not seen during training. Qualitatively, the real-face identities are well-preserved in the output faces. We also show the Arcface similarity as the quantitative metric and the results are correspondingly promising. Therefore, the proposed IDRetracor is logically comprehensive for input reconstruction.

To further assess the reliability of visual forensics, we design a semantic consistency verification experiment. Specifically, we randomly selected 1,000 fake faces from the test set and obtained their retraced faces. Subsequently, we perform face swapping between these retraced faces and their corresponding source faces used in the original fake face generation, yielding a set of new fake faces. If this set of new fake faces is consistent with the original fake faces, we can validate the semantic consistency of the retraced faces. As shown in Fig. 12, the experimental results verify the semantic consistency of the retraced faces.

In Sec. IV-A, we mentioned that the two public datasets have too few target IDs, namely, a maximum of 26 target IDs for Celeb-DF-v2 [19] and a mere single one for FaceForensics++ [24]. Given that the target ID in FaceForensics++ is obviously insufficient for training a retracing network, we skip FaceForensics++ and conduct training experiments on Celeb-DF-v2 (see Fig. 13 ) following the same setting as on the crafted dataset. The poor experimental results show that we cannot train a retracing network on these public datasets, thus demonstrating the necessity of crafting our target-fake dataset. Moreover, it is also impractical to craft datasets applying the manipulation methods used by Celeb-DF-v2 or FaceForensics++. That is because they are traditional one-to-one methods that require training a new model for each target ID, while we need a massive number of target IDs in the dataset for retracing.

Considering cross-dataset evaluation is important for real-world applications, we conduct the cross-dataset evaluation on FF-HQ [16] and CelebA-HQ [15] (trained on VGGFace2). The cross-dataset fake images are generated using target faces from FF-HQ and CelebA-HQ and exactly following the protocol that we apply for VGGFace2. Fig. 15 shows that our method can maintain promising performance in the cross-dataset evaluation both qualitative and quantitative, which further demonstrates the application potential of IDRetracor.

Here, we demonstrate that the framework of IDRetracor can be applied to other source IDs effectively. We craft datasets and train the corresponding IDRetracors for two other celebrities (i.e., Ariel Lin and Cristiano Ronaldo from VGGFace2). As shown in Fig. 16, the IDRetracor can also exhibit strong retracing performance for these two source IDs, where the identity and detailed facial characters are promisingly reconstructed. In Tab. III, we also provide the quantitative metrics on these identities. Due to the varying performances of face swapping methods when dealing with different source faces, there are minor differences in the retracing results on quantitative metrics. Nonetheless, IDRetracor can still maintain effective retracing performances, indicating that our framework can be applied to any source ID.