FaceCat: Enhancing Face Recognition Security with a Unified Diffusion Model

Pioneers attempt to exploit GANs (2016) and VAEs (2014) for representation learning. (2019) demonstrates GANs can learn a competitive representation for images in latent space. Over the recent biennium, diffusion models (2020; 2021) have garnered substantial achievements in the realm of generative learning. Therefore, recent works attempt to utilize diffusion models for representation learning to serve downstream tasks, such as image editing (2022), semantic segmentation (2021) etc. Despite achieving commendable performance, the majority of these studies lack investigation into integrating face security tasks and the robustness of diffusion models.

FAS (2022; 2023; 2023) and FAD (2021; 2018; 2022; 2021) have been developed to protect face recognition systems. In recent years, research in FAS has increasingly gravitated towards the exploration of 3D and multi-modal approaches (2020). FAD exhibits a preference for the development of attack-agnostic, universally applicable defense methods (2021; 2021). Moreover, some research (2023b; 2022) attempts to unify several face security tasks into a framework, aiming to enhance both practicality and generalization. UniFAD (2023b) attempts to unify digital attacks and FAS through k-means clustering. However, due to the inherent drawback of classification models, such methods primarily focus on the structural features of images. As a comparison, FaceCat has global structural and deep-level detailed features by virtue of its use of generative models.

For FAS and FAD tasks, we can define three input types: spoofing face $\bm{x}^{spoof}$, adversarial face $\bm{x}^{adv}$ and real face $\bm{x}^{real}$. As for a unified detector, it is only necessary to reject potential attack types without identifying whether $\bm{x}^{spoof}$ or $\bm{x}^{adv}$. Thus, we define $\bm{x}^{fake}\in\left\{\bm{x}^{spoof},\bm{x}^{adv}\right\}$. The decision function can be defined as:

$\displaystyle f_{\bm{\theta}}:\bm{x}\rightarrow\{0\text{ ($\bm{x}^{fake}$)},1\text{ ($\bm{x}^{real}$)}\},$

which classifies whether a sample $\bm{x}$ is fake or real. $f_{\bm{\theta}}(\bm{x}):\mathbb{R}^{d}\rightarrow\mathbb{R}$ is the unified detector parameterized by $\bm{\theta}$. Generally, the problem is formulated as a dichotomous classification as follows:

	$$\displaystyle\mathcal{L}_{c}(\bm{x},y)=\mathbb{E}_{p(\bm{x})}\left[-(y\log(f_{\bm{\theta}}(\bm{x}))+(1-y)\log(1-f_{\bm{\theta}}(\bm{x})))\right],$$
	$$\displaystyle\text{where }y\in\left\{0,1\right\},\bm{x}\in\left\{\bm{x}^{fake},\bm{x}^{real}\right\},$$		(1)

where $y$ is the ground truth, $\bm{x}\in\mathbb{R}^{d}$ is the input. In this paper, our method also considers unifying FAS and FAD as a dichotomous classification problem.

Traditional classification models may overly prioritize image structure at the expense of fine details, whereas FDM inherently contains rich global structural and deep-level detail features. Therefore, we aim to leverage it as a strong initialization for FAS and FAD. A brief overview of FDM reveals that its training process is structured in two stages: 1) The forward diffusion process gradually adds Gaussian noise to the data to obtain a sequence of noisy samples; 2) The backward generative process reverses the diffusion process to denoise images. The central objective is to predict the noise at time $t$, which is formulated as a simple mean squared error loss:

$$\begin{split}\mathcal{L}_{\text{simple}}=\mathbb{E}_{\bm{x}_{0},t,\bm{\epsilon}}[\|\epsilon_{\bm{\psi}}(\bm{x}_{t},t)-\bm{\epsilon}\|_{2}^{2}]\end{split},$$

(2)

where $\epsilon_{\bm{\psi}}$ is the diffusion models and $\bm{x}_{t}$ is the noisy image at time $t$ (noise level), i.e., $\bm{x}_{t}$ is obtained by adding noise $\epsilon$ to $\bm{x}$.

In this paper, we focus on using the Face diffusion model as a Catalyst (FaceCat) for enhancing FAS and FAD. Formally, given a FDM $\epsilon_{\bm{\psi}}$, Note that we only obtain the features from the middle layer through $\epsilon_{\bm{\psi}}$, not the predicted noise, i.e., the input $\bm{x}_{t}$ is processed through $\epsilon_{\bm{\psi}}$ to obtain a universal face feature with dimension $d^{\prime}$. To leverage this feature, a lightweight head $\mathcal{H}:\mathcal{R}^{d^{\prime}}\rightarrow\mathcal{R}$ is adopt. Therefore, the objective function of FaceCat can be formulated as:

$$\displaystyle\underset{\bm{\kappa}}{\min}~{}\mathbb{E}_{p(\bm{x}_{t})}\left[\mathcal{L}_{c}(\mathcal{H}_{\bm{\kappa}}(\epsilon_{\bm{\psi}}(\bm{x}_{t},t)),y)\right],$$

(3)

where $\mathcal{H}$ is parameterized by $\bm{\kappa}$ and $p(\bm{x}_{t})$ is the probability distribution of $\bm{x}_{t}$. Note that Equation 3 is only adopted to optimize the lightweight head $\mathcal{H}_{\bm{\kappa}}$. The rationale behind this is twofold: 1) The features of FDM are sufficiently general-purpose; 2) By not fine-tuning FDM, we reduce the training time. By following the objective (3), given a face image $\bm{x}$, we can get the probability of whether it is a real face.

As illustrated in Figure 3, different blocks of the diffusion model capture distinct facial features, with some layers representing detailed features and others abstract features. For facial security tasks, both types of features are beneficial to model performance. Thus, we propose leveraging a hierarchical fusion mechanism to integrate these diverse features:

$$f_{total}=\sum_{i=1}^{n}\mathcal{F}_{i}(f_{i}),$$

(4)

where $f_{i}$ denotes the input features to the $i^{th}$ block, and $\mathcal{F}_{i}(f_{i})$ represents the output features after processing by the $i^{th}$ block. $f_{total}$ is the features after fusion. For the lightweight head, aiming for minimal weight while ensuring performance, we utilize the first four layers of ResNet-18, complemented by an additional two downsampling layers.

Although some related works (2018) have also introduced the concept of multi-level features, their application has largely been empirical within traditional classification models. Given the unique architecture of diffusion models, which differs significantly from conventional classification models, previous experiences cannot be directly applied. Consequently, our investigation into the fusion of different blocks is detailed in Table 2. Observations indicate that optimal performance is achieved when blocks 5, 6, 7, 8, 12 are selected. Consequently, this configuration has been adopted for subsequent experiments.

The performance of FAS and FAD often suffers due to the constraints associated with the exclusive use of single-modal (RGB) data. Drawing inspiration from the ability of textual data to impart more detailed information in computer vision tasks, we try to design a text-guided multi-modal alignment strategy so that text information can be used in face security models.

Our text-guided multi-modal alignment is based on CLIP (2021). Given a batch of image-text pairs, we can predict the image’s class by computing a text-image cosine similarity. Formally, given an image $\bm{x}$, we can compute the prediction probability as follows:

$$p(\hat{y}|\bm{x})=\frac{\text{exp}(\text{cos}(\bm{\omega}_{i},\bm{e})/\tau)}{\sum_{j=1}^{K}\text{exp}(\text{cos}(\bm{\omega}_{i},\bm{e})/\tau)},$$

(5)

where $\tau$ is a temperature parameter and $\hat{y}$ is the predicted class label. $\bm{e}$ is the image embedding of $\bm{x}$ extracted by the image encoder, and {$\bm{\omega}\}_{i=1}^{K}$ represent the weight vectors from the text encoder. $K$ and cos($\cdot,\cdot$) denote the number of classes and cosine similarity, respectively.

Specifically, we craft some text prompts for this task along the lines of “a photo of a [class]”. The “[class]” is “[real face]” or “[fake face]”, all text prompts are presented in Table 3. This can be formalized as $\bm{t}=[V]_{1}[V]_{2}\ldots[V]_{M}[\text{real/fake face}]$, where each $[V]_{m}(m\in\{1,\ldots,M\})$ is a vector with the same dimension as word embeddings (i.e., 512 for the text encoder), and $M$ is a hyperparameter specifying the number of context tokens. Given the text encoder $g(\cdot)$, we can obtain $\bm{\omega}=g(\bm{t})$.

As for $\bm{e}$, we can acquire through the lightweight head $\mathcal{H}$. Assume that the feature layer of $\mathcal{H}$ is $\mathcal{H}^{f}:\mathcal{R}^{d^{\prime}}\rightarrow\mathcal{R}^{d^{\prime\prime}}$, where $d^{\prime\prime}=512$ which matches the dimension of word embeddings. Hence, Equation 5 can be re-expressed as:

$$p(\hat{y}|\bm{x}_{t})=\frac{\text{exp}(\text{cos}(g(\bm{t}),\mathcal{H}^{f}({\epsilon_{\bm{\psi}}}(\bm{x}_{t})))/\tau)}{\sum_{j=1}^{K}\text{exp}(\text{cos}(g(\bm{t}),\mathcal{H}^{f}({\epsilon_{\bm{\psi}}}(\bm{x}_{t})))/\tau)},$$

(6)

where $K=2$ and $\bm{x}_{t}$ is the noisy image. Additionally, we employ the more flexible focal loss (2017) as the optimization function instead of binary cross-entropy loss. Formally, given that $p$ denotes the probability in $p(\hat{y}|\bm{x}_{t})$ that the identified image is a real face, we adopt the notation $p_{c}$ to represent the probability of the target class:

$$p_{c}=\begin{cases}p&\text{if }y=1,\\ 1-p&\text{otherwise.}\end{cases}$$

(7)

Referring to the standard $\alpha$-balanced focal loss (2017), the text-guided multi-modal alignment can be formulated as follows:

$$\text{TG}(p_{c},y)=-\alpha_{c}(1-p_{c})^{\gamma}\log(p_{c}),$$

(8)

the parameter $\alpha_{c}$ is introduced to balance the data distribution, and its formal definition mirrors that of Equation 7, with $p$ replaced by $\alpha$. Since negative samples are approximately twice as many as positive samples, $\alpha$ is set to 0.75. To encourage $\mathcal{H}^{f}$ to focus on hard samples during training, we set the focusing parameter $\gamma$ to 2.0.

Face data from different classes exhibits similar features, which is different from general classification tasks. This characteristic makes it difficult to distinguish between positive and negative sample features. Therefore, we adopt a triplet loss (2017) to supervise features, which is commonly utilized in deep metric learning, and has proven its effectiveness in optimizing the relative distances between samples in the embedded space. The mathematical form is expressed as follows:

$$\begin{split}\mathcal{L}_{\text{triplet}}(\bm{x}^{a},\bm{x}^{p},\bm{x}^{n})&=\max\left(0,\left\|\mathcal{H}^{f}({\epsilon_{\bm{\psi}}}(\bm{x}^{a}))-\mathcal{H}^{f}({\epsilon_{\bm{\psi}}}(\bm{x}^{p}))\right\|_{2}^{2}\right.\\ &\left.-\left\|\mathcal{H}^{f}({\epsilon_{\bm{\psi}}}(\bm{x}^{a}))-\mathcal{H}^{f}({\epsilon_{\bm{\psi}}}(\bm{x}^{n}))\right\|_{2}^{2}+m\right),\end{split}$$

where $\bm{x}^{p}$ and $\bm{x}^{n}$ are the positive sample ($\bm{x}^{real}$) and negative samples ($\bm{x}^{fake}$) respectively. $\bm{x}^{a}$ is an anchor sample, which is the same class as $\bm{x}^{real}$. $\left\|\cdot\right\|_{2}^{2}$ represents the squared Euclidean distance. The margin $m$ is a hyperparameter ensuring a safe gap between positive and negative pairs, preventing trivial solutions. Therefore, Equation 3 can be rewritten as:

$$\displaystyle\underset{\bm{\kappa}^{\prime}}{\min}~{}\mathbb{E}_{p(\bm{x}_{t})}\left[\text{TG}(p_{c},y)+\lambda\cdot\mathcal{L}_{triplet}(\bm{x}_{t}^{a},\bm{x}_{t}^{real},\bm{x}_{t}^{fake})\right],$$

(9)

where $\bm{\kappa}^{\prime}$ denotes the parameters of $\mathcal{H}^{f}$. $\lambda$ is the balancing factor. $\bm{x}_{t}^{a}$,$\bm{x}_{t}^{real}$ and $\bm{x}_{t}^{fake}$ represent the forms of $\bm{x}^{a},\bm{x}^{real}$ and $\bm{x}^{fake}$ with noises. Note that we also do not optimize the parameters of $g(\cdot)$. Following Equation 9, we can acquire a classifier that performs FAS and FAD simultaneously.

Evaluation metrics. We evaluate with the following metrics: Attack Presentation Classification Error Rate (APCER), Bona Fide Presentation Classification Error Rate (BPCER), and the average of APCER and BPCER, Average Classification Error Rate (ACER) for a fair comparison. Equal Error Rate (EER), Area Under Curve (AUC), and True Detection Rate (TDR) at a False Detection Rate (FDR) 0.2% (TDR @ 0.2% FDR). In addition, a visualization (2008) is also reported to evaluate the performance further. The specific training details can be found in Appendix C.

Effectiveness of the proposed method. To verify the efficacy of the proposed approach, we compare FaceCat with three baselines: 1) FAS: Flip (2023), CFPL (2024), DeepPixBis (2019), Depthnet (2018), FRT-PAD (2022); 2) FAD: EST (2022), FDS (2018), DFRAA (2021); 3) classification models: Resnet50 (2016), Inceptionv3 (2017), Efficientnetb0 (2019), ViT-b/16 (2020). These methods are either commonly used or represent the SOTA techniques. To fairly compare the proposed method, all these methods have been fine-tuned according to the proposed protocol. Table 4 shows the performance metrics for each method. Inspection of the table allows us to deduce the subsequent conclusions:

(1) FaceCat effectively improves the face recognition systems’ security. When conducting FAS and FAD simultaneously, all metrics achieve optimal performance, with ACER and EER being 1.24% and 1.68%, respectively. This can be attributed to: 1) the powerful representational capabilities of the diffusion model, which provide rich prior knowledge for FAS and FAD; 2) the proposed hierarchical fusion mechanism that effectively leverages the features of the diffusion model; 3) the text-guided multi-modal alignment technique that further enhances the performance of FaceCat by utilizing rich textual features.

(2) The proposed method with the text-guided multi-modal alignment strategy generally outperforms the variant without it across various metrics. This demonstrates the strategy can effectively improve the performance of the proposed method. We think it benefits from low-cost and effective text embedding, which facilitates face features learning under the nuanced supervision of multi-modal cues.

Evaluation on input transformation. To evaluate the robustness of FaceCat, common input transformations (are excluded from data augmentation), e.g., bit-depth reduction, Gaussian blur, and JPEG compression are employed to reduce the image quality. We compare our results with a widely recognized method, CDCN (2020). In Figure 5, when image quality degrades, the performance of CDCN in addressing diverse types of attacks diminishes, with its response to JPEG compression being the most notably affected. This indicates although the baseline can demonstrate decent performance under the proposed protocol, its robustness significantly decreases with the degradation of image quality. In contrast, FaceCat continues to maintain stable performance even in the face of degraded image quality, reaching up to 55.3% better than the baseline under JPEG compression. This can be attributed to the superior feature robustness of FaceCat, i.e., even with the loss of certain image information, the proposed method is still sufficient to achieve robust metrics in detecting such attack threats.

Advantages of FDM architecture. We conduct an ablation study to verify the effectiveness of FDM architecture, as shown in Table 6. Both MAE (2022) and SwAV (2020) represent SOTA self-supervised learning methodologies that have undergone pre-training on the FFHQ dataset prior to fine-tuning on our proposed protocol, ensuring consistency in model initialization. The FR-ArcFace (2019) is the face recognition model ArcFace, which utilizes a ResNet18 (2016) and is pre-trained for face recognition tasks. From the table, two key conclusions can be drawn: Firstly, even the employment of pre-trained face models can enhance the model’s capabilities in conducting FAS and FAD; however, FDM, with its dual focus on global structural information and intricate detail features, demonstrates superior performance over these pre-trained face models. Secondly, utilizing face recognition models as pre-trained systems yields inferior results compared to models pre-trained via self-supervised methods. This is attributed to the tendency of classification models to overemphasize structural features, resulting in a partial loss of feature information.

Value of $\bm{\alpha}$. In Table 6, we exploit the influence of different $\alpha$ of TG on the experimental results. $\alpha=0.75$ performs better than other settings. In joint defense scenarios, the quantity and diversity of real face samples are typically inferior to those of spoofing faces. This leads to an imbalanced focus of models on fake face samples. Hence, adjusting hyper-parameter $\alpha$ to mitigate sample imbalance significantly improves the model’s overall performance.

Feature visualization. The feature distribution in the test set on the proposed protocol is visualized in Figure 6 via t-SNE (2008), which consists of 6 types (live, PGD, makeup, silicone mask, facemask, transparent mask). Through the figure, it is evident from (b) that, in the absence of TG, facemask and PGD are not distinctly separable. Moreover, the live sample distribution from (a) is compact and the clusters exhibit improved separation for the proposed method trained with TG. These results illustrate the effectiveness of the text-guided strategy in FAS and FAD tasks. The main reason is that TG can leverage text embeddings to enhance the aggregation and distinction of image features.

The ablation study on the timesteps of diffusion and the real-world experiments are detailed in Appendix D and F.