Robust face anti-spoofing framework with Convolutional Vision Transformer

Depending on the feature extraction method used in neural networks, the backbones for FAS can be classified as CNN, SA-based ViT, and hybrid ViT with convolutional elements (Han et al., 2022).

CNN-based models can extract local features from images and are trained efficiently based on strong assumptions, such as spatial equivariance. For example, ResNet (He et al., 2016; Shao et al., 2019; Jia et al., 2020) and EfficientNet (Tan & Le, 2019; Kong et al., 2022) have been widely used to obtain local cues from images in FAS. However, the information from CNN is limited to the range of the receptive field, and it is difficult to aggregate information from other spatial locations (Raghu et al., 2021). The performance ceiling is unavoidable if the assumption of locality and invariance does not fit the task because the CNN is operated based on complex assumptions (d’Ascoli et al., 2021).

To overcome the limitations of CNNs, the ViT-based model designs the SA mechanism with a soft assumption. ViT divides an image into non-overlapped patches and transforms them into a linearly embedded sequence. Global dependencies between patches are encoded via SA (Dosovitskiy et al., 2020; Liu et al., 2021b). However, it is difficult to solve the FAS task using ViTs because a large dataset is necessary for pretraining ViT owing to a lack of prior knowledge (Raghu et al., 2021; d’Ascoli et al., 2021; Tu et al., 2022).

Therefore, hybrid models have recently been proposed to reflect the advantages of CNNs in ViT (Dosovitskiy et al., 2020; d’Ascoli et al., 2021; Tu et al., 2022). For example, ViT Hybrid uses a feature map extracted through a CNN as an input to ViT. ConViT indirectly reflects locality into the ViT structure by adding a modified SA module that operates similarly to the convolution in the initial stage (d’Ascoli et al., 2021). In addition, MaxViT is an advanced model of CoAtNet (Dai et al., 2021) that uses convolution layers directly in several stages of ViT (Tu et al., 2022).

The diversity of face images inputted into the authentication system is attributed to varying environmental conditions, such as background and camera angle, between individuals. This variation causes distribution shifts between the training and inference data, which degrades model performance. Therefore, the previous studies have suggested several methods to train the generalized feature space underlying some source domains and unseen but related target domains (Shao et al., 2019; Jia et al., 2020; Liu et al., 2021a; Wang et al., 2022b; Ming et al., 2022).

These studies have defined FAS as a binary classification problem that discriminates between real and spoof and adopted binary cross-entropy (BCE) as the loss function. However, recent studies have indicated that models trained via BCE are prone to attacks due to overfitting, thereby leading to decreased generalization performance. To address this issue, previous studies redefined the FAS task as a regression problem and probabilistically estimated its liveness score (Jiang et al., 2022; Kwak et al., 2023). Consequently, the regression-based FAS model improves the generalization ability because the converted continuous label contains more semantic information from both real and spoof images (Jiang et al., 2022). In addition, the expected liveness score-based regression neural network using the pseudo-discrete label encoding technique contributes to more robustness in unseen datasets than the model trained with BCE (Kwak et al., 2023).

We redefine the binary FAS classification as a regression-based method, as this approach is more suitable for learning generalized and discriminative representations (Jiang et al., 2022; Kwak et al., 2023). A sequence of class labels was transformed into a probability distribution using CutMix (Yun et al., 2019). CutMix is a technique used for data augmentation, but we generate a label with CutMix to discretize the pseudo-label and solve the FAS task with a regression-based loss function. Specifically, there are $N$ source domain datasets, which comprise face images $X\in\mathbb{R}^{H{\times}W{\times}3}$ with binary label $Y{\in}{\{0,1\}}$ that represents fake and real for each face image. The real and fake face images are denoted as $X_{r}$ and $X_{f}$, respectively, and $Y_{r}$ and $Y_{f}$ are the corresponding labels. Using CutMix, we swap some parts of $X_{r}$ to that of $X_{f}$ and denote this combined image as input image $X_{c}$. We define the discretized pseudo-label $Y_{c}$ as $m_{l}Y_{r}$, which is the single value $m_{l}$ selected from the set $M=\{0,\frac{1}{K},\frac{2}{K},…,\frac{K-1}{K},1\}$, where the interval $[0,1]$ is divided by a constant $K$.

In the second stage, we employ ConViT as a backbone $F$ for feature extraction to encode rich information to capture a cue for FAS. Specifically, global context and local cues are obtained through gated positional self-attention(GPSA) which is modified from SA. In general, an attention score A in original ViT is defined as the dot product of query embedding $Q$ and key embedding $K$. $W_{Q}$, $W_{K}$, and $W_{V}$ are the weight matrices for $Q$, $K$, and $V$, respectively, which are embeddings for the query, key, and value, respectively. The embeddings were inferred by multiplying the linearly projected input $X_{p}$ with the corresponding weight matrices. The attention score $A_{ij}$ implies the semantic relevance between patches $X_{p}^{i}$ and $X_{p}^{j}$ and can capture the long-range dependency between patches.

To reflect locality in the attention score, an inner product of the learnable embedding $v_{pos}$ and relative positional encoding $r_{ij}$ is added to $Q_{i}K_{j}^{T}$. As shown in Eq. (2), the gating parameter $\sigma$ manipulates the type of information on which to focus. A larger $\sigma$ at the initial stage makes this GPSA function as generalized convolution, which assigns higher weights to adjacent patches. After extracting the local features at the earlier layer, a smaller $\sigma$ is gated with more global information of patches far from each other at the upper layer.

The feature vector is derived from the input image $X_{c}$, by averaging the patch embedding $F(X_{c})$ extracted from our backbone $F$ and passed to both the score regressor $R$ and domain discriminator $D$, which contributes to a robust backbone for unseen domain data (Jia et al., 2020). As described in Eq. (3), $p_{k}^{i}$ is the probability vector of the real image in the $i^{th}$ input image, and the liveness score $\hat{Y}_{c}^{i}$ is calculated using the sum of the element-wise multiplication between $M$ and corresponding probabilities in $p_{k}^{i}$. The score regressor $R$ is trained to minimize the difference between the predicted score $\hat{Y}_{c}^{i}$ and groundtruth label $Y_{c}^{i}$ through the mean squared error. Subsequently, backbone $F$ is trained adversarially to extract generalized features, which makes it difficult for the discriminator to predict the domain label $Y_{D}$ of a feature. For adversarial learning, a gradient reversal layer (GRL) is included after the feature extractor to make it difficult for the discriminator to distinguish features from different domains. Additionally, the regressor and discriminator are composed of fully connected layers. In Eq. (4), $q(\cdot)$ is the probability vector of the discriminator and is equal to $D(F(X_{c}))$. Therefore, domain-invariant feature is extracted from backbone $F$ through adversarial learning, as described in Eq. (4). The final loss function is defined as in Eq. (5).

We conducted experiments on various FAS benchmark datasets, namely OULU-NPU(O) (Boulkenafet et al., 2017), MSFD-MSU(M) (Wen et al., 2015), REPLAY-ATTACK(I) (Chingovska et al., 2012), and CASIA-FASD(C) (Zhang et al., 2012). We have used a cross-dataset protocol to evaluate the performance. The cross-dataset protocol is a leave-one-out setting that is used to train a model with several source datasets and to test the generalization performance with an unseen target dataset. Half total error rate (HTER) and area under curve (AUC) were used as evaluation metrics (Yu et al., 2022).

We used CNN, ViT, and hybrid networks pretrained with ImageNet1K to prevent overfitting and achieve further improvement in DG (Parkin & Grinchuk, 2019; Ming et al., 2022). While we compared the FAS performance depending on differences in the feature extraction methods of the backbones, the structure of the score regressor and discriminator was kept the identical, and each feature extractor was trained with the same loss function as in Eq. (5).

We conducted a comparative experiment to determine the most robust FAS backbone among the candidate models. Specifically, we selected ResNet and EfficientNet as representative models for the CNN, ViT and swin transformer for the ViT series and ViT Hybrid, ConViT, and MaxViT for the hybrid series. First, Table 1 demonstrates that main feature extraction operation significantly affects the performance. When ConViT was used as a feature extractor, it achieved the best performance with an HTER of 11.7% and AUC of 93.9% compared to the CNN and ViT models. In addition to ConViT, ViT Hybrid had the second-highest average AUC. However, the ViT and swin transformer using only SA exhibit HTER scores of 23.6% and 15.1%, respectively, recording the highest error rates, which implies that the ViT is vulnerable to changes in the distribution of the evaluation data. When features were extracted from a ViT-based hybrid model containing convolution-like elements, performance in the unseen domain was typically superior to that of convolution or SA models alone. Specifically, the ConViT-based framework outperformed EfficientNet and ViT in 7.3%$p$ (=93.9%-86.6%) and 12.9%$p$ (=93.9%-81.0%) higher AUC scores, respectively.

Second, we found that convolution-like elements in the initial layer would be effective in improving FAS performance by comparing the hybrid models, as shown in Table 1. Specifically, MaxViT using convolution in most stages demonstrated poor performance, while ConViT, which indirectly adds convolutional SA in the early stage, and ViT Hybrid, which utilizes convolution in the input image, reported the lowest HTER values among the comparison groups. In addition, both MaxViT and swin transformer are similar in that they use window-based SA and the averaged AUC score of these models is about 86% compared to that of the other hybrid models, which are ConViT and ViT Hybrid, achieving 94%. Therefore, models using window-based SA show an approximately 8%$p$ lower AUC than the other hybrid models. Furthermore, we experimentally confirm that attention using a window shift is inappropriate for FAS; however, convolutional elements in an early stage of the network contribute to robust classification.

Table 2 presents a comparison of the proposed framework with the latest promising models. The ConViT or ViT Hybrid-based framework was ranked 1.5 and 3, respectively, by averaging HTER scores over four sub-protocols in a cross-dataset setting, showing the highest performance. In particular, the proposed ConViT-based model outperformed a previous model with ViT structure (Ming et al., 2022), exhibiting a 2.8%$p$ lower HTER. These results suggest that our ConViT-based model is the most effective as a generalized FAS methodology compared to the previously developed methodologies (Shao et al., 2019; Wang et al., 2020; Shao et al., 2020; Jia et al., 2020; Wang et al., 2021; Chen et al., 2021; Liu et al., 2021a; Wang et al., 2022b; Ming et al., 2022), and the extraction of rich representations from images significantly impacts improved performance in comparison to learning the spatio-temporal relationship in a video (Ming et al., 2022). Consequently, the results suggest that both locality and global dependencies within patches contribute to a more robust performance.