CASIA-SURF CeFA: A Benchmark for Multi-modal Cross-ethnicity Face Anti-spoofing

Face recognition systems are still dealing with ethnicity bias problems [17, 20, 31, 37]. As an effort in the direction of mitigating ethnicity bias in face recognition, Wang et al. [37] have recently released a face recognition dataset containing $4$ ethnicities to be used for algorithm design. However, there is no publicly available face anti-spoofing dataset with ethnic labels. Table 1 lists main features of existing face anti-spoofing datasets: (1) The maximum number of available subjects was $165$ on the SiW dataset [24] before $2019$; (2) Most of the datasets just contain RGB data, such as Replay-Attack [9], CASIA-FASD [46], SiW [24] and OULU-NPU [8]; (3) Most datasets do not provide ethnicity information, except SiW and CASIA-SURF. Although SiW provides four ethnicities, it has neither a clear ethnic label nor a standard protocol for measuring ethnic bias in algorithms. This limitation also holds for the CASIA-SURF dataset.

Static and Temporal Methods.

In addition to some works [27, 22, 6] based on static texture feature learning, some temporal-based methods [26, 29, 21] also have been proposed, which require from a constrained human interaction. However, these methods become vulnerable if someone presents a replay attack.There are also methods [7, 23] relying on more general temporal features by simply concatenating the features of consecutive frames [40, 4]. However, these algorithms are not accurate enough because of the use of hand-crafted features, such as HOG [42], LBP [25, 16], SIFT [30] or SURF [6]. Liu et al. [24] proposed a CNN-RNN model to estimate Photoplethysmography (rPPG) signals which can be detected from real but not spoof with sequence-wise supervision. Yang et al. [43] proposed a spatio-temporal attention mechanism to fuse global temporal and local spatial information. Although these face PAD methods achieve near-perfect performance in intra-database experiments, they are vulnerable when facing complex lighting environments in practical applications. Inspired by [13], we feed the static and dynamic images to SD-Net, which the dynamic image generated by rank pooling [15] instead of optical flow map [13]. Additionally, our SD-Net not only captures the static and dynamic features, but also static-dynamic fusion features in an end-to-end way.

Multi-modal Fusion Methods.

Zhang et al. [45] proposed a fusion network with $3$ streams using ResNet-18 as the backbone, where each stream is used to extract low level features from RGB, Depth and IR data, respectively. Then, these features are concatenated and passed to the last two residual blocks. Similar to [45], Tao et al. [33] proposed a multi-stream CNN architecture called FaceBagNet, which uses patch-level images as input and modality feature erasing (MFE) operation to prevent from overfitting. All previous methods just consider as a key fusion component the concatenation of features from multiple modalities. Unlike [45, 28, 33], we propose the PSMM-Net, where three modality-specific networks and one shared network are connected by using a partially shared structure to learn discriminative fused features for face anti-spoofing.

Single-modal Dynamic Image Construction.

Rank pooling [15, 38] defines a rank function that encodes a video into a feature vector. The learning process can be seen as a convex optimization problem using the RankSVM [34] formulation in Eq.1. Let RGB (Depth or IR) video sequence with $K$ frames be represented as $<{\bf I}_{1},{\bf I}_{2},...,{\bf I}_{i},...,{\bf I}_{K}>$, and ${\bf I}_{i}$ denote the average of RGB (Depth or IR) features over time up to $i$-frame. The process is formulated below.

where $\xi_{ij}$ is the slack variable, and $\delta=\frac{2}{K(K-1)}$. By optimizing Eq. 1, we map a sequence of $K$ frames to a single vector ${\bf d}$. In this paper, rank pooling is directly applied on the pixels of RGB (Depth or IR) frames and the dynamic image ${\bf d}$ is of the same size as the input frames. In our case, given input frame, we compute its dynamic image online with rank pooling using $K$ consecutive frames. Our selection of dynamic images for rank pooling in SD-Net is further motivated by the fact that dynamic images have proved its superiority to regular optical flow [36, 15].

Single-modal SD-Net.

As shown in Fig. 2, taking the RGB modality as an example, we propose the SD-Net to learn hybrid features from static and dynamic images. It contains $3$ branches: static, dynamic and static-dynamic branches, which learn complementary features. The network takes ResNet-18 [18] as the backbone. For static and dynamic branches, each of them consists of $5$ blocks (i.e., conv, res1, res2, res3, res4) and $1$ Global Average Pooling (GAP) layer, while in the static-dynamic branch, the conv and res1 blocks are removed because it takes fused features of res1 blocks from static and dynamic branches as input.

For convenience of terminology with the rest of the paper, we divide residual blocks of the network into a set of modules $\{{\cal M}^{t}_{\kappa}\}_{t=1}^{4}$ according to feature level, where $\kappa\in\{color,depth,ir\}$ is an indicator of the modality and $t$ represents the feature level. Except for the first module ${\cal M}^{1}_{\kappa}$, each module extracts static, dynamic and static-dynamic features by using a residual block, denoted as ${\bf X}^{t}_{s,\kappa}$, ${\bf X}^{t}_{d,\kappa}$ and ${\bf X}^{t}_{f,\kappa}$, respectively. The output features from each module are used as the input for the next module. The static-dynamic features ${\bf X}^{1}_{f,\kappa}$ of the first module are obtained by directly summing ${\bf X}^{1}_{s,\kappa}$ and ${\bf X}^{1}_{d,\kappa}$.

In order to ensure each branch learns independent features, each branch employs an independent loss function after the GAP layer [35]. In addition, a loss function based on the summed features from all three branches is employed. The binary cross-entropy loss is used as the loss function. All branches are jointly and concurrently optimized to capture discriminative and complementary features for face anti-spoofing in image sequences. The overall objective function of SD-Net for the $\kappa^{th}$ modality is defined as:

where $\mathcal{L}_{s}^{\kappa}$, $\mathcal{L}_{d}^{\kappa}$, $\mathcal{L}_{f}^{\kappa}$ and $\mathcal{L}_{sdf}^{\kappa}$ are the losses for static branch, dynamic branch, static-dynamic branch, and summed features from all three branches of the network, respectively.

The architecture of the proposed PSMM-Net is shown in Fig. 2(b). It consists of two main parts: a) the modality-specific network, which contains three SD-Nets to learn features from RGB, Depth, IR modalities, respectively; b) and a shared branch for all modalities, which aims to learn the complementary features among different modalities. For the shared branch, we adopt ResNet-18, removing the first conv layer and res1 block. In order to capture correlations and complementary semantics among different modalities, information exchange and interaction among SD-Nets and the shared branch are designed. This is done in two different ways: a) forward feeding of fused SD-Net features to the shared branch, and b) backward feeding from shared branch modules output to SD-Net block inputs.

Forward Feeding.

We fuse static and dynamic SD-Nets features from all modality branches and fed them as input to its corresponding shared block. The fused process at $t^{th}$ feature level can be formulated as:

In the shared branch, ${\tilde{\bf S}}^{t}$ denotes the input to the $(t+1)^{th}$ block, and ${\bf S}^{t}$ denotes the output of the $t^{th}$ block. Note that the first residual block is removed from the shared branch, thus ${\bf S}^{1}$ equals to zero.

Backward Feeding.

Shared features ${\bf S}^{t}$ are delivered back to the SD-Nets of the different modalities. The static features ${{\bf X}}^{t}_{s,\kappa}$ and dynamic features ${{\bf X}}^{t}_{d,\kappa}$ add with ${\bf S}^{t}$ for feature fusion. This can be denoted as:

where $t$ ranges from $2$ to $3$. After feature fusion, ${\widetilde{\bf X}}^{t}_{s,\kappa}$ and ${\widetilde{\bf X}}^{t}_{d,\kappa}$ become the new static and dynamic features, which are then feed to the next module ${\cal M}^{t+1}_{\kappa}$. Note that the exchange and interaction among SD-Nets and the shared branch are only performed for static and dynamic features. This is done to avoid hybrid features among static and dynamic information to be disturbed by multi-modal semantics.

Loss Optimization.

There are two main kind of losses employed to guide the training of PSMM-Net. The first corresponds to the losses of the three SD-Nets, i.e.color, depth and ir modalities, denoted as ${\cal L}^{color}$, ${\cal L}^{depth}$ and ${\cal L}^{ir}$, respectively. The second corresponds to the loss that guides the entire network training, denoted as ${\cal L}^{whole}$, which bases on the summed features from all SD-Nets and the shared branch. The overall loss $\cal L$ of PSMM-Net is denoted as:

We evaluate the performance of PSMM-Net on two multi-modal (i.e., RGB, Depth and IR) datasets: CeFA and CASIA-SURF [45], while evaluate the SD-Net on two single-modal (i.e., RGB) face anti-spoofing benchmarks: OULU-NPU [8] and SiW [24]. In order to perform a consistent evaluation with prior works, we report the experimental results using the following metrics based on respective official protocols: Attack Presentation Classification Error Rate (APCER) [2], Bona Fide Presentation Classification Error Rate (BPCER), Average Classification Error Rate (ACER), and Receiver Operating Characteristic (ROC) curve [45].

The proposed PSMM-Net is implemented with Tensorflow [3] and run on a single NVIDIA TITAN X GPU. We resize the cropped face region to $112\times 112$, and use random rotation within the range of [$-180^{0}$, $180^{0}$], flipping, cropping and color distortion for data augmentation. All models are trained for $25$ epochs via Adaptive Moment Estimation (Adam) algorithm and initial learning rate of $0.1$, which is decreased after $15$ and $20$ epochs with a factor of $10$. The batch size of each CNN stream is $64$, and the length of the consecutive frames used to construct dynamic map is set to $7$ by our experimental experience.

In this section, we investigate the performance biases of different ethnicities with two SOTA algorithms on the three ethnicities of CeFA. MS-SEF [45] is trained on CASIA-SURF for the multi-modal data while FAS-BAS [24] is trained for the RGB data on OULU-NPU. Then, the trained models are tested on CeFA. The results are shown in Table 3. Results show that both methods behave differently for the three ethnicities, i.e., East Asian ($11.4\%$) versus Center Asian ($19.6\%$) for MS-SEF and African ($14.2\%$) versus Center Asian ($26.1\%$) for MS-SEF under the ACER metric. In addition, both methods achieve relatively good results on East Asians (e.g., the values of ACER are $11.4\%$, $15.4\%$, respectively) because of most of the samples belong to East Asians on CASIA-SURF and OULU-NPU datasets. This indicates that existing single-ethnic anti-spoofing datasets limit the ethnic generalization performance of existing methods.

Here, we provide a benchmark for CeFA based on the proposed method. From Table 4, we can draw the following conclusions: (1) The ACER scores of three sub-protocols in Protocol $1$ are $0.6\%$, $4.4\%$ and $1.5\%$, respectively, which indicate the necessity to study the generalization of the face PAD methods for different ethnicities; (2) In the case of Protocol $2$, when print attack is used for training/validation and video-replay and 3D mask are used for testing, the ACER score is $0.4\%$ (sub-protocol 2_1). When video-replay attack is used for training/validation, and print attack and 3D attack are used for testing, the ACER score is $7.5\%$ (sub-protocol 2_2). The large gap between the results caused by the different PAI (i.e., different displays and printers). (3) Protocol $3$ evaluates cross-modality. The best result is achieved for sub-protocol 3_1 (ACER=$4.9\%$). (4) Protocol $4$ is the most difficult evaluation scenario, which simultaneously considers cross-ethnicity and cross-PAI. All sub-protocols achieve low performance, highlighting the challenges of our dataset: $24.5\%$, $43.2\%$, and $27.7\%$ ACER scores for 4_1, 4_2, and 4_3, respectively.

To verify the performance of our proposed baseline in alleviating ethnic bias, we perform a series of ablation experiments on Protocol $1$ (cross-ethnicity) of the CeFA dataset.

Static and Dynamic Features.

We evaluate S-Net (Static branch of SD-Net), D-Net (Dynamic branch of SD-Net) and SD-Net in this experiment. Results for RGB, Depth and IR modalities are shown in Table 7. Compared to S-Net and D-Net, SD-Net achieves superior performance showing that the learned hybrid features from static and dynamic images can alleviate ethnic bias. Concretely, for RGB, Depth and IR modalities, ACER of SD-Net is $12.6\%$, $6.1\%$, $6.4\%$, versus $17.2\%$, $7.7\%$, $9.4\%$ of S-Net (improved by $4.6\%$, $1.6\%$, $3.4\%$) and $19.9\%$, $9.4\%$, $11.3\%$ of D-Net (improved by $7.3\%$, $3.3\%$, $4.9\%$), respectively. It also shows that the performance of Depth and IR modalities are superior to the RGB modality because of the variability of lighting conditions interfering with feature learning of RGB samples.

Multiple Modalities.

In order to show the effect of analysing a different number of modalities, we evaluate one modality (RGB), two modalities (RGB and Depth), and three modalities (RGB, Depth and IR) on PSMM-Net. As shown in Fig. 2, the PSMM-Net contains three SD-Nets and one shared branch. When only RGB modality is considered, we just use one SD-Net for evaluation. When two or three modalities are considered, we use two or three SD-Nets and one shared branch to train the PSMM-Net model, respectively. Results are shown in Table 7. The best results are obtained when using all three modalities: $2.2\%$ of APCER, $2.2\%$ of BPCER and $2.2\%$ of ACER. These results show that multi-modal information has a significant effect in alleviating ethnic bias, mainly because of the smaller differences in skin color of different ethnicities in the IR modality.

Fusion Strategy.

In order to evaluate the performance of PSMM-Net, we compare it with other two variants: Naive halfway fusion (NHF) and PSMM-Net without backward feeding mechanism (PSMM-Net-WoBF). As shown in Fig. 3, NHF combines the modules of different modalities at a later stage (i.e., after ${\cal M}^{1}_{\kappa}$ module) and PSMM-Net-WoBF strategy removes the backward feeding from PSMM-Net. The fusion comparison results are shown in Table 7, showing higher performance of the proposed PSMM-Net with information exchange and interaction mechanism among SD-Nets and the shared branch.

CASIA-SURF.

The comparison results are show in Table 10. The performance of the PSMM-Net is superior to the ones of the competing multi-modal fusion methods, including Halfway fusion [45], single-scale SE fusion [45], and multi-scale SE fusion [44]. When compared with [45, 44], PSMM-Net improves the performance by at least $0.4\%$ for ACER. When the PSMM-Net is pretrained on CeFA, it further improves performance. Concretely, the performance of $TPR@FPR=10^{-4}$ is increased by $2.4\%$ when pretraining with the proposed CeFA dataset. The comparison results not only illustrate the superiority of our algorithm for multi-modal data fusion, but also show that our CeFA alleviates the bias of attack pattern to a certain extent.

SiW and OULU-NPU.

Results for these two dataset are shown in Table 10 and 10, respectively. We compare the proposed SD-Net with other methods without pretraining. Our method achieves the best results (a lower ACER value indicates better performance) on all protocols of the SiW and protocol $3$ and $4$ of the OULU-NPU. The experimental results show that our SD-Net combined with the dynamic image generated by the rank pooling algorithm can effectively capture features related to motion difference between the real face and the fake one.

Last but not least, using the proposed dataset to pre-train our baseline method significantly improves its ACER performance in most of protocols. In Protocol $2$ and $3$ of SiW, our method trained on the CeFA dataset performs the best among all models. Note that the STASN (Data) [43] used a large private dataset to pretrain. Similar conclusions can be drawn from the OULU-NPU experiment. These results demonstrate the effectiveness and generalization capability of the CeFA dataset, and suggest SOTA methods can be further improved by using our CeFA dataset for pre-training.

Dynamic images.

Given a video, we map a sequence of $7$ frames into a dynamic image by using rank pooling. Some samples are shown in Fig. 4(a). As for SiW and OULU-NPU datasets, the eye part (red box) of the real sample is more realistic than print or replay attack, while more speckles (orange box) caused by specular reflections are included in the replay attack. Our SD-Net can capture these discriminative dynamic features.

Misclassified Samples.

Some misclassified samples of our baseline on CeFA are shown in Fig. 4(b). Visually from static image, it is very difficult to distinguish the type of the sample from RGB and IR modalities. Furthermore, the depth modality of a 3D attack shows to be extremely similar to the real face.