Face Transformer for Recognition

Face Transformer model follows the architecture of ViT [1], which applies the original Transformer [17].

The only difference is that, we modify the tokens generation method of ViT, to generate tokens with sliding patches, i.e., to make the image patch overlaps, for the better description of the inter-patch information, as shown in Figure 1. Specifically, we extract sliding patches from the image $\bm{X}\in\mathbb{R}^{W\times W\times C}$ with patch size $P$ and stride $S$ for them (with implicit zero on both sides of input), and finally obtain a sequence of flattened 2D patches $\bm{X_{p}}\in\mathbb{R}^{N\times(P^{2}\times C)}$. $(W,W)$ is the resolution of the original image while $(P,P)$ is the resolution of each image patch. The effective sequence length is the number of patches $N=\lfloor\frac{W+2\times p-(P-1)}{S}+1\rfloor$, where $p$ is the amount of zero-paddings.

As ViT did, a trainable linear projection maps the flattened patches $\bm{X_{p}}$ to the model dimension D, and outputs the patch embeddings $\bm{X_{p}E}$. The class token, i.e., a learnable embedding ($\bm{X}_{class}=\bm{z}_{0}^{0}$) is concatenated to the patch embeddings, and its state at the output of the Transformer encoder ($\bm{z}_{L}^{0}$) is the final face image embedding, as Equation 2. Then, position embeddings are added to the patch embeddings to retain positional information. The final embedding

serves as input to the Transformer,

which consists of multiheaded self-attention (MSA) and MLP blocks, with LayerNorm (LN) before each block and residual connections after each block, as shown in Figure 1. In Equation 2, the output $\bm{x}$ is the final output of Transformer model.

One of the key block of Transformer, MSA, is composed of $k$ parallel self-attention (SA),

where $\bm{z}\in\mathbb{R}^{(N+1)\times D}$ is an input sequence, $\bm{U_{qkv}}\in\mathbb{R}^{D\times 3D_{h}}$ is the weight matrix for linear transformation, and $\bm{A}=softmax(\bm{qk}^{T}/\sqrt{D_{h}})$ is the attention map. The output of MSA is the concatenation of $k$ attention head outputs

where $\bm{U}_{msa}\in\mathbb{R}^{kD_{h}\times(D+1)}$.

The output $x$ of Equation 2, i.e., the final output of Transformer model, is supervised by an elaborate loss function [8, 9, 10] for better discriminative ability,

where $y$ is the label, ${P_{y}}$ is the predicted probability of assigning $\bm{{x}}$ to class $y$, $C$ is the number of identities, ${{\bm{W}_{j}}}$ is the $j$-th column of the weight of the last fully connected layer, and $\bm{b_{j}}\in{\mathbb{R}^{\text{C}}}$ is the bias. Softmax based loss functions [26, 8, 9, 10] remove the bias term and transform $\bm{{W_{j}}}^{T}\bm{{x}}=s\cos{\theta_{j}}$, and incorporate large margin in the $\cos{\theta_{y_{i}}}$ term [8, 9, 10]. Therefore, softmax based loss functions can be formulated as

where $f(\theta_{{y_{i}}})=\cos{\theta_{{y_{i}}}}-m$ in CosFace [9].

We apply two training databases, CASIA-WebFace and MS-Celeb-1M [7]. CASIA-WebFace is a sweet training database and contains 0.49M images from 10,575 celebrities, which can be seen as a relatively small-scale database compared with million-scale ones [7]. MS-Celeb-1M is a popular large scale training database in face recognition and we use the clean version refined by insightface [10], which contains 5.3M images of 93,431 celebrities. We choose CosFace [9] ($s$ = 64 and $m$ = 0.35) as the loss function for better convergence and recognition performance. The face images are aligned to 112 $\times$ 112. The Horizontally flip with a probability of 50% is used for training data augmentation.

For comparison, the CNN architecture used in our work is a modified ResNet-100 [12] proposed in the first version of ArcFace paper [10], which uses IR blocks (BN-Conv-BN-PReLU-Conv-BN) and applies the “BN [27]-Dropout [28]-FC-BN” structure to get the final 512-$D$ embedding feature. We also experiment with the recent proposed T2T-ViT [5]. The number of parameters, MACs and inference speed (Tesla V100, Intel Xeon E5-2698 v4) of these face recognition models are listed in Table I. Details are as follows. For ViT models, the number of layers is 20, the number of heads is 8, hidden size is 512, MLP size is 2048. For the Token-to-Token part of T2T-ViT model, the depth is 2, hidden dim is 64, and MLP size is 512; while for the backbone, the number of layers is 24, the number of heads is 8, hidden size is 512, MLP size is 2048. Note that, the “ViT-P10S8” represents the ViT model with $10\times 10$ patch size, with stride $S=8$, and “ViT-P8S8” represents no overlapping between tokens.

We use AdamW [29] and cosine learning rate decay following DeiT [4]. The models are trained from scratch without pre-training. With 1 warmup epoch, the initial learning rate is set as 3e-4, and we lower it to 1e-4 when the training accuracy reaches a stable stage (about 20 epochs).

We mainly report recognition performance of models on several mainstream benchmarks including LFW [18], SLLFW [19], CALFW [20], CPLFW [21], TALFW [22] CFP-FP [23], AgeDB-30 [24], and IJB-C [25] databases. LFW database contains 13,233 face images from 5,749 different identities, which is a classic benchmark for unconstrained face verification. Similar-looking LFW (SLLFW), Cross-Age LFW (CALFW), Cross-Pose LFW (CPLFW) and Transferable Adversarial LFW (TALFW) databases are constructed based on LFW database, to emphasize similar-looking challenges, cross-age challenge and cross-pose challenge, and adversarial robustness of face recognition. CFP-FP database is built for facilitating large pose variation and AgeDB-30 database is a manually collected cross-age database. IJB-C database contains both still images and video frames to address the unconstrained face recognition.

The experimental results are shown in Table II and Table III. We first find that in Table II, Face Transformer models trained on CASIA-WebFace database performs much worse than ResNet-100. Actually, we find that the accuracy of Face Transformer models trained on CASIA-WebFace can reach a high level as ResNet-100, while models cannot generalize well on test databases, which indicates that the scale of CASIA-WebFace may be not enough for Transformer models.

While things change when we use a much larger training database, MS-Celeb-1M. The performance of Face Transformer models demonstrate promising results on large-scale face training databases. The performance of Face Transformer is competitive compared to the ResNet-100 with similar number of parameters and MACs. Compared with “ViT-P8S8”, “ViT-P10S8” and “ViT-P12S8” have better performance, which demonstrates the overlapping patches can help in some degree. T2T-ViT also obtain good performance, while limited computer source, more hyper-parameters for T2T block remains to try. Another interest point is that, Transformer models obtain a little higher accuracy on TALFW database, which is a database with transferable adversarial noise. Since TALFW database is generated using CNNs as surrogate models, it seems that there is no significant specificality with Transformer in terms of adversarial robustness. It is interesting to explore the performance of combination of Face Transformer models and adversarial training.