Domain Adapting Ability of Self-Supervised Learning for Face Recognition

Because of the unique issues of face recognition, few researches focus on domain adaptation for face recognition. Some prior works, [4][5][6], adopt embedding alignment methods to be the core algorithm directly, like Maximum Mean Discrepancy (MMD), [14], or adversarial learning, [15], which does not really reduce the domain shift existing in face recognition. To compensate embedding alignment, pseudo labels generated by clustering on target domain are utilized to train another cluster distribution are proposed in [2] and [3]. Arachchilage et al. [10] focus on the task of video frames. With the identity consistency in a video, clustering can be more accurate. After clustering, triplets are mined to fine-tune the model with a modified triplet loss. However, clustering relies on prior knowledges of class number or cluster margin, which may be a non-trivial work when setting these parameters.

To leverage unlabeled data, some auxiliary tasks are designed in training losses based on the prior knowledges of data. For vision tasks, the simplest task is self-similarity. Chen et al. [16] augment an image, and adopt contrastive learning to minimizing their distances while maximizing the distances among different images in the embedding space. The method in [17] achieves better performance by adopting cosine similarity with stop-gradient operation. Its transfer ability is briefly illustrated, but it is limited. Therefore, we mainly refer this research to design an adapting loss for face recognition.

High recognition accuracy replies on a good embedding function. The purpose of an embedding function $f(x)$ is mapping images on a lower dimensional space where embeddings from same classes are closer while embeddings from different classes are farther. We can train an embedding function with an extra classifier, $\hat{y}(f(x))$, by cross entropy, which is simple and robust. The cross entropy loss is defined as:

$$L_{c}=-\sum_{i=1}^{N}y^{s}_{i}\log{\hat{y}(f(x^{s}_{i}))}$$

(1)

Since training dataset is usually large, to accelerate the convergence, focal loss [18] is adopted to encourage the learning of hard samples, so the loss can be modified as:

$$L_{c}=-\sum_{i=1}^{N}(1-\hat{y}(f(x^{s}_{i})))^{\gamma}y^{s}_{i}\log{\hat{y}(f(x^{s}_{i}))}$$

(2)

where $\gamma$ is a parameter for weighting down the loss caused by easy samples, and it is set to $2$ according to [18]. Also, metric learning policies, like [19][20][21], can be applied to train a better embedding function.

According to the studies in [16], minimizing the distances between the embeddings from random cropped and color distorted images can achieve the best result. However, applying color distortion may cause the racial bias, and we do not find obvious difference between using mirroring and random cropping in experiments. For each image, $x$, we only use it and its mirror, $x^{\prime}$, for training, so a self-supervised learning loss called SimSiam [17] can be expressed as:

$$L_{s}=\frac{1}{2}[D(h(z^{\prime}),\varphi(z))+D(h(z),\varphi(z^{\prime}))]$$

(3)

$$D(p,z)=-\frac{p^{T}z}{\|p\|\|z\|}$$

(4)

where $z=f(x)$, $z^{\prime}=f(x^{\prime})$, $\varphi$ is a stop-gradient operation, and $D(p,z)$ is a function for computing the cosine similarity between embeddings $p$ and $z$. Specially, $h$ is a head which remap the embeddings in another view, and its effectiveness has been discussed in [16] and [17].

To perform domain adaptation, we apply the loss on both source and target datasets and sum them up by a ratio which we call it adapting ratio, $\rho$. Thus, the loss is expressed as:

$$L_{a}=(1-\rho)L_{s}(x^{s})+{\rho}L_{s}(x^{t})$$

(5)

where $\rho$ controls the weights of SimSiam loss on two domains. For the purpose of domain transfer, it is reasonable to set the value of $\rho$ larger than $0.5$ a little bit. Since the loss only focus on self-similarity, it is necessary to use cross entropy loss to maintain the inter-class discrepancy. The total loss is defined as:

$$L=L_{c}+L_{a}$$

(6)

We call it SimSiam Adapting (SSA) Loss, and use it to do the experiments in the next section.

In the experiments, CASIA-WebFace [22] is utilized to be the source dataset. It contains 10,575 identities and 494,414 images which are the pictures of celebrities on Internet. As for the testing datasets, we use IJB datasets, including IJB-A/B/C [7][8][9]. IJB datasets contain mixture of images and videos in the wild. The conditions are all challenging due to blurry frames and large pose variations. IJB-A [7] contains 500 identities with 5,396 images and 20,412 video frames. IJB-B [8] extended from IJB-A contains 1,845 identities with 11,755 images and 7,018 videos, and 10,044 non-face images. IJB-C [9] extended from IJB-B contains 3,531 identities with 21,294 images and 11,779 videos, and 10,040 non-face images.

We follow the protocols in [7] to evaluate the performance of verification, open-set and close-set identification, and there metrics are True Positive Rate vs. False Positive Rate (TPR@FAR), True Positive Identification Rate vs. set False Positive Identification Rate (TPIR@FAIR), and top-K accuracy (Rank-K) respectively.

Cross entropy loss (2) is applied to train MobileFaceNet [13] from scratch by CASIA-WebFace dataset [22]. The training epoch and batch size is set to 50 and 128 respectively. The learning rate is set to 0.1 and is divided by 10 every 12 epoch. The trained model is the baseline for domain adaptation, and its performance is the baseline in the experiments.

We treat all images in IJB-A dataset [7] as the target dataset, so most images in IJB-B and IJB-C are not covered. As the unlabeled data from a target dataset are mixed in, the learning rate is decayed to be 0.0001 to protect the learned knowledge. The training epoch, batch size and learning rate schedule are preserved, but the epoch is counted based on the size of the target dataset, which induces less training iterations. For each batch, 128 images are sampled from each dataset.

The head, $h$, used in SimSiam loss is a two-layer neural network, and the number of hidden neurons is set to 32 which is 4 times less than the embedding dimension of MobileFaceNet [13]. According to [17], we add batch normalization and rectified linear unit in the hidden layer to guarantee its performance.

It is more efficient to use smaller dataset to evaluate the trained models in ablation study, so only IJB-A dataset [7] is adopted in this part of experiments.

To guarantee the effectiveness of SSA, it is compared with the state-of-the-arts focusing on domain adaptation for face recognition. We also use a margin penalty method (ArcFace) proposed in [19] to train our backbone to be another baseline. The comprehensive evaluations on IJB-A [7], IJB-B [8], and IJB-C [9] are listed in Table 3 and Table 4.

From Table 3, we can observe that SSA can successfully improve the baselines on all benchmarks under almost all FPRs especially under lower FPRs. With the guidance of ArcFace [19], the performance can be better. However, in the identification protocols, Table 4, the improvements only exists on open-set protocols. Such issue has been discussed in 4.3. Since the models are adapted on IJB-A only [7], the performances of on the open-set protocols of IJB-B [8] and IJB-C [9] are not that good, but it can be further refined by adapting more data from these datasets. Expect for close-set identification protocols, compared with the state-of-the-arts, our approach shows its good performance by not only the improvements but also the much lighter backbone.