Mitigating Domain Mismatch in Face Recognition Using Style Matching

Distributions can be matched by MMD [2] or adversarial learning [6, 7] to produce a distribution-confused embedding space. However, these approaches are not sufficient for face recognition since inter-class embeddings may not be separable in target domain.

To make up for the shortcomings of distribution matching, clustering is used to assign pseudo labels in the target domain. In Wang and Deng [3], a target-domain classifier is trained using pseudo labels. Following the low-density separation principle, Wang et al. [4] use mutual information loss to categorize target-domain embeddings. Leveraging frame consistency in video face recognition, Arachchilage and Izquierdo [5] better estimate positive and negative pairs of target images. After clustering and mining, the model is fine-tuned using a modified version of triplet loss. Similar work is described in Arachchilage and Izquierdo [8]. As clustering-based methods rely on the target domain priors, hyperparameters must be determined before clustering, which is difficult in practice.

In work on style transfer [21, 22, 23, 15], instance normalization is used for style normalization. This could also be a way to learn a domain-invariant model. Li et al. [58] remove bias by taking the mean and standard deviation from the target domain as batch normalization parameters. Qing et al. [59] combine instance and batch normalization (IBN) to improve cross-domain recognition. Nam et al. [60] propose batch-instance normalization (BIN), summing instance and batch normalization with an optimal ratio; they thus yield superior results by learning the ratio between two operators. In Qian et al. [61], the IBN layer is adopted as the domain adaptation layer, and the ratio between batch and instance normalization is determined by the domain statistics. Drawing from these studies, Jin et al. [62] propose style normalization and restitution (SNR), extracting domain-invariant features and preserving task-relevant information to further improve performance. However, these approaches necessitate changes to the network architecture that incur a far greater computational complexity.

The architecture from Zhang et al. [17] is modified as the domain discriminator, as shown in Fig. 2. This discriminator approximates human-level judgments to calculate a score that represents how similar an image is to the target domain. For image $x$, a feature stack is extracted from $L_{g}$ layers of a backbone network $\mathcal{F}(x)$ and unit-normalized in the channel dimension, denoted by $\mathcal{F}^{l}(x)\in\mathbb{R}^{H_{l}\times W_{l}\times C_{l}}$ for layer $l$. Each $\mathcal{F}^{l}(x)$ is scaled by vector $u^{l}\in\mathbb{R}^{C_{l}}$ and summed channel-wise. We average spatially to yield a scalar representation $r^{l}\in\mathbb{R}^{1}$ on each layer:

where $\otimes$ is a convolution operator. We then concatenate the representations to produce a perceptual feature $r\in\mathbb{R}^{L_{g}}$, which we pass through a single-layer perceptron (vector $v$) followed by a sigmoid activation to judge the input image. The domain discriminator $g(x)$ is expressed as

Note that since we use SqueezeNet [63] as the backbone network, in our case $L_{g}=7$.

We sample images from the source and target domains equally and use the following loss function (domain discrepancy loss) to train the domain discriminator:

in which we expect the discriminator to produce a low score (tending to 0) for a source image but a high score (tending to 1) for a target image. This score is used in the classification loss in Equation 1 to up-weight the loss caused by target-like images. The classification loss is modified as

where $\mathcal{L}^{*}_{c}$ is the loss for adapting the model based on PS.

According to studies on style transfer [21, 22, 23, 15], the stacks of mean and standard deviation on the feature maps describe the style of an image. For layer $l$ in the face recognition model, the style representation is designated as $\{\mu(f^{l}(x)),\sigma(f^{l}(x))\}$. The measures of styles from the two domains are defined as

where $p^{l}_{\mu}$ and $p^{l}_{\sigma}$ are measures of $\{\mu(f^{l}(x^{s})),\sigma(f^{l}(x^{s}))\}$, and $q^{l}_{\mu}$ and $q^{l}_{\sigma}$ are measures of $\{\mu(f^{l}(x^{t})),\sigma(f^{l}(x^{t}))\}$. We seek to minimize a certain distribution matching loss between $\{p^{l}_{\mu},p^{l}_{\sigma}\}$, and $\{q^{l}_{\mu},q^{l}_{\sigma}\}$.

In the literature on domain adaptation, MMD and adversarial learning are the main ways to match distributions. Although MMD is simple and necessitates no extra training parameters, it focuses on matching the region of high density [27], which may incur greater bias. Adversarial learning approaches, however, approximate relative entropy or the Wasserstein distance (optimal transport) by training the embedding function and the discriminator interactively. Although adversarial learning yields a better estimation of domain discrepancy, it requires extra parametric subnetworks, which complicates training. We strike a balance between the two methods by calculating the distribution matching loss using Sinkhorn divergence [24, 25, 26, 27, 28, 29, 30], which interpolates optimal transport and MMD, and has been successfully applied to generative models in Genevay et al. [64].

Similar to Genevay et al. [64], we align the style distributions of the source and target domains by minimizing

where $\hat{c}\overset{\textit{def.}}{=}[c(z^{\mathcal{P}}_{i},z^{\mathcal{Q}}_{j})]_{i,j}\in\mathbb{R}^{n\times m}$. Here, $z^{\mathcal{P}}_{i}$ and $z^{\mathcal{Q}}_{j}$ are samples from distributions $\mathcal{P}$ and $\mathcal{Q}$ respectively. Where $p>0$ is some exponent, $c(z^{\mathcal{P}}_{i},z^{\mathcal{Q}}_{j})=d(z^{\mathcal{P}}_{i},z^{\mathcal{Q}}_{j})^{p}$ is a cost equipped with a distance function $d$. We define $d$ as Euclidean distance, and set $p=2$. The entropy-regularized optimal transport, that is, Wasserstein distance, is $\mathcal{W}_{\varepsilon}(\mathcal{P},\mathcal{Q})$, in which factor $\varepsilon$ regularizes entropy. Thus $P$ can be treated as a mass-moving plan between $\mathcal{P}$ and $\mathcal{Q}$. As expressed in Cuturi [24], Cuturi and Doucet [25], and Genevay et al. [64], regularized optimal transport is equivalent to restricting the search space by using the scaling form of $P$:

where $K_{i,j}\overset{\textit{def.}}{=}e^{\frac{-\hat{c}_{i,j}}{\varepsilon}}$; the role of $\varepsilon$ here mirrors that of bandwidth in MMD. The optimization problem of Equation 8 turns into finding $a$ and $b$ that meet certain requirements. Based on the Sinkhorn algorithm [24, 25], $a$ and $b$ can be found by

where the fraction here is element-wise division. Starting with $b_{0}=\mathbf{1}_{m}$, we obtain feasible $a_{L}$ and $b_{L}$ after $L$ iterations. The budget $L$ is set to $10$ according to Genevay et al. [64]. Hence, Equation 8 can be modified as

where $\odot$ is an element-wise multiplication operator. Since $\hat{c}$ can be large, to prevent numerical explosions during training, we divide by $nm$ to normalize the total cost. However, such a discrepancy approximation tends to overfit. To account for this, we add intra-measure Wasserstein distance to penalize the inter-measure Wasserstein distance:

where $\tilde{\mathcal{W}}_{\varepsilon}(\mathcal{P},\mathcal{Q})$ is called Sinkhorn loss in Genevary et al. [64].

We adopt Sinkhorn loss in Equation 12 to match the style distributions of the source and target domains, so the style matching loss can be expressed as

where $L_{f}$ is the number of adaptation layers in $f(x)$.

In this protocol, to avoid bias, the numbers of positive and negative pairs are usually the same, such as in the Labeled Faces in the Wild (LFW) [35, 65] and YouTube Faces (YTF) [66] datasets. The dataset is divided into $K$ partitions. We evaluate the model on each partition by using the optimal threshold found by testing all the other partitions, and average the accuracies.

We determine the threshold for rejecting negative pairs that corresponds to the desired false positive rate (FPR) or false alarm rate (FAR), which yields a true positive rate (TPR) that represents the accuracy on positive pairs. In general, there are far more negative pairs than positive pairs to test the robustness of a model.

VGGFace2 [67] and CASIA-WebFace [68] are our training datasets; all facial images in both datasets were captured in the wild. Due to the large number of images for each identity in VGGFace2, it ensures better learning than CASIA-WebFace, even though it contains fewer identities. As in most of the relevant literature [6, 2, 3, 4], CASIA-WebFace is a primary source dataset in this paper. Following Wang et al. [3] and Arachchilage and Izquierdo [8], we initialize the backbone model on VGGFace2 after which we fine-tune the parameters on CASIA-WebFace.

We use LFW [35], CFP-FP [69], and AgeDB-30 [70] as the validation datasets; for each we use a 10-fold cross validation protocol. Their small size facilitates efficient model evaluations. When training a baseline model, we test it on these datasets after every epoch. After training, we pick the model whose average accuracy is the highest of all the generations as the baseline model.

IJB-A [31] is a joint face detection and recognition dataset that contains a mixture of still images and videos under challenging conditions, including blur as well as large pose and illumination variation. It provides 10-split evaluation protocols containing both verification and identification. We take the average of the 10-split evaluations as the final results. IJB-A was extended as IJB-B [32], which was further extended as IJB-C [33]. The evaluation methods in IJB-B and IJB-C are the same. In terms of verification, only one protocol is provided, but the positive-versus-negative pairs in IJB-B and IJB-C are about 10K/8M and 19.6K/15.6M respectively, which is more challenging than that of IJB-A (1.8K/10K), as listed in Table 2. For identification, both provide one probe set and two gallery sets for evaluation. We successively test each gallery set using the probe set and use their average as the final result. YTF [66] contains 3,425 videos from 1,595 identities, for a total of 621.2K frames. As with the validation datasets, its protocol follows 10-fold cross validation, in which each fold is composed of 250 positive and negative pairs. Furthermore, we cooperate with the library of National Chiao Tung University (NCTU) for surveillance application. A system was built to collect the data automatically. When a subject entered the library, the system recorded and labeled the video by the student or staff identity card. Due to personal information protection, the labels are anonymous. To eliminate the interference of head pose variation, we adopted the head pose estimation model in Hsu et al. [71] to filter out the frames with large head pose (pitch, roll, or yaw $>15^{\circ}$). This dataset is called NCTU-Lib, and its samples are shown in Fig. 3. The domain discrepancy between the training dataset and NCTU-Lib is large since almost all subjects in NCTU-Lib are Asian, which makes it more difficult to adapt the model.

If the multitask convolutional neural network (MTCNN) [72] detects a face in an image, we align the face to the fixed reference points using the eyes, mouth corners, and nose center. If it fails, we decide what to do based on the dataset. We ignore the training datasets and YTF since there is plenty of data for training and testing. For IJB-A/B/C, if three landmarks (eyes and nose center) are provided, we align these to the same reference points of the same landmarks, or we apply a square box whose side is half of the maximal side of the image to crop on the center. All processed images are resized to $128\times 128$ and cropped to $112\times 112$ for testing. When training, we augment the images by mirroring them with $50\%$ probability, and resize them to $128\times 128$ and randomly crop them to $112\times 112$. In evaluations, we follow Whitelam et al. [32] when fusing the features. That is, in a template, the frames of each type of media (still images or video frames) are averaged before all medias are averaged.

We used MobileFaceNet [73] as our backbone model. Equation 1 is the primary classification loss in this paper. As mentioned above, the model was first trained on VGGFace2 dataset [67]. We used stochastic gradient descent (SGD) with momentum to train all models. The training epochs, batch size, learning rate, and momentum were set to 50, 128, 0.1, and 0.9 respectively, and the learning rate was divided by 10 every 12 epochs. After that, when transferring to CASIA-WebFace dataset [68], the learning rate was set to 0.01 to warm up the classifier $\hat{y}$ by one epoch before fine-tuning all parameters. We then set the learning rate to 0.0001 to fine-tune the parameters, and again divided by 10 every 12 epochs. The other settings remained unchanged. The backbone model fine-tuned on CASIA-WebFace was then used as the baseline model denoted by softmax.

We assumed that the IJB-A/B/C datasets [31, 32, 33] are shared domain information. Only IJB-A was used as the target dataset when testing on these datasets. We used all images and frames in IJB-A to adapt the model. Similarly, when testing on the YTF dataset [66], we used all the frames of each video in YTF. During adaptation, the learning rate was also set to 0.0001 to fine-tune the baseline model; other settings remained unchanged.

Since the role of $\varepsilon$ is similar to bandwidth in MMD, we followed related studies [74, 2, 3] to set its value. Luo et al. [2] and Wang et al. [3] suggest setting the bandwidth to the median pairwise distance on source dataset. However, such an exhaustive search is not practical since training datasets are usually large. This can instead be estimated by a Monte Carlo method, but we propose a more efficient way to determine the value. Similar to the learning process in batch normalization, we estimated $\varepsilon$ by the mean pairwise distance between source and target samples in a mini-batch, and updated it every iteration by

where $\hat{\varepsilon}_{k}$ is the current estimation. Momentum $\rho$ was set to $0.9$. In the following experiments, we compare the dynamic $\varepsilon$ (the proposed method) and the fixed $\varepsilon$.

MobileFaceNet [73] is described in Table 3. For the following experiments, we mainly used four adaptation layers in shallow networks for SM. From shallow to deep, the dimensions of the feature maps were $56^{2}\times 64$ ($l=1$), $28^{2}\times 64$ ($l=2$), $28^{2}\times 64$ ($l=3$), and $14^{2}\times 128$ ($l=4$) respectively.

The softmax PS method in Tables 4 and 5 is the proposed PS method described in Equation 5. Compared with the baseline, it yields improved performance. We also adopted pure SqueezeNet [63] as a discriminator, denoted as softmax FF PS, in which FF stands for feed-forward. There is no significant difference between the two methods, but results show that softmax PS is robust to false alarms, which implies that low-level features are important for distinguishing identities at a finer granularity. To analyze the effect of PS, domain discriminator $g(x)$ is used to judge all images in both the source and target datasets. The distributions are shown in Fig. 4: the two domains are clearly separable, which is evidence of domain discrepancy. The purpose of PS is to mitigate such discrepancy by scoring samples in the source domain. With Equation 5, the distribution of the source dataset is flatter: note the reduced discrepancy for CASIA-WebFace (Weighted) in Fig. 4.

Compared with PS, all SM approaches yield improvements in most metrics under different protocols. We find no differences among the metrics when varying the adaptation layers $L_{f}$, but we do note a trend in which TPR or TPIR under lower FAR is better when more adaptation layers are applied. However, there is no clear benefit to a fixed $\varepsilon$. It is slightly better for some metrics but worse on others. Conversely, using a dynamic $\varepsilon$ is more convenient, and yields similar performance.

The performance of SM is further improved by PS. Improvements are more obvious on verification protocols. The improvements on identification protocols are slight but consistent. That is, most of the metrics show improvements. In particular, distinct improvements can be found in lower FAR (FPR and FPIR), which is an important concern in real-world scenarios. This implies that PS does help to generalize the learning of style matching on the target domain. Since most of the best metrics are achieved by $L_{f}=2$, we adopt this as our default setting.

To understand how the proposed methods affect the embedding space, we measure the embedding norm and inter-class and intra-class cosine similarities on the target datasets, as listed in Table 6. The norm measures the discriminative power of an embedding. PS slightly increases both the inter-class similarity and the embedding norm while preserving the intra-class similarity. SM separates embeddings from different classes by a large margin. Although intra-class similarity is decreased, the performance is also improved by larger inter-class similarity. In fact, the number of negative samples dominates in the testing protocols, which is a reasonable hypothesis in real-world applications; thus, slightly enlarging inter-class distances yields distinct improvements on accuracy. However, lower intra-class similarity saturates the performance at higher FAR, as shown in Tables 4 and 5. SM equipped with PS enjoys the benefits of both: not only is inter-class separation further enlarged, but intra-class similarity is also preserved slightly.