Open-Set Face Identification
on Few-Shot Gallery by Fine-Tuning

[14], one of the earliest works in OSFI, used their proposed Open-set TCM-kNN on top of features extracted by PCA and Fisher Linear Discriminant. [15] proposed their own OSFI protocol and showed that an extreme value machine [16] trained on the gallery set performs better than using cosine similarity or linear discriminant analysis for matchers. [17] trained a matcher composed of locality sensitive hashing [18] and partial least squares[19]. [20] applied OpenMax [21] and PROSER [22], two methods for open-set recognition of generic images, on top of extracted face features.

All previous works propose to train an open-set classifier (matcher) of some form, but all of them use a fixed encoder. To the best of our knowledge, we are the first to propose an effective fine-tuning scheme as a solution to OSFI.

[23] proposed to $l_{2}$-normalize the features such that the train loss is only determined by the angle between the feature and the classifier weights. [7] further extended this idea by applying a multiplicative margin to the angle between a feature and its corresponding weight vector. This penalized the intra-class features to be gathered while forcing inter-class centers (prototypes) to be separated. A number of follow-up papers such as [8, 9, 10, 11] modify this angular margin term in different ways, but their motivations and properties are generally similar. Therefore, in our experiments we only use CosFace loss [8] as a representative method. For comprehensive understanding of these loss functions, refer to [24].

Formally, in an OSFI problem, we assume the availability of an encoder $\phi$ pretrained on a large-scale face database (FR embedding model), which is disjoint from the evaluation set with respect to identity. The evaluation set consists of a gallery $G=\{(x^{G}_{i},y^{G}_{i})\}^{Cm}_{i=1}$ and a probe set $Q$. The probe set $Q$ is further divided into the known probe set $K=\{(x^{K}_{i},y^{K}_{i})\}$ and the unknown probe set $U=\{(x^{U}_{i},y^{U}_{i})\}$. $G$ and $K$ has no overlapping images $x$ but shares same identities $y\in\{1,...,C\}$ whereas $U$ has disjoint identities, i.e., $\mathcal{Y}^{U}\cap\{1,...,C\}=\O$. $m$ refers to the number of images per identity in $G$, which we fix to 3 to satisfy the few-shot constraint. We allow the encoder to be fine-tuned over the gallery set.

The evaluation of OSFI performance uses the detection and identification rate at some false alarm rate (DIR@FAR). FAR=1 means we do not reject any probe. Note that unlike the general case shown in [1], here we only consider rank-1 identification rate for DIR. Therefore, DIR@FAR=1 is the rank-1 closed-set identification accuracy.

Due to the few-shot nature of the gallery set where we fine-tune on, the initialization of model parameters and, in particular, of classifier weights is crucial to avoid overfitting. The most naive option is a random initialization of the classifier weight matrix $W$. Another commonly used strategy is linear probing [25], i.e., finding an optimized weight $W$ that minimizes the classification loss over the frozen encoder embeddings $\phi(x)$.

We experimentally find that, as seen in Fig. 2, both of these initialization schemes do not induce discriminative structure for the encoder embedding $\phi(x)$. In particular, during fine-tuning, each weight vector $w_{c}$ in the classifier acts as a center (or prototype) for the $c$-th class (i.e. identity). Fig. 2 shows that neither random initialization nor linear probing of $w_{c}$ derives optimally discriminative weight vectors $w_{c}$, resulting in low quality of class separation of gallery features.

Motivated from this issue, we propose to initialize by weight imprinting (WI), which induces the optimal discriminative quality for the gallery features:

where $\lVert\cdot\rVert_{2}$ is the $l_{2}$ norm, and the embedding feature $\phi(x)$ is unit-normalized such that $\lVert\phi(x)\rVert_{2}=1$.

As expected, Fig. 2 verifies that fine-tuning from the weight imprinted initialization achieves a much higher discriminative quality. This shows the superiority of weight imprinting compared to random initialization and linear probing.

Note that weight imprinting has been frequently used in FR embedding models [8, 9]. However, the critical difference is that those models utilize weight imprinting only to prepare templates before evaluation. In our case, the WI initialization is utilized particularly for fine-tuning.

Choosing the appropriate layer to tune is another important issue for fine-tuning. Moreover, due to the extremely small number of samples for each gallery identity, there is a risk of overfitting as suggested by the classical theory on the vc dimension [26]. In fact, a recent study [25] suggests that full fine-tuning hurts the pretrained filters including the useful convolutional filters learned from a large-scale database.

To minimize the negative effect of this deterioration, we fine-tune only the BatchNorm (BN) layers along with the classifier weight:

where $\theta$ refers to all parameters in the encoder $\phi=\phi_{\theta}$ and $\theta_{BN}$ and $\theta_{rest}$ respectively refers to BatchNorm parameters and the rest. During fine-tuning, $\theta_{rest}$ is fixed with no gradient flow. The loss function $\mathcal{L}$ can be a softmax cross-entropy, or widely used FR embedding model losses such as ArcFace [9] and CosFace [8].

Due to selective fine-tuning of only the BN layers (and classifier weight), the convolutional filters learned from the large-scale pre-train database are simply transferred. The BN-only training is thus computationally efficient as it occupies only 0.1-0.01% of the total parameters in the CNN. Nevertheless, its model complexity is sufficient to learn a general image task as guaranteed by [27].

The cosine similarity function is the most predominant matcher for contemporary face verification and identification. Denoting the probe feature vector as $p$ and the gallery prototypes as $\{g_{j}\}_{j=1}^{C}$, where $g_{j}:=\frac{1}{m}\sum_{y^{G}_{i}=j}\phi(x^{G}_{i})$ is the mean of all the normalized gallery feature vectors of class $j$, identification is performed by finding the maximum class index $c=\arg\max_{j=1,\dots,C}\cos(p,g_{j})$. On the other hand, in the extension to OSFI, the decision of accepting as known or rejecting as unknown can be formulated:

where $\cos(p,q)=\frac{p}{\lVert p\rVert_{2}}\cdot\frac{q}{\lVert q\rVert_{2}}$ is the cosine similarity between two feature vectors, $\tau$ is the rejection threshold.

Now, consider an example illustrated in Fig. III-C. The cosine matcher will assign the probe $u$ to the identity $i$ with the acceptance score $0.866$, which is fairly close to the maximum score $1$. This value alone might imply that the probe is a known sample as it is close to the gallery identity $i$. However, the probe feature vector is placed right in the middle of the identities $i$ and $j$. The in-between placement of $u$ suggests that the probe can be possibly unknown and thus should be assigned with a lesser value of the acceptance score.

Motivated by this intuition, we propose the Neighborhood Aware Cosine (NAC) matcher that respects all top-$k$ surrounding gallery features:

Here, $N_{k}$ is the index set of $k$ gallery prototypes that are nearest to the probe feature $p$, and 1 is the indicator function. The main goal of the NAC matcher is to improve the unknown rejection. Table III-C shows that known probe features are much closer to their closest prototype than the second-closest prototype, unlike unknown probes. By exploiting this phenomenon, the NAC matcher is able to assign a much smaller score to unknown probe, as shown in Fig. 4.

We use VGGFace2 [4] dataset for pretraining the encoders, and CASIA-WebFace [28] and IJB-C [29] for evaluation. Using MTCNN [30], we align and crop every images to 112x112 with equal parameters for all datasets. For VGGFace2, we remove all identities overlapping with the evaluation datasets. The evaluation datasets are equally split into two groups such that the number of known and unknown identities are equal. Then we randomly choose $m{=}3$ images of the known identities to create the gallery (G), and the rest are known probes (K). All images of unknown identities are unknown probes (U). Table II summarizes the statistics of the datasets we use. Note that we chose every known identity to have more than 10 images such that there can be at least 7 probe samples. Also note that IJB-C dataset consists of still images and video frames (video frames typically have poorer image quality). We sample the gallery from still images and probes from video frames, which makes this dataset much challenging. We note that the protocol devised here can be regarded as an extension of that in [15].

We choose VGG19 [2] and ResNet-50 [3] for the encoders with the feature dimension 512. We pretrain these encoders on the VGGFace2 dataset with CosFace with scale=32, margin=0.4 as loss function until convergence.

Then we fine-tune the encoder with different classifier initialization schemes and encoder layer configurations. When using the linear probing initialization, we train the classifier until the training accuracy reaches 95%.

We follow the encoder layer finetuning in Sec. IV-B. For the partial fine-tuning, we only train the last 2 convolutional layers (Conv-BN-ReLU-Conv-BN-ReLU). Table III shows the number of total and updated parameters for each fine-tuning scheme.

We fix the number of epochs to 20 and batch size to 128 for every method. We again use CosFace loss for consistency. For the optimizer we use Adam [32] with cosine annealing. The initial learning rate is set to 1e-4 for full and PA, and 1e-3 for partial and BN, which we find as the optimal learning rate for each method. For data augmentation, we use random horizontal flipping and random cropping with the random scale from 0.7 to 1.0. The cropped images are resized to the original size.

Since the gallery set is too small, we cannot afford a separate validation set to individually optimize $k$ for each dataset. Instead, we attempt to find a global value that has optimal performance regardless of the fine-tuning method, if one exists.

We first fine-tune the encoders with different layer configurations, which gives us five different encoders including one without any fine-tuning; pretrained, full, partial, PA, and BN. Then we search the best parameter k for the NAC matcher by grid search strategy, where the grid is [2,4,8,16,32,128,256,512,1024,$C$], and $C$ is the total number of identities. Note that $k=1$ refers to using cosine similarity instead of NAC, which we added for comparison. Since a single-value objective is preferred, we use the area under the curve (AUC) of the DIR@FAR curve instead of DIR value at different FAR values. We repeat this process with different datasets and encoder architectures.

The results are shown in Fig. 5. We did not include the results of CASIA-WebFace as it shows a similar trend. Excluding $k=1$ which is not NAC, the results show a smooth unimodal curve with a peak at $k=16$ or $32$. This shows that the NAC matcher indeed has a globally optimal $k$ value that is robust against different datasets, encoders, and fine-tune methods. Thus we choose $k=16$ ($k=32$ also gives similar results) as the global parameter throughout this paper.

Note that when $k=C$, NAC becomes equivalent to softmax function with cosine similarity logits. However, this is notably inferior compared to $k=16$, which implies that considering only the $k$-nearest is superior to considering every gallery prototype.

We compare the OSFI performances of the pretrained model (non-fine-tuned) with six different combinations of classifier initialization schemes and layer finetuning configurations: random+full, linear probing+full, WI+full, WI+partial, WI+PA, WI+BN. The matcher is fixed to cosine similarity. These correspond to row 4-9 in Table IV.

First, to compare different classifier initialization schemes, we fix the fine-tuning scheme to full. When using random initialization, rejection accuracy (DIR@FAR=0.001,0.01,0.1) and closed-set accuracy (DIR@FAR=1) severely drops. For linear probing, rejection accuracy improves while closed-set accuracy drops. Only WI clearly improves the encoder performance, supporting the superiority of weight imprinting.

Now we fix the classifier initialization to WI and compare different layer finetuning configurations. full clearly has the worst performance. While PA is better than partial in closed-set accuracy, partial clearly outperforms PA in rejection accuracy. BN outperforms all others in closed-set accuracy with a large margin but sometimes falls behind partial in rejection accuracy.

With the aid of the NAC matcher, our method WI+BN+NAC outperforms all other methods in every aspect. Compared to original, this gains 4.60%, 8.11%, 4.57%, 1.68% higher DIR in average with respect to FAR of 0.001, 0.01, 0.1, 1.0, respectively.

How do different layer finetuning configurations affect the final OSFI performance? To analyze this, we adopt three different metrics; inter-class separation, intra-class variance, and Davies-Bouldin Index (DBI) [33]. The definitions of the first two metrics are identical to that of Fig. 2. DBI is a metric for evaluating the clustering quality, where DBI $\approx 0$ means perfect clustering. We compute these metrics on the gallery features after fine-tuning, and the results are shown in Table V.

Here we can easily separate these configurations into two groups: full and partial vs PA and BN. The first group has similar inter-class separation with Pretrained and significantly smaller intra-class variance, which leads to small DBI. This is in stark contrast with the second group.

With this observation, we can conjecture the different optimization strategies of each group. The first group was able to easily reduce the training loss by collapsing the gallery features into a single direction (shown by the small angle between intra-class features). This was possible because both full and partial directly updated the parameters of the convolutional filters. On the other hand, all convolutional filters were frozen for both PA and BN. This constraint may have prevented these methods from taking the shortcut, i.e. simply collapsing the gallery features, and instead led to separating the embeddings of different identities. This explains why PA and BN have higher closed-set accuracy.

This can also explain the AUC gain ($\Delta$AUC) when using NAC instead of cosine similarity. Features become redundant when they collapse, and so does the prototype. Therefore the information from neighboring prototypes becomes less helpful in rejecting unknown samples, leading to the marginal gain from using NAC. This is why full and partial do not benefit from using NAC matcher.

Fig. 6 shows the OSFI performance of our method against the baseline (pretrained encoder with cos matcher) with respect to different gallery size. We can see that our method consistently improves upon the baseline, except for the extreme case where only one image is provided for each identity.