CCFace: Classification Consistency for Low-Resolution Face Recognition

FR is among the oldest and most surfed problems in computer vision and has matured over the years from utilizing handcrafted features and local descriptors toward using deep learning-based models [37]. Existing deep network architectures, such as CNNs and ViT variants [6, 8], are dominant feature extractors in this area [6, 25]. The critical issue is how to train the deep model using a large-scale dataset and prevent overfitting [38]. Deep FR training schemes are either proxy-less or proxy-based. The former utilizes a tuple of similar and dissimilar images. It encourages the network to map the faces with the same identity to close representations and faces with distinct identities to distant representations [37]. The fact that these loss functions directly supervise sample-wise similarities is aligned with the final objective of the FR system. However, the sampling process required for these methods becomes challenging in large-scale datasets, which leads to convergence problems [21]. Therefore, most of the research has been dedicated to classification-based loss functions, such as L-Softmax [27], SphereFace [26], CosFace [47], and ArcFace [6].

LR facial images lack the details of their HR counterpart, such as eyes and skin texture. Therefore, learning discriminative representations from LR samples are much harder, and applying general FR learning methods results in unsatisfactory performance [42]. Two main approaches have been surfed for LR face recognition: 1) construction-based and 2) projection-based methods [54, 48, 2, 19, 42, 10, 56].

Construction-based methods are based on super-resolving the LR input prior to recognition. Apart from the ill-posed nature of the FSR problem, the main scheme of FSR is not optimized for discrimination, resulting in the unsatisfactory performance of the subsequent FR task [37]. To alleviate this, Zhang et al. [54] use the face verification loss in the identity metric space to guide the reconstruction model toward preserving identity information. Several studies focused on dividing the FSR into different sub-tasks. Wang et al. in [48] dedicated one network to reconstructing global details and the other to enhancing local details. Cao et al. [2] used Reinforcement Learning (RL) to localize regions and a local network to attend to the specified regions. Current FSR methods produce visually appealing results; however, their integration with the FR model degrades the overall performance [42]. Also, these methods are computationally intensive [42].

The projection-based approaches try to map both HR and LR images to a unified embedding [10, 56]. To this end, various works focused on distilling well-constructed features from HR teacher to LR student module [56]. Zhu et al. [56] utilized the general KD approach of using soft-logits prediction of the teacher module to guide the student module for the task of LR classification. Numerous studies applied intermediate representation distillation from the HR network to improve LR performance [33, 10]. However, the consistency between HR and LR representations was not imposed, deteriorating the performance on cross-resolution scenarios. Among the facial parts, key parts are essential in FR, such as eyes and ears [22]. Kumar et al. in [22] force the model to generate key points using an auxiliary layer which guides the network toward focusing more on key facial characteristics. These methods mainly focused on improving the results of the LR images and did not maintain the performance of the original HR imagery. Also, guiding the LR network using the proxies of the HR network has not yet been explored for LR face recognition. Therefore, we investigate a prediction consistency knowledge distillation approach to improve the performance on LR images and preserve the model performance on HR images.

Knowledge distillation (KD), first proposed in [14], is a form of model compression that transfers knowledge from a robust teacher model into a small student model. Since its introduction, numerous distillation techniques have been developed [36, 44, 42]. FitNet [36] directly reduces the Euclidian distance between the student and teacher network. Combining representation learning with KD, CRD [44] utilizes contrastive objective to increase the mutual information between the teacher and student model. Despite the remarkable improvement in conventional KD, these methods are mainly designed for closed-set classification and are incompatible with FR. For instance, in [14] Kullback-Leibler (KL) divergence between soft logits of teacher and student network is used to distill knowledge. In an open-set problem such as FR, obtaining the discriminative feature is most important. Furthermore, consistency between features obtained from the teacher and student model is essential because cross-resolution embedding of a specific identity must be close to each other.

The majority of FR models consist of feature extractor $\mathcal{F}:\mathcal{I}\rightarrow\mathcal{X}$ mapping input faces $\mathcal{I}$ from image space to an embedding space $\mathcal{X}$. At the top of the feature extractor, there is a classifier $\mathcal{W}:\mathcal{X}\rightarrow\mathcal{\widehat{Y}}$ to predict the input identity from the embedding. Using gradient descent, both the feature extractor and classifier will be trained end-to-end. To this end, the widely used Softmax cross-entropy is applied on the predicted labels [18]:

where $\small{x_{i}\in\mathbb{R}^{d}}$ is the $\small{d}$-dimensional representation of $\small{i}$-th input. $\small{W\in\mathbb{R}^{d\times C}}$ represents the learnable matrix where each column $\small{W_{j}}$ is the proxy of $\small{j}$-th class. When both the $\small{x_{i}}$ and $\small{W_{j}}$ are mapped to a unit hypersphere, then the dot product $\small{W_{j}x_{i}}$ reflects the cosine of the angle between representation and proxy:

The introduction of angular penalty to the classification framework has been shown to be effective in increasing inter-class separability and intra-class compactness. SphereFace [26] argues that large-margin classification better aligns with open-set FR and introduces multiplicative angular margin for learning more discriminative features (period of the $cos(\theta_{y_{i}})$). CosFace proposes using a vertical shift of the function $cos(\theta_{y_{i}})$, which leads to more powerful feature discrimination and improved stability. ArcFace suggests using additive angular margin (phase shift of $cos(\theta_{y_{i}})$), which has more clear geodesic interpretation and also improves the performance. In Eq. 3, $\small{m_{s},m_{c}}$, and $\small{m_{a}}$ show the angular penalty introduced by SphereFace, CosFace, and ArcFace, respectively.

Here unlike conventional KD approaches, the teacher and student networks have the same architecture. Instead, the teacher network only sees the original images, and the student network the down-sampled instances. Given a training set $\small{D}$, the LR samples are obtained from down-sampling with varying interpolations ($s$): $\small{D_{LR}:{\{(I_{i}^{LR},y_{i}):I_{i}^{LR}=I_{i}\downarrow_{s})\}_{i=0}^{N}},s\in S}$. $N$ is the total number of samples. As presented in Fig. 4, $\small{f_{t}(.)}$ and $\small{f_{s}(.)}$, map the HR and LR faces to a $\small{d}$-dimensional embedding space; $\small{x_{i}^{HR}=f_{t}(I_{i})}$ and $\small{x_{i}^{LR}=f_{s}(I_{i}^{LR})}$.

As discussed in section 3.1, in the current SOTA FR objective functions, the similarity between features is guided through the cosine angle between features and softmax proxies. Therefore, the inter-class discrepancy increases if the proxies are well distributed on the hypersphere. Recently, numerous studies have been conducted on uniformly distributing the proxies on the hypersphere [24]. In Fig. 3, we show our studies on the inter-class similarity from the proxies’ viewpoint. We illustrate the maximum inter-class cosine value in two scenarios: 1) model trained on the original MS1MV2 dataset, 2) model trained on the down-sampled version of MS1MV2. This observation shows that naively training on the LR samples will result in poor discrimination (increase in inter-class similarity). The inter-class similarity is drastically increased in the model trained on LR images. Therefore, features being supervised through these proxies would not achieve preferred discriminability.

To overcome this issue, we propose sharing the teacher network’s proxies with the student model: $W^{HR}=W^{LR}=W$. However, the student model does not adjust the proxies and only uses them as a part of its forward propagation. Also, both networks are being trained using conventional large-angular margin loss function:

Generally, there are three types of samples on the original training dataset: 1) easy, 2) hard, and 3) unrecognizable [38]. Applying augmentation on different samples can produce different types of samples, whether unrecognizable or hard. Angular margin helps the model push the features toward the corresponding positive proxy, i.e., by reducing the value of $cos(\theta)$. Larger margin results in more loss and eventually more gradient to adjust the network weights. For a better illustration of the effect of angular margin on the loss and the magnitude of the gradient that the loss imposes on the network, we conduct a simple experiment. In a 2k classification problem, we changed the value of $cos(\theta_{y_{i}})$ from zero to one while the negative similarity scores with all of the classes are fixed to 0.1, $cos(\theta_{j})=0.1;j\neq y_{i}$. Reducing the Eq. 3 to ArcFace:

by deriving the gradient of Eq. 6 with regard to $x_{i}$:

Here, we describe the Asymmetric Cross-Resolution Learning (ACR) learning of CCFace in detail. To begin with, we first discuss the need for the ACR learing. Then we describe our asymmetric loss function, which is based on the state-of-the-art approaches for representation learning.

To increase the cross-resolution intra-class compactness, the network should consider different samples of an identity. Furthermore, to prevent LR representations from forming a cluster, the loss function should increase the inter-class discrimination among LR representations. To this end, we proposed ACR to promote discrimination among the LR images and increase the mutual information among the HR and LR counterparts of the same subject:

In Eq. 13, $N_{i}$ is a set of all negative LR samples presented in the mini-batch for $x_{i}$ ($N_{i}\equiv{n\in B:y_{n}\neq y_{i}}$), $P_{i}$ is a set of all positive HR samples ($P_{i}\equiv{p\in B:y_{p}=y_{i}}$) and $|P_{i}|$ is its cardinality. To maintain the performance on original HR data and since the HR representations have good intra-class compactness and inter-class discrimination, the detached HR features are used in Eq. 13. Therefore, Eq. 13 asymmetrically forces the LR representation to increase their mutual information with their HR counterparts [37]. Also, it penalizes the similarity between negative LR representations, which prevents them from forming a cluster.

We report our experiments based on using MS1MV2 as the training dataset [13, 6]. The MS1MV2 dataset is a refined version of the MS-Celeb-1M dataset with 85k identities and 4 million images. Images were down-sampled using random interpolation to construct the HR and LR pairs. For evaluation, we report our results on LFW, CFP-FP, AgeDB, CPLFW, and CALFW. To validate the proposed method’s performance on real-world LR faces, we also report our results on the TinyFace [4] and SCFace [46] datasets. We used an aligned version of these datasets in which samples are resized to 112 by 112 [20]. Also, we report our performance on IJB-B and IJB-C as benchmarks which contain both HR and LR samples.

IJB-B and IJB-C: IJB-B [50] contains around 21.8K images (11.8K faces and 10K non-face images) and 7k videos (55K frames). A total of 1,845 identities are presented in this dataset. Our experimental protocols follow the standard 1:1 verification, which contains 10,270 positive and 8M negative matches. There are 12,115 templates in the protocol, each of which consists of multiple images or frames. Consequently, a template-based matching process is used. Specifically, we average over the instances in a template to obtain the global feature vector for each template. IJB-C [29] is the extended version of IJB-B, including 31.3K images and 117.5K frames from 3,531 identities. The testing protocol of IJB-C is similar to IJB-B.

SCFace: SCface [16] is a challenging cross-resolution FR dataset. It contains HQ mugshot and LQ images captured by surveillance cameras. The images were taken from 130 subjects in an uncontrolled indoor environment using five video surveillance cameras at different distances $d_{i}\in\{4.2,2.6,1.0\}$ (meter); five images at each distance. Also, one frontal mugshot image for each subject is obtained using a digital camera, Fig. 7. In an experiment similar to [16], we employ frontal mugshot images as our gallery and samples taken by surveillance cameras as probes.

There are two primary approaches to validating an FR paradigm’s performance: 1) Recognition and 2) Verification. Recognition is a 1:N task where the network should calculate the similarity score of a given probe image with all the samples in the gallery and determine the identity of the probe image. Face verification is a 1:1 task where the network should determine whether a given pair of images represent the same identity. We report the verification results on the LFW, CFP-FP, AgeDB, CPLFW, IJB-B, IJB-C, and CALFW datasets (TAR@FAR from ROC for IJB-B and IJB-C [12]). The result for identification is reported on the SCFace and TinyFace datasets.

We followed the ArcFace setup for preprocessing [6]. All the images are resized to 112$\times$112, aligned to a canonical view, and pixel values are normalized to $[-1,1]$. To produce synthetic low-resolution samples, we randomly apply different down-sampling interpolations on the $I_{HR}$, including bilinear, nearest neighbor, and bicubic. Also, the size of the LR image is uniformly sampled from $\{(16\times 16),(20\times 20),(32\times 32)\}$. Together, there would be twenty-seven distinct augmentations. The experiments are conducted with ResNet100 as the backbone [6, 39] unless mentioned. The model is trained for 24 epochs with the Arcface loss. The optimizer is SGD, with the learning rate starting from 0.1 decreased by a factor of 10 at epochs {10, 16, 22}. The optimizer weight-decay is set to 0.0005, and the momentum is 0.9. During training, the mini-batch size on each GPU is 512, and the model is trained using two Quadro RTX 8000. After training is finished, we disregard the teacher model and only use the student model for evaluation.

In this section, we assess CCFace’s performance against the SOTA methods, such as CurricularFace, MagFace, and URL, using TinyFace, SCFace, IJB-B, and IJB-C datasets. Results in Table 1 demonstrate CCFace’s competitive performance against other methods. Showing the efficacy of alignment between teacher and student model. Since IJB-B and IJB-C contain both HR and LR samples, this performance shows the ability of the proposed method to generalize across a wide range of resolutions. Table 2 demonstrates the results on TinyFace. According to this result, CCFace can effectively boost the discrimination power over the LR data, which results in over 3 percent improvement in TinyFace data. To evaluate the performance of CCFace for cross-resolution scenarios, Table 3 shows the identification result on the SCFace dataset. CCFace could improve the results of the previous methods by more than one percent. Success across datasets with varying quality levels, from LR to HR, shows the efficacy in alignment between teacher and student embedding space, and maintaining the discriminability on both HR and LR images.

One of the major issues with angular-margin-based loss functions, which are dominant in FR, is that their integration with data augmentation is challenging [38]. In order to alleviate this issue, we use the output probability of the teacher network to tune the margin value for augmented samples; see section 3.3 for more detail. Table. 4 shows the student model’s performance with different augmentation probabilities. Increasing the augmentation occurrence boosts the FR performance on the LR faces considerably; however, negligible performance degradation is observed on high-quality benchmarks. Table. 4 also shows the performance of the student model without using an adaptive margin. Since the augmentation process increases the chance of unrecognizable instances, the performance gain is inconsistent.

Here, we analyze the impact of different elements within our training paradigm on performance. Our experiments utilize R18 as the backbone and CASIA as the training dataset. As shown in Table. 5, training a network solely with a single classifier results in poor performance on the LR samples. However, sharing the classifier between the teacher and student improves performance to some extent. Additionally, incorporating an adaptive margin into the model enhances performance on the LQ dataset. Notably, the application of ACR between the teacher and student significantly improves the cross-quality scenario.