Relational Deep Feature Learning
for Heterogeneous Face Recognition

Projection based methods involve learning to project features from two different domains into a discriminative common latent space where images with the same identity are close regardless of their domain. Lin and Tang [24] proposed a Common Discriminant Feature Extraction (CDFE) algorithm, in which two different domain features simultaneously learn common space to solve the inter-modality problem. With empirical separability and a local consistency regularization objective function, the model learns compact intra-class space and prevents the overfitting problem. Yi and Liao [25] suggested matching each partial patch of face images by extracting points, edges, or contours that are similar between domains. Lei et al. [26] designed a Coupled Spectral Regression (CPR) method for finding different projective vectors by representing relationships between each images and their embeddings. Different from this coupled method, which learns each domain’s representative vectors separately, Lei et al. [27] suggested the technique of learning the projection from both domains. Since target data neighbors should correspond to source data neighbors, Shao et al. [28] matched projected target and source data by using a reconstruction coefficient matrix, Z, in the common space.

With deep neural networks (DNNs) showing great improvement in face recognition performance, Sarfraz and Stiefelhagen [29] used a deep perceptual mapping (DPM) method in which DNNs learn projection of visual and thermal features together. In [30], Reale et al. used coupled NIR and VIS CNNs, initializing them with a pre-trained face recognition network to extract global features. Wu et al. [16] proposed a Coupled Deep Learning (CDL) method with relevance constraints as a regularizer and a cross-modal ranking objective function. However, these methods are difficult to train because they require extraction of domain-specific features with a small database.

Image synthesis based methods transform face images from one domain into the other so as to perform recognition in the same modality. Liu et al. [31] proposed a pseudo-sketch synthesis method which divides a photo image into a fixed number of patches and reconstructs each patch as a corresponding sketch patch. This patch-based strategy preserves local geometry while transforming the photo image into a sketch-style image. In [32], Wang et al. proposed a multi-scale Markov network, conducting brief propagation to transform multi-scale patches. Recently, with the widespread development of generative adversarial networks (GANs) [33], many studies have focused on generating visual face images from non-visual ones[14], [6]. In [13] Song et al. transformed NIR face images to VIS face images by pixel space adversarial learning using CycleGAN[34] and feature space adversarial learning with a variance discrepancy loss function.

These methods of transforming an image from one domain to another can be effective for visually similar domain, but do not fundamentally address the modality discrepancy that the data exhibits. In addition, due to issue of small amounts of unpaired HFR data, GAN-based methods struggle to create good-quality images, which affects performance.

Another approach is to use a feature extractor to reduce domain discrepancies and enable learning of domain-invariant features. Since the NIR-to-VIS face recognition task is heavily influenced by the light source in each image, Liu et al. [35] used differential-based band-pass image filters, relying only on variation patterns of skin properties. Lui et al. [17] also proposed a TRIVET loss function which applies Triplet loss [36] to cross-domain sampling to reduce the domain gap. He et al. [5] used a division approach, using two orthogonal spaces to represent invariant identity and light source information. The Wasserstein CNN in [15] is also divided into an NIR-VIS shared layer and a specific layer. The shared layer is designed to learn domain-invariant features by minimizing the Wasserstein distance between different domain data. Based on this approach, the DVR method [37] is proposed to disentangle the cross-domain facial representations with the identity information and within-person variations.

Several studies [38], [39], [40] used relational representations that allow projection heterogeneous data into a common space. In [38], Klare and Jain proposed a random prototype subspace framework to define prototype representations and learn a subspace projection matrix with kernel similarities among face patches. With their G-HFR method [39], Peng et al. employed Markov networks to extract graphical representations. This method finds the $k$ nearest patches of a patch in a probe or gallery image from the representation dataset and linearly combines them to obtain graphical representations. Since finding $k$ nearest patches from the representation process performance score relies heavily on the value of $k$, Peng et al. [40] proposed an adaptive spare graphical representation method which considers all possible numbers of related image patches. These methods found relations between representation dataset image patches in randomly selected pairs. Unlike these methods, our proposed RGM extracts domain-invariant features by considering global spatial pair-wise relations. We first apply a deep learning based relation approach to the HFR task with graph structured module.

We first treat the spatial feature vectors extracted through the backbone as initial graph node vectors with dimension $C$. Then we embed the node vectors into $d$-dimensional vectors using a transform matrix $\boldsymbol{W}_{1}\in\mathbb{R}^{C\times d}$. We experiment with the optimal value of this embedding dimension $d$ in Section IV-A2.

The feature vectors of the face image from the convolutional layers represent each face component (e.g., eyes, lips, and chin). In the RM, the feature vectors are simply concatenated to extract the relation through the shared fully connected layer. In RGM, after node embedding, we extract the directed edges of each node. Because the components that represent the face are the same for every classes, we generate a fixed number of component nodes rather than selecting nodes (64 nodes are used in this paper).

In Equation 2, the edge yielding the relationship between two node vectors $n_{i}$ and $n_{j}$ is obtained through the edge function $E_{w_{e}}(\cdot)$. Edge $E_{i,j}$ is a scalar value and is calculated as the weighted sum between node vector elements, where the weight $\boldsymbol{W}_{e}$ is a parameter obtained through learning. ${A}_{i,j}$ has a value in the range [0, 1] through the sigmoid function $\sigma({\cdot})$.

Then, as shown in Equation 3, each node vector propagates in inter-dependency with all other node vectors through the edges to become a propagated node vector ${\boldsymbol{n}_{i}}^{*}$. Each face component has different relations for each identity and updating the nodes with the relations can concentrate on component relational information rather than visual-domain features such as texture information.

As a point of comparison, Graph Attention Network (GAT)[48] adopt self-attention mechanisms by a learnable parameter $\alpha_{vu}=softmax(g(\boldsymbol{a}^{T}\left[\boldsymbol{W}^{T}\boldsymbol{n}_{v}||\boldsymbol{W}^{T}\boldsymbol{n}_{u}\right]))$ which only compute for neighbor nodes $u\in\mathcal{N}_{v}$ where an adjacency matrix($\mathcal{N}_{v}$) is used to define this neighbor. By updating nodes $\sum_{u\in\mathcal{N}_{v}}\alpha_{vu}{\bf W}^{T}\boldsymbol{n}_{u}$ within an adjacency matrix, where $g(\cdot)$ is a LeakyReLU activation function and $\boldsymbol{a}$ is a vector of learnable parameters. In the RGM, the adjacency matrix and $\alpha$ are learned simultaneously; also, the RGM uses a sigmoid activation function which looks at each value separately that allows for independent values of relations. Since the relation between the nodes is independent of the relations of other nodes, sigmoid activation is more relevant than Softmax which looks all values interrelated in phase and computes the sum of values to 1. We experiment with this activation issue in Section IV-F1.

After propagation, we apply the NAU as an activation function $\delta(\cdot)$ (see Figure 3). The NAU is serves as a node-adaptive activation function, this will be described in detail below. After we recalibrate nodes through the NAU, we use the weight matrix $\boldsymbol{W}_{2}\in\mathbb{R}^{d\times C}$ to re-embed the node vectors into the original input dimension and perform residual summation. Since the feature map from the backbone CNN contains general features of the face, we intended to use general feature and relational information together by residual terms [22, 46, 45]. After summation, we concatenate entire node vectors and embed them into the final representative embedding vector for comparison through the fully connected layer.

The CASIA NIR-VIS 2.0 database is one of the largest HFR databases, and is composed of NIR and VIS face images. It contains 725 subjects, imaged by VIS and NIR cameras in four recording sessions. We followed view 2 protocol, which training subjects and the corresponding testing sets are non-overlapped and the numbers of subjects are virtually identical. For evaluation, the gallery set comprises one VIS image per subject while the probe set contains several NIR images per subject. The prediction score is computed by similarity matrix over the whole gallery set and the identification accuracy and verification rate recorded.

We first experiment with different numbers of the node vector dimension $d$ to find the appropriate dimension for HFR. The experiment is conducted on LightCNN-9, in which there are 128 channels in the last convolutional layer. Figure 7 shows the results of training with the RGM node vector dimension $d=16,32,64,128$ and $256$. The identification accuracy and verification rate show better performance as the dimension increases and then drop off when it becames too large. We use a dimension of 128 in LightCNN and 256 in ResNet18, whose channel size at the last convolutional layer is $512$.

Table I shows ablation studies in three baseline networks on the CASIA NIR-VIS 2.0 database. In the table, the fine-tuned indicates the results of training only the fully connected layers while freezing the pre-trained feature extractor. For each network we attach our proposed RGM module, then experiment with NAU, and finally show the results of training with $C$-softmax. The ResNet18 fine-tuned is 88.37% in rank-1 accuracy. When the RGM extracts domain-invariant features focused on relational information, the performance improves to 96.33%. When the network trains with NAU and $C$-softmax, it shows additional performance improvement of 0.34% and 0.77% respectively which is higher than baseline with extra convolutional layer. Similarly, performance on LightCNNs are improved by 4.82% and 1.65% over fine-tuned accuracy.

In Table II, we compare our method (LigthCNN-29 backbone) with other deep learning-based HFR methods, namely HFR-CNN[59], TRIVLET[17], ADFL[13], CDL[16], WCNN[15], DSU[60], RCN[18] and RM[19]. All comparison methods use backbone pre-trained on MS-Celeb-1M or CASIA WebFace. Since RCN[18] performance in original paper used a backbone pre-trained on five large-scale databases, we reproduced RCN with LightCNN-29 pre-trained on MS-Celeb-1M as a backbone for fair comparison. The RM method, which extracts features by pair-wise relation embedding, performs better than other deep learning methods. Our method shows 0.38% performance improvement over RM and also yields results comparable with other domain-invariant based methods.

In Table III, we compare RGM with the attentional modules depicted earlier in Figure 2. We train under the same conditions, with the last feature map of LightCNN-9 and LigthCNN-29 passed through each module with cross entropy loss.The RM and graph-structured modules: GloRe [46] and RGM show higher performance than other methods, among which RGM shows 1.02% and 0.24% higher performance over second best performance in each backbone.

The IIIT-D Sketch database is designed for the sketch-to-photo face recognition task. We use the Viewed Sketch Database which comprises 238 subjects. Each subject has one image pair, a sketch and a VIS photo face image. Since there are only a small number of images for training, we train on CUHK Face Sketch FERET Database (CUFSF)[32] and evaluate on IIIT-D Sketch database, following the same protocol as in [16]. The CUFSF database includes 1,194 subjects from the FERET database [64], with a single sketch and photo image pair per subject. For testing, we use VIS photo images as the gallery set and sketch images as the probe set.

In IIIT-D database, where the domain discrepancy is large and the data is insufficient (only one pair image for each identity), the deeper the layer of the backbone, the lower the performance due to the overfitting problem [65]. As with the results on CASIA NIR-VIS 2.0 database, in Table IV, our approach improves further with the addition of RGM, NAU and $C$-softmax loss. However, when LightCNN-9 is the baseline, training with the original softmax performs 0.43% better than with the $C$-softmax. This is because the number of CUFSF and IIIT-D images is smaller than for the CASIA NIR-VIS database, so it is difficult to learn sufficiently with $C$-softmax loss and the margin values $m_{1}$ and $m_{2}$ need to be adjusted.

As we described in Table V, SIFT[9], MCWLD[61], VGG[62], CenterLoss[63], CDL[16] and RCN[18] are compared with our approach and all of these methods are use backbone pre-trained on MS-Celeb-1M (We reproduce RCN with LightCNN-9 pre-trained on MS-Celeb-1M for fair comparison). In particular, the sketch HFR database comprises artiest’s pictures, rather than the photos, making training based on deep learning difficult. Nevertheless, our method shows a rank-1 accuracy of 88.94%, the leading performance among deep learning and hand-crafted methods with same pre-trained on MS-Celeb-1M database condition.

We also apply the attention method and the graph method on LightCNN-9 and LightCNN-29 to the sketch-to-photo HFR task (Table VI). As with the NIR database, the B-CNN and DoubleAttention module show low performance making it difficult to reduce the sketch domain discrepancy via the self-attention method by simply multiplying feature vectors. In the Sketch database, which has a large domain difference and a small number of images, the RGM shows higher performance than second best performance by a larger amount of 9.79% and 4.68% on each backbone. The RGM, which extracts relational information with small parameters, prevents overfitting and outperforms even with a small database.

The BUAA-VisNir database is composed of NIR and VIS face images of 150 subjects. Each subject has nine NIR and VIS images including one frontal view, four different other views, and four different expressions (happiness, anger, disgust and amazement). These NIR and VIS images are paired and captured simultaneously. The training set comprises 50 subjects with 900 images. For testing, 100 subjects with one VIS image each make up the gallery set, with 900 NIR images in the probe set.

In Table IV, the performance with LightCNN-9 and 29 incrementally improves over baseline when the RGM module, NAU, and conditional margin loss $C$-softmax are added. With $C$-softmax loss, the performance improves 2.45% and 0.11% respectively. On the other hand, when NAU is added to the ResNet18 baseline, the performance decreases because it becomes more difficult to learn the global node correlation with fewer training set subjects. Our approach brings performance improvements 2.78%, 1.55%, and 2.23% over fine-tune in the three baselines.

Table VII compares our method with three other types of method (projection based, synthesis based and domain-invariant base method) on H2 (LBP3)[66], TRIVLET[17], ADFL[13], CDL[16] and WCNN[15]. Our method shows better performance than other domain-invariant feature methods such as WCNN, TRIVLET that focus on features themselves rather than relationships.

Oulu-CASIA NIR-VIS facial expression database[54] consists of 80 subjects with six different expressions and three different illumination conditions. Following the protocols in [15], we randomly selected 40 subjects and eight images from each NIR and VIS domain for six expressions. The train set and test set are 20 subjects each. For test set, all 960 number of VIS and NIR images of the 20 subjects are used as gallery and probe set.

TUFTS NIR and VIS database [55] consists of 100 subjects with large pose variations. The number of images per subject is less than 36, and each subject is photographed at nine equidistant positions around a semicircle with a fixed viewpoint. Since there are no protocols or comparison papers for training and test settings, we cropped all faces to 144x144 and use identities 1–25 as the test set and identities 26–100 as the training set.

The Figure 8(c) and 8(d) show visualization of relationship of face components in different poses in TUFTS database. In the figure, regardless of domain and pose, the strong relational components are similar within subject. The detailed explanation of the visualization will be provided in Section IV-F2.

Since Oulu-CASIA and TUFTS databases have too small identities (number of 20 and 25) which performances are 100% and 99.66% rank-1 accuracy respectively, we only conduct experiments with ablation studies or visualization. The performance on Oulu-CASIA database increases as our method is added, but saturated (Table IV). This is because it has fewer subjects, and it has multiple gallery images per subject, unlike other HFR databases which have only one gallery image per subject.

We compare our conditional margin loss ($C$-softmax) to other angular margin losses such as Normalized-softmax[51, 68], A-softmax[67], CosFace[52] and ArcFace[53] using LightCNN-9 as a baseline and training under the same conditions (the embeddings are normalized and scaled $s$ = 24); the margin for each loss follows each study. In Table VIII, on the CASIA NIR-VIS 2.0 database, performance of ArcFace and our C-softmax show better performance than the other losses at 97.97% and 98.03% because of the different margins for the class cosine similarity, as shown in Figure 5. ArcFace reduces the margin when the class cosine similarity is large or small and increases it near the midpoint (Figure 5(b)), while $C$-softmax increases the margin at larger class similarity values (Figure 5(c)). This helps to control classes with domain discrepancy because it effectively adjusts the margin between inter-classes.

In Table VIII, we also conduct experiments with LFW [9], a large-scale visual face database. For this purpose, we use CASIA WebFace [47] consisting of 10,575 subjects for the training dataset and perform evaluation in LFW. All the implementation details are same as ArcFace and the margin and scale factors of each loss function are set to the optimal values presented in the original paper. The performance of each loss function is reproduced and the value in parentheses is the performance written on each paper. The results show that $C$-softmax achieves best performance with 99.58%, followed by the CosFace and ArcFace. In addition, we conduct experiments on a small-CASIA WebFace database with 5,287 subjects, half the size of the CASIA WebFace. The performance gap between $C$-softmax and other losses is larger in small-CASIA WebFace than in CASIA WebFace. This result shows that $C$-softmax improves training effectiveness when we train on the dataset with a small number of classes.

As mentioned in Section III-B2, when obtaining a directed relation between nodes, all edge values are passed through an activation function. In this case, we use a sigmoid activation function instead of softmax because relation information for each node is independent of and should not be influenced by other nodes’ relations. Unlike sigmoids where $f_{sigmoid}(\boldsymbol{x}_{i})={1}/({1+e^{-\boldsymbol{x}_{i}}})$, softmax $f_{softmax}(\boldsymbol{x}_{i})={\boldsymbol{x}_{i}}/{\sum e^{-\boldsymbol{x}_{j}}}$ looks at the interrelation of all values. Table IX shows the results of experiments in which the activation function of RGM is varied. We use LightCNN-9 as a baseline with the CASIA NIR-VIS and IIIT-D Sketch databases. For these databases, the rank-1 accuracy is increased by 0.08% and 0.85% compared to softmax.

We visualize node relation $\boldsymbol{A}_{i,j}$ in Equation 2 which is the directed relations of nodes in the RGM. In Figure 8, the first row shows VIS and NIR pair, while the second row shows VIS and Sketch pairs. In each image, we select one reference node vector (red color) and visualize five strong relational node vectors (green color) of it. Regardless of domain, within a subject, the strong relational components in the gallery and probe images are similar. For example, the subject in Figure 8(a), the left eye has a strong relationship with right eye, nose and wrinkles; the subject in Figure 8(b), the left eye has a strong relationship with wrinkles, nose, and mouth; In the case of different subjects, when comparing (a) and (b), the strong relationship components with the left eye are different. Also, in case of pose variation, both gallery and probe in Figure 8(c), nostrils have a strong relationship with left eyebrow and eye, whereas right eye has a strong relationship with nose in Figure 8(d). These relationships are obtained by passing the gallery VIS image and the probe NIR or Sketch image separately to the RGM, revealing each identity has similar relationships in faces regardless of domain. Additional visualization results are presented in the Supplementary Material.

Nodes whose relational information is propagated through the RGM are node-wisely recalibrated through the NAU. Figure 9 shows the scale value computed for node-wise recalibration from the NAU ($s$ in Equation 4). Each row corresponds to samples of four subjects from test set; in each subject, the first row is a gallery and other rows are probe sets. Looking at the gallery and probe set, we can observe that the nodes are similarly focused for each subject. In other words, regardless of domain, the importance pattern of relation propagated nodes (output of RGM) are different across subjects and similar within each subject.