Locality-aware Channel-wise Dropout for Occluded Face Recognition

The occluded face completion methods are pixel-level approaches which aim to recover the occluded face regions. Considering the low-rank property of non-occluded images, early works attempt to solve the problem by using robust principal component analysis (PCA) to reconstruct the corrupted low-rank face images [19, 20, 21, 22, 23]. The work in [21] proposes the robust principal component analysis (RPCA) which improves the performance of removing shadows from face images. In [22], an important extension of RPCA, namely the low-rank representation (LRR) is presented to extend the recovery of corrupted images from a single subspace to multiple subspaces. Both RPCA and LRR assume that the occluded pixels are sparse, while real-world face images usually contain dense occlusions, which make the matrix non-sparse. To this end, [23] presents the double nuclear norm-based matrix decomposition to remove the dense occlusions.

Recently, more works resort to the deep learning for improving the occluded face completion [1, 2, 3, 4, 5, 6, 24]. In [1], the difference between the activation values of two stacked sparse denoising auto-encoders (SSDAs) are used to indicate occluded and un-occluded face regions. Then, the final occlusion-free image is reconstructed by transferring the encoding activations of the un-occluded region to the occluded region. In [3], the context encoder combines the auto-encoder architecture with context information of the occluded part to produce visually pleasing images. To further improve the context encoder, [4] introduces both global and local discriminators. Specifically, the global discriminator pursues the global consistency of the overall image and the local discriminator looks at a small area centered at the reconstructed region to judge the quality in details. To ensure the new generated contents more photo-realistic, a semantic parsing loss is developed in [5]. To refine local face textures, a 3D morphable model (3DMM) is utilized in [6] to further assist the learning of the local discriminator. In the unsupervised face normalization method (FNM)  [24], multiple local discriminators are integrated into a novel unsupervised framework. It generates impressive high quality faces which have dispelled various face variations including occlusions.

Overall, the above-mentioned face completion methods have shown promising results of transforming occluded faces to un-occluded ones. However, two major issues still remain. Firstly, except for a few methods (e.g., the FNM [24]), most previous methods require pair-wise training data (i.e., one occluded face and one un-occluded face of the same person). Unfortunately, such training sets containing a mass of pair-wise faces with natural occlusion are extremely rare. An alternative and commonly used approach is using synthetically occluded faces. However, these synthetically occluded faces, e.g., using manually designed occlusion templates or a black/white rectangle can not fully represent natural occluded faces. Secondly, the faces generated by GAN-based methods are usually visually pleasing. However, how to remove the occlusion while preserving the identity information still remains a challenging problem.

These approaches aims to remove the noise caused by occlusions with two pipelines, which either discards the occluded pixels before the face feature extraction, or discards the corrupted feature elements during the feature extraction.

Following the former pipeline, some works detect the occlusions first and then extract a feature representation from the non-occluded regions only. The early works [25, 8, 9, 10] usually employ a nearest neighbor classifier (NNC) or a support vector machine (SVM) to classify the occluded face regions. Since these methods are designed based on the traditional feature descriptors, it is non-trivial for them to obtain discriminative ability for face recognition in complex scenarios. Recently, the LPD [13] designs a neural network to locate the latent facial parts which are less affected by a specific occlusion (i.e., the respirator), and then extracts discriminative features from the selected latent part.

Following the spirit of the latter pipeline, the mask leaning methods [11, 12] locate and discard the corrupted feature elements rather than the occluded pixels. In [11], the MaskNet adaptively learns feature masks for occluded face images and automatically assigns lower weight to the hidden units activated by the occluded face regions. The PDSN [12] establishes a mask dictionary to represent the correspondence between the occluded facial block and the corrupted feature elements. During the testing phase, a segmentation network is employed to detect the occluded facial blocks, and then the corrupted feature elements are set to zero by retrieving the relevant dictionary items.

Although above-mentioned occlusion-aware discarding methods can alleviate the occlusion issue, completely discarding the occluded regions still takes the risk of reducing the system reliability. Firstly, precise and fine-grain occlusion detection is an essential prerequisite for these methods. However, such occlusion detection is non-trivial to obtain and a coarse detection will increase the risk of losing discriminative information or inducing unreliable information. Furthermore, even if the occluded face regions or the corrupted feature elements have been located perfectly, directly zeroing out the corrupted elements still incorporates a peculiar pattern into the final feature, which may harm the recognition accuracy. Moreover, due to the arbitrary occlusion, the feature vectors for occluded faces have unfixed number of valid elements. Thus, the conventional metrics for fixed-length vectors are not fully applicable. Although works on the instance-to-class distance [26] and the reconstruction-based similarity measurement [27] are proposed to tackle the above problem, the commonly used window sliding makes them relatively time-consuming. On the other side, directly ignoring occluded elements will potentially break the global cues of face images such as chin contours, which is also harmful to face recognition system.

Directly learning occlusion-robust feature is the most straightforward and effective way to handle occlusion face recognition. For this purpose, many efforts are devoted to seeking a feature space that is less affected by occlusion and meanwhile preserves the discriminative capability of distinguishing identities. Many works seek such feature space via the sparse representation classification (SRC) [28], in which the occluded image is represented by a linear combination of training samples plus a sparse constraint term accounting for occlusions. The LR-LUM [29] combines both the robust sparsity constraint and the low-rank constraint which outperforms previous methods in handing structured occlusion such as sunglasses and scarfs. Other works [30, 31, 32] extend the sparse representation by combining more discriminative feature descriptors. In [32], the JCR-ACF proposes a joint and collaborative representation with local adaptive convolution feature, which can improve the recognition accuracy by employing information from different local face regions. Although these SRC-based methods have made considerable progresses, they do not generalize well in practical scenarios since they requires that the test samples have identical identities with a pre-defined close-set.

Owing to leveraging massive training sets, recent deep learning methods reveal significant superiority on face recognition. Enlarging the training datasets with sufficient occluded faces may be an effective way to improve the occlusion robustness of face embedding. Unfortunately, such training sets of a mass of identities are extremely rare. As an alternative solution, synthetic image augmentation method has been studied by previous works [14, 15, 16, 17]. The work in [14] enriches the training set by synthesizing occluded faces with various pre-defined hairstyles and glasses templates. Although an accuracy gain has been witnessed, the diversity of the manually designed occlusion templates needs to be further improved. More recently, BFL [17] proposes an enhanced augmentation schema to randomly generate multi-scale spatially occluded samples and then modifies the loss to balance the impact of normal and occluded samples for training. To some extent, these augmentation methods improve the robustness of neural network, but the discrepancy between the synthetic and real occluded faces still limits the further improvements of the robustness.

Another line of researches [33, 34, 35] focus on the attention mechanism for robust feature extraction. The state-of-the-art method named InterpretFR [35] employs a Siamese network to compare the feature elements from a normal face with and without synthetic occlusion. Then it encourages the neural network to identify the input face solely based on the feature elements which are less sensitive to occlusions. However, it still need synthesizing occluded faces by using artificial occlusion templates and fails to generalize well on unseen occlusions. In contrast, our method can realistically simulate arbitrary occlusions via dropping out a random group of filter responses, leading to an improved performance under real world scenarios.

The proposed method attempts to learn occlusion robust face features by simulating occlusions during the training process. Different from previous methods which augment face images with synthesized occlusions, we propose to directly simulate the influence of arbitrary occlusions on intermediate features. As the occlusion essentially damages a group of neurons, we propose a revised dropout method, namely the locality-aware channel-wise dropout (LCD) to simulate occlusions by dropping a group of feature channels. Considering that, for the conventional neural networks, the occlusion usually affects the activations from most channels which is even impossible to simulate the occlusions by dropping several channels. Therefore, we first employ a spatial regularization to encourage each feature channel to respond to different face regions. In this way, the activations affected by the partial occlusion are more likely to be located in a single feature channel. Then, our LCD can simulate the occlusion by dropping out the entire feature channel, in which sense we name this method as locality-aware channel-wise dropout. Moreover, to improve the contributions of non-occluded regions for learning occlusion robust features, we design an auxiliary spatial attention module to reweight the feature channels. The whole framework shown in Fig. 1 is end-to-end trainable and can be easily applied on any existing convolution neural network.

In an occluded face image, the feature channels responded to the occluded regions will not contain discriminative information for face recognition. Besides, they potentially contains the noise from the occluded regions. To be free from this, we design an attention module to reweight the feature channels, making the network focus more on the feature channels which are related to the non-occluded regions. It is worth noting that, by utilizing the aforementioned spatial regularization, each feature channel is encouraged to respond to local and different face regions. In this sense, reweighting these feature channels actually performs as a spatial-wise attention approach, which is named as the spatial attention module (SAM).

For the input feature map $\mathbf{F}=[\mathbf{f}_{1},\mathbf{f}_{2},...,\mathbf{f}_{c}]$, our goal is to learn an attention vector $\boldsymbol{\theta}=[\theta_{1},\mathbf{\theta}_{2},...,\mathbf{\theta}_{c}]$ which controls the weight of each feature channel. As shown in Fig. 1, a light-weight module is designed to learn the attention vector $\boldsymbol{\theta}$. Specifically, we first employ a $1\times 1$ convolution layer and a fully-connected layer to obtain a global view of all input feature maps. Then, another fully-connected layer is utilized to extract the channel-wise attention vector. It is worth noting that previous attention block in SENet [36] utilizes a global average pooling as a global descriptor, while in our method the global pooling of dropped zero value feature maps will suffer from local minimum. To this end, the $1\times 1$ convolution layer in our proposed module is a necessary and effective information aggregation strategy to achieve attentions.

The refined feature map $\widetilde{\mathbf{F}}$ is obtained by channel-wise multiplication between each feature map $\mathbf{F}_{x}$ and its corresponding attention scalar $\mathbf{\theta}_{x}$:

Formally, the Algorithm 1 summarizes the training process using the locality-aware channel-wise dropout and the spatial attention module.

Both our method and DropBlock drop some activations in the intermediate feature layers, but they differ in two aspects. 1) The Dropblock drops partial regions within each feature channel for enhancing feature learning while our method employs spatial regularization to enforce each feature channel to respond to local face regions and further drop several feature channels to simulate partial occlusions. 2) Our method further proposes spatial attention module to improve the contributions of undamaged neurons to further promote the robustness to occlusion. Experimental results on IJB-C datasets show our method significantly outperforms Dropblock for occluded face recognition.

Both our method and the weighted channel dropout (WCD) drop channels but for different purposes and in different ways. Firstly, in terms of designing purpose, the WCD is for alleviating the over-fitting issue when fine-tuning neural network on small datasets, while our method aims to improve the robustness to partial occlusion for face recognition. Secondly, in terms of what channels to drop, the WCD drops out the feature channels which have relatively lower activation magnitudes, while in our method each feature channel is enforced to respond to features corresponding to some local facial region and thus can be dropped to simulate the feature corruption due to the occlusion of that region. Comparison experiments on IJB-C datasets show the superiority of our method in handling occluded face recognition problem.

Both our method and InterpretFR resort to occlusion robust feature learning for tackling face recognition under occlusions. However, we have two major differences. 1) InterpretFR synthesizes occlusions by putting black boxes on faces, which is unrealistic and the performance may severely degenerate under real world scenarios. Instead of simply utilizing fixed templates to synthesize occlusion, we propose the locality-aware channel-wise dropout to simulate more realistic occlusions, leading to performance improvement for occluded face recognition. 2) During training, InterpretFR masks out the elements of the final face representation sensitive to the occlusion, which may be sub-optimal as it only leverage remaining features for recognition without considering information recovering. Differently, we perform channel-wise dropout on the stage 3 of ResNet and design spatial attention module in the stage 4 to implicitly recover the absent information as verified by experiment in Section V-C, which is more favorable to tackle face recognition under occlusions.

The training images are collected from the MS-Celeb-1M dataset [39]. Since this dataset contains many labeling noise, we manually clean it and finally collect 3.7 Million images from 50K identities. The revised dataset is used as our training set for the experiments. We extensively test our method on three popular benchmarks, including IJB-C [40], MegaFace [41] and LFW [42].

IJB-C. As an extension of previous IJB-A [43] and IJB-B [44], the IJB-C [40] is a large-scale dataset which contains 117, 542 video frames and 3, 134 images. As 57% of the face images in IJB-C are natural occluded, this dataset is a commonly used benchmark for occlusion-robust face recognition. In this work, we employ two evaluation settings. First, we conduct comparisons on the holistic IJB-C dataset (including both occluded faces and non-occluded faces) for general evaluations. Second, to further verify the effectiveness of tackling occlusions, the occlusion subset of IJB-C (the occluded face images only) is employed for evaluations. All the two settings follow the standard IJB-C testing protocol. The true accept rate (TAR) and the false accept rate (FAR) are used as the evaluation metrics.

MegaFace. The MegaFace challenge 1 (MF1) benchmark [41] evaluates how the face recognition method performs with a huge scale of distracters. Specifically, the gallery set in MF1 contains one million face distractors, and the probe set Facescrub [45] contains 106,863 face images of 530 identities. In the testing pipeline, each Facescrub image will be added into the galley set and the remaining images of the same identity are exploited as probes. The rank-1 identification accuracy is used as the measurement of the face recognition performance. It should be noted that the un-cleaned version of MegaFace datasets are employed in the evaluation for the fair comparison with the state-of-the-art methods.

LFW. The LFW [42] is a well-known unconstrained face verification benchmark. It contains 13,233 images form 5,749 identities. In our work, we follow the standard 10-fold cross validation protocol to report the mean accuracy on the 6,000 testing image pairs.

In our experiments, the face images are aligned based on five facial landmarks detected by [46] and normalized to a size of 112$\times$112. We employ the ResNet-50 [47] as the baseline network and implement our method on the Tensorflow [48] platform. The Arcface loss [49] with the margin of 0.5 and the scale of 64 is utilized as the identification loss in our experiments. All the models in our experiments are trained on four NVIDIA TITAN XP GPUs by using SGD. The loss weight of the ArcFace loss is set to 1 and the loss weights of the filter orthogonal regularization and the response orthogonal regularization are 100 and 1, respectively.

We firstly investigate the best stage to deploy the channel-wise dropout. By integrating the channel-wise dropout into three different stages of the plain ResNet-50, respectively, we construct experiments under three settings. Specifically, the channel-wise dropout is conducted on the outputs of the last $3\times 3$ convolution layer in the stage specific to each setting.

Table I summarizes the results on the IJB-C occlusion subset of the three different settings. As seen, the channel-wise dropout in the stage 3 outperforms the baseline by an improvement up to 18.19% in terms of TAR when FAR = 0.0001. This setting works better than the channel-wise drop conducted in the shallow stage 2. The reason behind it is that features in stage 3 have larger receptive field than those in stage 2, which can well characterize facial components, leading to better simulation of partial occlusions on faces. Besides, conducting channel-wise dropout in the stage 4 performs severely worse than that in the stage 3, or even worse than the baseline model. We argue that employing several succeeding neural layers to compensate the damaged activations plays an important role for face recognition under occlusions. We conduct a further experiment to verify this analysis as bellow.

For a face recognition model, we compare the similarity of output features from stage 4 with and without conducting channel-wise dropout in stage 3, as shown in Fig. 3. The features similarity is evaluated by mean square error (MSE) as shown in Eq. 8.

where the $L$ denotes the length of the final feature vector and $f_{i}$ denotes the $i$-th elements within the feature vector.

We compare the baseline mode and our model trained with the channel-wise dropout integrated in the stage 3. The results are shown in Fig. 4. As seen, our model trained with channel-wise dropout integrated in the stage 3 achieves notably lower MSE than baseline, which means the final features of our method on “occluded face” are much closer to those on clean face, leading to occlusion-robust face recognition model. The result proves that after integrating the channel-wise dropout in the stage 3, the succeeding layers in the following stage 4 play a crucial role in compensating the absent of facial information caused by occlusions. This experimental results also explain why conducting the channel-wise dropout in the stage 4 leads to poor results in Table I. If the occlusion is simulated in the last stage, no succeeding layers are enforced to learn features robust to occlusions. Based on this analysis, the channel-wise dropout integrated into the stage 3 of the ResNet-50 are treated as the best setting for all the following experiments.

We conduct ablation studies to evaluate the effectiveness of each component in our method. The results on the IJB-C occlusion subset are shown in Table II. As seen, all the three components are beneficial to the performance improvement. Specifically, when only the channel-wise dropout is conducted, the TAR when FAR= 0.0001 is improved by 18.19%. Moreover, by jointly training with the spatial regularization, a further improvement of 6.83% is witnessed. Furthermore, by employing all the three components, our method achieves a significant improvement up to 36.58% in terms of TAR when FAR = 0.0001, which demonstrates the effectiveness of each component of our method. For simplicity, our method consisting of all the three components is abbreviated as LCD in the following experiments.