Harnessing Unlabeled Data to Improve Generalization of Biometric Gender and Age Classifiers

AI and computer vision has reached inflection point with advances in deep learning. However, these high accuracy rates are obtained at the cost of large number of parameters involved which in turn require large scale labeled (annotated) datasets for network training and learning optimal set of parameters.

The availability of large-scale annotated datasets may be limited due to the requirement of manual annotation by the human expert which is a cumbersome and time consuming operation. Further, due to the nature of field, large datasets may not be publicly available, for instance, in the medical field, due to the privacy and security concerns. Training using limited datasets impact the generalization ability of the deep learning models due to overfitting that arise due to small number of data points.

A number of studies have explored the automatic extraction of demographic attributes such as gender, age, ethnicity, etc. of an individual, known as soft-biometrics [1]. These attributes have been deduced from biometric data such as facial images, voice, gait, and hand or body images. Automated soft-biometric prediction has drawn significant interest in numerous applications such as surveillance, human–computer interaction, anonymous customized advertisement system, image retrieval system, continuous user authentication, subject re-identification, and in fusion with primary biometric modalities for performance enhancement [2].

Studies have been conducted to improve the prediction accuracy of soft biometrics in different biometric traits [3] using deep learning models. For face biometrics, studies have been conducted for deducing gender [4, 5, 6], age [4, 5, 7], and ethnicity [6, 8] from facial images. Soft biometrics such as gender [9, 10, 11, 12, 13], ethnicity [10, 12, 14], age [15, 16], and eye color [17, 12] have been predicted from periocular and iris in visible and near infrared spectrum.

Studies in biometric recognition have shown steady improvement in matching performance by combining soft biometric attributes with primary biometrics [18, 19, 1]. In [1], showed around $5\%$ improvement in identification system and around $4\%$ improvement in verification system by combining primary biometrics with predicted soft biometric traits such as ethnicity, gender, and height.

Recent advancements in soft biometric predictions are mainly based on deep learning for feature extraction and classification such as in  [4, 12, 11]. These deep learning methods require a large amount of training data to obtain high generalization accuracy rates. However, acquisition of large biometric datasets along with soft-biometric attributes such as gender, race and age, may invokes privacy and security issues.

One way to overcome this problem is by using semi-supervised learning [20]. Semi-supervised learning entails joint use of labeled and unlabeled data in improving the generalization performance of the classifier.

In this paper, we proposed a self-ensemble based semi-supervised learning method which improve the generalization ability of the deep learning model trained on limited labeled data and large amount of unlabeled data. The proposed method develops a joint loss function that uses categorical cross-entropy for the labeled samples while reducing the distance between the model outputs of two different perturbations on the unlabeled data. Figure 2 show the schema of the proposed self-ensemble model. Many of the semi-supervised methods propose to use noise to induce perturbations [21, 22]. However, using noise might affect the features which are used in predicting soft biometric attributes to be deduced from biometric traits. Therefore, we proposed to use data augmentation to generate different perturbations from unlabeled data.

The contributions of this paper are two-fold:

• •

  The proposal of self-ensemble based deep learning model which harness unlabeled data to improve the generalization ability of the soft-biometric prediction.

• •

  Experimental evaluations are conducted for gender prediction on eye region images from VISOB dataset and face images from CelebA dataset. We have also evaluated age-group prediction on face images from Adience dataset. We show the deep learning models can be trained using as low as $100$ labeled samples using the proposed semi-supervised learning method.

The rest of the paper is organized as follows: A brief explanation of semi-supervised learning with recent deep learning based advancements is provided in section-II. The proposed self-ensemble method is discussed in section-III. Experimental setup and datasets are presented in section-IV. Results are discussed in section-V and paper is concluded in section-VI.

Semi-supervised learning is a learning technique, which make joint use of labeled and unlabeled data for training a classifier, have a long history in machine learning research [20]. Specifically, the model trained on labeled data is used to generate pseudo-label for the unlabeled data. The pseudo-labeled unlabeled data is used for re-training the classifier for enhanced accuracy. For misclassification in psuedo-labeled data, the deep learning models have proven to be good at dealing with noisy labels [23].

These semi-supervised learning methods can be broadly divided into three types as follows:

(1) Self-training: In self-training based semi-supervised learning, a single model trained on a small labeled dataset is used to generate pseudo-labels for unlabeled dataset. The labeled data together with pseudo-labeled unlabeled data is used to re-train the model [24, 25]. Self-training is one of the earliest methods in semi-supervised learning and is used for performance enhancement of the classifier.

(2) Co-training: Co-training is a suitable semi-supervised learning approach that needs the dataset to be characterized by two different interpretations of features. In co-training, joint use of two-classifiers is used for pseudo-label generation and classifier re-training on unlabeled dataset. One way of performing co-training is one or more models $m_{i=1\to n}$ are used to predict proxy (pseudo-labels) labels for another model $m_{j\neq i}$[26]. Another way is by doing a majority voting on the predicted labels from two or more models to generate a proxy label for unlabeled dataset which is used for classifier re-training [27].

(3) Self-ensemble: This training process is similar to the co-training, where instead of using multiple classifiers to predict the pseudo-labels, only a single model with different perturbations of input samples is used. In self-ensemble, the model is trained using both labeled and unlabeled data during the training stage. Rasmus et al. [21] proposed to train model robust to noise by reducing the prediction error between noisy and clean input of unlabeled dataset while predicting targets for the labeled dataset in a supervised manner. The noise is induced by adding Gaussian noise to the input image as well as to the output of each layer in the model. Laine et al. [22] applied two different perturbations using Gaussian noise and augmentations to unlabeled samples and used dropout in deep learning model instead of Gaussian noise.

Our proposed method is based on self-ensemble technique, where instead of using noise as a perturbation in the previous models, we propose to use only data augmentations such as color jitter, flipping the image horizontally and using random translations. The proposed method is discussed in more detail in Section-III.

Block diagram of the proposed self-ensemble method is shown in Figure 2. In the proposed method, the deep learning model is trained consecutively in supervised manner on data $X_{i}$ with labels $T_{i}$ and in unsupervised manner by reducing output error between the pseudo-labels $z^{{}^{\prime}}_{j}$ and $z^{{}^{\prime\prime}}_{j}$ of two augmented samples $Z^{{}^{\prime}}_{j}$ and $Z^{{}^{\prime\prime}}_{j}$ generated from the unlabeled data $Z_{j}$. This is done so that the supervised part of the training will ensure the model learns to predict the correct target class for the given input while unsupervised part ensures the model learns to produce consistent output.

Data Augmentation for Perturbation: As we mentioned in Section I, the proposed method is based on self-ensemble based semi-supervised learning, where two output images were generated by inducing different kind of noise to the input image. However, inducing noise may distort the features in biometric traits, such as ocular and fingerprint images. We therefore propose to use data augmentation. In our experiments, we incorporated two different augmentations: firstly, we applied color jitter by changing the brightness and saturation of the image. Secondly, we randomly flipped the input image horizontally and finally, translation is applied by randomly cropping a region out of the image. This way, from data $Z_{j}$, we generate two perturbed samples $Z^{{}^{\prime}}_{j}$ and $Z^{{}^{\prime\prime}}_{j}$ to train the deep learning model for the unsupervised part. Using data augmentation around 108K unlabeled samples for VISOB dataset and 199k unlabeled samples for CelebA dataset were generated.

Proposed Deep Learning Model: In our experiments, we used a simple sequential convolution neural network (CNN) based on visual geometry group (VGG) [28] architecture. Table 1 shows the proposed deep learning model used in our experiments. In all our experiments, we used a single channel (grayscale) image of size $128\times 128$ pixels as an input, from which we extract $128$ features by taking the global average of the output of the last convolutional layer. This is followed by the final fully connected layer (dense layer) for predicting the soft-biometric attributes. The proposed deep learning model is relatively small with only $877.7K$ parameters as shown in Table 1.

Loss Function: For a given labeled data $X_{i}$ with targets $T_{i}$, the model predicts $T^{{}^{\prime}}_{i}$ as output, then we use cross-entropy loss as shown in equation-1 for supervised part of the model training.

Let $z^{{}^{\prime}}_{j}$ and $z^{{}^{\prime\prime}}_{j}$ be the predictions from the model for the given two perturbed samples, $Z^{{}^{\prime}}_{j}$ and $Z^{{}^{\prime\prime}}_{j}$, for given unlabeled input $Z_{j}$, then the model is trained in an unsupervised manner by taking mean square error (MSE) between two model predictions as shown in equation 2.

Finally, the proposed model is trained by joint combination of cross-entropy and mean square error loss functions as shown in equation 3.

Where $\alpha$ is coefficient for unsupervised loss (MSE) and in all our experiments, we used $\alpha=1.0$ based on empirical evidence.

Table II and Table III show gender prediction accuracy for supervised and self-ensemble model for face images from CelebA dataset and ocular images from VISOB dataset, respectively. The number of labeled and unlabeled samples and the obtained accuracy values using supervised as well as self-ensemble based models are also shown. It can be seen that using the low number of labeled samples, $100$ and $200$ samples, the supervised model performance is dropped with about $30\%$ accuracy in predicting the gender. Whereas, with the proposed self-ensemble model better capability at prediction the gender while at least $15\%$ improvement in total prediction accuracy (ACC%) could be obtained.

It can also be seen that the accuracy of the proposed self-ensemble learning with only $1000$ labeled samples obtained $81\%$ accuracy on VISOB dataset and $94.46\%$ on CelebA in gender prediction, which is very close to the model trained in a supervised manner with all the training dataset as labeled samples with $84.65\%$ and $97.91\%$ accuracy for VISOB and CelebA, respectively. The experimental results demonstrate that the self-ensemble model for gender prediction trained using both limited labeled data (as low as 100 samples) and unlabeled data (about 108K-199K) obtains performance equivalent to supervised model trained on all 108K-199K labeled samples.

Table IV show the age-group prediction on Adience dataset with the deep learning model trained using supervised learning (with about 11K samples) as the baseline. The comparative analysis is done with the proposed self-ensemble model for 5-fold cross-validation. It can be seen that the supervised deep learning model trained on 11K samples can obtain exact accuracy of $51.63\%\pm 3.66$ which already is $1\%$ higher than the model proposed in [4] with exact accuracy of $50.7\%\pm 5.1$. On top of that, by using our proposed self-ensemble learning (using 11K samples as labeled and unlabeled data equally divided), better exact accuracy of $55.55\%\pm 4.28$ which is $3.92\%$ more than the baseline, could be obtained.

Table V show the gender prediction performance comparison of the proposed self-ensemble method with the FixMatch [33]. Both the models were evaluated on face images from the CelebA dataset.

The experimental results on gender and age prediction from face and ocular images suggest the efficacy of harnessing unlabeled data towards improving generalization accuracy of the deep learning models.

The generalization ability of the deep learning models is proportional to the use of large scale and representative datasets for model training. However, labeling the large scale dataset require human operator and is very expensive and time consuming. In this paper, we proposed a self-ensemble based semi supervised learning model which could be trained using labeled data in a supervised manner and can consecutively harness the unlabeled data in an unsupervised fashion using multi-objective loss function. A case study on gender and age-group prediction using face and ocular images from CelebA, Adience and VISOB datasets suggest increase in the generalization ability of the proposed deep learning model when trained with small amount of labeled data (as less as 100 samples) along with large unlabeled dataset. The deep learning models with the ability to harness unlabeled data also have applications in few shot learning [34], bias mitigation towards unrepresentative sub-population in the dataset [35], and model adaptation to the dynamic and temporal variations of the operational data [36]. As a part of future work, comparative analysis of the proposed self-ensemble method, implemented using different backbone architectures such as ResNet and MobileNet, with the other semi supervised learning methods like MixMatch [37], Noisy Student [38] and FixMatch [33] will be performed.