Unsupervised Deep Learning Image Verification Method

As shown in Fig 1, an autoencoder comprises an encoder and a decoder. The encoder functions to compress the input face image vector $x$ into a lower-dimensional space, while the decoder reconstructs it back to its original form. Conventional training involves minimizing the Mean Square Error (MSE) between the input $X$ and the reconstructed $X^{\wedge}$.

In this work, we proposed the training of an autoencoder to minimize the loss function $MSE\left(X^{\wedge},N\right)$, illustrated in Figure 1, where $N$ represents a face image vector similar to $X$ and $X^{\wedge}=$ decoder $(\operatorname{encoder}(X))$. We put forth a method for the automatic selection of similar face image vectors. For each training input face image vector $X$, multiple similar face image vectors can be taken into consideration. Thus, each images in the dataset could have different number of similar images. In Section 4, we will conduct a comparative analysis between the results obtained from our proposed training approach and the conventional training method, which involves reconstructing the same training face image vectors.

All the training face image vectors are assessed against one another using the cosine scoring technique. For each face image vector, a group of similar face image vectors is chosen. A simple method for selecting these similar face image vectors involves setting a threshold on the cosine scores between the training face image vectors. Face image vectors with scores surpassing this threshold are designated as similar face image vectors. It’s worth noting that this approach may result in a varying number of similar face image vectors for each training face image vector.

Another approach is to adopt a constant value, denoted as k, and choose k similar face image vectors for each training face image vector. This method ensures a consistent number of similar face images for every training face image vector, resulting in a balanced training approach.

Figure 2 provides a visual representation of the process for selecting similar face image vectors when a constant $k$ number of neighbors is considered. Let $X_{i}$ be a training face image vector, where $i=(1,\ldots,n)$ and $n$ represents the total number of training face image vectors. Initially, we compute cosine scores for all training face image vectors relative to one another. Subsequently, we pick the top $k$ face image vectors with the highest scores to serve as similar face image vectors for each $X_{i}$. These selected similar face image vectors are denoted as $N_{ij}$, where $N_{ij}$ is the $j^{\text{th }}$ similar of $i^{\text{th }}$ training face image vectors.

Thus, we have a total of $n\times(k-1)$ training data, which automatically generated on the fly which address the issue of having large training data. We run extensive experiment to determine the the values of threshold and $k$. Thus, the autoencoder is able to learn information about the intrinsic variability of face image vectors, without necessarily using actual face image labels.

Consequently, we accumulate a total of $n\times(k-1)$ samples for the autoencoder training process. The specific values for the threshold and $k$ are established through experimental investigation, and a detailed discussion on these parameters will be provided in Section 4. Through this approach, the autoencoder effectively assimilates information regarding the session variability inherent in face image vectors, all without the dependency on actual face image labels.

After the autoencoder is trained with the chosen neighbor face image vectors, we proceed to convert the testing face image vectors into a fresh face image vector representation, utilizing the autoencoder, as illustrated in Figure 1. At this stage, we extract the target face image vectors from the output of the autoencoder. In the experimental trials, these extracted vectors have demonstrated an enhanced discriminatory capability for face image vectors, all achieved without reliance on face image labels. Leveraging these extracted vectors, we carry out the experimental trials using the cosine scoring technique.

For the training of our model, we utilized the Keras deep learning library [29]. The architecture of the autoencoder employed in this work comprises three hidden layers. This design maintains symmetry between the encoder and decoder sections. Specifically, both the first and third hidden layers contain an identical number of neurons, as illustrated in Fig. 2. Each of the encoder input and decoder output layers consists of 112 by 112 neurons. In the second layer of both the encoder and decoder, there are 800 neurons. The encoder’s output layer is composed of 300 neurons, determining the size of the resulting embedding vector. Within the DNN, the dimension of the face image embedding layer was fixed at 400, while the classification layer encompassed 1000 neurons.

The training of the autoencoder was conducted over 400 epochs or until the error stops decreasing and showed no further decrease. In all layers of the autoencoder, we applied the Rectified Linear Unit (ReLU) activation function, with the exception of the final layer, which utilized a linear activation function. The learning rate was set to 0.03, incorporating a decay rate of 0.0002. We implemented a logarithmically decaying learning rate, spanning from $10^{-2}$ to $10^{-8}$. The batch size was configured at 100.

The CelebA dataset, introduced by Liu et al. [9], encompasses a vast collection of over 200,000 images featuring 10,177 celebrities. This dataset incorporates diverse elements such as pose variations and background clutter, providing a comprehensive representation of real-world scenarios. To facilitate model training and evaluation, the dataset is partitioned into distinct sets for training, validation, and testing purposes. In this study, we leveraged the training and test segments of the CelebA dataset as our unlabeled dataset for training the autoencoder. In contrast, supervised training was conducted using the validation portion of the CelebA dataset.

The Labeled Faces in the Wild dataset (LFW) [30] comprises a collection of 13,233 images featuring 5,749 distinct individuals. For testing purposes, this database is randomly and uniformly divided into ten subsets. Within each subset, 300 pairs of matched images (depicting the same person) and 300 pairs of mismatched images (depicting different persons) are randomly selected. In essence, this testing protocol employs a total of 3,000 matched pairs and 3,000 mismatched pairs [30].

In Tables I and II, approach refers to the method where face image vectors are extracted using a conventionally trained autoencoder. Table I presents a performance comparison between our proposed method and the baseline, where a threshold is set to select a specific number of neighbor face image vectors. All vectors are then evaluated using the cosine scoring technique. We experimented with different threshold values to fine-tune the system and achieve the best results. The table clearly illustrates that our proposed method outperforms the baseline system. As we decrease the threshold value, the system’s performance improves. The optimal Equal Error Rate (EER) of 8.24% was achieved with a threshold set to 0.2. This represents a 51% relative improvement over the baseline method. Consequently, this bridges the performance gap between cosine and Probabilistic Linear Discriminant Analysis (PLDA) scoring techniques.

By performing a score-level fusion of the baseline and our proposed method, using a threshold of 0.2, we achieved a further enhancement in performance, as evidenced by a reduced Equal Error Rate (EER) of 7.12%. The optimal weights for the fusion were determined empirically, resulting in values of 0.55 and 0.45 for the baseline and our proposed method, respectively. This fusion strategy effectively leverages the strengths of both approaches to yield superior results.

In Table II, we present a performance comparison between our proposed method and the baseline, utilizing a constant value of k for the selection of similar face image vectors. We conducted experiments with various values of k to optimize the system for achieving the best results. The table illustrates that employing a fixed value of k has demonstrated improvement compared to both the threshold-based approach and the baseline. Beginning with k set to 1, an Equal Error Rate (EER) of 15.21% was attained. This signifies a 17% relative improvement over the baseline system by considering only the nearest neighbor for each image vector. As we increment the value of k, the system’s performance continues to improve. The most favorable EER of 8.03% was achieved when k was set to 11, leading to a remarkable 56% relative improvement over the baseline method. This achievement effectively bridges the performance gap between cosine and PLDA scoring techniques. Moreover, this approach enables a balanced training regimen, ultimately enhancing the system’s performance. It is worth noting that a further increase in the value of k may incorporate very distant neighbors, potentially degrading the system’s performance.

By combining the scores from the baseline and our proposed method, both utilizing k set to 11, we achieved a further improvement in performance, specifically in terms of the Equal Error Rate (EER). This fusion approach yielded an EER of 7.01%, which is in close proximity to the results obtained using Probabilistic Linear Discriminant Analysis (PLDA) with actual face image labels. The optimal weights for the fusion were fine-tuned through experimental iteration and were ultimately set to 0.49 and 0.51 for the baseline and our proposed method, respectively. This fusion strategy effectively leverages the strengths of both approaches to yield highly competitive results.

As depicted in Table III, conducting a score-level fusion between Probabilistic Linear Discriminant Analysis (PLDA) and cosine scoring, utilizing the proposed method with k set to 11 (yielding the best results), leads to an Equal Error Rate (EER) of 6.23%. This showcases an impressive relative improvement of nearly 7% over the PLDA approach alone. The optimal fusion weights were determined to be 0.04 for the baseline and 0.96 for the proposed method, respectively. This fusion strategy effectively capitalizes on the strengths of both approaches, resulting in a highly competitive performance.

Despite the slightly higher accuracy of methods like ArcFace, GroupFace, Marginal Loss, and CosFace, it’s noteworthy that our proposed method achieves competitive results with a significantly smaller training dataset (around 200K images). In contrast, many state-of-the-art methods rely on much larger training sets, often in the millions of images. This demonstrates the efficiency and effectiveness of our approach, particularly in scenarios where acquiring a massive dataset may not be feasible or practical.

The experimental result of the proposed method become particularly notable when compared to state-of-the-art approaches. For instance, ArcFace, which attains 99.82% accuracy, utilized a substantially larger dataset of 5.8 million labeled images. In contrast, our method achieved comparable performance with significantly fewer training images (around 200K). This underscores the efficiency and effectiveness of our approach, especially in scenarios where obtaining a massive dataset is challenging.