Mobile Contactless Palmprint Recognition: Use of Multiscale, Multimodel Embeddings

Despite the fact that government agencies do not typically maintain palmprint databases, palmprint recognition has been an area of significant research interest in recent years due to its high amount of discriminative features (large number of minutiae, palm creases, hand geometry, etc.), ease of use, and lower privacy concerns compared to other biometric modalities. In this section, for the sake of brevity, we limit our discussion to the recent advancements in the field of contactless palmprint recognition.

Early palmprint recognition methods relied on handcrafted features extracted from a variety of local cues such as principal lines and landmark points for region of interest extraction [11, 12, 13] and achieved reasonable accuracy on benchmark datasets of the time. However, many of these methods [14, 15], operated on high resolution images (500 ppi or more), obtained from contact-based flatbed scanners collected under controlled settings (e.g., laboratory environment, low degrees of freedom due to flat imaging surface, etc.). However, contactless palmprint imaging from a smartphone is typically limited to lower resolutions (e.g., $<$ 200 ppi) and captured in a much less constrained scenario, introducing six degrees of freedom between the position of the user’s palm and the capture device.

To overcome the large intra-class (intra-palm) variability introduced by contactless imaging, SOTA palm recognition systems have moved toward deep learning-based features (i.e., embeddings) for matching [16, 17]. Many of these systems extract a single, global embedding, while some (such as the BEST algorithm [18]) utilize multiple local embeddings to accumulate similarities throughout the captured palmprint image. Despite the impressive accuracy achieved by the BEST algorithm across several benchmark datasets, the speed of matching a single global embedding for each candidate image pair is a much more efficient operation (consisting of an inner product between two fixed-length vectors) compared to aggregating multiple features from several locations in the input images. However, a cascade of both global and local feature matching provides an opportunity to leverage the speed of a single, fixed length embedding with the added reliability of local matching.

The variability in 3D hand positioning during contactless palmprint acquisition results in decreased intra-class similarity among multiple captures of the same palm. Some differences are localized, particularly near extremities, such as the lower palm region affected by thumb positioning. Therefore, relying on a single global embedding may not suffice for an accurate and robust contactless palm recognition system. Additionally, contactless palmprint images exhibit high inter-class (inter-palm) similarity due to the absence of fine-ridge details and low contrast from large standoff (distance of capture). Relying solely on local feature matching can lead to numerous false matches among visually similar palmprint images sharing common principal line structures. These challenges motivated our novel framework that combines global and local features for enhanced palmprint matching accuracy.

Recent advancements in deep learning have witnessed the rise of vision transformer (ViT) architectures [22], rivaling convolutional neural networks (CNNs) for various computer vision applications, including biometrics [23, 24, 25]. Studies suggest that CNNs and ViTs encode different and complementary features for the same task [26], motivating a fusion of both architectures for improved accuracy [27, 28]. Building on this, we employ a fusion of a CNN and a ViT to extract a single global embedding for each palmprint image. Furthermore, multi-resolution patches are incorporated into the ViT architecture to embed varying scales of local features.

As part of our end-to-end pipeline for contactless palmprint recognition, we incorporate a novel palmprint enhancment method that utilizes domain knowledge during training to improve the downstream recognition performance. In contrast to previous palmprint enhancement approaches that rely on hand-crafted image filtering techniques, we leverage deep learning to enhance the region of interest in palmprint images. The enhancement model is trained to address degradation such as occlusion, low contrast, and noise in the form of tattoos, while emphasizing the prominence of principal lines in the enhanced palmprint image.

To demonstrate the effectiveness of our method, we adopt cross-database evaluation protocols, encompassing cross-sensor and time-separated training and test datasets. Additionally, a large-scale palmprint image database captured via mobile phone cameras in an operational setting (a charitable hosptal in Dayalbagh, India) was collected and will be made available to the research community. This database spans multiple sessions with time intervals ranging from five to thirteen months, as well as images of both adults and children (between the ages of 9-48 months).

Lastly, unlike widespread deployments by Amazon, Tencent, and Fujitsu, where templates and matching reside in a central server, Palm-ID consolidates the entire recognition process within a mobile device. This integration includes image capture, feature extraction, template generation, and matching, all while upholding user privacy and security. To accelerate processing on embedded devices, a non-linear dimensionality reduction model reduces the embedding (template) size to 516 bytes while maintaining an average matching accuracy of TAR=96.97% at FAR=0.01% across five test datasets, comparable to the original average accuracy of 96.96% at a dimensionality of 772 bytes.

In summary, the key contributions of this research are:

• •

  Development of a state-of-the-art palmprint recognition pipeline using deep learning methods, which integrates a diverse set of global and local features for matching.

• •

  Introduction of a novel palmprint enhancement module that leverages domain knowledge to highlight and preserve relevant features for recognition.

• •

  Implementation of an efficient and accurate palmprint recognition system through a learned, non-linear dimensionality reduction model.

• •

  Introduction of a simple and effective palmprint image quality estimation method for optional rejection and re-capture of poor-quality samples.

• •

  Adoption of cross-database and time-separated evaluation protocols, including a newly collected database that will be made publicly available to the research community.

• •

  Development of a mobile-app for every stage of the end-to-end recognition pipeline, including data collection, region of interest extraction, feature extraction, and matching; all taking place on-device. This app will be made publicly available upon publication.

After successfully capturing a palmprint image, the obtained image is fed into the keypoint extraction module denoted as $K(\cdot)$. This module predicts nine keypoints positioned around the palmar boundary. Example keypoint predictions and the corresponding Region of Interest (ROI) extractions are depicted in Figure 3. The $K(\cdot)$ module is comprised of a ResNet-50 model trained via an MSE loss using keypoints generated by a commercial-off-the-shelf (COTS) palmprint SDK from Armatura [6] as ground truth. Leveraging these nine keypoints, the ROI is precisely localized through a homographic transformation involving nine meticulously selected destination points in a uniform coordinate system. These destination points were hand selected to encapsulate the most discriminate and stable regions of the palm.

During training, a range of data augmentations (translation, rotation, scaling, blurring, and perspective transforms) are randomly applied to build a robust model with high accuracy, particularly on images with substantial pose variations commonly encountered in contactless palmprint images. Following the homographic transformation, an ROI with dimensions $224\times 224$ is cropped and employed as input for the enhancement network. The network is trained with the Adam [29] optimizer for 200 epochs with a batch size of 32, learning rate of 0.001, and weight decay of $2e^{-5}$. The final model was selected according to the highest accuracy obtained on the validation set.

In the context of contactless palmprint recognition, where the capture scenario is unconstrained, various sources of noise may be present in acquired images, potentially compromising matching performance. Notably, there is a significant likelihood of encountering very low-contrast images, arising from diverse factors such as a substantial stand-off distance between the palm and the capture device, motion blur, and over-saturation. Other degradations, including extreme perspective distortions and occlusions, may also be observed. While perspective distortions can be partially alleviated through non-linear region of interest extraction and robust feature extraction, as previously outlined, we specifically address the sensitivity to occlusions and low contrast by implementing a palmprint enhancement module through learning.

One straightforward method for improving palmprint quality involves image sharpening techniques such as high-pass filtering, Discrete Fourier Transform (Jain et al. [32]), Gabor filters (Cappelli et al. [33]), and others. The goal is to accentuate the distinctive edge features of palm creases. However, a limitation of these approaches is that the parameters of the enhancement modules are tuned to a specific dataset and in practice do not generalize to newly acquired palmprint images from different devices and environments. Thus, rather than design a hand-crafted approach, which relies heavily on correct hyper-parameterization, we turn to deep learning for palmprint image enhancement.

Motivated by the recent application of DNNs for latent fingerprint image enhancement [34], we employ the efficient SqueezeUNet architecture for palmprint enhancement. We guide the network to remove certain degradations (low contrast and occlusions) present in the palmprint images by adding a series of augmentations to the ground-truth, high-quality palmprint images and training the network to recover the original images (see Fig. 4) via an MSE loss. The augmentations that we add are specific to the contactless palmprint recognition scenario and include Gaussian blurring, random down-sampling, and overlaying text (tattoos or handwriting), colored lines, and henna-like patterns on the surface of the palm. Various examples of these augmentations are shown in Figure 5. The network is trained with the Adam optimizer for 75 epochs with a batch size of 64, learning rate of 0.001, and weight decay of $2e^{-5}$. A validation set was used during training the save the model with highest validation accuracy.

To create henna patterns (tattoos), often found on the palms of individuals in several countries especially during festivals and marriage¹¹1 https://en.wikipedia.org/wiki/Mehndi, we employ the Stable Diffusion implementation [35] from the Diffusers library [36]. Utilizing the textual prompt “colorful henna flower sketch on white background,” we generated 200 henna patterns. After a manual filtering process to eliminate poorly generated samples, we retained 143 high-quality henna patterns. These patterns are then integrated into the original palmprint ROIs to produce authentic-looking henna palmprints. The procedure involves obtaining a mask outline for each pattern, randomly resizing and cropping the mask, assigning a random RGB color, and blending the colorized mask with the original image using a random transparency level within the [0.1, 1] range, where 0 represents full transparency.

To create random text and marker patterns, a comparable method is employed, but without utilizing Stable Diffusion. In this case, random English language sentences are generated using the Faker library in Python, employing five distinct font styles. Additionally, random line markings are generated by using the OpenCV function “polylines,” incorporating variations in size, thickness, and placement within the image.

We once again turn to deep learning for contactless palmprint feature extraction. Diverging from previous contactless palmprint recognition studies, we depart from a single deep network architecture approach. Instead, we leverage recent research [26] that advocates for the synergistic qualities of combining Vision Transformers (ViT) and Convolutional Neural Networks (CNNs). Specifically, we incorporate both the small version of ViT and the ResNet50 architectures in our method.

Given the variability in stand-off distance, leading to differences in scale and resolution across captured palmprint images within various databases, capturing features at diverse scales is crucial for robust contactless palmprint recognition. ViT and CNNs inherently process information at distinct scales across their networks, with ViTs aggregating global information at earlier layers compared to CNNs. To further enhance scale invariance, we encode multi-scale patches into our ViT architecture. Both our ViT and ResNet50 networks take, as input, palmprint ROI images resized to 224x224 pixels. For ViT, we use patch sizes of 16 and 32 as input, demonstrating the multi-scale patches in Figure 6. An ablation study in Section III-E quantitatively demonstrates the advantages of incorporating both patch sizes in our ViT architecture, as well as the synergies achieved by combining CNN and ViT over using each network individually.

The ResNet50 network is trained with an Adam optimizer for 150 epochs with a batch size of 1024, learning rate of 0.001, and weight decay of $2e^{-5}$. The ViT model is first trained with a patch size of 16 using an AdamW [37] optimizer for 350 epochs with a batch size of 1024, learning rate of 0.001, and weight decay of $2e^{-5}$. The ViT model is then finetuned for 50 epochs using both patch sizes of 16 and 32 as input features with a learning rate of 0.0001. Both ViT and ResNet50 models are trained with the ArcFace classification loss [38] and the best models were saved based on highest validation accuracy.

Similarity scores are computed by taking the dot product between L2-normalized embeddings. To be more specific, we concatenate each normalized 384-dimensional feature embedding from Vision Transformer (ViT) and ResNet50 into a unified 768-dimensional embedding. Before the concatenation, each of the embeddings, denoted as $Z_{v}$ and $Z_{r}$, are normalized to unit length. Given that the dot product between the concatenated embeddings ranges from [0,2], we re-normalize the scores to fall within the range [0,1] via the following equation:

Figure 6 provides an illustration of dimensionality reduction through the DeepMDS++ model. The codebase used is adapted from [39], relying on DeepMDS [40] for the task of reducing the dimensionality of image embeddings. DeepMDS transforms the ambient representation into a minimal intrinsic space, effectively reducing dimensionality while preserving essential discriminative features.

Distinct dimensionality reduction models are trained for ViT and ResNet embeddings instead of using a single model for the concatenated embeddings. Training a single reduction model on concatenated embeddings proved challenging due to the variance in latent space between the two representations. The best results were obtained when compressing each of the individual embeddings from a dimension of 384 to 256, resulting in a concatenated embedding of dimension 512. Although experimentation with even smaller embedding sizes was conducted, we ultimately settled on 512 for our final model as it exhibited no decline in accuracy compared to the original embedding size. Experiments with other embeddings sizes are given in section III-D.

In order to significantly reduce the storage size of the embeddings, we employ compression by converting the precision from floats (4 bytes) to unsigned integers (1 byte). Initially, we calculate the maximum and minimum float values within the embedding and conduct a min-max normalization, ensuring the values fall within the range [0, 1]. Subsequently, each value is scaled to fit within [0, 255], and the precision is truncated from 32 bits to 8 bits by converting from float to unsigned integer. Lastly, we convert the minimum and maximum values to 16-bit floats and store them alongside the compressed embedding, resulting in a total storage requirement of $n+4$ bytes, where $n$ represents the dimension of the uncompressed embedding. During the matching process, the embeddings undergo decompression by converting them back to floats within the range [0,1]. The values are then re-normalized using the stored maximum and minimum values. Empirically, compressing and decompressing the features in this manner demonstrated no loss in accuracy throughout the experiments.

Assessing the quality of contactless palmprints is a crucial practical consideration for any deployed biometric system. Consequently, we have opted to integrate a quality metric into our end-to-end pipeline, providing the flexibility of an optional reject for processing palmprint images with low quality. However, deriving an objective measure of palmprint quality proves challenging due to the myriad factors influencing image quality, such as pose, contrast, and occlusions, which can impact downstream recognition tasks. Rather than devising a manually crafted metric or training a distinct quality prediction network, a task requiring accurate ground-truth labels, we leverage the L2 norm of our feature embeddings as a surrogate for image quality. This approach, as highlighted in Kim et al. [41] within the domain of face recognition, demonstrates a strong correlation with image quality. Through the adoption of this straightforward yet effective quality measure, we demonstrate a large decrease in the error rate of our palmprint recognition system with a small percentage of rejected samples.

Moreover, we conduct a comparative analysis, juxtaposing the L2 norm metric against two baseline quality measures - one from the Armatura SDK and the other employing the Laplacian of Gaussian (LoG) approach for image quality prediction. Our results reveal the superiority of our metric across all datasets that we have used here. Figure 11 showcases examples of low and high-quality images predicted by our quality assessment method.

To fulfill the goal of a real-world deployment of the proposed Palm-ID, we built an Android application that is capable of enrollment and search in real-time.

The app has three modes of operation which are listed below:

1. 1.

   Enrollment Mode: This mode enrolls the palmprint image and saves the templates on device.

2. 2.

   Authentication Mode: This mode is intended for 1:1 verification where the user inputs an ID and then proceeds to scan a palmprint to determine if it matches to the enrolled palmprint corresponding to the input ID.

3. 3.

   Search Mode: This mode searches the query palmprint template against all registered identities and returns the ID with the highest similarity score. To account for an open-set scenario, a similariy threshold is set to determine if the returned match scores are above an acceptable level to avoid false positives.

Screenshots from the Palm-ID mobile application are shown in Figure 7. This app will be made publicly available to promote easier data collection efforts in the field.

We present authentication results on three publicly available contactless palmprint databases: i) CASIA Palmprint Image Database, ii) IITD v1 Palmprint Database, and iii) NTU Controlled Contactless Palmprint Database v1 (NTU-CP-v1). Additionally, we assess performance on two time-separated extensions, APDB-2-3 and CPDB-2-3, of our newly collected MSU Palmprint databases. The authentication performance of our proposed model is compared to three state-of-the-art methods: Godbole et al. [17], Matkowski et al. [21], and a COTS palmprint SDK by Armatura. We directly use the publicly available SDK for the Matkowski et al. method, rather than re-implement the algorithm ourselves. Similarly, we use the Armatura SDK out-of-the-box, without knowledge of the matching algorithm or training databases used to develop it. For Godbole et al., we re-train the network on the same training and validation set as the proposed Palm-ID method. Other baseline methods have either not made their code publicly available or use a percentage of the test databases for training, whereas we have reserved the entirety of each test database for evaluation.

Our model achieves a True Acceptance Rate (TAR) of 99.08% at a False Acceptance Rate (FAR) of 0.01% on NTU-CP-v1, surpassing the next best performance of 98.94% by Armatura. Furthermore, our method demonstrates competitive TARs of 99.53% on CASIA and 99.93% on IITD v1, both at FAR = 0.01%, compared to 100% and 99.72% by Armatura. For the time-separated datasets APDB-2-3 and CPDB-2-3, the proposed model significantly outperforms all other methods in our comparison, showcasing its efficacy in less-constrained and longitudinal recognition scenarios.

To enhance recognition system accuracy further, we perform score-level fusion of our model and Armatura. We assign a weight of 0.7 to the similarity score generated by our method and a weight of 0.3 to the similarity score from Armatura. The improvement in performance underscores the complementary nature of features extracted by both models and suggest potential for future improvements in our approach. The detailed authentication results are provided in Table III.

Open-set search scenarios may incur two types of recognition errors. Firstly, in a nonmated search, the matching algorithm may return an identity from the gallery with a similarity score above an acceptance threshold which does not correspond to the correct identity of the probe subject, referred to as Type I error or false positive. Secondly, in a mated search, the algorithm may return an incorrect identity from the gallery instead of the correct enrolled identity, resulting in a Type II error or false negative. The False Positive Identification Rate (FPIR), representing the Type I error rate, quantifies the fraction of nonmated searches where enrolled identities are inaccurately returned at or above the specified threshold (T). On the other hand, the False Negative Identification Rate (FNIR), corresponding to the Type II error rate, measures the fraction of mated searches where the enrolled mate falls outside the top R ranks or the comparison score of the correct mate is below T. FPIR and FNIR rely on both the size of the enrolled gallery (N) and the number of top candidates considered (R). For our experiments we use N=4,377 and R=1. Table V compares the FNIR (%) at FPIR=1% of the proposed Palm-ID matcher against the baseline method proposed in [17]. Across all five evaluation datasets, Palm-ID signifcantly outperforms the baseline, returning an average FNIR of 4.42% compared to 32.23%.

For closed-set identification, results are usually reported in terms of rank R retrieval rate. This accuracy metric signifies the fraction of mated searches where the enrolled mate is at rank R or better, irrespective of the comparison score. Rank-1 retrieval rate of the proposed Palm-ID matcher is compared against the baseline method of Godbole et al. in Table IV. Palm-ID obtained an average rank-1 retrieval rate of 98.24%, compared to Godbole et al. which obtained 95.94%.

In addition to accuracy, practical considerations, such as model latency and template size, are important factors for real-world deployment. Table VII provides a comparison of these efficiency metrics for our model against other methods. Our model excels in template extraction and large-scale search efficiency, due to the proposed dimensionality reduction and feature compression techniques. The detailed tradeoff between efficiency and accuracy is given in Table VI, illustrating the performance variations as the embedding size of the proposed model changes. The results indicate that reducing the embedding size by a factor of four (from 772 bytes to 196 bytes) only decreases the average TAR by 0.51 percentage points.

For a detailed examination of the contribution of each individual component in our Palm-ID system, we present a comprehensive ablation study in Table VIII. A comparison between rows 1 and 2 highlights a significant improvement with the incorporation of multi-scale patches in the ViT architecture compared to a single patch size of 16, thus advocating for the utilization of multi-resolution local features for robust palmprint recognition. Additionally, the fusion with ResNet50 results in further improvements in three out of the five datasets. Lastly, it is observed that the inclusion of the learned enhancement model played a pivotal role in boosting performance, particularly on lower-quality, time-separated datasets (APDB-2-3 and CPDB-2-3), along with marginal gains for CASIA and NTU-CP-v1 datasets.

Figure 9 illustrates both successful and unsuccessful comparisons by our method. Depicted in row (a) are some correctly matched genuine pairs that are in close proximity to the match threshold, suggesting their challenging nature for our model. Nevertheless, our system appears capable of effectively handling numerous challenging scenarios, including those with slight occlusion, blurriness, and perspective distortions. In the middle row (b) of Figure 9, are some failure cases representing genuine pairs that failed to match by falling just below the match threshold. Finally, row (c) showcases examples of genuine comparisons that were incorrectly classified well below the match threshold. These samples contain over-saturated images and occlusions which are challenging to be matched.

Lastly, we introduced a palmprint quality estimation metric to identify low quality images. In particular, the L2 norm of the Palm-ID embedding is used as the quality value, providing a more accurate estimate of the objective quality of the image compared to the quality metric implemented by Armatura’s SDK and the variance of the Laplacian of Gaussian (a common method for estimating blur in an image). The error-reject trade-off curve in Figure 10 demonstrates the quantitative advantages of the L2 norm over the other methods. A visual comparison of low and high-quality predictions by Armatura, the variance of the Laplacian of Gaussian, and our method is presented in Figure 11.

The variance of the Laplacian of Gaussian (LoG) metric was computed on the Region of Interests (ROIs) rather than the full palm images, as this was found to be less susceptible to background noise. Furthermore, we crop 32 pixels from all four sides of the ROI to ensure that the LoG is not influenced by minor occlusions, such as the edges of shirt sleeves, etc. For Armatura’s metric, we directly use the quality scores output by the system.

In most datasets, our quality metric performs the best, except for CASIA and IITD. In the case of IITD, there are only a few mis-classifications so the difference in performance among the three metrics seems to be negligible. However, in the case of CASIA, the poor quality samples seem to primarily stem from a few poor contrast, blurry samples, which the LoG metric is primarily designed to detect.