IFViT: Interpretable Fixed-Length Representation for Fingerprint Matching via Vision Transformer

A number of studies [3, 4, 5, 6, 25] have investigated the feasibility of learning deep fixed-length fingerprint representations in fingerprint recognition. CNN-based approaches are among the first that were widely employed [24, 3, 4, 18]. As the pioneering work, Cao et al. [24] design an Inception V3-based approach for fingerprint indexing and matching using the extracted fixed-length representation [24]. To improve the recognition performance, the domain knowledge e.g. minutiae information is further introduced in deep networks. One of the representative works is DeepPrint [3], which could achieve competitive performance compared to the state-of-art minutiae matchers. The learning of fixed-length representation is also taken into account in contactless and contact fingerprint matching [4]. Different from the aforementioned approaches that are based on classification, similarity metric learning has been also used to learn fixed-length representations. Lin et al. develop a Siamese CNN-based framework for accurately matching contactless and contact-based fingerprint images [18].

More recently, the application of the ViT for learning fixed-length fingerprint representation has also drawn attention [25, 19]. Grosz et al. [25] propose the first use of the ViT in extracting fixed-length fingerprint representation and demonstrate that the independent performance of ViT in fingerprint recognition rivals that of CNN-based models. Tandon et al. [19] propose a convolutional ViT network based on both global representation (i.e. complete fingerprint image) and local representation (i.e. minutiae information) to complement semantic information in fingerprint recognition.

However, most of above-mentioned approaches encounter the limitation of interpretability during the fingerprint matching process. The difficulty lies in the lack of ability of these learned fixed-length representations to determine feature points of the fingerprint pair utilized for matching. They lack the same level of interpretability exhibited by the minutiae-based matching method as the latter can visually provide pixel-wise minutiae correspondences in the matching result.

Establishing deep learning models with acceptable interpretability has become an important task for numerous studies when dealing with computer vision-related tasks in recent years [12, 26]. As another prevalent biometric technology for identifying individuals, facial recognition has received significant attention, leading to considerable research on interpretable deep learning models for face matching and indexing [27, 28]. In contrast, there is hardly any research on interpretable fingerprint recognition technology. The need for interpretable fingerprint recognition approaches based on deep learning models is equally significant and warrants more consideration.

The minutiae-based fingerprint matching approaches have held a dominant position over the past few decades [29, 30]. This is attributed not only to its reliable performance in matching but also to its remarkable interpretability. The minutiae expressed by their origin (i.e. $x$ and $y$ coordinates) and angle are extracted first and then minutiae correspondences in the input fingerprint pair could be established and visualized. Similar to minutiae matching, Cao et al. [5] introduce the concept of virtual minutiae using deep learning models to sample directed points located on fingerprint images to ensure an adequate number of key points within the latent fingerprint area and to provide interpretable correspondences. Gu et al. [16] propose a coarse-to-fine matching scheme considering using undirected sampling points as key points to produce interpretable correspondences on the given latent fingerprint pair. However, while identified key points e.g. minutiae or virtual minutiae could be used to obtain the average similarity employed for fingerprint matching, they lack the benefit of the fixed-length representation extracted by deep models. Performing pairwise comparisons of fingerprints is computationally demanding which varies with the number of detected key points. On the other hand, Cui et al. [31] also define the task of fingerprint dense registration for obtaining pixel-wise displacement measures between two fingerprints with nonlinear skin distortion. The learned correspondences could contribute to the improvement of the subsequent matching performance as well as providing interpretable pixel-wise correspondences of feature points.

Despite the aforementioned studies being able to provide interpretable pixel-wise correspondences of feature points, none of them is integrated with learning fixed-length fingerprint representation during the matching process, which limits the reduction of computational complexity and applicability in other downstream tasks e.g. encryption [3]. In addition, the CNNs employed in these studies have limited receptive fields, which is not conducive to finding correspondences of feature points within a large global context [32]. A network called AFRNet is devised by Grosz et al. [17] for learning discriminative fixed-length fingerprint representations in the indexing task while obtaining correspondences between local fixed-length fingerprint representations that can be used for refining global representations in low-certainty situations. However, similar to previous studies, the CNN branch of AFRNet constrains the achievable number of correspondences, limiting its interpretability. On the other hand, the correspondences are patch-based instead of pixel-wise and are not a direct output of the neural network. They are obtained using a brute-force algorithm that demands significant computational resources, making the process time-consuming.

Employing gradients of the prediction output to identify parts of an input image that exert the most influence on the fingerprint matching result is also considered. Chowdhury et al. [33] train a multi-scale dilated Siamese CNN using fingerprint patches and demonstrate the significance of minutiae for fingerprint matching using Grad-CAM [34]. However, it could only reveal general areas that CNN focuses on during the inference stage, but could not establish further detailed pixel-wise correspondences.

In contrast to previous studies, we propose IFViT for fingerprint matching with excellent matching performance and interpretability. It combines the benefit of fixed-length representations extracted by deep models and interpretable pixel-wise correspondences of feature points in matching results. The ViT-based model is established to obtain interpretable dense pixel-wise matches. These matches could be employed for effective fingerprint alignment and improve interpretability in the subsequent matching procedure, allowing feature points used for fixed-length representation-based fingerprint matching to be visualized.

The architecture of the proposed IFViT is shown in Fig. 2. It contains two main modules, an interpretable dense registration module and an interpretable fixed-length representation extraction and matching module. IFViT is trained with a large dataset composed of millions of fingerprint images across several benchmark datasets including plain, slap and rolled fingerprints.

Given the input fingerprint pair $I_{1}$ and $I_{2}$, a specific CNN, ResNet-18 [35], is employed to extract local features $F_{1}$ and $F_{2}$. The ViT is then adopted to learn the position and context-dependent features from $F_{1}$ and $F_{2}$, followed by a differentiable matching layer [36] to produce dense pixel-wise correspondences of feature points for the given fingerprint pair. Alignment is subsequently carried out based on these correspondences to obtain an aligned fingerprint image $I^{\prime}_{1}$ with reference to the other input $I_{2}$. The aligned fingerprint pair then undergoes fingerprint enhancement implemented by FingerNet [37], a state-of-the-art minutiae extractor consisting of orientation estimation, segmentation, Gabor enhancement and minutiae extraction, to obtain the enhanced fingerprint pair $I^{\prime}_{e1}$ and $I_{e2}$. To take full advantage of the complementary local and global representation of the input fingerprint pair, we identify the corresponding Region of Interests (ROIs), including $I^{\prime}_{ro1}$ and $I_{ro2}$ in original fingerprint images, and the overlapped regions $I^{\prime}_{oe1}$ and $I_{oe2}$ in the aligned enhanced fingerprint images. Subsequently, those ROIs are passed into the ViT that is trained in the interpretable dense registration module but is finetuned with the new fully connected layer to obtain interpretable dense pixel-wise correspondences of feature points and discriminative fixed-length fingerprint representations, simultaneously. The learned local and global fixed-length representations are individually computed for their similarity, and combined to derive the final matching score.

Dense fingerprint registration seeks to establish pixel-wise correspondences of feature points between two fingerprints that are subjected to nonlinear skin distortion or displacement [38]. These learned correspondences could then be used to fit a spatial transformation model for aligning fingerprints which is an essential component in most fingerprint recognition systems [3]. Since the quality of the alignment may greatly affect the performance of subsequent fingerprint matching process, it motivates us to develop a robust dense registration model. We particularly consider the characteristics of the ViT due to its ability to capture global context.

Given the pair of fingerprint images $I_{1}$ and $I_{2}$, firstly we use a CNN to learn translation-equivariant local features which also reduces the input length to the ViT module to ensure a manageable computation cost. In this study, the ResNet-18 is chosen as the backbone. After obtaining feature maps $F_{1},F_{2}\in\mathbb{R}^{H\times W\times D}$ from the CNN, a ViT model is built to extract local features that are both position-dependent and contextually relevant. Specifically, ViT performs a linear projection at first based on $N$ patches $f_{i}\in\mathbb{R}^{1\times 1}$ in feature maps $F_{1},F_{2}$ and then transforms them into 1D tokens ${z_{i}\in\mathbb{R}^{d}}$ as shown below.

where $E$ represents the linear projection. The position embedding $p\in\mathbb{R}^{N\times d}$ is also added to tokens to provide the positional information. Subsequently, the token is passed into the encoder consisting of a sequence of $L$ transformer layers. As a critical part in the encoder, the attention layer receives the token in three different vectors: query ($q$), key ($k$) and value ($v$). The vectors originating from diverse inputs are subsequently consolidated into three different matrices, namely $Q$, $K$ and $V$. The query $Q$ retrieves information from the value $V$ based on the attention weights calculated from the dot product of $Q$ (corresponding to each value $V$) and the key vector $K$. The process is denoted as:

In addition to the self-attention layer, the cross-attention layer inspired by [39, 32] is also introduced in our ViT model to enable cross-image communication in the fingerprint pair similar to the way humans look back-and-forth when matching images. In terms of the self-attention layer, the input features $f_{i}$ and $f_{j}$ are the same (either $F_{1}$ or $F_{2}$). However, for the cross-attention layer, the $f_{i}$ and $f_{j}$ are either ($F_{1}$ and $F_{2}$) or ($F_{2}$ and $F_{1}$). The self-attention layer and cross-attention layer are interleaved in our transformer by $M$ times. After passing through the encoder layer, the transformed features denoted by $F_{1}^{\prime}$ and $F_{2}^{\prime}$ are obtained.

In terms of identifying the dense pixel-wise correspondences of feature points in the given fingerprint pair, a differentiable matching layer using the dual-softmax operator is adopted [36]. Firstly, we compare the feature similarity for each pixel in $F_{1}^{\prime}$ with respect to all pixels in $F_{2}^{\prime}$ by calculating their correlations as follows:

where correlation matrix $C$ denotes the correlation value of each coordinate between $F_{1}^{\prime}$ and $F_{2}^{\prime}$. $\frac{1}{\sqrt{D}}$ is a normalization factor to prevent the dot-product operation from producing large values.

Subsequently, we normalize $C$ using the softmax operation to obtain the matching confidence matrix $P$ as in equation (4). The utilization of a softmax-based approach not only enables the end-to-end training process but also provides pixel-level accuracy.

The candidate matches in $P$ are selected if the corresponding confidence score is higher than a given threshold $T_{1}$. We use Mutual-Nearest-Neighbor (MNN) criteria to further filter outlier matches and obtain the final selected matches $M$:

Loss function. For the interpretable dense registration module, the loss function for training the ViT that produces the dense pixel matches is negative log-likelihood loss, which is computed based on the matching confidence matrix $P$. The pre-trained ViT model for feature matching in LoFTR [32] is employed to accelerate the convergence for training. Considering there is no available public fingerprint database that provides ground truth labels of matches of fingerprint pairs, we construct several noise models to simulate diverse types of noises and corruptions appearing in the real fingerprint images to improve the model’s ability for recognizing correspondences of feature points in low-quality or low-texture fingerprint images. In this case, the fingerprint images enhanced by FingerNet are not used but instead, we solely rely on original fingerprint images and their corresponding corrupted image to train the model. Since the relation of each pixel position between original and synthetic corrupted fingerprint images could be directly obtained, we are able to determine ground-truth labels $G^{gt}$ for the confidence matrix. Three types of noises are simulated in this study including sensor noise, dryness and over-pressurization, as depicted in Fig. 3. Sensor noise could be approximated by the Perlin noise [40], whilst dryness and over-pressurization could be simulated by performing the dilation and erosion operation, respectively. To ensure rotation invariance, all corrupted fingerprint images are rotated randomly within $\pm$60 degrees. This rotation parameter is determined based on the domain knowledge regarding the maximum degree to which a user would rotate their fingers while positioning them on the reader platen [3]. Furthermore, to minimize incorrect matches, ground-truth labels are also created for the registration of imposter fingerprint pairs. The loss function of dense registration $\mathcal{L}_{D}$ based on negative log-likelihood loss over the grids in $G^{gt}$ is minimized during the training stage:

After performing the dense registration on the given fingerprint pair, we establish the second critical component of our method, namely the interpretable fixed-length-representation extraction-and-matching module. In this module, we combine both local and global representations of fingerprints to improve the matching performance. It can additionally provide interpretable dense pixel-wise correspondences of feature points similar to the previous interpretable dense registration model in the matching results.

Fingerprint Alignment. Recognizing that the outcome of the alignment would significantly influence the subsequent matching module’s performance, we rotate $I_{1}$ at five angles (-60°, -30°, 0°, 30°, 60°) to determine the best matches $M_{max}$ for alignment. Subsequently, we compute the homography matrix $H$ using the RANSAC algorithm and apply the obtained $H$ directly to $I_{1}$ to produce the aligned $I_{1}^{\prime}$. Compared with the Spatial Transformer Network (STN) which has been widely used recently for fingerprint alignment, e.g., in [3, 6], our proposed ViT-based interpretable dense registration module for fingerprint alignment can provide dense pixel-wise correspondences of feature points for the given fingerprint pair, which is conducive to understanding the criteria used to determine when fingerprint alignment succeeds or fails.

Obtaining Local and Global Representation. Although the global fixed-length fingerprint representation has been demonstrated to be effective for fingerprint recognition [3, 41, 42], it may produce high similarity on some global similar ridge-flow structures from different identities by mistake, therefore leading to wrong matches [19]. On the other hand, local representation, e.g. fingerprint patches, may also face matching failures, primarily because some factors like distortion and dryness could negatively impact the effective fingerprint information, especially for minutiae [19]. Fig. 4 shows a case susceptible to matching failures when considering either only the local or global representation. Therefore, combinations of both representations should be considered to compensate for the semantic shortcomings of each other.

In this study, we fuse global representation extracted from overlapped regions of enhanced fingerprints with local representation derived from the corresponding ROI of original fingerprints. The motivation for considering the ROI from original fingerprints instead of enhanced fingerprints as the local representation is that we observed a certain level of prediction error when using FingerNet for enhancement. The FingerNet performs well in enhancing the global ridge-flow structures of fingerprints but is prone to introducing patch-like artifacts, resembling a mosaic, in low-quality local areas in fingerprints. This could potentially lead to the loss of some crucial information.

Specifically, we introduce the Sobel operator to calculate the gradient of the fingerprint image’s intensity in the horizontal ($G_{x}$) and vertical ($G_{y}$) directions, and then compute the magnitude of the gradient $|G|$ for each pixel based on equation (7). The box filter with 25×25 kernel size is used to obtain the integral of the gradient magnitude. Threshold $T_{2}$ which equals 15 percent of the maximum integral of the obtained gradient magnitude is further empirically determined for segmenting the general fingerprint area. We perform detection for contours on the segmented regions and conduct connected component analysis, resulting in the respective mask of an effective fingerprint area for $I_{e1}^{\prime}$ and $I_{e2}$. After taking the intersection of masks, they are separately applied to $I_{e1}^{\prime}$ and $I_{e2}$, producing effective overlapped fingerprint areas $I_{oe1}^{\prime}$ and $I_{oe2}$ (i.e. global representation). The flowchart for extracting overlapped regions from the fingerprint pair is illustrated in Fig. 5. Although the FingerNet can also directly output the mask of a fingerprint, the generated mask frequently includes irrelevant background around the fingerprint, as shown in $I_{e1}^{\prime}$ and $I_{e2}$ of Fig. 5. For the model to focus only on the effective fingerprint information, the mask generated by FingerNet is not considered in this step. Furthermore, we extract ROIs (90×90) from the overlapped region of original fingerprints as the local representation.

Learning Interpretable Fixed-length Representation. For the interpretable fixed-length representation extraction and matching module, we propose two ViT-based Siamese Networks $f(\cdot)$ with shared weights which learns a similarity metric for the input fingerprint pair and outputs a matching score. Specifically, the input consists of the overlapped fingerprint $I_{oe1}^{\prime}$ and $I_{oe2}$ (i.e. global information) as well as the corresponding ROIs $I_{ro1}^{\prime}$ and $I_{ro2}$ extracted from original fingerprint images (i.e. local information). We utilize the interpretable dense registration module trained previously in the fingerprint matching module to generate pseudo labels of fingerprint pairs for learning correspondences of feature points. We combine the previously trained ViT-based Siamese model as the backbone with a new fully connected layer, which is then retrained with labelled fingerprint pairs. Thus, it offers interpretable dense pixel-wise correspondences of feature points $M$ in matching results, and the 256-dimensional fixed-length representations $R_{1}$ and $R_{2}$.

Loss Function. In this module, dense registration loss $\mathcal{L}_{D}$ is employed again to learn dense pixel-wise correspondences of feature points for the fingerprint pair from ViT’s output. The cosine embedding loss $\mathcal{L}_{E}$ shown in equation (8) is chosen for training the model to transform identified correspondences of feature points into the 256-dimensional discriminative fixed-length representation.

To minimize and maximize the distance of learned intra-class and inter-class representation separately, we further introduce the ArcFace loss $\mathcal{L}_{A}$ in equation (9) to optimize the network via classification task, constituting the total loss $\mathcal{L}_{T}$ shown in equation (10).

where $y_{e}\in\{-1,1\}$ represents the ground truth of the cosine embedding loss $\mathcal{L}_{E}$ and $y_{c}$ is the true class label of the ArcFace loss $\mathcal{L}_{A}$, respectively. $N$ is the total number of the training samples while $K$ is the number of classes. $\theta_{y_{c}^{i}}$ denotes the cosine of the angle between the learned representation ($R_{1}$ or $R_{2}$) and the weight vector for the corresponding label $y_{c}$. In terms of $\theta_{k}$, it calculates the cosine of the angle between the learned representation ($R_{1}$ or $R_{2}$) and the weight vector for all classes except the $y_{c}$. Additionally, the margin $m$ is set to 0.4 for both $\mathcal{L}_{E}$ and $\mathcal{L}_{A}$ in this study and the rescaling factor $s$ is set as 64. $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ in the $\mathcal{L}_{T}$ are empirically set as 0.5, 0.1 and 1 to balance the matching performance and interpretability, respectively.

Rather than introducing the domain knowledge (i.e. minutiae information) [6, 4, 3], we hope that the fixed-length representation learned from the IFViT could utilize prior information related to dense pixel-wise correspondences of feature points in the fingerprint pair, to deal with the situation where it is challenging to learn minutiae information from low-quality fingerprints.

Matching Score Computation. For evaluation, the learned fixed-length global representation $(R_{1}^{g}$, $R_{2}^{g})$ and fixed-length local representation $(R_{1}^{l}$, $R_{2}^{l})$ of an input fingerprint pair can be extracted. The matching scores, namely $t_{1}$ and $t_{2}$ based on the $(R_{1}^{g}$, $R_{2}^{g})$ and $(R_{1}^{l}$, $R_{2}^{l})$ are calculated as shown in equation (11) and normalized to the range [-1, 1]. The total matching score $S$ is finally obtained by weighted fusion of the individual scores ($t_{1}$ and $t_{2}$) as shown in equation (12), where the weights $\alpha_{1}$ and $\alpha_{2}$ are determined by linear regression learning.

Dense Registration Module. Regarding the training of the interpretable dense registration module, the FVC2002 (DB1, DB2 and DB3) [43], NIST SD301a (A, B, C, E, J, K M and N) [44], a subset of NIST SD302a (A, B, C, D, E, F, U, V, L and M) [45] and a subset of MOLF (DB1 and DB2) [46] are used. The total number of fingerprint images in these datasets is 25, 090, and after applying three noise models previously proposed, a total of 100, 360 training images can be obtained. They are combined to form 100K fingerprint pairs (75K for genuine pairs and 25K for imposter pairs) for training.

Matching Module. In terms of the interpretable fixed-length representation extraction and matching module, we employ FVC2002 (DB1, DB2 and DB3) [43], NIST SD301a (A, B, C, E, J, K M and N) [44], a subset of NIST SD302a (A, B, C, D, E, F, U, V, L and M) [45], a subset of MOLF (DB1 and DB2) [46] and a recently released synthetic fingerprint dataset called PrintsGAN [47] as the training dataset and evaluate the model on FVC2004 (DB1, DB2 and DB3) [48], NIST SD4 [49], the remaining test partitions of NIST SD302a (last 200 fingers) and MOLF (last 200 fingers). There is no overlap between fingers selected for training and testing sets in NIST SD302a and MOLF. Furthermore, to strike a balance between model performance and computational demand, we only use the first impression from different fingers when creating imposter fingerprint pairs; otherwise, the size of the training dataset could even reach 100 M. Additionally, in PrintsGAN, only 2, 500 fingers are selected as a part of our training set and mainly used for model pre-training. In total, our aggregated training dataset contains 3.6M pairs (7.2M images). Table I provides the details of all databases and their usage in this study.

In this study, the interpretable dense registration module and the interpretable fixed-length representation extraction and matching module in IFViT are implemented based on Pytorch and trained independently on four NVIDIA GeForce RTX 3090 GPUs.

In terms of the interpretable dense registration module, a total of 200K fingerprint images (100K fingerprint pair) are used for training. The input images are resized to 128 × 128 to accelerate the convergence process. The module is trained over 100 epochs with a learning rate of $1\times 10^{-3}$. For the interpretable fixed-length representation extraction and matching module, we initially pretrain it employing 6.7M synthetic images from PrintsGAN with a learning rate of $1\times 10^{-3}$. Then, 575K real fingerprint images are used for fine-tuning this module over 70 epochs with a learning rate of $1\times 10^{-4}$. Both of these two modules are optimized by Adam with a weight decay of $2\times 10^{-4}$ and a batch size of 128.

Due to the lack of ground truth labels for pixel-wise correspondences of feature points in real fingerprint pairs, it is challenging to evaluate the performance of the fingerprint dense registration algorithm directly through the positions of correspondences. Instead, we execute alignment using correspondences of feature points produced from the registration algorithm and evaluate the alignment performance based on the similarity of aligned enhanced fingerprint pairs ($I^{\prime}_{e1}$ and $I_{e2}$) and corresponding ROIs ($I^{\prime}_{ro1}$ and $I_{ro2}$) in original fingerprint pairs. Notably, we do not select overlapped regions from enhanced fingerprint pairs ($I^{\prime}_{oe1}$ and $I_{oe2}$) for similarity evaluation due to the potential risk of alignment failure, which could result in an extremely limited overlap region. It might lead to similarity computations being based on invalid background areas, finally leading to unreasonable similarity outcomes.

We select the Mutual Information (MI) metric that has been widely used for image registration in the last decades [50, 51] to measure the similarity in aligned fingerprint pairs. Higher MI observed in the fingerprint pair from the same finger indicates superior performance by the registration algorithm. In this study, we compare the alignment performance of the proposed module with two methods: a prevalent open-source algorithm, Scale-Invariant Feature Transform (SIFT), employed for recent registration in fingerprint matching [33], and the Spatial Transformer Network (STN), an alignment module widely adopted in many fingerprint recognition tasks [3, 17, 6]. The STNs trained in DeepPrint and AFRNet [3, 17] are introduced due to their state-of-the-art performance on fingerprint benchmark datasets.

We report MI values based on different methods in Table II. As we can observe 1) The performance of the interpretable dense registration module in the proposed IFViT is more accurate than other methods with better alignment performance. This is evident in its highest MI value on ROIs of original fingerprint pairs and aligned enhanced fingerprint pairs from the same finger across most datasets. 2) The MI values on aligned enhanced fingerprint pairs are generally higher than those on ROIs of original fingerprint pairs. This is primarily attributed to the MI value being influenced by many factors e.g. the quality disparities in original fingerprint pairs, and the variations in fingerprint appearance caused by various sensors. However, following enhancement by FingerNet, all enhanced fingerprint images adhere to uniform quality standards, reducing their inconsistencies. In this context, the MI value of the latter provides a more precise indication of fingerprint alignment performance. 3) Relative to MI values of unaligned fingerprint pairs, MI values of aligned fingerprint pairs using the SIFT algorithm have shown a marked decline. This is due to the propensity of SIFT to generate incorrect correspondences of feature points, which in turn result in erroneous affine transformations during the alignment procedure. It compromises the structural integrity of original fingerprint images, as illustrated in Fig. 6 (b). 4) The STN in AFRNet and DeepPrint show a comparable level of alignment performance. Notably, STN in AFRNet has superior MI values on ROIs of original fingerprint pairs in the FVC2004 DB1 and NIST SD302a. However, it has slightly worse MI values for enhanced fingerprint pairs compared to the proposed dense registration module. Besides, due to the limitation of STNs in these models to only perform rotation and translation, they cannot achieve pixel-wise alignment for cross-sensor fingerprint pairs, as shown in Fig. 6 (c) (d).

We further explore the influence of fingerprint alignment from various methods on the matching performance based on the trained proposed model, and report their Equal Error Rate (EER) in Table III. The results indicate that the quality of fingerprint alignment significantly impacts the model’s matching performance. When considering pixel-wise correspondences of feature points established by the proposed dense registration module, the aligned fingerprints demonstrate the lowest EER in the ROIs from original fingerprint pairs and the overlapped region extracted from aligned enhanced fingerprint pairs across diverse datasets. In comparison to the other methods, our approach is more robust, even for cross-sensor fingerprint pairs, resulting in a considerable number of precise correspondences of feature points between them as shown in Fig. 6 (a).

We compare the matching performance of the proposed IFViT with two recent well-known works in fingerprint recognition called DeepPrint and AFRNet [3, 17]. Aside from the minutiae points in the DeepPrint network, which are generated by FingerNet as an alternative for creating minutiae maps, the employed model structures are as consistent as possible with those reported in their studies.

From the EER reported in Table IV and the full Receiver Operating Characteristic (ROC) shown in Fig. 7, it can be observed that the proposed IFViT outperforms the other two approaches across six benchmark databases, achieving the lowest EER. The reason for this discrepancy is that we observe a decrease in the performance of DeepPrint and AFRNet when there is an imbalance in the quantity of fingerprints collected from different manners (e.g., rolled, plain, and slap) on the training set. In the context of a limited number of fingers, the developed IFViT can consider both the ROIs of the original fingerprint pairs and the overlapped regions from enhanced fingerprint pairs to reduce the disparity among fingerprint images, thus improving the model’s generalization capacity. Particularly with cross-sensor fingerprints from the MOLF and NIST SD302a, predicting them poses a challenge for both DeepPrint and AFRNet. But for IFViT, it can notably reduce the EER by integrating the dense registration module for pixel-wise precise alignment and combining local and global representations within the matching module.

In addition to demonstrating the interpretability of fingerprint alignment shown in Fig. 6. We further investigate the interpretability of the proposed matching module. Based on the implementation reported by AFRNet [17], we evaluate the interpretability of AFRNet by computing the correspondences of feature points in the fingerprint pairs. In addition, similar operations are also performed on the CNN output of DeepPrint [3] to compare with our method on original fingerprint images. As shown in Fig. 8, when matching fingerprint pairs from the same identity, the matching module in the proposed IFViT can offer more reasonable and superior interpretability. DeepPrint can only focus on local regions and establish limited number of correspondences. AFRNet, due to its adoption of self-attention in ViT that can capture long-range dependencies and global context, similar to the proposed IFViT, can provide more correspondences in the global context, but it is challenging to provide reasonable ones based on the matching result.

Fig. 9 further illustrates the interpretability of IFViT during the matching procedure by using the fusion of local and global representations. It can calculate the matching score for the given fingerprint pair while simultaneously establishing interpretable dense pixel-wise correspondences of feature points across both the ROI extracted from the original fingerprint ($I_{ro1}^{\prime}$ and $I_{ro2}$) and the overlapped region from the enhanced fingerprint ($I_{oe1}^{\prime}$ and $I_{oe2}$) in the matching result. This aids in understanding the reasons behind the decision made by the deep learning model during fingerprint matching. In the example provided in Fig. 9, the imposter pair establishes fewer correspondences of feature points during prediction compared to the genuine pair, with the vast majority of them being incorrect.

To further observe which feature points of fingerprints are possibly used in fixed-length representation-based matching, we provide examples of learned correspondences with a high confidence level. Fig. 10 (a) showcases that feature points of fingerprints at different levels can be identified around learned dense correspondences by IFViT. These features range from the global-level feature (level-1) such as the loop, to local-level features (level-2) represented by minutiae composed of ridge endings and bifurcations, down to the very fine-level feature (level-3) typified by the pore. This implies that, although we do not explicitly integrate domain knowledge into the model, to some extent, the model can infer it on its own. Moreover, we also discover that as the resolution of fingerprint images decreases, such as in cases where pores cannot be clearly distinguished, or when the matching region is limited, there is a noticeable shift towards emphasizing the minutiae in learned dense correspondences as shown in Fig. 10 (b). This is aligned with our understanding that minutiae are one of the most important features adopted in fingerprint matching algorithms.

Instead of using techniques such as Grad-CAM or Saliency Map [33, 6], which rely on gradient computation to identify general regions contributing to predictions in the input fingerprint pair, IFViT can provide superior interpretability at the pixel level. It allows for the detailed examination of specific fingerprint feature points that are used and transformed into a fixed-length representation for matching, offering insights into the decision-making process of black-box deep learning models for both successful and failed fingerprint matching cases.

To justify the incorporation of each component (i.e. fingerprint alignment, local representation and global representation), we conduct an ablation study by sequentially introducing each module into the matching pipeline. The outcomes, as shown in Table V, demonstrate improved matching performance with the addition of each component. It highlights the importance of the proposed alignment module in improving fingerprint matching performance, especially in cases involving fingerprint pairs with limited overlapping areas. Furthermore, the combination of local and global representations can compensate for the deficiencies in semantic information present in each, thereby achieving an improved recognition rate.

The inference time with the proposed IFViT primarily involves three stages: fingerprint enhancement by FingerNet, registration and alignment, and matching. Running on an Intel(R) Xeon(R) Gold 6133 CPU @ 2.50GHz with the NVIDIA GeForce RTX 3090 GPU, it takes approximately 463ms for inference based on a given fingerprint pair (116ms for enhancement, 187ms for registration and alignment, and 160ms for matching). To further improve the computational efficiency, the deep convolutional neural network (i.e. ResNet-18) for local feature extraction in IFViT could be replaced by other shallower networks. Also, the FingerNet could be substituted with alternative fingerprint enhancement techniques with lower computational complexity. We do not consider optimization of them, as this falls outside the scope of this study.