Fingerprint Feature Extraction by Combining Texture, Minutiae, and Frequency Spectrum Using Multi-Task CNN

Before processing with the proposed method, the input fingerprint image is enhanced to increase the recognition accuracy. In this paper, we employ the fingerprint enhancement method based on the intensity gradient, where we use the implementation of FastEnhanceTexture [1]. In the first step of the proposed method, we align the rotation of fingerprint images, which causes a decrease in accuracy in feature extraction. In this paper, we use STN [12] for end-to-end learning of CNNs. The STN estimates the transformation parameters in the localization network and transforms the image based on the estimated parameters. Fig. 2 shows the architecture of the localization network used in the proposed method. Although it is possible to correct the translation and rotation using the STN, we have confirmed experimentally that the correction of the rotation angle alone is sufficient. As a result, the number of parameters to be estimated is reduced and the training is stabilized.

Frequency, texture and minutia features are extracted from the rotation-corrected fingerprint image using ResNet [9]. The frequency features are extracted using CNN as shown in Fig. 3, where ResBlock is the same as the one used in ResNet [9]. The input is two channels of the real and imaginary parts of the frequency spectrum obtained by Discrete Fourier Transform (DFT) of the fingerprint image. In addition, the following preprocessing is applied to extract the frequency features that represent the features of the fingerprint image. The DC component is much larger than the other frequency components and represents the gain inherent to the sensor. In order to reduce the effect of DC component, we apply normalization so that the average of the pixel values is zero, and then perform DFT. Since the energy of the fingerprint image is concentrated in an ellipse of the low-frequency band, the high-frequency region contains only perturbations such as noise and aliasing. In order to consider only the inherent frequency band of fingerprints similar to the BLPOC [11], only the region containing elliptical frequency band is extracted as an input. The CNN network architecture for extracting texture and minutiae features is shown in Fig. 4. Note that the CNNs that extract texture and minutia features share weights up to the middle. The gray-scale fingerprint image is input to the CNN as one channel. In order to extract minutia features using CNN, we introduce a minutia map [6] which can represent the positions and angles of minutiae. The minutia map is obtained with a minutiae map generator as shown in Fig. 5, which is branched from a minutia feature extractor.

Texture features are extracted from the entire image through training of the CNN to be effective for fingerprint recognition. On the other hand, Li et al. [16] demonstrated that using texture features in the local area around the minutiae is more accurate than using features extracted from the entire fingerprint image. Based on this idea, we extract highly discriminative texture feature by using the location information of minutiae for texture feature extraction. In the proposed method, we employ a Minutia Attention Module (MAM) inspired by Spatially Attentive Output Layer (SAOL) [14] for extracting texture feature that takes into account local regions around the minutiae. Instead of the Global Average Pooling (GAP) Layer in the texture feature extractor, MAM is used with the minutia map as an attention mask to extract features that aggregate texture information around the minutiae. Fig. 6 shows the architecture of MAM used in the proposed method.

Let $\bm{X}^{l}\in\mathbb{R}^{C^{l}\times H^{l}\times W^{l}}$ be the feature map output from the $l$-th layer of the texture feature extractor, where $H^{l}$, $W^{l}$, and $C^{l}$ are the height, width, and the number of channels, respectively, of the feature map output from the $l$-th layer ($1\leq l\leq L$). When using the GAP layer and Fully-Connected (FC) layer, the texture feature $\bm{t}_{\rm tex}$ is given by

$$\bm{t}_{\rm tex}={\rm GAP}(\bm{X}^{L})^{\intercal}\bm{W}^{\rm FC},$$

(1)

where ${\rm GAP}(\bm{X}^{L})\in\mathbb{R}^{C^{L}\times 1}$ indicates the spatially aggregated feature vector by GAP, $\bm{W}^{\rm FC}\in\mathbb{R}^{C^{L}\times K}$ indicates the weight matrix of the output FC layer, $(\cdot)^{\intercal}$ indicates the transposition, $C^{L}=1,024$, and $K=512$. On the other hand, when using MAM instead of GAP, the texture feature with attention, $\bm{t}^{\rm att}_{\rm tex}$, is given by

$$\bm{t}^{\rm att}_{\rm tex}={\rm MA}(\bm{X}^{L})^{\intercal}\bm{W}^{FC},$$

(2)

where ${\rm MA}(\bm{X}^{L})\in\mathbb{R}^{C^{\prime}\times K}$ is defined by

$${\rm MA}_{c^{\prime}}(\bm{X}^{L})=\sum_{i,j}A_{ij}(Y_{c^{\prime}})_{ij},$$

(3)

for each $c^{\prime}$, where $c^{\prime}$ indicates the class index $(1\leq c^{\prime}\leq C^{\prime})$, $\bm{A}\in[0,1]^{H^{L}\times W^{L}}$ indicates the attention mask, $\bm{Y}\in[0,1]^{C^{\prime}\times H^{L}\times W^{L}}$ indicates the spatial logits, which is obtained by applying softmax to $\bm{X}^{L}$, and $(Y_{c}^{\prime})_{ij}$ indicates the $(i,j)$-th element of $c^{\prime}$-th feature map of $Y_{c}^{\prime}$. Note that the size of $\bm{W}^{FC}$ in Eq. (2) is changed to $\mathbb{R}^{C^{\prime}\times K}$ since the training is performed as 1,000-class classification problem, i.e., $C^{\prime}=1,000$ in this paper.

This section describes the loss functions to be trained in the proposed method and deep metric learning to improve the accuracy of fingerprint recognition.

We use the MSE loss for estimating the minutia map. Let the ground-truth and estimated minutia maps be $\bm{H}_{g}$ and $\bm{H}_{e}$, respectively. The loss function for minutia map estimation is defined by

$$L_{\rm map}=\sum_{i,j,k}\rho\left\{H_{g}(i,j,k)-H_{e}(i,j,k)\right\}^{2},$$

(4)

where $(i,j,k)$ indicates $(x,y)$ coordinates and $k$ channel, respectively, and $\rho$ is a constant ($\rho=100$ in this paper). The loss functions for texture, minutia, and frequency features are defined by the cross-entropy loss and are denoted by $L_{t}$, $L_{m}$, and $L_{f}$, respectively. The loss function of the proposed method $L_{\rm all}$ is given by

$$L_{\rm all}=L_{t}+L_{m}+L_{f}+{\lambda}_{\rm map}L_{\rm map},$$

(5)

where $\lambda_{\rm map}$ is the weight for $L_{\rm map}$. Note that the weight parameters for STN are also optimized so as to minimize $L_{\rm all}$.

The proposed method employs deep metric learning in training to improve the accuracy of fingerprint recognition. In this paper, we use AdaCos [22], which has been demonstrated to be effective in face recognition. The use of AdaCos makes it possible to increase the cosine similarity of the genuine pairs and to decrease that of the impostor pairs during training. Let the class weight consisting of the center feature vector for each class be $\bm{W}$. The probability $P_{i,y_{i}}$, which the $i$-th input image is classified into the class label $y_{i}$ of $\bm{W}$ is given by

$$P_{i,y_{i}}=\frac{\exp\left(\tilde{s}_{d}^{(t)}\cdot\cos\theta_{i,y_{i}}\right)}{\exp\left(\tilde{s}_{d}^{(t)}\cdot\cos\theta_{i,y_{i}}\right)+\displaystyle\sum_{k\neq y_{i}}\exp\left(\tilde{s}_{d}^{(t)}\cdot\cos\theta_{i,k}\right)},$$

(6)

where $\cos\theta_{i,y_{i}}$ indicates the cosine similarity between the $i$-th input feature and the corresponding class label $y_{i}$ of $\bm{W}$, $\tilde{s}_{d}^{(t)}$ indicates the scaling parameter, $k$ indicates the class label of $\bm{W}$ except for $i$, and $t$ indicate the number of updates. In the case of using AdaCos, the final loss function $L$ is given by

$$L=-\frac{1}{N}\sum_{i=1}^{N}\log P_{i,y_{i}}.$$

(7)

Applying AdaCos to the proposed method, $L_{t}$, $L_{m}$, and $L_{f}$ are calculated by Eq. (7).

In this paper, data extension is important since the training is performed only on a public dataset that is not large. The general data augmentation methods are random contrast and random noise. In the proposed method, we introduce random deformation and random morphology as new data augmentation methods specialized for fingerprint images. During the acquisition process, nonlinear deformation may be added to the fingerprint image and the ridge pattern of the fingerprint may be collapsed. In order to represent deformation of fingers, the fingerprint image is deformed according to the fingerprint deformation model [7] by random deformation. In order to represent collapse of ridge patterns, the morphological filters of dilation and erosion are applied to a part of the fingerprint image. Note that the size of the filter for random morphology is $0.02\sim 0.2$% of the image size according to random erasing [23]. Fig. 7 shows examples of fingerprint images with data augmentation.

The matching score is calculated between fingerprint features extracted by the proposed method. Let the fingerprint features extracted two fingerprint images be $\bm{t}_{1}$ and $\bm{t}_{2}$, respectively. The matching score $S$ is calculated by

$$S=\bm{t}_{1}^{\intercal}\cdot\bm{t}_{2}.$$

(8)

The proposed CNN model is trained as follows. We use 12,000 fingerprint images (DB1, DB2, and DB3) out of 19,200 fingerprint images in MOLF in the experiments. Note that we do not use DB4 and DB5 consisting of latent fingerprint images since we only use the scanned fingerprint images in the experiments. The 12,000 fingerprint images are divided into 9,000 (1,000 fingers $\times$ 3 images $\times$ 3 DBs) for training and 3,000 (1,000 fingers $\times$ 1 image $\times$ 3 DBs) for validation. We perform multi-task learning for identifying 1,000-class labels and detecting minutiae. RMSProp is used as an optimizer and the weight decay for preventing overfitting is set to $10^{-5}$. The learning rates for feature extractors and STN are set to $10^{-3}$ and $5^{-4}$, respectively. Fingerprint images are resized to $256\times 256$ pixels as an input. The size of the minutia map is $128\times 128$ pixels with 6 channels. The effective frequency band is set to 50% of the size of frequency spectrum. The weight for the minutia map is set to ${\lambda}_{\rm map}=10$. The probability of random noise, random contrast, random deformation, and random morphology is set to 80%, 80%, 50%, and 50%, respectively. We use the minutia information detected by VeriFinger SDK 10.0 [5] as the ground truth of the training data.

The performance of the proposed method is evaluated using the public fingerprint image dataset, FVC2004 [2] DB1 and DB2. DB1 and DB2 consist of 800 fingerprint images (100 fingers $\times$ 8 images). Fig. 8 shows the example of fingerprint images from the genuine pairs in FVC2004 DB1 and DB2. DB1 has many low-quality fingerprint images that contain little overlap in the fingerprint region, nonlinear deformation in the same subjects, collapsed ridges due to wetness, or interrupted ridges due to drying. The DB2 has a lot of collapsed ridges, which makes it difficult to detect minutiae in many fingerprint images. We evaluate the matching scores of 2,800 genuine pairs and 368,500 impostor pairs, which are all the combinations of the dataset. The verification accuracy is evaluated by Equal Error Rate (EER), where is defined as the error rate where False Acceptance Rate (FAR) and False Rejection Rate (FRR) are equal.

The effectiveness of refinement techniques introduced in the proposed method, i.e., frequency feature, deep metric learning, data augmentation, and MAM, is evaluated by the following ablation study using FVC2004 DB1 and DB2. In this experiment, we compare the verification accuracy of the proposed method with Verifinger SDK 10.0 [5] and the deep learning-based method [8] to demonstrate the effectiveness of the proposed method. Note that we replaced Inception modules used in [8] to ResNet modules used in the proposed method since the number of classes in the training data used in this experiment is quite small compared with the original implementation of [8], where 382,914 classes are included in the private training data. Table 1 shows the summary of experimental results. In FVC2004 DB1, the EERs of the proposed method have been improved by adding refinement techniques. The EER of the proposed method (D), which employs all the refinement techniques, is lower than that of VeriFinger and Engelsma [8]. In DB2, the accuracy of the proposed method has been also improved by adding refinement techniques. The proposed method (C) is more accurate than VeriFinger and [8]. On the other hand, the proposed method (D) with MAM had more errors than the proposed method (C) without MAM. This is because the fingerprint image in FVC2004 DB2 has many areas with collapsed ridges and blurred minutiae as shown in Fig. 8, and therefore, it may not be possible to apply proper attention to them.

We evaluate the accuracy of the proposed method in neonate fingerprint recognition, which is an example of fingerprint images that are extremely difficult to extract minutiae from. In this experiment, we use the neonate fingerprint image dataset used in [15]. This dataset contains neonate fingerprint images taken with a 1,920-ppi sensor at 2, 6, and 18 hours of age. The number of genuine and impostor pairs is 664 and 9,632, respectively. The EERs of VeriFinger, the proposed method (C), and (D) are 42.6%, 34.9%, and 38.4%, respectively. In VeriFinger, wrinkles and pores are extracted as minutiae as shown in Fig. 9. The EER of VeriFinger is high due to the use of many wrong minutiae for fingerprint matching. On the other hand, the proposed method is more accurate than VeriFinger SDK in the neonate fingerprint recognition since the proposed method uses texture and frequency features in addition to minutia feature for matching. Minutiae extracted from neonate fingerprint images using the proposed method are shown in Fig. 9. This figure shows that the proposed method was able to extract relatively correct minutiae, however, the accuracy of the proposed method (C) with MAM was reduced due to the small number of minutiae.

We present qualitative evaluation of the minutia extraction using the proposed method. Fig. 10 shows the minutiae extracted using MINDTCT [3], MinutiaeNet [18] , VeriFinger, and the proposed method (D) for fingerprint images with quality values of 96 and 45 in VeriFinger. For the fingerprint image with a quality value of 96, the same minutiae were extracted by all the methods. In the case of the fingerprint image with a quality value of 45, there are many false positives for MINDTCT, and there are many undetected for MinutiaeNet, while the proposed method (D) is able to extract the majority of minutiae extracted by VeriFinger.

The effectiveness of the proposed method is verified using a saliency map that visualizes the effect of each pixel in the input image on the extracted features. We use Guided-back propagation [20] to make a saliency map from CNN weights. Fig. 11 shows the saliency maps of texture and minutia features. While CNN for extracting minutia feature focuses on the entire fingerprint image, CNN for extracting texture feature focuses on the core region. CNN with MAM focus not only on the core region, but also on the region around minutiae. Fig. 12 shows the saliency maps of frequency feature. CNN for extracting frequency feature focuses on the frequency bands inherent in the fingerprint pattern and we confirmed that the individual frequency features are extracted.