Palm Vein Recognition via Multi-task Loss Function and Attention Layer

K Simonyan proposed the VGG deep learning network, A Zisserman in 2014 [18], using an architecture with very small (3x3) convolutional filters for a comprehensive evaluation of networks with increasing depth, and introducing a 1*1 convolutional kernel in the convolutional structure of VGG, without affecting the input-output dimensionality, and introducing nonlinear transformations to increase the expressiveness of the network and reduce the computational effort. They demonstrate that increasing the depth to 16-19 weight layers can significantly improve the existing technical configuration.

VGG-16 has 16 layers, divided into 13 convolutional layers and 3 fully connected layers. The first time, after two convolutions of 64 convolutional kernels, one pooling is used, the second time, after two convolutions of 128 convolutional kernels, another pooling is used, and two more repetitions of three 512 convolutional kernels are convolved and then pooled. Finally, three full connected layers are used to obtain the results. The schematic structure is shown below[51].

At the end of the network, VGG uses a set of fully connected nodes for the output, where the final output size is the number of categories of images in the dataset. In the middle of each fully connected layer, we add a dropout layer to randomly deactivate a certain percentage of neurons, which avoids the occurrence of model overfitting [42]. Finally, the obtained results are fed into the $Softmax$ function (1) to find the probability distribution, and finally, the loss is calculated and the error is back-propagated by the cross-entropy loss function (2).

The softmax function can effectively map the output to a probability distribution space and is particularly effective for solving classification problems.

As more and more machine learning application scenarios emerge and existing supervised learning that performs better requires large amounts of labeled data, so transfer learning is receiving more and more attention. The goal of transfer learning is to apply the knowledge or patterns learned on a domain or task to a different but related domain or problem [22].

For this paper’s palm vein recognition task, we use a VGG-16 network pre-trained on the ImageNet[21] dataset for transfer learning. imageNet is pre-trained on more than one million images containing more than 20,000 categories. The data volume of the palm vein dataset involved in this paper is small, because there is still some difference between object recognition and palm vein recognition. Considering the above, we freeze the first 30 convolutional layers in VGG-16 with gradient, i.e., they do not generate gradient information during the training process, thus the parameters will not be updated. Instead, the remaining 34 convolutional layers are trained with fully connected layers, which greatly shortens the training time and increases the convergence rate while ensuring prediction accuracy.

When a human sees a set of information, human neurons will automatically scan the global image based on experience and the current target, and obtain the target area that needs to be focused on, that is, the focus of attention. More attention is then devoted to this region to obtain more details about the target and to suppress other useless information. The following figure is a schematic diagram of the application of the attention mechanism in image processing. For such a journal screenshot, attention is focused on the image and the caption, respectively, under our life experience. [23].

A brief description of the attention layer is as follows. First, two hidden states (Encoder and Decoder) are created, which are equivalent to the input and output of the structure. Subsequently, a dot product is performed with each hidden state in Decoder and Encoder, and the result is noted as $Score$. Then all the scores are sent to the softmax layer (1), so that the higher the original score of the hidden state, the higher its corresponding probability, thus suppressing that invalid or noisy information.

Next, the hidden state of each Encoder is multiplied by the $Score$ after softmax, and then all aligned vectors are accumulated and the context vectors are sent to the Decoder to obtain the decoded output. the internal structure of the Attention layer is as follows.

For the palm vein recognition task, we want the neural network to focus more on the features and patterns of the hand, and give a relatively low weight to secondary factors such as shadows and illumination. Therefore, we use the spatial Attention mechanism before image input and the channel Attention mechanism for image feature extraction [24]. Another advantage of the Attention layer is that it is a plug-and-play adaptive module, which does not change the dimensionality of the image and does not change the subject model. It can also be applied to the training of transfer models before the input or after the output of the subject model.

In this paper, we build and train the model based on the Pytorch framework [26], and optimize the model using the Adam algorithm [25]. Adam algorithm is different from the traditional stochastic gradient descent which maintains a single learning rate to update all the weights, and the learning rate does not change during the training process. In contrast, the Adam algorithm designs independent adaptive learning rates for different parameters by computing the gradient’s first-order moment estimates and second-order moment estimates. This allows the loss to drop faster in the early learning period without jaggedness in the late learning period due to excessive learning rate.

Meanwhile, we use the (4)$L_{2}$ penalty term for the linear output layer, which enables the network to output a relatively sparse feature vector, which is more helpful for the subsequent similarity calculation. The $L_{2}$ penalty term can limit the size of the secondary elements in the weights, and in terms of optimization difficulty $L_{2}$ penalty is more convex compared to $L_{0}$ and $L_{1}$ penalties [32], so the optimization is less difficult and the results obtained are better. Also, the inclusion of the regular term helps to avoid overfitting because it compresses part of the parameters to a number close to 0, making the model simpler.

Where $\left\|w\right\|_{2}$ is the L2 penalty term, which is easier to compute than the L1 parametrization, and the derivative values are defined at each point. Including the L2 parametrization is achieved simultaneously to facilitate optimisation and obtain a sparse solution. The expression of the loss function after adding the canonical term is as follows.

After feature extraction by the above deep learning network, for the input $i$th palm vein image, we can get the feature vector $V_{i}$. Subsequently, to determine whether the $i$th and $j$th images are from the same person, we use the cosine similarity function for matching comparison, whose expression is shown in (5).

This function accepts two feature vectors extracted from the neural network and obtains the similarity by employing a generalised cosine operation. When $s>\alpha$, we decide that the two palm vein pictures originate from the same person, and reject them in the opposite case. Obviously, in the most ideal case, for any $V_{i}$ and $V_{j}$ coming from the same person, there should be:

And for any $V_{i}$ and $V_{j}$ from different persons, there should be:

But in practical scenarios, we can relax this judgment condition. We usually use the threshold $\alpha$ to determine whether it is the same person. It follows that as long as the difference between the similarity $s_{i}$ for any pair of palm veins from the same person and the similarity $s_{j}$ for any pair of palm veins from different people is large enough, then we can always find a suitable $\alpha$ that allows us to distinguish whether a given input is the same person or not. That is, the difference between the similarity values computed by the similarity function from the same person and different persons should be as large as possible, i.e., equation (8) should be as large as possible. We call this equation matching goodness of fit, which is counted as $MG$.

This facilitates us to pick a better threshold $\alpha$ more easily, making the rate of wrong rejections and wrong acceptance decrease.

Our loss function is shown in (4) for the traditional classification task. That is, we solve the problem posed by (8) by solving the (4)-style. However, there is no research to prove that there is a relaxation optimization relationship between the two, and it may not be effective if the classification model is more indirect for the matching task. Therefore, we would like to add (8) to the loss function for optimization. The reconstructed loss function is shown as follows. It can be seen as a linear combination of a classification task and a matching task.

Although we want the value of equation (8) to be the largest for all data (i.e., $1-MG$ is the smallest), it is very detrimental to training because it takes too much time to compute $MG$. However, since we divide the training batches randomly, each training batch $1-MG$ minimum is equivalent to the overall $1-MG$ minimum.

The parameter $\theta$ controls the weight between two tasks, $\theta=1$ for a pure classification task and $\theta=0$ for a pure matching task. It is worth noting that the premise of computing the batch $MG$ is that there must be both data of the same category and data of different categories in each batch. To achieve this, we may need to increase the data in each batch, which inevitably leads to a decrease in training accuracy. We will elaborate on the specific parameter selection in Chapter 3.

To fit the input size of the convolutional network, we first cropped the image to 224*224 centered on the palm to be the ROI(region of interest). The images were sourced from the same person by recording 6 consecutive images and 6 images 10 days later. That is, there are 12 pictures from the same hand each. Therefore, we use the former as the training set, counting as the set $TR$ for the classification task. And every 9 photos are taken out as the validation set, which is counted as the set $V$. For the matching process, in order to simulate the entry-comparison process in real application scenarios, we extract the last 10 sets of photos (i.e., 10 different palms) from the dataset as the test set for zero-experience learning matching to test the matching effect, and match the photos in the two datasets one by one to form a 60*60 dataset, which is counted as The set $TE_{m}$.

Image enhancement is mainly used to highlight certain information or to meet specific needs. Techniques for image enhancement generally fall into two main categories: null domain image enhancement and frequency domain image enhancement algorithms, in addition to image enhancement algorithms that combine fuzzy theory and genetics.

Space domain image enhancement is the direct processing of image pixels and is essentially based on a grey-scale mapping transformation. Commonly used methods include histogram equalization, spatial filtering, and grey scale transformations.

The essence of frequency domain image enhancement is convolution. It mainly uses the Fourier transform or wavelet transform to transform the image to the frequency domain, then frequency domain enhancement, and finally inverse transformation back to the null domain to obtain the desired image. Common methods include low-pass filtering, high-pass filtering, etc.

Enhancement of palm print images can eliminate noise in the original image, balance illumination effects and enhance contrast. This paper selects four image enhancement methods: histogram equalization, Laplace operator, Constrained Contrast Adaptive Histogram Equalisation algorithm (CLAHE), and log transform. Two samples were randomly selected on two datasets respectively and processed as shown in the following figures .

Among them, the log transform was the least effective on the TJU-P dataset, while the laplacian, clahe, and hist effectively removed the noise from the original images. On the PolyU M_N dataset, the laplacian operator processed results do not differ much from the original image, while the remaining methods each highlight distinct features after enhancement.

We compared several optical enhancements mentioned in II. B. 2) and tested them in both datasets with the following results.

On the TJU-P dataset, the enhanced palm prints of the images by three different ways were compared with the results of the original images after 15 training sessions. And the batch size is 64, the learning rate is $5\times 10^{-5}$ and the optimiser is Adam. The results for loss and matching accuracy are shown in Figure 9 and Figure 9.

From the experiments, it can be seen that after image enhancement by histogram equalization, loss converges the fastest and has the highest recognition rate improvement; CLAHE has the second highest effect; Laplace operator processing has the least significant effect, and the accuracy rate is even lower than the recognition rate of the unprocessed image.

On the PolyU M_N palm vein dataset, only these two image enhancement methods were used for comparison as hist and CLAHE were shown to achieve better results through the above experiments. The parameters were chosen: the number of iterations was 15, the batch size was 8, and the learning rate and optimiser were the same as before. The comparison results are shown below in Figure 11 and Figure 11.

As can be seen from the figure: on the PolyU M_N palm vein dataset, the enhancement effect of image enhancement using hist and clahe is almost the same, and hist is slightly better in terms of convergence speed and accuracy.

We then tested the effect of introducing attentional mechanisms on the experimental results. Using the Poly-U M_N dataset as an example, we tested the matching accuracy with and without the inclusion of the attention mechanism. The results are presented in Figure 12.

The results show that the attention mechanism’s introduction largely improves convergence speed. It also improves the accuracy of the model at convergence.

The attention mechanism can be used to weight different spatial regions or channels, as can be seen from the weighting of channels by the attention mechanism in the Figure 13, where the weight of some channels is even set to 0. This also serves the purpose of feature selection. Since we found that the attention mechanism performed well in the model, we will use attention as the underlying structure in all subsequent experiments.

Batch size = 32 is set, 20 rounds are trained, and a regularization strength of $\lambda=0.001$ is used. The resulting classification accuracies and loss functions for the training set, validation set and test set are shown in Figure 15 and Figure15, and the training was stopped at the 7th round due to the fast convergence rate.

The training took a total of 173 seconds for 7 rounds, and it can be found from the figure that the model converged well. The accuracy is 100% on the training set and 99.85% on the validation set. We also tried different combinations of parameters, and the results are shown in the table 1.

The results show that with the regularization strength of $\lambda=0.001$, the final loss and accuracy are not sensitive to the number of batches and training sessions. However, for $\lambda=0.01$ the model appears to be underfitted.

At the same time, we found that too large batch size will greatly affect the training effect. Although a large batch size helps to speed up training, with the support of a powerful GPU, the time consumption here is negligible.

Next, the matching effect is tested on the dataset $TE_{m}$. Also, to get a better matching effect, we test thresholds of 0.6, 0.65, 0.7, 0.75, and 0.8, calculate their AUCs and compare them. Figure 16 shows the comparison graph of AUC, and it can be found that the best result is obtained when the threshold is 0.6. Figure 18 shows the prediction when the threshold is 0.6, where yellow is the correct prediction and blue is the wrong prediction.

The overall correct rate was 95.08%. Among them, 279 pairs were correctly accepted, 121 pairs were incorrectly rejected, 56 were incorrectly accepted, and 3144 were correctly rejected. Although the overall accuracy rate is very high, the acceptance rate for positive samples is very low, even less than 50%. Considering that the classification accuracy has reached 100%, we need to modify the loss function in the training to further improve the matching accuracy.

Meanwhile, we used the (8) equation to calculate the goodness-of-fit under the matching task, i.e., the difference between the average similarity from the same sample and the average similarity from different samples were calculated separately. The experimental results show that the average similarity of samples from the same palm is 0.7700, and the average similarity of samples from different palms is 0.2531, with a difference of 0.5169, which is the $MG$. We find that the correct acceptance rate of this method is low and far from the standard for practical use, so we will try our improved objective function next.