SWIS: Self-Supervised Representation Learning for Writer Independent Offline Signature Verification

Most researchers have leveraged supervised learning methods [rantzsch2016signature, dey2017signet, ruiz2020off, wan2021learning, parcham2021cbcapsnet, bhunia2019signature] for offline signature verification. While handcrafted feature analyses have comprised the bulk of studies in this domain [bhunia2019signature, alaei2017efficient, hafemann2017review, banerjee2021new], various deep learning-based methods have also been proposed, particularly dwelling on metric learning approaches [rantzsch2016signature, dey2017signet, ruiz2020off, wan2021learning]. Dey et al. [dey2017signet] introduced a contrastive loss based convolutional Siamese network for handwritten signature verification. Ruiz et al. [ruiz2020off] combined synthetic signature generation with siamese networks for the verification task. The authors of [shariatmadari2019patch] proposed a hierarchical CNN to learn features from patches of genuine signatures. Zhu et al. [zhu2020point] sought to tackle intra-writer variations by introducing a point-to-set metric for offline signature verification. Zois et al. [zois2017parsimonious] and Berkay et al. [berkay2018hybrid] explored sparse dictionary learning and hybrid two-channel CNNs respectively for signature verification, whereas the authors of [wan2021learning] proposed two triplet losses, each to tackle random and skilled forgeries respectively. Other works include an interval symbolic representation and fuzzy similarity measure [alaei2017efficient] based handcrafted feature engineering method; region-based metric learning [liu2021offline]; a neuromotor equivariance inspired model [diaz2016approaching]; a recurrent neural network architecture [ghosh2021recurrent] and a graph neural network based approach [roy2021offline]. Nevertheless, all the aforementioned works are fully supervised methods and therefore, share the common bottleneck of data scarcity. To this end, we demonstrate the first use of self-supervision for offline signature verification.

Self-supervised learning [jing2020self] aims at developing a pre-training paradigm to learn a robust representation from an unlabelled corpus for generalization to any given downstream task. Widely studied in recent years, several pretext tasks have been proposed, such as solving jigsaw puzzles [noroozi2016unsupervised], image colorization [zhang2016colorful], super-resolution [ledig2017photo] and cross-modal translation [bhunia2021vectorization], to name a few. Contrastive learning based self-supervised pre-training [chen2020simple, he2020momentum] has also gained popularity, which aim at learning similarity between augmented views of the same image while distancing views from different images. SimCLR [chen2020simple] and MoCo [he2020momentum] are some of the state-of-the-art contrastive learning-based self-supervised algorithms in literature. In a different approach, the authors of [zbontar2021barlow] aimed at simultaneously maximizing similarity and minimizing redundancy between embeddings of two distorted views of an image.

In signature images, it is essential to capture the stroke information from the different authors as well as to learn the variations in the signatures of the same individual. This allows the model to learn representations which not only discriminates one author from another, but also helps differentiate between genuine and forged signatures of an individual. To feed the stroke information without any human supervision, we divided the signature images into patches of dimensions $32\times 32$ with an overlap of 16 pixels from a signature image reshaped to $224\times 224$. This gives 169 patches from a single image of dimensions $32\times 32$. As the base encoder we choose ResNet-18 [he2016deep]. When the patches are passed through the encoder, we obtain an output of $1\times 1\times 512$ from each patch. We arrange the patches into a grid of $13\times 13$ such that the output from a single image after rearranging is $13\times 13\times 512$. After applying global average pooling, we obtain an output feature vector of dimension $1\times 512$. This feature vector is then passed through a non-linear projector with 1 hidden layer and output dimension $512$ to obtain the final output.

The main objective behind dividing the signature image into overlapping patches is to extract stroke information without human annotations of the same. To facilitate stroke information learning, we intend to decorrelate the output dimensions of the encoder such that each output gives linearly uncorrelated information. Linearly uncorrelated information means that the encoder discards redundant features by minimizing the cross-covariance between each dimension and extracts only meaningful information from the input by creating a bottleneck of information flow from the input space to the feature space. We normalize the feature vectors such that they lie within an unit hypersphere $\mathcal{S}^{D}$, where $D$ is the dimension of the feature vector. The diagonal terms of the cross-covariance matrix are optimised such that it equates to 1. This allows for the uncorrelated dimensions to have zero mean and unit variance.

For forming positive pairs, we augment a single signature image in two randomly chosen augmentations. The augmentation details are mentioned in Sec. IV-B. The images are then divided into patches as mentioned before and then passed through the encoder and the projector.

Thus, the proposed loss function has the form:

where $z_{k}^{i}$ is a scalar value at $i$-th dimension of the $k$-th centered and normalized feature vector $z_{k}$. Thus, the pre-processing steps before feeding the feature vector $z_{k}^{i}$ to the loss function are as follows

It is to be noted that $z^{i}_{k}$ and ${z^{\prime}}^{i}_{k}$ are not same and are obtained from the each element of a positive pair. Thus, the proposed loss function does not optimize the terms of a cross-covariance matrix in the true meaning of the term. We can refer to this matrix as a Pseudo cross-covariance matrix, and in Figure 1, we illustrate how this matrix is formed using the feature vectors. The diagonal elements (coloured white) constitute the second term of the loss function $\mathcal{L}_{C}$, whereas the rest of the non-diagonal elements constitute the first one.

From eq. 1, we can see that optimizing the proposed loss function allows us to decorrelate the dimensions of the output by converting the pseudo cross-covariance matrix into a diagonal matrix with positive diagonal values. This allows for positive correlation between the output in the same dimension for the samples in the positive pairs. It also linearly decorrelates one dimension from another such that the representations learnt by the encoder discard redundant information and maximize the useful information. In other words, we can describe the representation learning process as learning linearly decorrelated generative factors of the input. One thing to note is that, the cross-covariance and auto-covariance terms are not used in their true sense as the two samples used in a positive pair are not identical.

The model architecture used in the pretraining phase is given in Figure 2. The diagram shows the input that is fed to the ResNet [he2016deep] encoder. The input is reshaped to $169\times 32\times 32\times 512$ before passing it through the encoder. Figure 2 also shows an example of the input that is used in the pretraining phase.

For predicting whether a signature is forged or genuine, we are not training any siamese neural network model for the downstream task. Instead, we take 8 reference signature for each user and use these reference signatures to train a Support Vector Machine (SVM) classifier with radial basis function kernel. Since, the objective in our work is writer-independent signature verification, we assume that the user for which the signature is being verified is known. If the signature of a particular user is forged, it is expected to be mapped far from the cluster of the genuine reference signatures. Thus, we assume that the forged signature will be mapped outside the decision boundary of that particular user. If the user is predicted correctly and the signature is genuine, we count it as a correct prediction. Similarly, if the predicted user and the ground truth user is not the same and the signature is actually forged, then also it is counted as a correct prediction. In all the other cases, the prediction is considered as wrong.

By using a SVM classifier, we depend on the feature extraction capability of the pretrained encoder to express the input in terms of its linearly decorrelated generative factors. Whereas all the contemporary state-of-the-art supervised algorithms use siamese type architecture or supervised contrastive learning framework for the offline signature verification task. Our approach also reduces the downstream inference time by a several orders of magnitude.

In this work, we used two datasets, namely, BHSig260 [pal2016performance] and ICDAR 2011 [alvarez2016offline]. BHSig260 dataset contains signatures from 100 writers for Bengali and 160 writers for Hindi signatures. For each writer of both the languages, there are 24 genuine and 30 forged signatures. Among the 100 writers in the Bengali subset, we randomly select 50 writers for the training set and the rest 50 are used for testing. For the Hindi subset, we randomly selected 50 writers for self-supervised pretraining and the rest 110 writers were left for testing. Similarly, for ICDAR 2011 Signature Verification dataset, there are signatures for Dutch and Chinese languages. The subset of the Dutch signatures contains signatures from 64 writers. Among the 64, 10 writers are included in the training set as reference and the rest 54 are included in the Test set. The divisions are however given in the dataset. The Dutch subset contains 362 signatures in the training set and 1932 signatures in the test set. Among the 1932 signatures in the test set, 646 signatures are used as reference signatures for the writers and the rest as the unknown samples. Similarly, for the Chinese subset, the reference or training subset and the test set, both contain signatures from 10 writers. The number of signatures in the training and test set in the Chinese subset is 575 and 602, respectively. Among the 602 signatures in the test set, 115 signatures are used as reference signatures and the rest are used as the unknown samples. In the test set, however, there are 8 reference genuine signatures for each writer. To adhere to this structure, we randomly selected 8 genuine signatures from the test set of BHSig260 dataset for each writer and used it as the reference set, for both Bengali and Hindi languages.

The train and test set divisions used in our work is described in the table below:

For the pretraining phase, we used different number of epochs for different datasets. The models were trained by optimizing the loss function given by 1 using LARS [you2017large] optimizer. We used a learning rate of 0.1 and a momentum value of 0.9. The batch-normalization and bias parameters were excluded from weight normalization. We decayed the learning rate following a cosine decay schedule with a linear warmup period of 10 epochs at the start. The decay was scheduler for 1000 epochs irrespective of the number of training epochs.

For the ICDAR datasets, we pretrained the model for 500 epochs. Whereas for the BHSig260 dataset, the pretraining was carried out for 200 epochs only. For both the datasets, the batch size used was 32.

To ensure that the pretrained models learn generalized and robust features, we applied several augmentations, such as, color jittering, affine transformation and random cropping to $224\times 224$. The images obtained after augmentation were normalized to the range $[-1.0,+1.0]$.

As not all images in the datasets contain perfectly cropped signature images, we needed to crop the signature images such that the input to the encoder contained is a tightly bounded signature image. To achieve this objective, we performed Otsu’s thresholding [otsu1979threshold] followed by finding the bounding box with least area containing all non-zero pixels around the centre of mass of the image. After this preprocessing step, the images were divided into 169 patches of dimension $32\times 32$ and fed to the encoder for training.

The downstream task we considered in our work is the writer-independent classification of signatures into two classes: genuine or forged. The predictions were obtained using the procedure described in Section III-C. The results obtained by the proposed model in the downstream task on the datasets ICDAR 2011 and BHSig260 signature verification datasets are given in Table II. These two datasets contain large number of signatures in different languages to validate the effectiveness of the proposed model.

In this section, we show how the proposed loss function fares at training the encoder to learn representations from the data. As shown in Table II, in spite of trained in a self-supervised manner, the proposed framework performs satisfactorily on both the multilingual datasets. Table II also presents the comparative results of one of the state-of-the-art self-supervised algorithms (SimCLR) on the same data. From Fig. 3, we can see that the proposed algorithm performs better at producing distinct clusters for ICDAR 2011 Chinese and BHSig260 Bengali dataset, whereas the plots for ICDAR 2011 Dutch and BHSig260 Hindi datasets look equally well-clustered for both the proposed model and SimCLR. It should be mentioned here that the SimCLR algorithm was trained for 1000 epochs on the ICDAR 2011 dataset (both, Dutch and Chinese), whereas the proposed model was trained for only 500 epochs on the same datasets. Both the methods were trained for 200 epochs on the BHSig60 dataset.

To further validate our proposed self-supervised pipeline for offline signature verification, we compare its performance with some fully supervised methods in literature. The results have been tabulated in Table III. We observe that the proposed framework performs competitively against the fully supervised works on the BHSig260 datasets, outperforming [pal2016performance] by a large margin on the Bengali signature dataset. Moreover, the low FAR and FRR values obtained by the proposed method on the signature datasets affirm its potential in separating forged signatures from the genuine ones.