Adversarial Training for Face Recognition Systems using Contrastive Adversarial Learning and Triplet Loss Fine-tuning

Since 2014, when the observation was made that applying small perturbations to inputs can cause dramatic shifts in model outputs[16], the field of adversarial machine learning has been elevated to the forefront of computer vision research, with numerous techniques for generating so-called adversarial attacks being published each year. One such technique, FGSM[8], is based upon the model gradients, and serves as the foundation for other techniques such as PGD[13]; this class of adversarial example generation technique generally applies per-pixel perturbations budgeted according to some other techniques, such as DeepFool[14]and C&W [2], similarly apply pixel-level changes, often while simultaneously reducing the overall amount of applied perturbation and still maintaining an impressive rate of misclassification. Though many of these pixel-level perturbation techniques are generalizable from the task of image classification to broader applications, there has been some research into adapting these techniques specifically to suit the domain of facial recognition [1].

On the other hand, and particularly in the field of facial recognition, another popular method of generating adversarial attacks involves applying perturbations not at the level of the individual pixel, but instead at the level of salient features within the image (that is, the mouth, eyes, etc.). AdvFaces [6], for example, is a genetic algorithm-based approach which generates so-called “perturbation masks” to be applied to face images, resulting in changes in facial features that range from imperceptible to visually apparent; crucially, these perturbations can be generated in either an untargeted (obfuscating) or targeted (impersonating) manner. SemanticAdv [15] generates similar perturbations by employing a semantic segmentation strategy and applying transformations to salient semantic regions (hairlines, for example). In a similar vein, GFLM [4] applies geometric scaling to salient image regions, with horizontal and vertical transformation hyperparameters controlling the level of applied perturbation.

Crucially, all of the aforementioned techniques are supervised adversarial techniques; that is, they require labeled datasets to successfully generate perturbations. The burgeoning field of self-supervised adversarial generation is a relatively recent and active area of research [10].

Arguably the most thoroughly and successfully researched adversarial defense strategy is that which involves adversarial training, or retraining the attacked model against adversarial examples in order to improve its adversarial robustness (all of the adversarial training citations there are a bunch). This increase in robustness is generally accompanied by a notable decrease in performance against benign inputs, leading to a well-established trade-off between robustness and accuracy [19][17]. Furthermore, adversarial training itself raises concerns with respect to its scalability as a practice, particularly regarding the transferability of adversarial robustness; models trained against one set of adversarial examples may not be resistant to examples generated with a different technique, resulting in the need for repeated retraining [12].

Though adversarial training is certainly the most popular avenue of exploration in improving adversarial robustness, other techniques have been suggested. One such technique involves applying random resizing and padding during input evaluation, and in doing so achieves an impressive degree of robustness against standard adversarial attack techniques [18], though this method suffers from a weakness to translation-invariant adversarial attack strategies [7]. In the domain of facial recognition specifically, another common avenue of defense involves applying a pre-processing purification technique to the adversarial input, aiming to systematically remove perturbations, though these techniques can often be subject to exceptionally poor classification rates against benign images due to their tendency to “purify” benign images (removing perturbations where there are none) [5].

We first introduce the concept of contrastive loss, which serves as the foundation for self-supervised contrastive adversarial learning. Let sim($u$, $v$) be a measure of the similarity (e.g., cosine similarity) between $u$ and $v$. Consider a positive pair of images, or two pairs belonging to the same labeled class, ($i$, $j$). The contrastive loss for this pair, then, can be computed as

where $\mathbbm{1}_{[k\neq{i}]}\in\{0,1\}$ represents an indicator evaluating to $1$ iff $k\neq i$, $\tau$ represents temperature, and $z_{i}$ and $z_{j}$ represent model output for inputs $i$ and $j$ [3].

Self-supervised contrastive learning aims to learn general features on a dataset without labels. This is done by focusing on the similarity between images within a class and increasing the distance between classes. For each image in a dataset, augmented, positive versions are generated through cropping, resizing, recoloring, etc. These augmented versions of the image will ideally have the same classification as the original image. By using a contrastive loss function, the difference between the model’s encodings of an image and its augmented versions is minimized.

In the example of SimCLR, a contrastive learning framework, the method uses random cropping followed by resizing the original image size, randomized color distortions, and random Gaussian blurs, as the three sequential augmentations to generate similar examples. Once the extra images are created, ResNet-50 is used to create the vector representations of the inputs. The projector used is two-layer multi-layer perception (MLP), and the loss function is Normalized Temperature-scaled Cross Entropy Loss (described in the previous section).

As opposed to class-wise attacks, instance-wise attacks generate perturbations for single inputs to cause the model to misclassify it as a different sample. This type of attack is used for unlabeled sets of data, which would lack the class distinction for a class-wise attack. These attacks attempt to maximize the self-supervised contrastive loss between the perturbation and the anchor image. For our work, we employ instance-wise adversarial attacks with PGD during adversarial contrastive pre-training, following the methods proposed in [9].

A key aspect of our methodology is the exploration of self-supervised contrastive adversarial training as a pre-training technique applied to facial recognition systems. In this stage, we generate instance-wise adversarial attacks using PGD with contrastive loss, and apply contrastive learning as a vehicle for adversarial training. Specifically, we aim to apply contrastive loss to a clean image, $x$, and its instance-wise adversarial example, $x_{adv}$, for all images in the dataset.

Algorithm 1 describes the proposed method in detail. The technique is fundamentally similar to the A2S technique proposed in [9], but with the crucial difference that it is employed here as pre-training technique.

Augmented images that are generated in self-supervised contrastive learning are called positive images, as they should be associated with the base, or anchor image. Any images belonging to or formed from a differing class are negative images. To calculate triplet loss on a model, multiple triplets are formed and compared after they are fed through the model. Each triplet consists of an anchor image, a positive image, and a negative image. Intuitively, the encodings of the anchor and positive images would ideally be much more similar than the negative image is to either. The triplet loss function returns the difference of the distances between the anchor and positive images and the anchor and negative images, or zero if the difference is negative.

In this work, we apply triplet loss-based adversarial fine-tuning following self-supervised adversarial pre-training. Triplet loss-based adversarial fine-tuning is a supervised technique, and involves gathering an anchor image, positive image, and negative image for each input in the dataset. In order to incorporate adversarial training into triplet loss-based learning, we propose a new technique for generating anchor images for each input in the dataset.

First, for each image $x$, a PGD adversarial example, $x_{adv}$, is generated. In this stage of adversarial training, PGD adversarial examples are generated using triplet loss. In order to form the anchor for triplet learning, $a$, $x_{adv}$ is concatenated to $x$, with the resulting output serving as the anchor. To our knowledge, ours is the first work to investigate generating adversarial examples against facial recognition systems by leveraging triplet loss in this manner.

Algorithm 2 describes our technique in detail.

CelebFaces Attributes Dataset (CelebA) consists of over 200,000 images of celebrities, each with name labels and binary attributes for the appearance of various body part types (mustache, wavy hair, etc.) and clothing (glasses, hat, etc.). For initial training purposes, the whole dataset is used, but for later experiments with contrastive learning, the dataset is filtered to include only classes (individual people) for whom multiple images are available. This is because triplet loss, discussed in a later section, requires at least two images for each class. After this filtering process, 9,136 remain available, with 8,156 designated for training and 980 for validation.

To establish a baseline by which to compare our proposed methodology (adversarial pre-training with contrastive loss followed by triplet loss-based adversarial fine-tuning), we first examine the effect of triplet loss adversarial training alone on model robustness. In order to generate adversarial examples, an $\ell_{\infty}$-norm PGD attack with triplet loss is used, with $\epsilon=8/255$, $\alpha=2/255$, and maximum iterations set to 7. The model, already pre-trained on clean images from our dataset, is then adversarially trained using the triplet loss technique described in Algorithm 2 for 250 epochs. A batch size of 128 is used for training.

In this stage, we examine the effectiveness of applying self-supervised contrastive adversarial pre-training, and then fine-tuning afterwards in a supervised manner using triplet loss-based adversarial training. For contrastive adversarial pre-training, instance-wise adversarial examples are generated using an $\ell_{\infty}$-norm PGD with contrastive (as opposed to triplet) loss, maintaining all the same hyperparameters described above. The model is then trained for 500 epochs in a self-supervised contrastive manner before being fine-tuned for 50 epochs using triplet loss-based adversarial training. This is analogous to performing 500 epochs of Algorithm 1 followed by 50 epochs of Algorithm 2, both described above. Again, a batch size of 128 is used, both for the contrastive adversarial training step and for the triplet loss adversarial fine-tuning step.

As a final point, [9] observes that introducing even small amounts of labels (1%, 10%, etc.) to an otherwise unlabeled dataset can dramatically improve the results of contrastive adversarial training. Following this observation, we examine the effect of performing semi-supervised adversarial contrastive training followed by triplet loss adversarial fine-tuning. To achieve this, we recreate the experiment setup from the prior section, but reintroduce labels during the contrastive adversarial learning stage. All other hyperparameters remain consistent with those described in the previous section.

There is no accuracy metric for the contrastive loss for the experiment, but we did see that in training for 500 epochs, the loss began to stabilize around 150 epochs into the training.

Both Triplet Adversarial Training and Pre-Training + Fine-Tuning considerably outperformed the Standard model in terms of RA, while slightly lagging behind in terms of SA. This is generally expected when comparing models with and without adversarial training. When comparing our approach with the Triplet Adversarial Training baseline, we found that our results were very similar to the baseline, despite training for much less through fine-tuning. We firmly believe that through further fine-tuning, our model can easily outperform the baseline in this scenario, but we were unable to in the interest of time.

With the introduction of a subset of labels, our model completely surpasses both FaceNet and the Triplet baseline, achieving roughly 30% higher accuracy across all datasets.