Selective Amnesia: On Efficient, High-Fidelity and Blind Suppression of Backdoor Effects in Trojaned Machine Learning Models

A backdoor attack aims to induce strategic misclassification in a DNN model through training data poisoning [11, 65, 6] or training process manipulation [62, 16]. The risk of a backdoor attack can be mitigated through detection: discovering triggers using SGD [66], leveraging images carrying triggers [81] and others [72], as mentioned earlier. A prominent example is Neural Cleanse (NC) [66], which searches for the pattern with the anomalously small norm that causes any image with the pattern to be classified into a target label. Once discovered, the pattern can be used to unlearn the model’s backdoor, by training it on the correctly labeled inputs carrying the pattern. Unlike NC, which depends on trigger discovery to unlearn a backdoor from a DNN model, Fine-pruning [39] and Neural Attention Distillation [38] are designed to directly remove a backdoor from a Trojaned model, without using any information about the trigger. More specifically, Fine-pruning prunes less informative neurons and then finetunes the model [39], in an attempt to directly erase the backdoor effect; NAD first finetunes a given model to create a teacher model, which is combined with the original model (the student) through distillation to unlearn a hidden backdoor [38].

In multi-task learning (MTL), several learning tasks are solved jointly at the same time, which could outperform the training alone on individual tasks [67, 80, 79]. More specifically, consider several supervised learning tasks in MTL $\tau_{t},t\in[T]$ where $\tau_{t}=\{x_{t_{i}},y_{t_{i}}\}^{n_{t}}_{i=1}$, with $T\in\mathbb{N}^{\star}$ the set of positive integers. Suppose $\mathcal{X}$ is a feature space of interest, $\mathcal{X}\subseteq\mathbb{R}^{p}$, and $\mathcal{Y}$ is the space of labels, $\mathcal{Y}\subseteq\mathbb{N}$, then $X_{t}\subseteq\mathbb{R}^{n_{t}\times p}$ ($p$ is the feature dimension) represents the dataset of the task $\tau_{t}$ and $x_{i}^{t},i=1,\ldots,n_{t}\in\mathcal{X}$ is a sample with its corresponding label $y_{i}^{t}\in\mathcal{Y}$. The goal of MTL is to learn a function: $f_{\omega}:\mathcal{X}\times\mathcal{T}\rightarrow\mathcal{Y}$ with $\omega\in\mathbb{R}^{q}$ being the parameters that fit the prediction as accurate as possible. As a special case of MTL, Continual Learning (CL) [73, 47] solves a stream of supervised learning tasks $\tau_{1},\tau_{2},...,\tau_{T}$. The goal is to train a predictor with the best accuracy on each task assuming that the training data for previous tasks are no longer available after the tasks are accomplished. In our paper, we utilizes the continual learning theory to model the unlearning process used by SEAM. Specifically, we consider the backdoor unlearning as an independent process after the given model (benign or backdoored) has already been trained.

Our research shows that fine-tuning a Trojaned model cannot guarantee removal of its backdoor, particularly when the clean dataset for fine-tuning is small ($<$10% of the model’s original training data). This has been acknowledged in the prior research [39], even with its effort to prune suspicious neurons to improve the efficacy of fine-tuning.

Fundamentally, we believe that fine-tuning is limited in its potential to suppress the backdoor effect in general: consider a well-trained but backdoored model, with its ACC close to 1; fine-tuning the model on the clean dataset under its primary (overt) task will have little impact on its weights, given that its loss is already small, and therefore will not significantly interfere with its covert backdoor task.

A natural solution here is to explicitly forget the backdoor information without undermining the model’s capability to solve its primary task. This purpose can be served, effectively, by the idea of Catastrophic Forgetting(CF), as discovered in our research. CF has long been considered to be a problem for artificial neural network (NN) [34], causing an NN to completely and abruptly lose memory of the previously learnt task when learning the information about a new task. The problem is known to be a major barrier to continual learning [24], since the weights in an NN for storing the knowledge of one task will be changed to meet the requirements of the subsequently learnt task. This long-standing trouble, however, was leveraged in our research to enhance the trustworthiness of a DNN. More specifically, prior research shows that a CF can be induced by a new task with similar input features as the old task but different classification outputs [12]. This property was utilized by us to build a novel pipeline in which a forgetting task is first run to cause the maximum CF (Section 4.3), and then a recovery task is performed to revive the primary task without wakening the backdoor. In this way, we can achieve a selective amnesia on an infected DNN to remove its backdoor effect.

At the center of our blind unlearning pipeline is the forgetting task for inducing a CF on a given DNN model. The task is meant to achieve the following goals: 1) ensuring a large CF and 2) helping a quick and selective recovery from the CF. For 1), we need to largely preserve input feature extraction but incur significant impacts on classification, so as to interfere with the original tasks, including the backdoor task; for 2), we hope that the changes to the weights of the DNN, as caused by the interference, can be easily and selectively reversed, so the primary task can be recovered. To this end, we designed a random-labeling task that assigns a random class label to each output of the model. This simple approach utilizes the features the original model discovers on the input and intermediate layers but causes a large loss that leads to significant changes to the weights of the layers close to the output. Further, such changes can be done with a few rounds of updates on the weights through stochastic gradient descent (SGD), so the impacts on the primary task can be quickly reversed.

Our research shows that a very small $\mathcal{D}_{for}$, as randomly selected from the clean dataset for $f(\cdot)$, is adequate for almost completely removing backdoor effect from the model, that is, resulting in a negligible ASR (Section 5.1). The threshold $Acc_{for}$ was set in our experiment to $\min(2/\mathbf{C},0.6)$, where $\mathbf{C}$ is the number of classes in the primary task. The recovery dataset $\mathcal{D}_{rec}$ includes only correctly labeled data, which could be a subset of the testing data for the primary task. This ensures that the final model $\tilde{f}$ after the recovery step achieves an accuracy close to that of the input model $f(\cdot)$, while keeping ASR exceedingly low (approaching 0). Note that there is no overlap between $\mathcal{D}_{rec}$ and $\mathcal{D}_{for}$. In our experiments, we set the threshold $Acc_{rec}$ for the recovery step to $0.97$ of the input model’s accuracy.

We use the NTK theory for continual learning in our theoretical analyses. We consider a sequence of tasks $\tau_{1},\tau_{2},\ldots,\tau_{T},T\in\mathbb{N}^{*}$, in which each is a supervised learning task with its input and output in the same high dimensional spaces, respectively. Consider the NN trained on the training data labeled for each of the $T$ tasks in the sequential order. Prior research [12] utilizes NTK to approximate a NN trained for a target task $\tau_{T}$ with the model trained for a source task $\tau_{S}$, where tasks $\tau_{S}$ and $\tau_{T}$ are any two tasks in a sequence, and $\tau_{S}$ occurs before $\tau_{T}$. The NTK approximation from $f_{\tau_{S}}^{\star}(x)$ to $f_{\tau_{T}}^{\star}(x)$ is expressed as: $f_{\tau_{T}}^{\star}(x)\approx f_{\tau_{S}}^{\star}(x)+\left\langle\phi(x),\omega_{\tau_{T}}^{\star}-\omega_{\tau_{S}}^{\star}\right\rangle,$ where $\omega^{\star}_{\tau_{S}}$ is the final vector of weights after the training on task $\tau_{S}$, $\left\langle\cdot\right\rangle$ represents the inner product in the kernel space. $\nabla_{\omega}f_{0}(x)=\phi(x)\ \in\mathbb{R}^{q}$ is defined as the NTK feature map for an input sample $x$, where $q$ is the number of weights in the neural networks. Note that in prior research [12] and  [4], the NTK is defined on the Taylor expansion around the initial state of the DNN ($f_{0}$), but it can be extended to Taylor expansion around any state of the DNN to approximate a subsequent state of DNN in sequential training.

Following, we first present the definition of CF, which is defined over the transition from a source task to a target one, as measured from a given dataset. Then we utilize NTK to describe the CF measurement, so we can separate an NN’s feature representation from classification. This is important for analyzing the impacts of SEAM on CF (e.g., how the forgetting task interferes with classification).

Corollary 2 is extended from the Lemma 1 in paper [12]. This new form of the CF measurement lays the foundation for the analysis on both the forgetting and the recovery tasks. $U,\sigma$ and $V$ represent the left singular vector, singular value, and right singular vector respectively after SVD. Their subscripts represent the tasks they are corresponding to. In addition to the residual for the influence on classification, the term $||O_{\tau_{S,T}}||^{2}_{2}=||V_{\tau_{S}}^{\top}V_{\tau_{T}}||^{2}_{2}$, which is positively related to the CF, is considered to be a good similarity metric between $\tau_{S}$ and $\tau_{T}$ in prior studies [12]: intuitively, a large $||O_{\tau_{S,T}}||^{2}_{2}$ indicates that the angle between the representation vectors produced by the models (for $\tau_{S}$ and $\tau_{T}$) on the same input samples (i.e., $X_{\tau_{S}}$) is small (i.e., these two tasks are similar). Given a fixed residual, a large similarity leads to a large CF, i.e., a high impact of $\tau_{T}$ on $\tau_{S}$. In Eq. 3, $U_{\tau_{S}}$, $\Sigma_{\tau_{S}}$ and $V_{\tau_{S}}^{\top}$ result from the SVD of kernel matrix over the dataset $X_{\tau_{S}}$; the subscript of $\tau_{S}$ is used here to denote the dataset $X_{\tau_{S}}$ instead of the source task $\tau_{S}$. Notably, once the model representation (i.e., the feature map) is fixed for a neural network, the term $O_{\tau_{S,T}}$ is dependent on the dataset $X_{\tau_{S}}$ (on which the CF is evaluated) as well as the training dataset $X_{\tau_{T}}$ for the target task. For the rest of the paper, we will refer to this term as the ”task similarity” following the terminology in the previous studies [12] even though it is not directly related to the two tasks involved in the transition, but to their training datasets.

Using NTK to model CF, we are able to prove that the random-labeling step of SEAM maximizes the residual, which is proportional to the CF, on a given training dataset. This result demonstrates that our forgetting task is the best we could do to disrupt the backdoor task in the absence of the information about the backdoor (trigger, source and target classes, etc.). Following, we elaborate the analysis.

Note that this residual is independent of the sample $X$ on which the CF is evaluated. Therefore, we have the following theorem:

Eq. 4 also implies that the direct fine-tuning of the model $f_{\tau_{P}}^{*}$ using the clean dataset $D_{C}$ (equivalently, using the primary task as the competitive task) will not cause effective unlearning of the backdoor, which is consistent with the findings reported by previous studies [39]. Specifically, during the fine-tuning, the true labels $Y_{c}$ of the clean samples in $D_{c}=\{X_{c},Y_{c}\}$ are used, and thus the residual $\tilde{y}_{\tau_{F}}={Y}_{\tau_{F}}-f^{\star}_{\tau_{P}}(X_{\tau_{F}})$ is very small, because the model $f^{\star}_{\tau_{P}}$ is expected to correctly predict most of the true labels of the clean data for the primary task. Therefore, the CF induced by fine-tuning is significantly lower than the CF incurred by the competitive task of SEAM trained on randomly labeled training data. Intuitively, since the initial model $f_{\tau_{P}}^{*}$ already makes a correct prediction on most samples in the clean dataset, fine-tuning does not change the model significantly and thus the backdoor task may not be effectively unlearnt.

Our analysis using the NTK model of the forgetting task shows that SEAM can maximize the disruption (CF) of an unknown backdoor task for a given set of clean data for training the forgetting task. Following we further report our analysis on the recovery task using NTK.

According to the Theorem 5, if $X$ is a subset of the testing dataset for the primary task (i.e., following the same distribution as $D_{C}$), the representations of the samples in $X$ and $D_{C}$ are highly similar, and thus $||O_{X,C}||^{2}_{2}$ is large, then the CF is large, which indicates the recovered model gives very different output from the initial model for the task $\tau_{F}$ (i.e., it effectively recovers the primary task). In contrast, if $X$ is a dataset for the backdoor task, $||O_{X,C}||^{2}_{2}$ is small and thus the CF is small, which indicates the recovered model does not recover the backdoor task.

Here we present our experimental results on the efficacy of SEAM using the aforementioned datasets. Our approach is found to be highly effective in preserving the ACC of the original model and in the meantime suppressing the effects of backdoors when they are present in the model. In our experiments, we ensure that the testing set used to measure the performance of SEAM does not have any overlap with the clean dataset for inducing CF to the model and recovering its preliminary task, and also the clean set is selected independently from the training dataset (except our study on natural backdoor). In practice, however, the defender could utilize her testing dataset for evaluating the functionality of the model to blindly unlearn the backdoor from the model through SEAM.

For the two representative backdoor attacks, Reflection involves selecting a set of candidate images and utilizing reflection transformation to generate triggers, and further injecting them into the training dataset of the victim model. In our experiments, following the steps provided by the Reflection paper [45], we sampled 200 images from the target class as candidate images and selected the reflection-transformed images with the highest ASRs on a dummy dataset (the training set in our experiments to give the adversary advantage) as triggers, and further pasted such triggers onto 10% of the selected images and injected them into the training dataset for each victim model. The TrojanNet attack trains a simple NN to recognize whether a trigger appears on the input images and then merges the NN and the victim model [61]. Again, following the experimental setting given by the prior research [61], we utilized a 4x4 square trigger with 11 white pixels and 5 black pixels, and a 4-layer perceptron NN as the trigger recognition network, and further integrated the network into the victim DNN by combining the outputs of their penultimate layers through interpolation. Note that both Relection [45] and TrojanNet [61] requires a source class label as their inputs for generating Trojaned models. We randomly selected a label as the source for each attack.

In our experiments on SEAM against Reflection and TrojanNet over MNIST, GTSRB and CIFAR10, we utilized 0.1% of the training data of each dataset for the forgetting step (label randomization) and 10% of its training data for the recovery step (retraining the randomized model on the primary task). Further, we leveraged the testing data provided by these datasets to measure ACC, and added triggers to all the testing inputs from the source class of each attack to measure ASR. Table I presents our experimental results. Here, A high ACC and a high ASR before SEAM characterize an effective backdoor attack. A high ACC and a low ASR after the unlearning, that is, a high FID, indicate effective suppression of the backdoor effect from a victim model. On all three datasets, SEAM is found to achieve a good FID against both attacks. The minimum FID is $95.24\%$ for the Trojaned Vgg16 on CIFAR10 under TrojanNet, which is caused by the remaining ASR after unlearning ($3.67\%$). Note that the ACC of Vgg16 on CIFAR10 is among the lowest before the unlearning operation, with and without the backdoor, which indicates the limitation of the model itself and could have an impact on the effectiveness of our approach on the model.

The datasets of TrojAI Round 3-4 contain a large number of models (3198 models) in various model architectures and with different triggers (Section A.1). In our experiment, we utilized a very small set of clean data provided by the competition organizer for the forgetting and the recovery steps, which is only $0.1\%$ of the data for training each model; we also followed the competition’s documents [28] to generate the testing data (a separate dataset about $10\%$ the size of the training data, with those with the source label also used with the trigger for measuring ASR). Table IV illustrates the experiment results. With this unprecedentedly small set of clean data, still SEAM achieved a FID of $89.85\%$ and $86.75\%$ for Round 3 and 4. Actually, only 3-5 clean samples from each class are provided for each model (Section 5.2). No existing technique, up to our knowledge, could utilize such a small set of clean data for meaningful unlearning (see Table II).

In our experiments, again, we utilized the small set of clean data ($0.1\%$ of the training set) provided by the competition organizer for the forgetting and the recovery steps; we also followed the competition’s documents [28] to generate the testing data. Table IV illustrates the experimental results. As we can see from the table, SEAM achieves an average FID of $88.30\%$, $89.16\%$ and $92.65\%$ on the datasets of Round 5 and Round 6 and Round 7 respectively. The relatively higher Fidelity in the last round could be attributed to the higher ACC of the infected models in the round ($93.60\%$ vs. an average $90.15\%$ of the models in other rounds), and the availability of more clean training data: in the sentiment analysis (Round 5-6), one clean sentence released by the competition organizer can only be used as one instance in the training set (for both the forgetting and recovery operations), while the same sentence can be broken down to multiple entity-related terms for unlearning backdoors in an NER model, the Round 7 task.

We further analyzed the efficiency of SEAM from two aspects: execution time and clean data size.

Then we take a close look at the forgetting and recovery steps. The time complexities of these steps are $\mathcal{O}(\mathbf{N}_{for})$ and $\mathcal{O}(\mathbf{N}_{rec})$, where the former is the number of epochs for training the random-labeling task of the forgetting step, and the latter is the number of epochs for training the primary task for the recovery step. We compared the execution time of SEAM on infected ResNet18 models on three datasets (MNIST, GTSRB and CIFAR10) with these models’ original training time. As we can see from the results in Table VI, on average, SEAM takes less than 7% of the time for training a model from scratch to suppress the model’s backdoor effect while preserving its legitimate classification capability.

We found that, on MNIST/GTSRB/CIFAR10, the forgetting step takes much less time than the recovery step, since 1) the forgetting typically needs less than 10 epochs while the recovery requires at most 100 epochs, and 2) the dataset for the forgetting is just $0.1\%$ of the training dataset, while the dataset for the recovery was set to $10\%$ in our experiments on MNIST/GTSRB/CIFAR10. On Round 3-7 of TrojAI competition, the recovery dataset is only $0.1\%$ training dataset that largely accelerated the recovery step with the cost of lightly reduced Fidelity. The effect of clean data size will be evaluated later.

To understand what makes SEAM so efficient, we looked into the changes of each layer after the forgetting step and the recovery step. Specifically, for a VGG16 model, we measured the similarity of the same layer between the original model (the backdoored one before unlearning), the randomized model (after the forgetting step) and the recovered model (after the recovery step). Here we use the centered kernel alignment (CKA) [35] as the metric, which ranges in $[0,1]$ with $CKA=1$ being identical and $CKA=0$ being totally different. Fig. 2 demonstrates the CKA results for a VGG16 model on a clean dataset. As we can see here, for each layer, the closer it is located to the output, the more different can been seen between the original model and the randomized model. This indicates that many features of the original model has been preserved during the forgetting step, particularly those on the shallow layers of the model, so the recovery step only needs to restore the features on the layers more toward the output layer, thereby allowing a faster unlearning and recovering.

To understand the impact of the size of recovery datasets $\mathcal{D}_{rec}$ on the effectiveness of unlearning, we ran SEAM on $\mathcal{D}_{rec}$ of different sizes (randomly drawn from CIFAR10). The results are presented in Fig. 3. We observe that the execution time of SEAM goes down along with the increase of the size of $\mathcal{D}_{rec}$ against Reflection attacks and TrojanNet attack. while the Fidelity changes gently. Again, we confirm that indeed SEAM only needs an exceedingly small amount of clean data ($0.1\%$, about 5 images per class) to achieve a decent unlearning effect (Fidelity of $86\%$).

We compared SEAM with Neural Cleanse (NC) [66], Fine-Pruning (FP) [39], fine-tuning, naive continuous-training, and Neural Attention Distillation (NAD) [38], five representative unlearning techniques. As mentioned earlier, NC is a detection-based approach that first recovers the trigger from a backdoored model and then removes the backdoor from the model through unlearning (i.e., retraining the model on the trigger-carrying inputs with the correct labels). FP is a blind unlearning approach that prunes a backdoored model and then fine-tunes it on a clean dataset, in an attempt to remove the backdoor effect. Fine-tuning is a method that tunes the parameters of a backdoored model’s last two layers on a small set of clean data using gradient descent; continuous-training keeps on training the model on a small set of clean data, hoping to weaken its backdoor effect when it is infected. NAD is a blind unlearning approach that uses a clean dataset to distillate a new clean model from the victim model.

For a fair comparison, we implemented SEAM under the TrojanZoo framework [49] and utilized the implementations of NC and FP provided by the TrojanZoo team. We also kept the unlearning datasets for NC, FP and SEAM to the same size, i.e., $10\%$ of the training datasets for MNIST/GTSRB/CIFAR10 and $0.1\%$ for the TrojAI competition. Also, for Fine-pruning, we followed the FP paper [39] to set the prune ratio to 0.82 of all neurons (i.e., maximum $82\%$ neurons would be pruned) and the fine-tuning epochs to 300 for MNIST, GTSRB and CIFAR10, 1000 for TrojAI competition Round 3-7.

Table II and Table IV show the comparison of Fidelity among these solutions against the Reflection and TrojanNet attacks on MNIST/GTSRB/CIFAR10 and TrojAI datasets. We observe that SEAM achieves a high and stable Fidelity against both attacks on all eight datasets, while FP performs well against both attacks on MNIST and GTSRB, and NC only does well on GTSRB. The failure of NC against the Reflection attacks on MNIST and CIFAR10 could be attributed to its limitation in finding large triggers, as the Reflection attack may use an entire clean image as the trigger. The failure of FP against the Reflection attacks on CIFAR10 could be due to the difficulty in reducing ASR when trigger-relevant neurons have not been completely pruned.

A larger Fidelity gap between SEAM and NC/FP can be observed on Round 3-7 datasets of the TrojAI competition. The small clean dataset ($0.1\%$ of training data) available for unlearning significantly reduces the efficacy of NC/FP. Specifically, on Round 3-7, the maximum Fidelity achieved by NC is $67.27\%$ on the Round 5 data, and for FP, it is $72.81\%$ on the Round 4 data, which are far below the performance of SEAM on the same datasets: $88.30\%$ on Round 5 and $88.75\%$ on Round 4. Table II and Table IV also compare the execution times of these solutions. On MNIST/GTSRB/CIFAR10, SEAM takes on average $1/6$ of the execution time used by NC and $1/4$ by FP. On Round 3-7 of the TrojAI competition, SEAM takes on average just $1\%$ of the execution time for NC and $1/50$ for FP. The high efficiency of SEAM can be attributed to the fact that the recovery step only needs to restore the features on the layers close to the output (Section 5.1).

SEAM also significantly outperforms NAD and two simple baselines (fine-tuning and continuous training), especially when the clean data is scarce. In Table V, we present the Fidelity results on the ResNet18 models for comparing SEAM, NAD and the two baselines on different sizes of clean datasets against the TrojanNet attack.

As we can see from the table, NAD could not reasonably reduce ASR while maintaining a decent ACC when the size of the clean dataset goes down to 1% and further to the 0.1% of the training dataset size (note again that the clean data is not a subset of the training data). On the other hand, SEAM maintains its high effectiveness in unlearning, achieving 85% to over 91% Fidelity with a small set of clean data (0.1% of the training data size), in line with its performance on the TrojAI datasets, where for each model, only 10 samples are available for each class (Table IV).

Compared with another naive baseline – retraining the whole model from scratch on clean data, SEAM also demonstrates superior performance. We compared the performance of ResNet18 models recovered by SEAM and those trained from scratch on three datasets (MNIST, GTSRB and CIFAR10). As shown in Table VI, the models recovered by SEAM achieve on average 17% higher ACC than the models trained from scratch on the same clean dataset (10% of the whole training data size), and approach the ACC of the models trained from scratch on the whole training dataset. Also unlearning through SEAM takes about 45% of the time for the training from scratch to converge on 10% of the training data size (with much lower ACC) and only less than 7% of the time for training on the whole dataset.

In this section, we investigate several possible evasion methods against SEAM, including the Label Consistent (LC) backdoor [65], the Latent Separability (LS) backdoor, the Natural Backdoor (NB), the Entangled Watermarks (EW) [32] and the evasion polluting the recover dataset with trigger-carrying inputs.

To find out the robustness of SEAM against LC, we performed a set of experiments on CIFAR10. Table VII shows that our approach successfully reduces the ASR from $>90\%$ to $<10\%$ and recovers the ACC to the level similar to that of the original model. In the meantime, indeed LC weakens the effectiveness of SEAM by causing a higher ASR after the recovery step, compared with the results of running SEAM against the Reflection and TrojanNet attacks on CIFAR10 (Table I). Note that unlike other data poisoning attacks, LC requires information about the target model’s representation space, which can only be estimated through model transferability. It is still less clear how likely transferring a backdoor in this way could succeed.

To evaluate the performance of SEAM against NB, we performed experiments on CIFAR10. Specifically, we first recovered the NB of the target model through trigger inversion [66]. Then, we ran SEAM on the target model using a forgetting dataset and a recovery dataset, with 0.1% and 1% of the training data considered to be “clean” (without the recovered trigger), respectively. The results are shown in Table VII. We observe that SEAM reduces the ASR of the NB, with the maximum reduction of 33.44% on VGG16. To further weaken the effect of NB, we utilized $10\%$ of the testing dataset (which amounts to 1% of the training data in size but has no overlap with its content), for recovery. Then, we leveraged the remaining $90\%$ of the testing data to measure the ACC and the ASR of the unlearned model. The results are presented by the NB-t row in Table VII, which indicates that the use of the clean data not in the training set for recovery could be more effective in suppressing NB.