Progressive Poisoned Data Isolation for Training-time Backdoor Defense

As the pioneering work in backdoor attacks, (Gu, Dolan-Gavitt, and Garg 2017) manipulated the training set by injecting mislabeled poisoned samples with a specific trigger to execute the attack. (Liu et al. 2017) selected an input region along with neurons which exhibit sensitivity to changes in that region, and the attacker aimed to activate these neurons. Many other works such as (Chen et al. 2017; Nguyen and Tran 2020) have proposed techniques to enhance the stealthiness of backdoor attacks, thereby reducing the effectiveness of backdoor detection mechanisms. Various forms of the triggers have been developed, including sinusoidal strips (SIG) (Barni, Kallas, and Tondi 2019), reflection (ReFool) (Liu, Bailey, and Lu 2020), and warping-based (WaNet) (Nguyen and Tran 2021). In particular, WaNet used a smooth warping field to generate backdoor images with inconspicuous modifications. LIRA (Doan et al. 2021) alternated between trigger generation and backdoor injection to learn visually stealthy triggers. Adversarial Embedding (Tan and Shokri 2020) sought to enhance the latent indistinguishability of backdoor attacks by minimizing the distance between the latent distributions of backdoor inputs and clean inputs through adversarial regularization. BppAttack (Wang, Zhai, and Ma 2022) combined image quantization and dithering to implant the trigger into the dataset. Combined with style transfer technique, (Cheng et al. 2021) proposed an image style trigger attack. Similarly, (Jiang et al. 2023) presented a backdoor attack method using color space shift. Backdoor attacks have been extended to various applications, including those outlined by (Saha et al. 2022) for self-supervised learning (Grill et al. 2020), (Li et al. 2023) for pretrained models and (Ma et al. 2022) for object detection (Redmon and Farhadi 2018). Concurrently, certain backdoor attacks have been specifically designed for distinct network architectures, exemplified by (Yuan et al. 2023) for vision transformers (Dosovitskiy et al. 2021) and (Chou, Chen, and Ho 2023) for the diffusion model (Ho, Jain, and Abbeel 2020). A handful of studies, such as that by (Hayase and Oh 2023), have also explored more efficient methods to implant backdoors, with their work focusing on neural tangent kernels to significantly reduce the poison rate. Additionally, (Shi et al. 2023) have explored the vulnerability of the latest ChatGPT to backdoor attacks.

To combat backdoor attacks, many defense algorithms have been proposed to train clean models on poisoned datasets. (Li et al. 2021) observed that poisoned samples typically have a lower training loss compared to benign samples. They proposed a two-stage process: firstly, isolating a few samples with the lowest losses, and secondly, unlearning the backdoor on these isolated samples. (Huang et al. 2022) was the pioneer in utilizing self-supervised learning and semi-supervised learning in backdoor defense. (Guo et al. 2023) found that poisoned samples demonstrated scaled prediction consistency when pixel values were amplified, and they identified poisoned samples by tracking the predictions of these scaled images. (Liu et al. 2023) revealed that poisoned models exhibit consistent performance on benign images under various image corruptions while performing divergently on poisoned images. (Jebreel, Domingo-Ferrer, and Li 2023) observed that poisoned samples and benign samples exhibit significant differences at a crucial layer, and using the feature difference at this critical layer could aid in distinguishing poisoned samples. (Wang et al. 2022) pinpointed backdoor neurons by investigating the linearity of the models, and performed a statistical test to identify poisoned samples.

We consider the same threat model as in prior defense methods (Li et al. 2021; Huang et al. 2022; Gao et al. 2023; Wang et al. 2022), which assumed the backdoor injection is performed at training stage and the attacker can only poison the dataset rather than control the training process. Formally, given a training set $\mathcal{D}=\{\mathcal{D}_{b},\mathcal{D}_{p}\}$ including poisoned part $\mathcal{D}_{p}$=$\left\{(\hat{x}_{i},\hat{y}_{i})\right\}_{i=1}^{n_{p}}$ and benign part $\mathcal{D}_{b}$=$\left\{(x_{i},y_{i})\right\}_{i=1}^{n_{b}}$, the training process can be regarded as a dual-task learning on these two parts:

where $\theta$ represents the parameters of model $f$, and $\ell$ denotes the objective loss (e.g., cross-entropy loss).

Fundamentally, the objective of defense is to train a clean model on poisoned dataset $\mathcal{D}$. This model should have a low ASR on the poisoned test set, while maintaining a high inference performance on the clean test set. Considering the capabilities in previous works, we assume that the defender has access to the poisoned dataset $\mathcal{D}$ and is aware of the presence of poisoned data within it but lacks knowledge about the type of the attack or the poison rate $\eta$ which is the ratio of the number of poisoned samples to the total data size.

We propose to utilize the isolation results of an existing isolation method as a setup, furnishing a precise prior for the subsequent isolation based on distributional discrepancies. By adopting the loss function from (Li et al. 2021):

the samples with a slower loss descent rate will be trapped around a threshold $\gamma$, while keeping other samples away. After training for a few epochs, we select p lowest loss samples as poisoned subset $\mathcal{P}_{p}$ and q highest samples as benign subset $\mathcal{P}_{b}$. Considering the low poison rate $\eta$ (usually $\eta<20\%$), we set p as 1% and q as 20%.

After acquiring the final predicted poisoned set $\hat{\mathcal{D}}_{p}^{T}$ and benign set $\hat{\mathcal{D}}_{b}^{T}$, we propose to optimize the $\theta$ of $f$ via a selective training strategy rather than conventional training. Specifically, we only execute gradient descent when $f$ identifies the samples within $\hat{\mathcal{D}}_{p}^{T}$ as their ground-truth labels, while employ standard gradient descent on $\hat{\mathcal{D}}_{b}^{T}$. The training process can be formulated as:

where the indicator function $\mathds{1}(\cdot)$ returns 1 when the condition is true and 0 otherwise. Compared with a similar solution in ABL, in which it continues to apply gradient ascent on the isolated samples, our strategy is capable of mitigating the impact of gradient ascent on the inference performance.

Dataset $\mathcal{D}$: Following convention (Li et al. 2021; Wang et al. 2022; Huang et al. 2022; Gao et al. 2023), we conduct experiments over the CIFAR-10 (Krizhevsky, Hinton et al. 2009) and a subset of ImageNet (Deng et al. 2009) datasets.

Due to space limit, more details regarding the experiments are deferred to the appendix. Specifically, the isolation results of PIPD on poisoned dataset with different poison rates, and the impact of clustering method on isolation results are included in Appendix C.1 and C.2, respectively.

The ASR and CA results are delineated in Table. 1. As the data illustrates, our defense method consistently achieves superior CA and low ASR values in all attack settings. Specifically, PIPD demonstrates the highest average CA at 94.5% and the lowest average ASR at 0.43%. For the more complex ImageNet, PIPD still exhibit superior performance, where the average ASR is 0.14% and the average CA is 85.87%.

As a comparision, we also present the defense performances of other defense methods. Both the DBD and ASD methods employ semi-supervised learning, and they exhibit relatively low ASRs. This is attributed to their treatment of a significant portion of data as unlabeled, whereby the removal of labels from poisoned data effectively prevents backdoor injections. The CA for the NONE method is the highest among them, and simultaneously, its ASR is also the highest. It’s noteworthy that the NONE method employs two mechanisms to reduce ASR: one through isolating samples and another through resetting neurons. A high CA suggests that it isolates an insufficient number of samples, leaving ample residual data for training. However, the elevated ASR indicates that the neurons it resets are not necessarily related to the backdoor. The CA results for the ABL method are suboptimal. We attribute this to the detrimental effects of excessive unlearning on the model’s performance. Benefiting from the progressive isolation process with selective training strategy, our PIPD leads to the best performance over all testing datasets in both CA and ASR. These results confirm the effectiveness of our proposed defense method in training a clean model using a poisoned dataset.

We now elucidate the isolation quality of our algorithm and juxtapose it with established isolation methods. Table. 2 reports the isolation quality. Our method demonstrates the capacity to yield the highest TPR and lowest FPR across all attacks. Notably, apart from the Refool method, the TPR values with other attacks reach at 100%, signifying that most of the poisoned samples are isolated. Also, the FPR for all attacks are both 0%, except from a 0.42% for the Refool. It can be seen that, by using the proposed PIPD, benign samples are rarely classified as poisoned. In comparison, ABL, which records the second lowest FPR of 1.07% among these methods, is still 1.01% higher than PIPD. Concurrently, the average TPR value of ABL lags ours by 9.31%. It should be noted that DBALFA requires a 10% benign test set for setup, yet its TPR falls short of ours by 2.12%, even though it exhibits the highest TPR amongst other methods. The FPR values of DBD and ASD are comparably high, which is the consequence of treating 50% of the dataset as poisoned. These findings solidify our method’s capability to deliver low ASR and high CA for the model.

In this experiment, we explore the scenario where the poisoned subset, derived from the pre-isolation phase, encompasses a number of benign samples. The proportion of the poisoned subset is 1%, assembled by randomly selecting diverse ratios of poisoned to benign samples. We define the fault ratio as the percentage of benign samples mixed in the poisoned subset. Specifically, we scrutinize settings where the fault ratio stands at 20%, 40%, 60%, and 80%. Table. 3 presents the isolation performance of our PIPD approach. It has been observed that as more benign samples are mixed into the poisoned subset, the TPR displays a downward trend, while the FPR shows an upward trajectory. This implies that an inferior quality of the poisoned subset will compromise the final isolation quality.

This section aims to testify if the progressive isolation process effectively mitigates the issue of high FPR. In this experiment, we choose a poison rate of 20% for illustration. As demonstrated in Fig. 4, the preliminary isolation process yields suboptimal results, specifically an average FPR value of around 0.3 with these attacks. However, with the progressive refinement of the isolation results, the FPR decreases constantly. These outcomes strongly endorse the effectiveness of our progressive strategy in addressing the high FPR issue. Additionally, we evaluate the CA and ASR under different $T$s, and the results are displayed in Table. 4. The isolation process of the first iteration failed to defense against BadNet and Blended with a high ASR of 99.81% and 98.73%. By gradually increasing $T$, the ASR values decline constantly. Meanwhile, the CA values increase when $T$ becomes larger, indicating that the number of benign samples being misclassified is decreasing and the FPR is effectively reduced. We conjecture that five iterations is enough for defending most attacks; while increasing $T$ merely prolongs the isolation time.

This section studies the effect on defense performance with selective training strategy, especially compared with the unlearning strategy in ABL. For fair comparison, we use the same isolation results and only replace the training strategy. As can be seen from Fig. 5, our approach consistently maintains a high CA, during the training process as the ASR declines. In contrast, regarding the ABL method, as the ASR decreases, the CA also decreases sharply. We ascribe this to excessive unlearning damaging the inference performance of the model on clean test set. In our method, the model does not perform unlearning on poisoned samples that are no longer predicted as the target label. This averts the detrimental effects of excessive unlearning on the CA performance.

The details of datasets and DNN models in our experiments are summarized in Table 5. Specially, we randomly choose 30 classes from ImageNet to construct a subset due to the limitation of the computational time and costs.

We mainly consider 9 state-of-the-art backdoor attacks, including: BadNets (Gu, Dolan-Gavitt, and Garg 2017), Trojan (Liu et al. 2017), Blended (Chen et al. 2017), Dynamic (Nguyen and Tran 2020), WaNet (Nguyen and Tran 2021), SIG (Barni, Kallas, and Tondi 2019), Refool (Liu, Bailey, and Lu 2020), and CL (Turner, Tsipras, and Madry 2019). In addition, we evaluate one recently proposed adaptive attack termed Adaptive-Blend (A-Patch) (Qi et al. 2023).

We use two standard data augmentation techniques (horizontal flip and random crop with padding $4\times 4$ for CIFAR-10 and $16\times 16$ for ImageNet) during model training. We follow the default settings suggested in the original papers and the open-source codes for most attacks; these include the trigger pattern, trigger size, and backdoor label. The backdoor label of all attacks is set to class 0. Table 6 summarizes the detailed settings of these attacks.

We experiment with four training-time defense methods: ABL (Li et al. 2021), DBD (Huang et al. 2022), NONE (Wang et al. 2022), and ASD (Gao et al. 2023), and two backdoor detection methods: DBAFLA (Jebreel, Domingo-Ferrer, and Li 2023) and SCALE-UP (Guo et al. 2023). When evaluating isolation result on DBALFA, we use 10% benign test set, as specified in their work. Regarding DBD and ASD, when evaluating the isolation quality, we will mark the data determined as unlabelled by them as poisoned, since the labels of this part of data were lost during the training process.

We use the open-source PyTorch codes for ABL^‡‡‡https://github.com/bboylyg/ABL, DBD^§§§https://github.com/SCLBD/DBD, ASD^¶¶¶https://github.com/KuofengGao/ASD, NONE^∥∥∥https://github.com/RU-System-Software-and-Security/NONE, DBAFLA^******https://github.com/najeebjebreel/dbalfa, and SCALE-UP^††††††https://github.com/junfenggo/scale-up to reproduce the results of them.

We use four metrics for evaluation: 1) Clean Accuracy (CA) - the model’s performance on the clean test set; 2) Attack Success Rate (ASR) - the model’s performance on the poisoned test set; 3) True Positive Rate (TPR) - calculated as the ratio of isolated poisoned inputs to the total number of poisoned inputs, and 4) False Positive Rate (FPR) - the ratio of benign inputs incorrectly isolated as poisoned to the total number of benign inputs. These metrics were adopted from the DBALFA. Higher TPR and CA values, and lower FPR and ASR values generally suggest a more efficient isolation process and an improved training-time defense method, respectively.

This section investigates the effectiveness of our method against varying poisoning rates. In the following experiments, we set the poisoning rates for five distinct attacks at 5%, 10%, 15%, 20%, and 30%. We use a ResNet-18 and CIFAR-10 dataset for the illustration, the results are detailed in Table. 7. As the poison rate gradually increases, the TPR begins to decline while the FPR starts to rise. This suggests that the difficulty of isolation intensifies with the escalation of the poison rate.

This part of our study evaluates the impact of clustering algorithms used in the isolation process on the quality of isolation. Again, we experiment on the ResNet-18 trained on poisoned CIFAR-10 with 5% poison rate attacked by various attacks. We adopt different clustering algorithms replacing the Fisher-Jenks algorithm. We list the TPR and FPR by using K-means, Bisecting K-Means and Mini-Batch K-Means, in addition to our default Fisher-Jenks in Table. 8. As can be noticed, all TPR and FPR results with different clustering algorithms are similar. This implies that the isolation quality is not sensitive to the clustering algorithms adopted.