Active Learning Under Malicious Mislabeling and Poisoning Attacks

The performance of a deep learning model depends on the size of a labeled dataset, model complexity, and so on. Though we generated about 2.5 quintillion bytes of data each day from a variety of cyber and physical systems [1], a large portion of this data is unlabeled. To use it for supervised learning requires human labelers to take time and effort to provide the labels for each data point. Because the 2.5 quintillion bytes of data per day results in enormous unlabeled data, we need to find a way to effectively select a portion of data for annotating. Active learning was born from such needs.

Either experienced labelers (such as doctors for medical data) or ordinary labelers (such as graduate students for image data) can hand-annotate the data depending on a classification task. Despite the level of expertise, all labelers can make a mistake while labeling. Fard, et al. [2] has shown that deep learning networks are robust against minor uniform mislabeling, randomly mislabeling one class as any other class. However, systematic mislabeling, consistently mislabeling a class as another class, is more far-reaching [2]. In this paper, we consider both human error and malicious mislabeling in active learning.

In addition to the mislabeling, another security concern is a data poisoning attack [3]. Despite state-of-the-art performance, deep learning models are easily fooled by data poisoning attacks and evasion attacks [4]. The main difference between data poisoning attacks and evasion attacks is the assumption of an attacker’s ability. For a data poisoning attack, an adversary tries to manipulate some training data in order to change a decision boundary and cause the misclassification of a specified test instance on a targeted attack. Contrarily, an adversary tries to generate an adversarial example by introducing a subtle perturbation to a specified test instance in order to mislead a learning algorithm and evade detection in an evasion attack. In this paper, we propose a novel active learning method that mitigates the effect of data poisoning attacks by utilizing adversarial examples.

The key contributions of this paper are in the following:

• •

  We present a new heuristic methodology for robust active learning that utilizes the adversarial examples as a measure of uncertainty and a minimax algorithm for data selection. Furthermore, the crafted adversarial examples can be added to a training set without incurring an extra labeling budget.

• •

  We develop a confidence score updating scheme to check the performance of each labeler. This scheme significantly reduces the effectiveness of a malicious mislabeling attack. Specifically, the proposed active learning method achieves the desired accuracy of 89% using only one-third of the labeled data, whereas the baseline active learning method performs poorly (50%).

• •

  We empirically demonstrate that the proposed methodology is highly efficient for defending against malicious labeling and targeted poisoning attacks.

The remainder of this paper is organized as follows. In section II, we discuss existing active learning methods. We describe a threat model and assumptions in section III. In section IV, we describe in detail the proposed approach, followed by its evaluation in section V. Section VI discusses our conclusion and provides future work.

This section provides a brief description of existing active learning methods. Though gathering large quantities of unlabeled data is easy and cheap, obtaining the label for them is usually time-consuming and expensive. Active learning reduces the number of labeled instances needed for training a model effectively. Active learning is an iterative process that consists of training and querying. First, a model is trained based on an initially labeled training set. After the initial model is trained, a query strategy is used to select a new subset of data for label querying. Then, an oracle, e.g., a human annotator, labels these data, and the model is retrained with currently available labeled data. The process repeats until a stopping criterion is met, such as the reach of the desired accuracy. A variety of active learning algorithms have developed over the years. They can be broadly categorized as uncertainty-based active learning and diversity-based active learning.

The uncertainty-based active learning algorithms focus on selecting instances based on an uncertainty principle [5], [6], [7]. The uncertainty principle states that a training instance that is most uncertain to a model contains the most new information for improving the model. There are many ways of measuring uncertainty. For instance, the distance between an instance and the classification boundary can be used to measure the uncertainty. As an instance is closer to the classification boundary, it is more difficult for a classifier to make a high confidence classification. This provides a measure of uncertainty. Another measure of uncertainty is the final confidence provided by a trained model, as the lower confidence provided by a model indicates a relatively high uncertainty in its performance. An alternative way is to measure the difference between the two highest confidence levels provided by the model. The idea behind this way is that if the model is confident in its prediction, it should only provide a very high confidence value for one class and low confidence values for other classes. On the contrary, if a model is unsure of its prediction, it generally provides relatively high confidence values for two or more classes. Other measures of the uncertainty include entropy [8], smallest predicted class probability [9], and so on; See, e.g., [6] for an overview of classic active learning methods.

However, recent work on data poisoning attacks, e.g., [10], [11], [12], has shown that uncertainty sampling may help attackers in achieving their goals. Since poisoning instances are usually near the decision boundary, they are more likely to be selected by the uncertainty sampling approaches.

Another class of active learning algorithms focuses on selecting the most diverse and representative dataset. For instance, Yang et al. [13] proposed an optimization-based active learning algorithm that imposed a diversity constraint objective function. The core-set approach [14] is a diversity-based approach using the core-set selection. Batch Active learning by the Diverse Gradient Embeddings (BADGE) algorithm [15] selects a subset of instances whose gradients with respect to the parameters in the output layer are most diverse. In this research, we present a new active learning method that incorporates both the representative sampling and uncertainty sampling ideas.

In this paper, we study the effect of a mislabeling attack and a data poisoning attack on the active learning of deep neural networks. There is a large amount of data generated every day. However, these data need to be properly labeled before we can apply supervised machine learning algorithms. This is not always feasible with a restricted labeling budget. Crowdsourcing is not reliable as humans make mistakes as well, and the adversary can intentionally provide misleading labels. Furthermore, the data obtained could be poisoned.

In this work, we assume that the class labels provided by a human labeler are not perfect. That is, we consider the effect of human errors. We assume each benign labeler mislabels an instance $p_{r}\in(0,1)$ of the time. Moreover, it is equally likely that an instance (whose true class is $c$) is mislabeled as any other class $\bar{c}\in C/{c}$, where $C$ is the set of classes. Each independent labeler will provide the true label with probability $1-p_{r}$ and the remaining probability $p_{r}$ is evenly distributed to other class labels. That is, if an instance $x$ belongs to class $c$, then a human labeler will label it

where $u$ is a uniform random number between 0 and 1, and $\bar{c}$ is randomly selected from the set $C/{c}$. However, under a malicious mislabeling attack, an attacker consistently mislabels instances of a target class $c_{t}$ as if it belongs to a benign class $\bar{c}_{t}\in C/c_{t}$. As shown in [2], biased labeling usually severely reduces the classification accuracy of the target class but has little effect on the non-target classes.

Furthermore, we consider a data poisoning attack. The attacker’s objective is to inject a data poisoning instance $p$ such that

where C is a cost function that measures the performance of a model $F$ on a target set $D_{t}$, and the parameters $\theta_{i}^{*}$ of the model $F$ at iteration $i$ are determined by minimizing a loss function $L$ used to train a poisoned training set $D_{i}$ as follows:

The poison instance $p$ can be generated by solving Equation 1 based on [16]. The targeted poisoning attack (also known as the feature collision attack) proposed in [16] assumes an attacker has no knowledge of the training data but has knowledge of the model architecture and its parameters. In this research, we extend this poisoning attack to an active learning setting, where the attacker can inject poison instances into the data pool. Specifically, the attacker can generate a poison instance $p$ by solving the following optimization problem:

where $f_{i}$ is the output of the penultimate layer of a network obtained at iteration $i$ of the active learning process, $\beta\in[0,1]$ is a similarity parameter that indicates the relative importance of the first component $\left\lVert f_{i}(x)-f_{i}(t)\right\rVert^{2}_{2}$ to the second component $\left\lVert x-b\right\rVert^{2}_{2}$, $t$ is a target instance, and $b$ is a base instance. The research problem of this study is to propose a novel defense method against such an attack under the assumption that the oracle does not always provide correct/true labels.

In this section, we propose a robust active learning method that reduces the effectiveness of malicious mislabeling and data poisoning attacks. Fig. 1 shows the framework of our proposed active learning method. The four main components of this framework are (1) training, (2) performance checking, (3) querying, and (4) artificial data crafting. The training step is the same as regular training on a labeled dataset. Initially, a set of $m$ data points $D$ is randomly selected to query for labels. Then, the model is trained using the labeled dataset $D$. To check the performance of the model and the existence of malicious instances, a representative subset $S$ of data with labels is set aside and secured so that adversary does not know it. This set $S$ is updated every $q$ iterations to make sure it represents the diversity of the current dataset, where $q$ is an application-specific parameter that depends on the dynamic of the dataset. If the underlying distribution of the data does not change over time and there is no problem with its security, $q=\infty$ and no update is needed.

Besides, for each queried instance, we can generate both adversarial examples and data augmentation. In this way, the training set is increased without incurring an extra query budget. The various adversarial crafting techniques and data argumentation techniques are useful in this case. For each center $c$, $N_{c}$ instances are selected. There is a total of $\sum_{c=1}^{n}N_{c}$ data points $S^{\prime}$ from $n$ centers. Then, a group of human labelers provides the labels for these data points. Next, this newly labeled dataset

where $O(x)$ is the label provided by an oracle/human labeler for an instance $x$, and $D^{\prime}$ is added to the previously obtained labeled dataset $D$; i.e., $D=D\cup D^{\prime}.$ Next, we train the model $F_{i}$ on the new dataset $D$. The performance of the updated model is tested on a secured set $S.$ If the performance degrades by a threshold $\epsilon$, the newly obtained dataset $D^{\prime}$ is examined by $h$ ($h>0$) independent labelers to see if any instance is poisoned or mislabeled. The probability $P_{h}$ that at least one labeler out of $h$ ($h>0$) independent labelers provides correct annotation is

where $P_{1}$ is the average performance of a human labeler. To find the number $h$ of independent labelers to ensure that $P_{h}$ is large enough, saying $0.9995$, we solve equation 5 and obtain

For instance, [17] showed the human annotation performance for the CIFAR-10 dataset is 93.91%. Hence, three independent labelers are enough to ensure that at least one labeler provides correct annotation. If the performance does not degrade more than $\epsilon$, the model is accepted and stopping criterion is checked. If the stopping criterion is met, the process stops. Otherwise, the process repeats. The stop criterion can be (1) the minimal perturbation of all adversarial examples is larger than a threshold $\tau$, (2) the desired classification accuracy is reached, or (3) the labeling budget is exhausted.

In the following subsections, we provide additional discussions for performance checking, querying, and artificial data crafting.

We empirically evaluate the proposed defense method on a subset of the CIFAR-10 dataset [18]. The baseline model for comparison is active learning based on random selection. Our experiments were conducted in the high-performance computing system within the campus private cloud at the University of South Florida. In the set of experiments, we wish to address the following: (1) Does our proposed method perform as well as the standard active learning method when there is no attack? We try to address this question since a few papers (e.g., [19], [20]) have shown the trade-off between the performance of the active learning algorithms with attacks and their performance without attacks. (2) Is our proposed method effective under a malicious mislabeling attack? and (3) Is our proposed method effective under a poison attack? That is, are certain vulnerabilities (e.g., to specific inputs) learned when we perform the adversarial active retraining?

Recent work has shown that machine learning models are vulnerable to adversarial examples and data poisoning attacks. In this work, we utilized these adversarial examples to generate artificial data for training without incurring an extra labeling budget. Furthermore, we proposed a confidence score updating system to check the trustworthiness of each labeler and a performance check on a secured set to monitor the potential mislabeling attack. The experiment results showed that our proposed active learning method is robust against malicious mislabeling and data poisoning attacks. Though we only evaluated the performance of the proposed active learning method on binary classification in the campus private cloud, the method can be applied to a multi-classification task in a distributed system of a public cloud as well. Although most classic single-query active learning strategies do not work as well as a random sampling strategy on deep neural network and many uncertainty-based active learning strategies also perform poorly under data poisoning attacks, it would be interesting to compare the proposed method with the state-of-the-art batch active learning method for CNNs, core-set [14] in the future. Furthermore, our evaluation is based on a single dataset. It would be interesting to check the effectiveness of the method on other datasets. Last, the artificial crafting step introduces additional computation cost. Hence, future work will also investigate the scalability of the proposed method. We can parallelize the process and utilize the scalability in cloud computing for this purpose. Besides a private cloud, we can also deploy our proposed method on the public cloud to not only achieve scalability but also to increase reliability and flexibility.

We acknowledge the AFRL Internship Program to support Jing Lin's work and the National Science Foundation (NSF) to partially sponsor Dr. Kaiqi Xiong's work under grants CNS 1620862 and CNS 1620871, and BBN/GPO project 1936 through an NSF/CNS grant. This material is based on research sponsored by the Air Force Research Laboratory under agreement number FA8750-20-3-1004. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or NSF.