Poisoning Attacks and Defenses in Federated Learning: A Survey

The notion of FL was first introduced by Google and is characterized as a distributed machine learning (ML) paradigm to collaboratively train a global ML model on datasets that are distributed across a number of clients (e.g., mobile devices) while protecting the privacy of clients [1]. In essence, there are two different types of parties in FL, i.e., a number of clients and a central server/cloud, wherein each client maintains a local model trained on the local dataset. In contrast, the central server maintains a global model, which is the aggregation of locally trained models.

In general, FL employs the stochastic gradient descent (SGD) to minimize a loss function and the models are updated iteratively (due to the use of SGD). In each iteration, there are three stages as follows: (1) selection of clients; (2) local model updates; and (3) aggregation by the central server. In particular, firstly, the central server selects a number of clients and shares the current global model. Secondly, each client retrains the current global model on the local dataset and sends the latest local model back to the central server. Finally, the central server aggregates all the local models in order to obtain the latest global model and such aggregation schemes include mean aggregation, byzantine-robust aggregation, etc [4].

The term ‘poisoning attack’ in FL refers to an attack that involves attackers intentionally tampering with the training data or model parameters to manipulate the model aggregation in order to disrupt the integrity and availability of the global learning model. In FL, because the participants’ data and the training processes are invisible to the central server, poisoning attacks might be easy to carry out by some clients.

Poisoning attacks can be divided into two types, i.e., DPA and MPA, or can be divided into targeted and untargeted attacks [4]. We have considered the division in terms of data and poisoning attacks and a taxonomy of these attacks is illustrated in Figure 1(a). The location of poisoning attacks may vary and it can be on the client’s side, communication channel, or sometimes on the central server. Nevertheless, the common poisoning attacks usually take place on the client side (i.e., modifying the data, models, or both) as illustrated in Figure 1(b).

In FL, poisoning attacks work by corrupting the data of one or more clients or the locally trained models in an effort to interfere with the global model. For DPA, it is assumed that the attacker has the access to the data of at least one client and the capacity to alter it, wherein the poisoned data is an amalgam of cleaned (original labels) and modified data. Furthermore, the MPA directly poison the local model updates sent by the participating clients to the global server by modifying the weights of the local models.

With the assumption that the FL participants may change the raw data on their devices, the authors of [5] proposed a DPA model in terms of label flipping attacks. Under this assumption, a complex neural network model is employed to investigate the FL application in the presence of label-flipping attacks. Furthermore, the efficacy of the attacks is shown with respect to varying percentages of malevolent participants using two well-known image datasets, CIFAR-10 and Fashion-MNIST. It can be observed that the attack is effective even with a small ratio of malicious participants and can also be targeted, i.e., having more impact on a subset of targeted data labels.

Xie et al. suggested a distributed backdoor attack, called DBA [6], which employed a composite global trigger to perform the attacks as opposed to the straightforward label-flipping attack [5] carried out by several attackers. The local models are trained on a poisoned dataset after each adversary picks a unique local trigger to poison the training dataset. The server then receives the updates from the attackers for model aggregation. The attackers employ the local triggers to create a global trigger at the inference step rather than using them directly. It is demonstrated that the global trigger has the highest attack success rate of any local trigger, even if it never appears in the training phase.

Furthermore, [7] theoretically demonstrated that backdoor attacks are inevitable in the presence of adversarial samples in FL and intuitively proposed a new family of backdoor attacks, known as edge-case backdoor to manipulate the model on the input resides on the tail of the distribution. Therefore, this makes it difficult for the defense strategies to detect such attacks as the attacker’s access is restricted to developing an untrained feature map instead of a fully trained model.

In [8], the vulnerability of FL towards a MPA is discussed. The MPA can be performed by using model replacement, wherein one or more malicious clients attempt to replace a benign model with a malicious one, leading the model to misclassify future inputs. Moreover, the study discusses the countermeasures and concludes that the attackers can easily bypass such measures. The main idea of FL is to take advantage of the diversity of the clients in terms of non-iid training data, including uncommon or low-quality data. However, using secure aggregation by the central server to filter out anomalous contributions runs counter to this idea.

In contrast to [8], Bhagoji et al. proposed [11] to carry out the attack even when the global model is not yet converged. They specifically nurture the malicious updates for $lambda$ times to counteract the benefits of benign clients. Two stealth metrics that the server might examine were suggested. The first is that the server may determine whether the attacker’s update aids in model training, i.e., enhances the performance of the overall model. The server may also examine if the submitted update differs from earlier updates, which is the second possibility. They then improved the loss function based on those two covert metrics to prevent anomaly identification in order to increase the attack’s robustness. Finally, the effectiveness of the model is evaluated with a number of assumptions (single-shot, repeated, and pixel-pattern attack). It was observed that the model can achieve $100\%$ accuracy on an attacker’s task in just a single training round.

A number of defense strategies are present in the literature, this study divides the state-of-the-art into three defense types, namely anomaly detection, robust aggregation, and perturbation mechanism.

A feedback-based defense strategy, known as BaFFle, is put forward in [13] to eliminate the backdoored model updates. BaFFle employs each participant to validate the global model for each round of FL by quantifying the validation function with their own private data to report whether the model is backdoored or not to the central server. The validation function compares the misclassification rate of a distinct class with the previous global model. Then, the central server utilizes the feedback via the validation function to decide whether to accept or reject the global model update based on the misclassification rate.

The discussed defense strategies work fine in the presence of a small number of malicious participants. The performance of these strategies degrades with an increase in malicious participants. Zhang et al. [10] proposed the malicious client detection algorithm, named FLDetector, wherein the idea is to detect the malicious participants and eliminate them from the global model updates. In FLDetector, the central server predicts the participants’ model updates in each iteration based on historical updates and flags the participant as malicious if there is an inconsistency between the current update and the predicted one in multiple iterations. The performance evaluation illustrates promising results when compared with the current state-of-the-art. Nevertheless, the impact of eliminating the malicious participants on the main task accuracy is not discussed.

Firstly, we compare attack strategies with the following fours characteristics (Table I)

As can be seen in Table I, the majority of the attack strategies utilize iid datasets for DPA (except [6]) in order to distinguish between the benign and malicious updates [5] [7]. In contrast, the majority of MPA use datasets that are non-iid, making them beneficial in real-life FL environments.

The experimental evaluation in terms of overall model accuracy (MA) and attack success rate (ASR) is carried out for a few selected attacks models [5][6][8][11] as depicted in Figure 2 for both DPA and MPA. MA measures the model accuracy on benign samples, whereas, ASR represents the probability (success rate) of attacks in misclassifying the target labels.

We have trained the global model on CIFAR-10 for attack scenarios and MNIST for defense strategies with a total of 100 participants and considered the common simulation scenarios among all the compared models. For DPA, we train the global model with a varying number of malicious participants. For the MPA, the global model is trained with a varying number of iterations with a poisoning rate of $10\%$.

The results for DPA are shown in Figure 2a and 2b. It can be seen that [6] can achieve higher ASR (Figure 2(a)) than [5] and also maintains high MA (Figure 2(b)). The reason for lower ASR and lower MA for [5] is the use of complex CIFAR-10 datasets and the increase in the malicious ratio. Moreover, the results for MPA are depicted in Figure 2c and 2d. It can be seen that the ASR for [8] decreases with an increase in the iterations as the benign participants dilute the impact of backdoor. However, the ASR and MA of [11] remains stable with the iterations.

In general, the current studies are well-suited to poisoning attack scenarios with the fixed assumption. Nevertheless, the predefined conditions in the state-of-the-art attack strategies may not be suitable for real-world applications with dynamic characteristics.

Table II illustrates the comparison of defense strategies by employing a number of aspects discussed below.

In Figure 3, we have compared the four defense strategies studied in [5][12][13][10] in terms of MA with respect to varying percentages of malicious participants the number of iterations for DPA and MPA, respectively. As depicted in Figure 3(a), the MA for [5] decreases as the number of malicious participants increases, and maintains the accuracy of around $80\%$. Nevertheless, the accuracy of [12] remains stable with an average MA of $85\%$. Furthermore, the MA for defense strategies for MPA increases as the number of iterations increases for both [10] and [13]. Nonetheless, the convergence time in terms of defense mechanism for [10] is better than [13].

As a whole, the current research on defense strategies for poisoning attacks is well-defined for the specific scenarios. However, intelligent malicious clients can still learn the behaviour of these defense strategies and poison the FL systems.