Network-Level Adversaries in Federated Learning

FL considers a set of $n$ clients, each with a local dataset $D_{i}$, and a server $S$. Clients train locally on their own datasets and the server requests model updates, rather than data, building an aggregated global model iteratively over time [18]. We consider here the Federated Averaging training algorithm designed for cross-device settings [1]. In each round $1\leq t\leq T$, the server randomly selects a subset of $m\leq n$ clients to participate in training, and sends them the current global model $f_{t-1}$. Each selected client $i$ trains locally using dataset $D_{i}$, for a fixed number of $T_{L}$ epochs. The server updates the global model $f_{t}$ by mean aggregation of local updates $U_{i}$: $f_{t}=f_{t-1}+\eta\frac{\sum_{i=1}^{m}U_{i}}{m}.$ Data privacy can be further enhanced in FL by secure aggregation performed via Multi-Party Computation (MPC) [19].

In network-level attacks, based on their network-level capabilities, the adversary can observe communication sent over the network and prevent the machine learning algorithm from receiving the information needed to learn a good model. The basic action for a network-level attack is dropping traffic. We consider a smart attacker that selectively drops traffic to maximize the strength of the attack while minimizing detection. The attacker has several decisions to make: (1) what clients to select to drop their contributions, (2) when to start the attack given that federated learning is an iterative protocol, and (3) how many packets (i.e., local models) to drop. We focus on a targeted dropping attack in which the attacker selects a set of clients contributing highly to the target class for dropping their contributions. We compare that with an attack which drops the traffic of a subset of randomly chosen clients.

To summarize, we consider an adversary that has either the Comm_Plain, Comm_Enc or Comm_Enc_Limited network capability, has knowledge of the global model at each round, and targets a particular victim population of interest. We also consider an adversary that additionally has Poison_Model poisoning capabilities.

FL protocols such as Federated Averaging [1] usually abstract the communication protocol between the server and clients. In practice, clients will communicate with the server either directly through a TCP connection, or through a multi-hop overlay network of hop-by-hop TCP or customized transport services. Finally, the last hop connecting the client to the Internet is often in the form of wireless communication. Network infrastructure and routing protocols facilitate all this communication through physical or logical connectivity. By exploiting the underlying routing protocols and network topology an attacker can position themselves to observe and interfere with data of interest; they can then further impact the accuracy of the global model and create an effect similar to that created through a data or model poisoning attack.

Packet dropping attacks can be achieved in multiple ways, and have been studied for network-level adversaries who perform physical-layer attacks in wireless networks [10], router compromise [12], or transport-level attacks [14]. Each of these methods will require different attacker capabilities. For example for physical layers attacks for wireless clients, the attacker needs to be in their proximity - this is both a powerful attack since jamming is usually difficult to defend against, and limiting since it will be difficult for the attacker to attack geographically distributed clients without significant resources. In the case of routing, while a router might be more difficult to compromise, in practice the impact can be bigger as it might have control over the traffic of multiple clients. Last, but not least, if customized overlays are used for communication, compromising nodes in the overlay is slightly easier as they are typically hosts, and the number of clients that can be impacted depends on the scalability of the service.

The key insight of our attack is that only a small number of clients contribute to the accuracy of a target population of interest. Thus, the most effective and difficult to detect strategy is to drop the traffic just for those highly contributing clients. However, the attacker will first need to identify such clients. We design a Client Identification algorithm aimed at determining the set of clients whose model updates lead to the largest improvements in target population accuracy. Our method, described in Algorithm 2, supports both plain and encrypted network communication, and is very effective even if the attacker has limited observability of the network. We demonstrate in Section V that if the adversary has high success at identifying the clients contributing to the task of interest, then the targeted dropping attack impact is much higher compared to random dropping.

The main intuition is that the adversary computes the loss of the model before and after the updates on the target class for a number of rounds $T_{N}$. Rounds in which the loss decreases correspond to an increase in accuracy on the target class. The adversary tracks all the clients participating in those rounds, and computes a loss difference metric per client. In the plain communication case, the adversary will determine exactly which clients decrease the loss, while in the encrypted communication case the adversary only observes the aggregated updates and considers all clients participating in the round to be collectively responsible for the loss improvement.

In more detail, to measure each client’s contribution to the target population, the adversary computes the loss difference between successive model updates on the target population (using a representative dataset $D^{*}$ from the population):

Note that the second term $\ell(f_{t}^{j},D^{*})$ is the loss of the local update of client $j$ in round $t$ for Comm_Plain, but becomes the loss of the global model $\ell(f_{t},D^{*})$ when the adversary only has access to aggregated updates in Comm_Enc or Comm_Enc_Limited. The value $L_{t}^{j}$ measures the decrease in the target class loss before and after a given round $t$.

To more reliably estimate client contributions, especially with aggregation of the model updates, the values of $L_{t}^{j}$ are accumulated over time. The adversary maintains a list for each client $j$ of the values $L_{t}^{j}$ for rounds $t$ in which it participated. This allows the adversary to update the list of clients most relevant to the target class accuracy at every round. Empirically, computing the mean of $L_{t}^{j}$ for each client $j$ across all rounds it participates in works well at identifying the clients contributing most to the target class.

The targeted dropping attack starts with the adversary performing the Client Identification procedure, by running Algorithm 2 during the first $T_{N}$ rounds of the protocol, in which training happens as usual. The adversary identifies in expectation $k_{N}$ clients contributing updates for the target class. After Client Identification is performed, the network adversary drops contributions from the $k_{N}$ clients in every round in which they participate after round $T_{N}$. As the adversary can expect an increase in identification accuracy by monitoring more rounds of the protocol, they can repeat the Client Identification procedure in each subsequent round, and, if necessary, update the list of selected clients. We assume that the adversary drops the identified clients’ updates, and the server simply updates the model using all received updates. If the Client Identification protocol is not completely accurate, the attacker might drop traffic from non-target clients but we found this has a minimal impact on the trained FL model.

Algorithm 2 uses several parameters: the number of clients to identify $k_{N}$ and the number of rounds to wait before identification is successful $T_{N}$, which we analyze here.

In order to amplify the effectiveness of the targeted dropping attack, the adversary may also collude with, or inject, malicious clients, whose presence in training is designed to further compromise the performance on the target distribution. Following Bagdasaryan et al. [4] and Sun et al. [7], we use a targeted model poisoning attack known as model replacement. Writing $\theta_{t}$ for the parameter of the global model $f_{t}$, in this attack, the adversary seeks to replace $\theta_{t}$ with a selected target $\theta_{t}^{*}$ (as is done in [4, 7], we use the $\theta_{t}^{*}$ resulting from a data poisoning attack on a compromised client’s local training procedure). The poisoned clients’ local training sets are sampled identically to clients with access to the target class, with the difference of changing the labels of target class samples to another, fixed class. The update that the adversary sends is $\theta_{t}^{*}-\theta_{t-1}$. The server aggregation then weights this update with an $\eta/m$ factor. In a model replacement attack, a boosting factor $\beta$ is applied to the update, so the update which is sent is $\beta(\theta_{t}^{*}-\theta_{t-1})$, increasing the contribution to overcome the $\eta/m$ factor decrease.

We use three well known datasets (EMNIST, FashionMNIST, and DBPedia-14), which we adapt to the non i.i.d. setting by controlling the class distribution across clients. In all cases, the target population is represented by samples of an entire class selected from the dataset. Our datasets have balanced classes, and we use classes 0, 1, 9 for our evaluation. All reported results are averages of 4 trials, with different randomness seeds. The list of parameters used in our FL protocol is shown in Table I.

EMNIST [32] is a handwritten digits recognition dataset with 344K samples. We use approximately 100K images of numbers, partitioned equally, without overlap, among 100 clients. To enforce heterogeneity, we allocate samples from the target victim class to $k$ fixed clients and vary $k$. For those $k$ clients, we allocate $\alpha_{T}=50\%$ of the local dataset to be target class points, while the remainder is sampled from the remaining classes according to a Dirichlet distribution with parameter $\alpha_{D}=1.0$. For clients without the target class, we sample 100% of the local training set from a Dirichlet distribution with $\alpha_{D}=1.0$, following [33]. We train a convolutional model (two 2D convolution layers and two linear layers, with learning rate $\eta_{L}=0.1$) for 100 rounds, selecting 10 clients at each round, using mean aggregation with a learning rate $\eta=0.25$.

FashionMNIST [34] is an image classification dataset, consisting of 70K gray-scale images of 10 types of clothing articles. It is more complex than EMNIST, but smaller. Thus, we increase the number of training rounds to $T\in\{200,300\}$, reduce the number of clients $n$ to 60, and set $\alpha_{T}$ to 0.6. We set the local dataset size to 400. We use a convolutional model similar to the one used before, and fix all other parameters.

DBPedia-14 [35] is an NLP text classification dataset consisting of 560K articles from 14 ontological categories, such as ”Company”, ”Animal”, ”Film”. DBPedia-14 is a relatively complex dataset, so we use the same $k,T$, and $\alpha_{T}$ as in FashionMNIST, and a local dataset size of 1000. The model is trained starting from pre-trained GloVe embeddings [36], followed by two 1D convolution layers and two linear layers, and optimized with Adam, with $\eta_{L}=0.001$.

To demonstrate the potential severity of a dropping attack, we evaluate the best possible case, where the adversary has perfect knowledge of a subset of $k_{N}$ target clients from the beginning of the protocol, and drops every update originating from them throughout training. The results in Table II demonstrate the power of update dropping, and provide a baseline to compare our full attack pipeline against. We also evaluate the effect of an adversary that selects victim clients uniformly at random.

Table III shows the average number of targeted clients correctly identified by Algorithm 2 for encrypted communication with plain as comparison. Client Identification works very well for plain communication, and maintains a reasonable performance even when the adversary only sees aggregated updates (Comm_Enc). For instance, on DBPedia an average of 11.75 out of 15 clients are identified at 70 rounds under Comm_Enc. Interestingly, for FashionMNIST and DBPedia, it is easier to identify target clients than for EMNIST (where an average of 7 out of 15 clients are identified at round 70). One hypothesis for this phenomenon is that classes are more distinct in complex datasets, leading the global model to forget the target class in rounds with no participating target clients, resulting in significant loss increases for those rounds.

We also use Table III to select parameters for the targeted dropping attack and validate our analysis from Section III-C. In the Comm_Enc scenario, we see that more rounds are necessary for successful identification, as expected from Section III-C. To identify between $k/3$ and $2k/3$ of the target clients, we need to wait between 30 and 50 rounds, and, in many cases, the identification accuracy tends to plateau in successive rounds. Thus, we select round 30 as the starting point for the dropping attack in our Comm_Enc experiments.

We measure our targeted dropping attack’s performance in Table IV. Under Comm_Plain, it significantly compromise target population accuracy, closely approximating the perfect knowledge adversary for increasing values of $k_{N}$. We also observe that our attack in the Comm_Plain scenario vastly outperforms the strategy of randomly dropping the same number of clients, in all situations. Our attacks’ performance decreases when the adversary’s knowledge is limited in Comm_Enc, which is expected from the reduced Client Identification performance. We still observe a significant advantage in using our identification pipeline, over the random selection baseline. For instance, with $k_{N}=5$, the target accuracy on DBPedia drops drastically from 53% to 6%. Moreover, these results highlight the trend mentioned in Section V-C: on more complex tasks, such as FashionMNIST and DBPedia, the high identification accuracy leads to noticeably larger attack performance, than in EMNIST. Given the targeted nature of our attack, in all considered scenarios, dropping the victim clients generally leads to very little degradation of the overall model performance – on average 3.88% for Comm_Plain and 1% for Comm_Enc.

Tables VI, VIII, X, show the accuracy of the global model on the full test set under the same settings as the other experiments we run.

We compared the effects of the model poisoning strategy and targeted dropping on the EMNIST dataset for the cases of perfect knowledge, Comm_Plain, and Comm_Enc in Figures 1a, 1c, and 1e respectively. These heatmaps show that, for different levels of $k_{N}$ (number of dropped clients) and $k_{P}$ (number of poisoned clients), the model replacement attack is more effective than targeted dropping, even when the adversary has perfect knowledge. The results, however, are significantly different when Clipping, a standard poisoning defense [7, 37, 38], is applied to local model updates. Figures 1b, 1d, and 1f show the same set of experiments repeated with a clipping norm of 1. These heatmaps highlight that, while clipping lowers the impact of model poisoning, the combination of targeted dropping and model poisoning still results in very noticeable targeted performance degradation. For instance, under Comm_Plain, the original model accuracy on class 0 is 0.8. This becomes 0.59 when dropping 7 clients out of 15, and 0.66 when poisoning 7 clients, while combining both attacks leads to 0.4 accuracy.

By repeating the experiments with $T=50$, we can observe that it appears that targeted dropping is more effective than poisoning at low round counts, as shown in Fig. 2 When the model is in its early phase of learning, it needs to see some examples of the target class to begin learning it, and dropping significantly delays the arrival of these examples. Later in training, however, the difference between poisoning and dropping decreases.

We model a targeted, resource-limited, adversary, Comm_Enc_Limited, by restricting their ability to observe the clients participating to the protocol to a fixed-size subset of the clients, chosen before the attack starts, sampled from a Dirichlet distribution with concentration parameter $\alpha_{V}$. $\alpha_{V}$ controls how likely it is for the sampled visible set to include clients containing data of the target population, with larger values of $\alpha_{V}$ leading to higher likelihood. We run the attack on FashionMNIST with a similar setup as in Table IV under the Comm_Enc conditions. The results reported in Figure 3a show that, as expected, higher visibility fractions, and larger $\alpha_{V}$ lead to smaller target accuracy values. The network adversary is still quite effective even when observing a small fraction of the clients, for instance achieving a $\approx 43.6\%$ relative target accuracy decrease when observing 20 out of 60 clients with $\alpha_{V}=2$. Similarly, Figure 3b shows that adding poisoning always leads to better accuracy degradation.

In Tables V,  VII, we evaluate the defense strategies under Comm_Plain and Comm_Enc, using targeted dropping and poisoning attacks. We set the number of dropped ($k_{N}$) and poisoned clients ($k_{P}$) to $2k/3$ for EMNIST, and $k/3$ for FashionMNIST and DBPedia. These parameters result in strong attacks, as the target accuracy is below 4% under targeted dropping and poisoning, when no mitigation is used. UpSampling is successful in mitigating targeted dropping, and, when combined with Clipping, achieves high accuracy against the powerful combined attack. For instance, on EMNIST’s class 1 with $k=9$, the update dropping attack causes accuracy to decrease from 92% to 25% under Comm_Plain, and UpSampling manages to restore the target accuracy to 76%.

The UpSampling defense is very effective under the Comm_Enc setting, due to knowledge asymmetry, i.e. the attacker receives aggregated updates, while the defender observes individual updates. Thus, UpSampling, defending against the dropping attack, improves the target model accuracy by an average (over the 3 classes) of 6.33% on EMNIST, 10.66% on FashionMNIST, and 24.66% on DBPedia for $k=15$ compared to the original accuracy. Even under both targeted dropping and model poisoning attacks, the combined UpSampling defense and Clipping results in an average decrease of 5% on EMNIST and average increase of 9% on FashionMNIST and 16.66% on DBPedia over the 3 classes compared to the original accuracy. Interestingly, classes with lower original accuracy benefit more from the UpSampling strategy, with improvements as high as 34% (on DBPedia for class 9, original accuracy is increased from 60% to 94% with UpSampling under targeted dropping).

We observed that under-represented classes (in terms of number of clients holding samples from those classes) are impacted more by our attacks. To alleviate this problem, the server could identify new clients with data from the populations of interest, and add them to the set of clients participating in the FL protocol. In cross-device FL settings, servers typically have access to a large number of clients, and can make decisions on expanding the set of participating clients to improve accuracy on under-represented populations.

In Table IX, we present attack and defense results for this challenging setting. Similarly to the previous tables, we set the parameters to $k_{N}=k_{P}=2k/3$ for EMNIST, and $k_{N}=k_{P}=k/3$ for both FashionMNIST and DBPedia. While under this setting there is essentially no difference in the effectiveness of the attacker models we consider, the defensive performance varies due to the limited knowledge. The results we observe for the UpSample and Clip + UpSample defensive mechanisms are, as we would expect, somewhere in between the Comm_Enc and Comm_Plain scenarios. For instance for EMNIST, with $k=15$ on class 1 we obtain an average accuracy level of 0.84, considerably higher than the 0.51 obtained under Comm_Plain, but also not as high as the average of 0.94 obtained with Comm_Enc, which is essentially the same result obtained without any attack.