DeSMP: Differential Privacy-exploited Stealthy Model Poisoning Attacks in Federated Learning

However, traditional untargeted poisoning attacks mainly utilize the hyperparameters of the targeted model to scale up the effectiveness of the malicious model [11]. To attain the goal of poisoning, the adversary may use explicit boosting that deforms the weights’ distribution, however, then it can be easily detected by the server through simple server-side model checking [19]. Hence, untargeted model poisoning attacks in a stealthy manner remain an open problem in FL [20]. Moreover, since an FL system usually consists of a huge number of clients and only a portion of clients are chosen for any particular round [21], the odds of impacting the global model accuracy significantly by a single malicious contribution is very low. This leads us to the question- “How can the adversary perform an untargeted model poisoning attacks in a stealthy but persistent fashion?”. Motivated by this, in this paper, we investigate the DP mechanism as a tool to conduct such adversarial poisoning attacks in FL. In the rest of the paper, the ‘false data injection (FDI)’ attack and ‘model poisoning’ attack is mentioned interchangeably.

In this paper, we show that the DP mechanism is creating a new attack avenue for stealthy false data injection (FDI) or model poisoning attacks in a DPFL environment. We name this attack model as ‘DP-exploited stealthy model poisoning’ (in short, DeSMP) attacks. Particularly, we make the following contributions:

• •

  We demonstrate that DP, as a privacy-preserving tool, is opening a new backdoor for untargeted model poisoning attacks in the FL setting. Our proposed attack strategy (DeSMP) is stealthy and persistent in nature.

• •

  To tackle the proposed DeSMP attack, we develop a reinforcement learning (RL)-based defense strategy. The proposed RL-based defense approach intelligently selects the differential privacy level for the clients’ model update. It also minimizes the attack vectors and facilitates attack disclosure.

Section II of this paper covers preliminaries of FL and a brief review of the related works while section III outlines the research problem and threat model. Section IV formulates the proposed DeSMP attack and defense model and their working principle. In section V, we analyze and evaluate the effectiveness of our proposed model. Finally, in section VI, we conclude the paper with some future research directions.

Nonetheless, while deploying the DP mechanism, researchers [2, 21] have suggested using norm clipping or early stopping methods to compensate for the high level of random differential noise and prevent the model to be completely unusable. Once a pre-defined testing criterion (e.g., model accuracy is greater than a threshold or privacy budget exceeds) is met, the server finalizes the global model and stops the training procedure; otherwise, the training process re-initiates.

In the proposed setting, we assume that the adversary can manipulate the model updates regardless of the attack vectors (i.e., through man-in-the-middle or server-side attack formation). However, the adversary cannot directly change the models that are already on the server. He has white-box access (i.e., full knowledge of the global and local model parameters). The adversary might have partial knowledge of the training and testing data (i.e., distributions of the data); however, this is not a strict requirement in our threat model. In addition, we assume that the adversary has the knowledge of the imposed DP mechanism and privacy budget ($\varepsilon$). This assumption is particularly important and realistic as many researchers including Dwork et al. [28] emphasize the necessity of publishing the privacy budget in order to increase the trustworthiness of the system.

To achieve this goal, the adversary carefully selects the tolerance value ($\gamma$) and draws noise from the optimal attack distribution, $f_{a}^{*}$ as represented by (2). In other words, the adversary replaces the benign Gaussian noise mechanism, $\mathcal{N}(\theta,\sigma^{2}\mathcal{S}^{2})$ by malicious noise adding mechanism $\mathcal{N}_{a}(\theta+\sqrt{2\gamma}\sigma,\sigma^{2}\mathcal{S}^{2})$ following (2). Here, $\theta$ represents the mean value or location parameter of the Gaussian distribution while $\sigma^{2}\mathcal{S}^{2}$ indicates the scaling factor of the same distribution. By controlling the value of the tolerance level ($\gamma$), the adversary can control the attack impact level ($\mu_{a}$) and shift the mean value further from the actual value (i.e., $\theta$ to $\theta+\sqrt{2\gamma}\sigma$). In short, increasing/decreasing the value of $\gamma$ increases/decreases the level of noise and vice versa. Nevertheless, since the attack distribution $(f_{a}^{*})$ follows the same statistical properties of a benign distribution $(f_{0})$, the adversarial noise $(\mathcal{N}_{a})$ as well as the poisonous weights will not be very different statistically from other weights. More specifically, unless the adversary chooses a very large $\gamma$, the proposed DeSMP attack will achieve stealthiness while remaining persistent. We empirically observe and evaluate the proposed DeSMP attack on the FL models in section V.

For MNIST, the training (Tr) and validation (Val) loss of three random clients (C1, C2, and C3) in an arbitrary communication round has been depicted in Fig. 2(a). It can be inferred that, in each incremental epoch, the training and validation loss is decreasing. Also, from Fig. 2(b), we can see that the DPFL algorithm converges after a few communication rounds. The final accuracy value after round $30$ is around $0.95$. Another important thing to notice is that the privacy budget ($\mathcal{B}$) is spent very quickly if the $\varepsilon$ is small and the model can not converge properly. Therefore, it is significantly important to select the privacy loss and budget level (i.e., $\varepsilon$ and $\mathcal{B}$) intelligently and in an optimal way so that the model possesses the desired level of privacy and utility. Likewise, for Power consumption dataset, we verify the DPFL approach and find similar results. The cost of applying DP (i.e., the ‘privacy cost‘) over the global model loss varies with $\varepsilon$. Thus, more privacy leads to more loss for both the classification and regression problems.

In Fig. 2(c) and (d), We can further observe the impact of our proposed approach. Fig. 2(c) reflects the outcome of the proposed DPFL model on some randomly selected MNIST image samples whereas 2(d) depicts the adversarial outcomes through our proposed (DeSMP) model for the same samples. Due to the stealthy adversarial noise with $\gamma=0.10$, only the image of digit ‘4’ has been predicted wrongly as digit ‘7’ while the other digits are predicted correctly. Since we are considering the untargeted model poisoning attack, the adversarial action through the proposed DeSMP model may alter the image label differently each time. However, as the overall accuracy does not degrade too much with a low $\gamma$, the malicious action becomes stealthy and goes unnoticed by the anomaly detectors.

Likewise, the impacts of adversarial action exploiting the DP noise for Power consumption dataset have been illustrated in Fig. (3). It can be observed that the DeSMP attack is also increasing the loss with respect to the increase in $\varepsilon$ and $\gamma$ value. However, for the regression problem, if the raw training data across all the clients are similar and identically distributed, then the attack requires adding more noise (i.e., small $\varepsilon$ and large $\gamma$) in order to achieve the desired level of attack impact. For instance, we can see from Fig. 3(a) that even after adding the DP mechanism with different $\varepsilon$, the model converges after a sufficient number of communication rounds. It is also desirable since the privacy preserves and the utility remains satisfactory. However, in the presence of an adversary. the loss starts to increase. This phenomena can be observed in Fig. 3(b)-(d). Moreover, as $\varepsilon$ starts to decrease (i.e., privacy increases) from $8.0$ to $1.0$, the attacker obtains more attack opportunities. From Fig. 3(b), it can be inferred that the shifting from $(\varepsilon,\gamma)=(8,0.01)$ to $(\varepsilon,\gamma)=(8,0.10)$ is increasing the loss by $7$ times (i.e., $0.01$ to $0.07$) whereas in Fig. 3(c), shifting from $(\varepsilon,\gamma)=(4,0.01)$ to $(\varepsilon,\gamma)=(4,0.10)$ is increasing the loss by more than $20$ times (i.e., increasing from $0.01$ to more than $0.20$). Moreover, comparing the red FDI-DP curveS of Fig. 3(b) and (c), it can be perceived that decreasing $\varepsilon$ by half (from $8.0$ to $4.0$) is increasing the test loss by almost $3$ times ($0.07$ to $0.20$) when tolerance level is relatively high ($\gamma=0.10$).

Therefore, the attack impact ($\mu_{a}$) increases significantly with the increment of the attacker’s tolerance level, $\gamma$, and the model turns to a sub-optimal model. Eventually, through the DeSMP attack, at a very low $\varepsilon$ and high $\gamma$, the DPFL model becomes unusable and initiates denial-of-service. Nevertheless, the proposed DeSMP model can also be tailored to conduct more devastating attacks while maintaining stealthiness through hyper-parameter tuning and selectively choosing the FL-parameters that are mentioned in Table II.

Fig. 4 illustrates the accumulated reward of the defending agent for learning rate $(\alpha=0.1)$ and discount factor $(\chi=1)$ for two distinct datasets (MNIST and Power consumption). The trend in the figure illustrates that the agent learns optimal policy over episodes, and it converges after sufficient episodes are executed. Since we define the reward function such that it takes care of federated loss $(f_{l})$, attacker loss $(m_{l})$, and privacy loss $(\varepsilon)$, this convergence finds the optimal trade-off policy for the privacy, security, and utility of the system. Since the agent outputs an action (or $\varepsilon$) for each state, we can calculate the standard value of federated loss $(f_{l}^{s})$ for that state. Therefore, if the practical or real-time observed federated loss $(f_{l}^{p})$ differs from the expected (or standard) one, we can infer whether the attack is launched or not. Specifically, if the $f_{l}^{s}$ is less than $f_{l}^{p}$, we can infer that the large scale (large $\gamma$) FDI attack is launched; otherwise, the system is not compromised or the degree of FDI attack scale (low $\gamma$) is very low.