Practical and General Backdoor Attacks against Vertical Federated Learning

The goal of $P_{K}$ is to establish a strong link between the trigger and the attack target. Whenever the trigger is presented in the input instance, the victim model should predict the target label. Meanwhile, the attacker should ensure the clean data are classified correctly to maintain the model utility. Formally, the attacker optimizes the following objective function:

where $f_{i}^{c}=Concat\{\mathcal{F}_{\theta_{1}}(x_{i}^{1}),...,\mathcal{F}_{\theta_{K}}(x_{i}^{K})\}$, $f_{i}^{p}=Concat\{\mathcal{F}_{\theta_{1}}(x_{i}^{1}),...,\mathcal{F}_{\theta_{K}}(x_{i}^{K}+\mathcal{T})\}$ are the clean and poisoned feature representations, respectively. $\mathcal{T}$ is the injected trigger. The first term ensures that the victim model behaves normally on clean data, and the second term achieves the backdoor behavior. We showcase various backdoored samples in Supplement A.1.

We assume $P_{K}$ strictly follows the VFL protocols: uploading feature representations, receiving gradients, and updating its local model. The data accessible to $P_{K}$ are: own local data $\{x_{i}^{K}\}_{i=1}^{N}$ and the corresponding gradients $\{g_{i}^{K}\}_{i=1}^{N}$ returned from the active party. Moreover, $P_{K}$ has no knowledge of the data, the model, and any intermediate information owned by other parties. The adversary cannot interfere with the normal interactions between the active party and other passive parties.

We randomly select $n$ samples per batch (denoted as $x_{nset}^{K}$, $|nset|=n$) to detect whether they are from the target class. To increase the gradient similarity between $x_{t}^{K}$ and the data from the target class in $x_{nset}^{K}$, we replace $x_{j}^{K}$ with $x_{t}^{K}$ (for all $j\in nset$), so the only difference between $x_{t}$ and $x_{nset}$ is the data held by the other clients. Then we upload the replaced data to the top model.

After obtaining the returned gradients of $x_{nset}^{K}$ (denoted as $g_{nset}^{K}$), we compute the cosine similarity between $g_{nset}^{K}$ and $g_{t}^{K}$. (Note that $g_{t}^{K}$ is updated when $x_{t}^{K}$ is uploaded.)

Intuitively, the higher the similarity, the more likely they have the same label. To illustrate how this works, we consider a simple example where the top model consists of only one linear layer.

where $W$ is the top model parameters, $O$ is the output of sample $i$, and $C$ is the number of classes for classification. Here we assume the true label $y_{i}$ of sample $i$ is class $c$, where $c\in[1,C]$. As we can see, the gradient of the cross-entropy loss w.r.t. the feature representation of sample $i$ is:

Therefore, the gradients received by $P_{K}$ have the following properties: data from the same class will return similar gradients, resulting in a high positive cosine similarity. While samples from different classes will return dissimilar gradients, resulting in a low cosine similarity.

Finally, we set a threshold $\alpha_{thre}$. If the similarity between $g_{j}^{K}$ (for $j\in nset$) and $g_{t}^{K}$ is higher than $\alpha_{thre}$, we consider $x_{j}^{K}$ to be from the target class, named source data. We terminate the SDD process until enough source data are found or all training data have been considered. Compared with the state-of-the-art label inference method [5] which requires many auxiliary labeled data, our SDD is more practical.

A successful backdoored model should give the target prediction as long as the malicious trigger is present, despite the existence of the background clean features. To make the trigger a higher priority than the other clean features in the decision-making phase, we attempt to replace the source data with the data randomly selected from the same batch. In this way, the source data will contain clean features of different classes. Then we add the trigger on the perturbed source data and put them into the training process to achieve backdoor injection.

There are mainly two reasons why SDP module enhances the decision-dependency between the trigger and attack target: (a) The source data inferred by SDD have the same label (attack target). And their features are replaced by randomly selected data after the SDP module. This makes the source data equipped with different features but the same label, causing the model more difficult to learn from these features. (b) After the SDP process, we add the same trigger to the source data, thus they have the same trigger and the same target label, which makes the model more likely to establish the decision-dependency between the trigger and attack target.

We evaluate BadVFL on the following datasets: CIFAR-10 [16], ImageNet [25], Breast Histopathology Images (BHI) [23], and IMDB [20]. The first three are image datasets, and IMDB is a text dataset. We describe the datasets in detail in Supplement A.2. To make these datasets suitable for the VFL scenario, as the common setting in VFL [18, 13, 5], for CIFAR-10 and ImageNet, we split the data into two parts along the middle line so that each party holds half. For BHI, there are multiple examination image patches per patient, and we distribute the patches of each patient with the same label to each party in a round-robin manner. For IMDB, we split each sample (a paragraph for a movie review) into two parts and distribute them to each party.

We experiment on three classic deep neural networks to get feature representations, namely ResNet18 [8] for CIFAR-10 and BHI, VGG16 [28] for ImageNet and LSTM [11] for IMDB. As for the top model used for feature combination and classification, following previous works [12, 13], we adopt a linear combination of these features and then apply a nonlinear transformation (e.g., softmax) to make the prediction. To verify the stability of BadVFL, we also conduct experiments with multi-hidden layers.

For image datasets, the models are trained by SGD optimizer for 200 epochs. The initial learning rate is $0.01$, multiplied by $0.1$ per $50$ epoch. For text dataset, the optimizer is Adam with an initial learning rate of $0.001$. In all experiments, the poisoning rate $\eta=\frac{|D_{poisoned}|}{|D_{train}|}$ is $1\%$, as the common setting for backdoor attacks [7]. And we set the replacement number $n=5$ and the threshold $\alpha_{thre}=0.6$ in SDD for all datasets.

We adopt Test Accuracy Rate (TAR) and Attack Success Rate (ASR) to evaluate BadVFL performance. Specifically, TAR is the probability that the clean data are classified correctly, measuring the impact of backdoor attacks on the main task. ASR is the probability of predicting the poisoned data as the target label, which measures the attack efficacy.

For image datasets, the trigger we applied following Gu et al. [7], which is a white square located in the center of the image. We apply $4\times 4$, $20\times 20$, and $5\times 5$ trigger size for CIFAR-10, ImageNet and BHI, respectively. For IMDB, we insert the word ‘[START]’ into the middle of the sentence as the trigger.

To ensure our attack remains consistently effective, for each class, we construct BadVFL with randomly selecting 3 different target data from the dataset and get their average as the final result. To suppress the effect of non-determinism, all experiments are averaged across multiple runs. The results are shown in Table 1. Observe that the triggers are successfully injected as the poisoned models have small TAR difference compared with the benign models and high ASR. Specifically, BadVFL achieves above $93\%$ ASR in all datasets with negligible TAR drops.

Figure 3 depicts the category distribution of the source data obtained by SDD module for different target class. The color-coded values in row $i$ and column $j$ represent the number of inferred source data from class $j$ when target data from class $i$. As we can see, for most cases, SDD module can correctly identify the source data which are truly from target class. However, for some target class, such as class $3$ in CIFAR-10, the SDD mistakenly identifies few samples from class $5$ as its source data. This is because the data from class $3$ (cat) have the similar features with data from class $5$ (dog). And few false detection results have little influence on the attack effectiveness.

We compare BadVFL with the state-of-the-art method GR [18], which attacks VFL system by replacing the gradient of local triggered samples with the gradient of the target data when updating the local bottom model. Table 1 shows the comparison results. Apparently, our method outperforms GR on all metrics and achieves a significant boost. In more details, GR hurts worse on the main task accuracy. This is because the adversary replaces the triggered data’s feature with random vectors and sends them to the active party, to prevent the active party establishing a new mapping between the triggered data with their true label. Moreover, the attack performance of GR strongly depends on the selection of target data.

One straightforward attempt to defense BadVFL is adding noise to the exchanged information. We experiment Gaussian noise [35] with variance from $10^{-5}$ to $10^{-2}$. Because adding noise inevitably affects the gradient similarity calculation, we select the 1% highest similarity results as the source data in each iteration instead of a fixed threshold. The results are shown in Figure 5a. As we can see, the BadVFL performance monotonically decreases with the increasing of noise scales. In details, when variance is 1e-4, the ASR on ImageNet severely deteriorates and TAR drops by nearly 20%, resulting in a good defense performance but seriously compromising the model utility. For CIFAR-10, setting variance to 1e-3 successfully defends against BadVFL, where the ASR drops to 30% with negligible TAR drops. However, in practice, it is non-trivial to figure out an appropriate noise scale that guarantees security while maintains model utility.

Another effective defense strategy is pruning the gradients with small magnitudes to zero [17]. We evaluate different level of sparsity from $0.75$ to $0.1$. As shown in Figure 5b, interestingly, for CIFAR-10, BHI and IMDB, BadVFL maintains considerable high TAR and ASR with the increasing of compression rate. As for ImageNet, the gradient compression can successfully mitigate the backdoor attacks in VFL, but introducing significantly TAR drops and destroying the model utility.

To further validate the influence of trigger position on attack effectiveness, we plot Figure 6 to show the BadVFL attack performance with three possible locations. For image classification task, we experiment the trigger located in “up left" (u-l), “center", and “bottom right" (b-r) of the image to analyze the effectiveness. For text classification task, we conduct the experiments with the trigger located in the “initial", “middle" and “end" of the text.

As shown in Figure 6, we notice that for image classification task, the center location has a significant advantage over the other two locations, because the center area contributes more to model classification and its original features are blocked.

For text classification task, the initial and end position have a slight advantage over the middle position, due to the training mode of the Recurrent Neural Networks. Nevertheless, no matter where the triggers are, BadVFL can always succeed in injecting the backdoor into the model with negligible main accuracy drops.

As discussed in Section 4, if we directly add triggers to the source data and put them into training process, the model may not learn the trigger but the clean feature of the source data. Thus, we evaluate the importance of SDP and the results are shown in Table 2. Specifically, we consider the three following cases: (a) without perturbation, (b) replace the source data with data selected from the same batch, and (c) replace the source data with data selected from the whole dataset.

We observe that there is no significant difference between the cases where the attacker replaces the source data with the data from same batch, or from the whole training dataset. However, when the attacker does not perturb the source data and directly adds the trigger on them, the BadVFL performance drops significantly. This is especially prominent in the case of the CIFAR-10 and ImageNet dataset.

We investigate the critical factor $\eta$ which affects the number of poisoned data in the training process. As shown in Figure 7a, the ASR becomes worse when $\eta>3\%$. This is because the attacker uploads more “wrong” features (perturbed source data) as $\eta$ increases, resulting in the top model depends more on other clean features in decision-making.

We depict the impact of replacement number $n$ on the BadVFL performance in Figure 7b. There is no significant different among varied $n$. Moreover, because a small $n$ makes the poisoning process more stealthy, we set $n=5$ as default for all datasets.

The $\alpha_{thre}$ in the SDD module can control the final source data set of backdoor attacks. The results are shown in Figure 7c. Specifically, when $\alpha_{thre}<0.6$, the larger $\alpha_{thre}$ makes BadVFL more effective. This is caused by the fact that the larger $\alpha_{thre}$ can infer more accurate source data which are truly from target class. When $\alpha_{thre}>0.6$, the BadVFL converges more stably. Hence, we set $\alpha_{thre}=0.6$ for all datasets.

Another critical factor that might affect the BadVFL performance is the size of trigger. As shown in Figure 7d, we can observe that BadVFL achieves a stable performance among different trigger sizes.