Kick Bad Guys Out! Conditionally Activated Anomaly Detection in Federated Learning with Zero-Knowledge Proof Verification

Federated Learning (FL) [43] enables clients to collaboratively train machine learning models without sharing their local data with other parties. Due to its privacy-preserving nature, FL has attracted considerable attention across various domains in real-world applications [31, 14, 49, 39, 8, 16]. Even though FL does not require clients to share their raw data with other parties, its collaborative nature inadvertently introduces privacy and security vulnerabilities [10, 5, 38, 34, 56, 15, 54, 36, 66]. Malicious clients can harm training by submitting corrupted model updates to disrupt global model convergence [20, 15], or by planting backdoors that cause the global model to perform poorly on certain data [3, 2, 57].

Existing literature on defenses in FL comes with certain inherent limitations, making them unsuitable for real-world FL systems [7, 62, 24, 47, 33, 12, 35, 53, 23, 46, 52]. Some strategies require prior knowledge of the number of malicious clients within the FL system [7], while in practice adversaries would not announce their malicious intentions before attacking. Other defense strategies mitigate impacts of potential malicious client submissions by leveraging methods that inevitably alter the aggregation results, such as re-weighting the local models [24], modifying the aggregation function [47], and removing local models that tend to be poisoned [7]. However, in practical FL systems, attacks happen infrequently. While introducing the aforementioned defenses can mitigate the impact of potential malicious clients, the performance loss caused by the inclusion of them can outweigh the defense gain, as most real-world training cases are benign and these defenses largely compromise the model quality for all benign cases. Moreover, existing defense mechanisms are deployed om FL servers without any verification for their execution. As a result, clients are unable to verify whether the defense mechanism was executed accurately and correctly, leaving them reliant on server’s integrity and undermining trust in real-world FL systems.

Motivated by these, a successful anomaly detection approach should simultaneously satisfy the following: i) detectability: it should be capable of detecting potential attacks and responding solely when such threats are likely to occur; ii) identifiability: if an attack is detected, the strategy should further identify the malicious client models and mitigate (or eliminate) their adversarial impacts without harming the benign ones; and iii) verifiability: the defensive mechanism should be integrated with a verification mechanism to ensure the correct execution of the defense mechanism, such that clients can trust the FL system without relying solely on the server’s goodwill.

This paper proposes a two-stage defense for anomaly detection that filters out malicious client models in each FL training round with challenges in real-world FL systems addressed. On the first stage, our approach detects potential existence of malicious clients in the current FL round based on cross-round detection. The potential presence of malicious clients activates the second stage, named cross-client detection that evaluates the degree of evilness of each local model and filters out malicious ones based on the intuition of 3$\sigma$ Rule [48]. Our mechanism integrates a robust verification protocol that utilizes Zero-Knowledge Proof (ZKP) [26] to guarantee integrity and honest execution of the proposed defensive mechanism on the FL server. We overview our mechanism in Figure 1. Then, we compare our approach with state-of-the-art ones, including Krum [7], RFA [47], Foolsgold [24], NormClip [53], Bucketing [35], Coordinate-Wise Median [63], and Trimmed Mean [63] in Table 1. Our contributions are listed below:

i) Real-world applicability. Our method is designed to meet practical requirements of defenses in real-world FL applications. As far as we know, we are the first to close the significant gap between theoretical research and its real-world applicability in FL security.

ii) Utility and practicability. Our method is free from any unrealistic prior information, nevertheless it can still detect and eliminate the impact of malicious client models without harming the benign ones. By this means, our method proves it applicability and effectiveness in real-world FL systems where attacks happen rather rarely.

iii) Conditional activation. We propose a two-stage detection method that first identifies suspicious models and then, if necessary, triggers a double-check of the local models, thereby avoiding unnecessary accuracy loss caused by introducing a defense mechanism.

iv) Accuracy preservation. Our method preserves accuracy in attack-free situations, which is essential due to the infrequent occurrence of attacks in real-world scenarios.

v) Identifiability. Our approach removes malicious local models with high accuracy without harming the benign models or modifying the aggregation function.

vi) Verifiability. To foster trust in FL systems, we leverage ZKPs, enabling clients to independently verify the correct execution of the proposed defense mechanism on the server without relying solely on the server’s goodwill.

We propose a two-stage anomaly detection mechanism to identify and filter out malicious local models on the server. This mechanism is executed at each FL round after the server collects local models from the clients. The server first performs a cross-round check that leverages some cache, which we call reference models, to assess the likelihood of the presence of any malicious clients. Note that at this stage, the server does not remove any local models. If potentially malicious clients are detected, the server subsequently conducts a cross-client detection to analyze each local model and assess its degree of evilness. Based on this evaluation, the server identifies and excludes the malicious models from aggregation.

Our method incorporates a verification module to enhance trust and privacy within the FL system. Ideally, the verification module should have the following features: i) client-side verification: it should enable clients who may have concerns about the integrity of the FL server to verify the correct execution of the defense mechanism, without solely relying on the server’s goodwill; and ii) privacy protection: it should not necessitate the clients to access inappropriate knowledge, such as local models from other clients, thus preserving privacy and integrity in the FL system.

We incorporate ZKPs to ensure that the clients can verify the integrity of the defense without accessing the other local models. We utilize zkSNARKs [6] that offer constant proof sizes and constant verification time regardless of the size of computation. Such property is crucial for applications where the verifier’s (i.e., an FL client) resources are limited, e.g., real-world FL systems. We design ZKP circuits as in Figure 3. Details of implementations are in Appendix §A.4.

ZKP for Anomaly Detection. Most of the computations in Algorithm 1 and Algorithm 2 are linear and can be compiled into an arithmetic circuit easily, e.g., computing cosine similarity between two matrices of size $n\times n$ requires a circuit with $O(n^{2})$ multiplication gates and one division. While it is difficult to directly compute division on a circuit, it can be easily verified with the prover providing the pre-computed quotient and remainder beforehand. Similar to [60], we can utilize Freivalds’ algorithm [22] to verify matrix multiplications. In general, the matrix multiplication constitutes the basis of the verification schemes used for the proposed mechanism. Naively verifying a matrix multiplication $AB=C$ where $A,B,C$ are of size $n\times n$ requires proving the computation step by step, which requires $O(n^{3})$ multiplication gates. With Freivalds’ algorithm, the prover first computes the result off-circuit and commits to it. Then, the verifier generates a random vector $v$ of length $n$, and checks $A(Bv)\stackrel{{\scriptstyle?}}{{=}}Cv$. This approach reduces the size of the circuit to $O(n^{2})$. We exploit this idea to design an efficient protocol for the square root computation in Algorithm 2. To verify that $x=\sqrt{y}$ is computed correctly, we ask the prover to provide the answer $x$ as witness and then we check in the ZKP that $x$ is indeed the square root of $y$. Note that we cannot check $x^{2}$ is equal to $y$ because the zkSNARK works over a prime field and the square root of an input number might not exist. So, we check if $x^{2}$ is close to $y$ by checking that $x^{2}\leq y$ and $(x+1)^{2}\geq y$. This approach reduces the computation of square root to 2 multiplications and 2 comparisons.

The zero-knowledge property of ZKPs allows public verification of prover’s (i.e., the FL server) integrity in case of the server being untrustworthy. By incorporating ZKPs, we provide a public verifiable approach for each client to ensure FL server’s integrity which is essential for building and maintaining trust in FL systems. This ensures that clients can verify the correctness of the defense without needing to rely solely on the server’s goodwill. This is also secure in case there exists adversarial clients, as the ZKP itself reveals nothing about the prover’s witness, i.e., private data, models, and/or thresholds the server uses during the proposed anomaly detection approach.

Setting. A summary of datasets and models for evaluations can be found in Table 2. By default, we employ CNN and the non-i.i.d. FEMNIST dataset ($\alpha=0.5$), as the non-i.i.d. setting closely captures real-world scenarios. We utilize FedAVG in our experiments. By default, we use 10 clients for FL training, corresponding to real-world FL applications where the number of clients is typically less than 10, especially in ToB scenarios. We also vary the number of clients from 10 to 100 in Exp 5, and validate the utility of our approach in a practical application using 20 edge real-world devices; see Exp 11. We conduct our evaluations on a server with 8 NVIDIA A100-SXM4-80GB GPUs, and validate the correct execution with ZKP on Amazon AWS with an m5a.4xlarge instance with 16 CPU cores and 32 GB memory. We implement the ZKP system in Circom [17].¹¹1We provide a link to our code in Appendix §LABEL:code.

Selection of attacks and defenses. We employ two byzantine attacks and two backdoor attacks that are widely considered in literature, including a random weight Byzantine attack that randomly modifies the local submissions [15, 20], a zero weight Byzantine attack that sets all model weights to zero [15, 20], the label flipping backdoor attack that flip labels in the local data [55], and a model replacement backdoor attack  [3] that intends to use a poisoned local model to replace the global model. We utilize 5 baseline defense mechanisms that can be effective in real systems: $m$-Krum [7], Foolsgold [24], RFA [47], Bucketing [35], and Trimmed Mean [63]. For $m$-Krum, we set $m$ to 5, which means 5 out of 10 submitted local models participate in aggregation in each FL training round.

Evaluation Metrics. We evaluate the effectiveness of cross-round check using cross-round detection success rate, defined by the proportion of rounds where the algorithm correctly detects cases with or without an attack relative to the number of total FL rounds. A 100% cross-round success rate indicates that all FL rounds that potential attacks might have happened are detected, and none of the benign cases are identified as “attacks” by mistake. We evaluate the quality of cross-client detection using modified Positive Predictive Values (PPV) [21], the proportions of positive results in statistics and diagnostic tests that are true positive results. Let us denote the number of true positive and the false positive results as $N_{\mathit{TP}}$ and $N_{\mathit{FP}}$, respectively. Then we have $\mathit{PPV}=\frac{N_{\mathit{TP}}}{N_{\mathit{TP}}+N_{\mathit{FP}}}$. In our setting, client submissions that are detected as “malicious” and are actually malicious are defined as True Positive, i.e., $N_{\mathit{TP}}$, while client submissions that are detected as “malicious” even though they are benign are defined as False Positive, i.e., $N_{\mathit{FP}}$. Since we would like the PPV to reveal the relation between $N_{\mathit{TP}}$ and the total number of malicious local models across all FL rounds, we use the total number of malicious local models across all FL rounds, denoted as $N_{\mathit{total}}$, and compute a modified PPV as $\frac{N_{\mathit{TP}}}{N_{\mathit{TP}}+N_{\mathit{FP}}+N_{\mathit{total}}}$, where $0\leq\mathit{PPV}\leq\frac{1}{2}$. Ideally, PPV is $\frac{1}{2}$, where all malicious local models are detected, i.e., $N_{\mathit{FP}}=0$ and $N_{\mathit{TP}}=N_{\mathit{total}}$. The details are in Appendix A.2.

Exp 1: Selection of importance layer. We utilize the L2-norm of the local models to evaluate the “sensitivity” of each layer. A layer with a norm higher than most of the other layers indicates higher sensitivity compared to others, thus can be utilized to represent the whole model. We evaluate the sensitivity of the layers of CNN, RNN, and ResNet-56. The results for RNN, CNN, and ResNet-56 are deferred to Figure 9(a), Figure 9(b), and Figure 9(c) in Appendix §A.5, respectively. The results show the sensitivity of the second-to-the-last layer is higher than most of the other layers. Thus, this layer includes adequate information of the whole model and can be selected as the importance layer.

Exp 2: Impact of the similarity threshold. We evaluate the impact of the similarity threshold $\gamma$ in the cross-round check with 10 clients in each FL round, where 4 of them are malicious. Ideally, the cross-round check should confirm the absence or presence of an attack accurately. We evaluate the impact of the cosine similarity threshold $\gamma$ in the cross-round check by setting $\gamma$ to 0.5, 0.6, 0.7, 0.8, and 0.9. As described in Figure 4(a), the cross-round detection success rate is close to 100% in the case of Byzantine attacks. We observe that, when the cosine similarity threshold $\gamma$ is set to 0.5, the performance is satisfactory in all cases, with at least 93% cross-round detection success rate.

Exp 3: Selection of the number of deviations ($\lambda$). We set $\lambda$ to 0.5, 1, 1.5, 2, 2.5, and 3, and utilize $\mathit{PPV}$ to evaluate the impact of the number of deviations, i.e., the parameter $\lambda$ in the anomaly bound $\mu+\lambda\sigma$. To evaluate a challenging case where a large portion of the clients are malicious, we set 40% clients malicious in each FL round. Given that the number of FL rounds is 100, the total number of malicious submissions $N_{\mathit{total}}$ is 400. We evaluate our approach on three tasks, as follows: i) CNN+FEMNIST, ii) ResNet-56+Cifar100, and iii) RNN + Shakespeare. We observe in Figure 4(b), that when $\lambda$ is 0.5, the results are the best. Especially for the random weight Byzantine attack, we see that the $\mathit{PPV}$ is exactly 0.5, indicating that all malicious local models are detected. In subsequent experiments, unless specified otherwise, we set $\lambda$ to 0.5.

Exp 4: Varying the percentage of malicious clients. We use random Byzantine attack and set the percentage of malicious clients to 20% and 40%. We also include a baseline case where all clients are benign. As shown in Figure 4(c), the test accuracy remains relatively consistent across different cases, as in each FL training round, our approach filters out the local models that tend to be malicious to minimize the negative impact of malicious client models on aggregation.

Exp 5: Varying the number of FL clients. We explore the impact of the number of clients under the random Byzantine attack. We set the number of clients to 10, 40, 70, and 100, and set the percentage of malicious clients to 40%. The results, as described in Figure 4(d), indicate that in all cases, our approach has high utility and can filter out malicious clients with high accuracy.

Exp 6: Evaluations on Byzantine attacks. We compare our approach with the state-of-the-art defenses using 10 clients, and set one of them as malicious in each FL round. We include a “benign” baseline scenario with no activated attack or defense. The results for the random weight Byzantine attack (Figure 5(a)) and the zero weight Byzantine attack (Figure 5(b)) demonstrate that our approach effectively mitigates the negative impact of the attacks and significantly outperforms the other defenses, by achieving a test accuracy much closer to the benign case.

Exp 7: Evaluations on backdoor attacks. We compare our approach with the state-of-the-art defenses using 10 clients, where one of them is malicious in each FL round. Considering that the label flipping attack is subtle and manipulates local training data and produces malicious local models that are challenging to detect, we set the parameter $\lambda$ to 2 to produce a tighter boundary. The results for the label flipping attack and model replacement backdoor attack are shown in Figure 5(c) and Figure 5(d), respectively. Results show that our approach is effective against backdoor attacks, with the test accuracy much closer to the benign case compared to the baseline defenses.

Exp 8: Evaluations on different attack frequencies. We configure attacks to occur only during specific rounds to evaluate the effectiveness of the proposed two-stage approach. The total number of attack rounds is set to 10, 40, 70, and 100, respectively. We then fix the number of attack rounds to 40 and compare our approach with the state-of-the-art defenses. The results in Figure 7(a) and Figure 7(b) show that our method effectively mitigates the impact of the adversarial attacks, ensuring minimal accuracy loss and robust performance even under different attack rounds.

Exp 9: Evaluations on different tasks. We evaluate the defenses against the random mode of the Byzantine attack with different models and datasets described in Figure 2. The results in Figure 7(c), Figure 7(d), and Figure 9(d) in §A.5 show that our approach outperforms the baseline defenses by effectively filtering out poisoned local models, with a test accuracy close to the benign scenarios. Moreover, some defenses may fail in some tasks, e.g., $m$-Krum fails in RNN in Figure 9(d), as those methods either select a fixed number of local models or re-weight the local models in aggregation, which potentially eliminates some local models that are important to the aggregation, leading to an unchanged test accuracy in later FL rounds.

Exp 10: Evaluations of ZKP verification. We implement a prover’s module which contains JavaScript code to generate witness for the ZKP, as well as to perform fixed-point quantization. Specifically, we only pull out parameters of the importance layer to represent the whole model to reduce complexity. We report the results in Table 3.

Exp 11: Evaluations in a real-world setting. To validate the utility and scalability of our approach in real-world applications, we utilize 20 real-world edge devices to demonstrate how our anomaly detection mechanism performs under practical constraints and settings. The device information is shown in Figure 10 in Appendix §A.6. In each FL round, we designate 5 devices as malicious. The FL client package is integrated into the edge nodes to fetch data from our back-end periodically. Due to the challenges posed by real-world settings, such as devices equipped solely with CPUs (lacking GPUs), potential connectivity issues, network latency, and limited storage on edge devices, we select a simple task, i.e., using the MNIST dataset for a logistic regression task, to run FL training for 10 rounds, and use our proposed anomaly detection method to prevent against the random weight Byzantine attack. The training process is shown in Figure 11 in Appendix §A.6 , and the total training time is 221 seconds. The CPU utilization and network traffic during training are shown in Figure 13 and Figure 13 in Appendix §A.6, respectively. We also included a benign case and an attack-only case for comparison. The results are shown in Figure 14. Results show that despite the presence of malicious clients and the limitations of edge devices, our approach successfully identifies and mitigates the impact of malicious local models.

Detection of attacks. [65] employs $k$-means to partition local models into clusters that correspond to “benign” or “malicious”. While this approach can efficiently detect attacks, it requires some pre-training rounds and relies much on historical client models, thus might not be as effective when there is limited information on past client models. For example, their implementation [64] sets the starting round to detect attacks to different training rounds, e.g., 50 when the datasets are MNIST and FEMNIST, and 20 when the dataset is CIFAR10. While this approach is novel, it is not suitable for real FL systems, as attacks may happen in earlier rounds as well.

Defense mechanisms in FL. Robust learning and the mitigation of adversarial behaviors in FL has been extensively explored [7, 62, 24, 47, 33, 35, 53, 23, 46, 52, 63, 15, 28, 61, 40, 11]. Some approaches keep several local models that are more likely to be benign in each FL round, e.g., [7, 28, 63], and [61], instead of aggregating all client submissions. Such approaches are effective, but they keep fewer local models than the real number of benign local models to ensure that all malicious local models are filtered out, causing misrepresentation of some benign local models in the aggregation. This completely wastes the computation resources of the benign clients that are incorrectly removed and thus, changes the aggregation results. Some approaches re-weight or modify local models to mitigate the impacts of potential malicious submissions [24, 35, 53, 23, 46, 52], while other approaches alter the aggregation function or directly modify the aggregation results [47, 35, 63, 15]. While these defense mechanisms can be effective against attacks, they might inadvertently degrade the quality of outcomes due to the unintentional alteration of aggregation results even when no attacks are present. This is especially problematic given the low frequency of attacks in practical scenarios.

We present a novel anomaly detection approach specifically designed for real-world FL systems. Our approach utilizes an early cross-round check that activates subsequent anomaly detection exclusively in the presence of attacks. When attacks happen, our approach removes malicious client models efficiently, ensuring that the local models submitted by benign clients remain unaffected. By leveraging ZKPs, our approach enables clients to verify the integrity of the removal performed by the server.