A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guarantee

The key insight of the scoring system is to distinguish between the honest and malicious gradients by calculating their Euclidean distance. The Euclidean distance of two honest gradients is lower than that of an honest gradient and a malicious gradient. Based on this insight, the scoring system is designed as comparing the Euclidean distance of two gradients, where the clients who upload the honest gradients can obtain higher scores while the clients who upload the malicious gradients will obtain the lower score. Assume that the local gradient on the $k$-th training client at round $t$ is denoted as $\hat{g}_{k}^{t}=g_{k}(\mathbf{w}_{k,\tau}^{t},\mathcal{B}_{k,\tau}^{t})$, and the local gradient on the the $c$-th committee client at round $t$ is expressed as $\hat{g}_{c}^{t}=g_{c}(\mathbf{w}_{c,\tau}^{t},\mathcal{B}_{k,\tau}^{t})$. The score $\mathcal{P}_{k}^{c}$ of the $k$-th training client assigned by the $c$-th committee client is computed as follows:

Since $\hat{g}_{k}^{t}$ and $\hat{g}_{c}^{t}$ are local gradients generated from different clients, we assume that $\hat{g}_{k}^{t}\neq\hat{g}_{c}^{t}$ for any $k\in S_{b}^{t},c\in S_{c}^{t}$. We define

as the final score of the $k$-th training client. The scoring principle is mainly based on the Euclidean distance between the local gradients $\hat{g}_{k}^{t}$ and $\hat{g}_{c}^{t}$. Another insight remains that Byzantine attackers usually replace some local gradients with malicious gradients, which directly increases the Euclidean distance between these gradients and the honest gradients. As a result, when the proportion of malicious clients is within a tolerable range, the score of the malicious training clients is expected to be lower than the honest training clients. However, in a scenario without Byzantine attacks, the score represents the degree of heterogeneity of clients. A higher score means a higher degree of heterogeneity.

We design the selection strategy for determining which uploaded gradient is used to update the global model. Based on our scoring system, we can achieve a secure aggregation by accepting the gradients with high scores, since the malicious gradients obtain low scores. Besides, the convergence analysis and experimental result show that the opposite strategy performs better in a non-attack scenario. Therefore, two opposite selection strategies are designed for different considerations as follows:

• •

  Selection Strategy I. The selection strategy I selects several local gradients with relatively higher scores to construct a global gradient for the update of the global model. In other words, we hope that the local gradients which are similar to the committee gradients in the Euclidean space will participate in the construction of the global gradient. In practice, we sort the local gradients according to their scores and only accept the top $\alpha\%$ of them. We design the selection strategy for the consideration of robustness. Those malicious gradients and honest gradients are far from each other in the Euclidean space, and choosing the gradients close to the committee gradients under the condition that the committee clients are honest can achieve a more robust aggregation process. However, it is difficult for those clients with obvious differences to be selected to participate in the aggregation procedure, which makes it hard for the global model to learn the comprehensive data characteristic, resulting in a decline in performance. Overall, selection strategy I is suitable for FL scenarios with Byzantine attacks.

• •

  Selection Strategy II. The selection strategy II selects several local gradients with relatively lower scores to construct a global gradient for the update of the global model. That is, we hope that the local gradients which are different from the committee gradients in the Euclidean space will participate in the construction of the global gradient. Similar to the selection strategy I, we sort the local gradients according to their scores and only accept the bottom $\alpha\%$ of them in practice. We design the selection strategy II for the consideration of convergence rate and performance of the global model. Constructing a global gradient by selecting those local gradients with obvious differences allows the global model to learn more comprehensively, gaining a faster convergence and better performance of the global model. However, this strategy can not provide robustness guarantees, mainly since that the malicious gradients will be preferentially used in global learning, leading to a sharp drop in the performance of the global model. Overall, selection strategy II is suitable for FL scenarios without Byzantine attacks.

The election strategy is designed for guaranteeing the honesty of committee clients. The committee members reach a consensus on their decisions, and the committee’s decision depends on the majority of the committee members. However, the malicious clients mixed into the committee may interfere with the committee’s decision-making. Therefore, the committee must guarantee that its honest members are more than malicious members. Otherwise, the committee can not filter out the malicious gradients. We sort the training clients according to their scores and then select these training clients closed to the middle position as the committee clients for the next round. We design the election mechanism based on the following two considerations:

• •

  Robustness. According to the above analysis, as malicious training clients will get lower scores than honest training clients, they are expected to locate at the end of the sorted sequence. Therefore, choosing the training client closed to the middle or upper position prevents the malicious training clients from becoming the committee clients in the next round, thereby guaranteeing the security of the global model. Although there may still be a small number of malicious clients mixed into the committee members when the proportion of malicious clients is relatively large, it is difficult to affect the judgment of the whole committee because of their exceeding small proportion of the committee members.

• •

  System Stability. Those training clients with the highest scores are not chosen to be the new committee members in order to avoid the system from relying too much on the initialization of committee members. When we choose those training clients with the highest scores to form a new committee, the learning direction of the global model will be completely determined by the initial committee members. It is because those clients with a large Euclidean distance between the local gradient and the committee gradients are not only difficult to be selected to participate in the aggregation procedure, but also lost the opportunity to run for the next round of committee members. This is in line with our intuition that “committee members should be those who can represent the majority”.

In our decentralized framework, the committee clients must reach a consensus to complete the scoring, aggregation, selection, and election process. Thus we design a committee consensus protocol, which is inspired by the pBFT[39]. The designed committee consensus protocol(CCP) is as follows(at round $t$):

1. 1.

   After the scoring process, each committee client obtains the scores of the training clients. Then every committee client broadcast its scores to the other committee clients. Each committee client is able to calculate the total score of every training client by Eq. (8).

2. 2.

   A committee client $p$ is selected as the primary committee client by random, while other committee clients are regarded as the replicate committee clients.

3. 3.

   Each committee client decides its aggregation set $S_{a,c}^{t}$ according to the selection strategy. Since all scores are broadcasted among committee clients, the aggregation sets among honest committee clients are the same.

4. 4.

   The primary committee client creates a request $\langle$ Request, $S_{a,p}^{t}$, operation, timestamp $\rangle$ to ask whether its $S_{a,p}^{t}$ is correct. Then the primary committee client broadcasts the request to all the replicate committee clients. The operation in the request is to aggregate the local gradients uploaded by aggregation clients as Eq.(5)

5. 5.

   All the replicate committee clients execute the request. Each of them checks whether the $S_{a,p}^{t}$ is the same as its own $S_{a,c}^{t}$. If so, it performs the aggregation process as Eq. (5). After aggregation, it checks whether the result is consistent with the request. If so, the executing result $\langle$ Reply, timestamp, $S_{a,p}^{t}$, response $\rangle$ is returned to the primary committee client.

6. 6.

   The primary committee client checks whether it has received at least $\lfloor C/2\rfloor+1$ identical results from replicate committee clients. If so, the consensus is reached; otherwise, the primary committee client should be reassigned and steps 3)-6) should be repeated.

7. 7.

   Similarly, steps 3)-6) are repeated to reach consensus on new committee client set $S_{c}^{t+1}$(However, the aggregation client set $S_{a}^{t}$ in step 3)-6) should be replaced by new committee client set $S_{c}^{t+1}$).

8. 8.

   The primary committee client broadcasts the global model to all other clients. The next training round is ready to start.

First, we introduce the assumptions used for convergence analysis.

Assumption 1 and 2 are standard conditions in FL setting[40][41][42][43] and many common machine learning optimization algorithms meet these assumptions, such as the $\ell_{2}$-norm regularized linear regression, logistic regression, and softmax classifier [28]. Assumption 3 is a form of bounded variance between the local objective functions and the global objective function[44], and Assumption 4 is fairly standard in nonconvex optimization literature[45][46][47][48]. They are widely used in the FL convergence analysis, such as Li et al. [28] and Cho et al. [49]. Assumption 5 is used to constrain the optimal objective function deviation between the aggregation client set and the committee client set caused by the committee mechanism. For the need of convergence proof, we define $S_{c}^{*}$ as the set which contains $C$ clients with the smallest optimal local objective function $F_{k}^{*}$.

We introduce two related definitions for the convenience of analysis as follows.

Li et al. [28] proposed this definition, which is widely used in the convergence analysis of FL on Non-IID dataset[50]. In the Non-IID FL scenarios, the $\Gamma$ remains nonzero and its value reflects the heterogeneity of the data distribution. In the IID FL scenarios, with the growth of $K$ the $\Gamma$ gradually goes to zero.

$\varphi(S_{a}^{t},\mathbf{w})$ changes with the changes of $S_{a}^{t}$ and $\mathbf{w}$ during training. An upper bound $\varphi_{max}$ and a lower bound $\varphi_{min}$ are defined to obtain a conservative error bound in the convergence analysis:

We analyze the convergence of Algorithm 3 in this section and find an upper bound of $\mathbb{E}[F(\overline{\mathbf{w}}^{(T)})]-F^{*}$, which denotes the convergence error of the global model after $T$ rounds:

where $L$, $\mu$ and $\sigma$ represent the constant light indicated in the Assumption 1, 2 and 3. $G$ and $\kappa$ represent the upper bound values defined in Assumption 4 and 5 .$\Gamma$ denotes the heterogeneity among the clients according to the Definition 1. These are all constant while $\varphi_{min}$ and $\varphi_{max}$ will be different depending on the selection strategy. The impact of selection strategy on $\varphi_{min}$ is analyzed below. CMFL affects the performance of the global model by altering $\varphi_{min}$. The proof of the theorem is shown in Supplement.

The impact of selection strategy on $\varphi_{min}$. According to the definition, the value of $\varphi$ is positively correlated with $\sum_{k\in S_{a}^{t}}p_{k,S_{a}^{t}}F_{k}(\mathbf{w})$, which represents the average local loss of the model over the aggregation client set $S_{a}^{t}$. Under our framework, the lower bound of $\varphi(S_{a}^{t},\overline{\mathbf{w}}^{t})$ defined as $\varphi_{min}$ affects the convergence rate of the global model, where $\overline{\mathbf{w}}^{t}$ represents the global model at round $t$. In general, those clients whose data set distribution is similar to that of the majority have a relatively low local loss, while others have a relatively high local loss. We call the former universal clients and the latter extreme clients. The tendency to choose a universal client for aggregation results in low $\varphi_{min}$, and the tendency to choose an extreme client for aggregation results in high $\varphi_{min}$. The building of the election strategy ensures that committee members can represent the majority, and the scoring system is designed as the clients similar to committee members can get higher scores, so clients with high scores are more likely to have a low local loss. When we adopt selection strategy I, we get a low $\varphi_{min}$. Instead, when we adopt selection strategy II, we get a high $\varphi_{min}$.

Effect of $\varphi_{min}$ on convergence rate. Note that a higher $\varphi_{min}$ results in faster convergence at the rate $\mathcal{O}(\frac{1}{T\varphi_{min}})$. That is, adopting the selection strategy I make the convergence of the global model slows down, while the selection strategy II accelerates the convergence of the global model, which is verified in the following experiments. However, considering the reality of Byzantine attacks, the first selection strategy is a more appropriate choice.

There are five phases related to computation complexity as follows:

• •

  Local Training. Each training client and committee client performs SGD locally before they upload their local gradients. The computation overhead of $k$-th client is $\mathcal{O}(|\mathcal{D}_{k}|\cdot|\mathbf{w}|)$, where $|\mathcal{D}_{k}|$ is the number of data samples, $|\mathbf{w}|$ is the model size. Assumed that $|\mathcal{D}|$ represents the average value of $|\mathcal{D}_{k}|$ over all clients, the total computation overhead is $\mathcal{O}((n+C)|\mathcal{D}|\cdot|\mathbf{w}|)$. Because each client performs local training in parellel, the computation time of local training is $\mathcal{O}(|\mathcal{D}|\cdot|\mathbf{w}|)$.

• •

  Scoring. In the phase of scoring, each committee client assigns a score to each training client. According to the scoring system, the computation overhead of scoring for one committee client is $\mathcal{O}(n\cdot|\mathbf{w}|^{2})$. The total computation overhead of scoring is $\mathcal{O}(n\cdot C\cdot|\mathbf{w}|^{2})$. As each committee client performs scoring operation in parallel, the computation time of scoring is $\mathcal{O}(n\cdot|\mathbf{w}|^{2})$.

• •

  Aggregation. In the phase of aggregation, only $m$ local gradients are used for aggregation, so the computation overhead of aggregation for one committee client is $\mathcal{O}(m\cdot|\mathbf{w}|)$. According to the CCP, each committee client performs aggregation so the total computation overhead of aggregation is $\mathcal{O}(m\cdot C\cdot|\mathbf{w}|)$. Since the aggregation process is performed in parallel, the computation time of aggregation is $\mathcal{O}(m\cdot|\mathbf{w}|)$.

• •

  Selection. In the phase of selection, each committee client sorts the received local gradients and selects $m$ local gradients for aggregation. The computation overhead is $\mathcal{O}(nlogn)$, which can be ignored since it is much smaller than the computation overhead of the above three phases.

• •

  Election. In the phase of the election, each committee client determines its new committee client set based on the sorted local gradients. The computation overhead is $\mathcal{O}(1)$, which can be ignored because it is much smaller than the computation overhead of the above four phases.

Assumed that each client owns the same maximum bandwidth, and $r$ represents the max transmission rate, the communication time of transferring data $s$ is computed as $T_{transmission}=s/r$. There are three phases related to communication complexity as follows:

• •

  Gradient Uploading. After local training each training client uploads its local gradient to all the committee clients, so the communication overhead of uploading for one client is $\mathcal{O}(C\cdot|\mathbf{w}|)$. The total communication overhead of uploading is $\mathcal{O}(n\cdot C\cdot|\mathbf{w}|)$. Since each training client performs the uploading process in parellel, the communication time of uploading is $\mathcal{O}(C\cdot|\mathbf{w}|/r)$.

• •

  Global Model Downloading. After global aggregation, the primary committee client broadcasts the global model to all training clients in the next round. The total communication overhead of downloading is $\mathcal{O}(n\cdot|\mathbf{w}|)$, thus the communication time of downloading is $\mathcal{O}(n\cdot|\mathbf{w}|/r)$.

• •

  Broadcasting. Committee clients should perform CCP to reach consensus, in this phase the primary committee client broadcast $S_{a}^{t}$ and $S_{c}^{t+1}$ to other committee clients, which occur communication overhead. However, this communication overhead can be ignored because it is much smaller than the communication overhead of uploading and downloading.

Besides, system heterogeneity is a widespread problem in the field of Federated Learning. The differences in the computational performance of clients lead to various time consumptions, which becomes the bottleneck limiting the efficiency of the Federated Learning system. Some clients even drop out during the training process. There are some methods to alleviate the performance degradation caused by system heterogeneity, such as setting a maximum waiting time or a minimum number of received gradients. Indeed, system heterogeneity is an important issue in Federated Learning, and more works need to be carried out to address it.

We evaluate CMFL over three datasets with a Non-IID setting, including FEMNIST, Sentiment140, and Shakespeare.

• •

  FEMNIST-AlexNet.[51] FEMNIST (Federated Extended MNIST) is a real-world distributed dataset formed by a specific division of the EMNIST dataset. This dataset is used to train a model for handwritten digit/character recognition tasks, which contains 805263 images of $28\times 28$ pixels, divided into 64 categories, each category represents a type of handwritten digit/character ($0-9$ and $a-z$). The FEMNIST dataset divides the EMNIST dataset into 3550 parts in a specific way and stores them on each client to simulate a real federated learning scenario. We use the convolutional neural network AlexNet as the basic experimental model for this image classification dataset.

• •

  Sentiment140-LSTM. [52] Sentiment140 is a real-world distributed dataset which focus on the text sentiment analysis task, including 1,600,000 tweets extracted using the Twitter API. The data in this dataset has been annotated (0 = negative, 2 = neutral, 4 = positive). It can be used to discover the sentiment of a brand, product, or topic on Twitter. We regard each Twitter account as a client and choose a two-layer LSTM binary classifier as the basic experimental model. The LSTM binary classifier includes 256 hidden units with pretrained 300D GloVe embedding[53].

• •

  Shakespeare-LSTM. [54] Shakespeare is a real-world distributed dataset built from The Complete Works of William Shakespeare. Each speaking role corresponds to a different device, and we choose a two-layer LSTM classifier as the basic experimental model, which contains 100 hidden units with an 8D embedding layer. There are 80 classes of characters in dataset. The model takes a sequence of 80 characters as input, embeds each character into a learned 8-dimensional space, and outputs one character for each training sample after 2 LSTM layers and a densely connected layer.

The Non-IID levels of these two datasets are shown in Table II.

Global Setup: At the beginning of each communication round, $10\%$ of the idle clients are activated to participate in the training, while the rest clients wait for the next selection. At one communication round, each client runs one iteration of SGD for local training. Besides, the initial committee clients are selected from all clients at random. In order to better demonstrate the experimental effect, we set different learning rates $\eta$ in different datasets.

Machines: We perform our experiment on a commodity machine with Intel Core CPU i9-9900X containing a clock rate of 3.50 GHz with 10 cores and 2 threads per core. And we utilize Geforce RTX 2080Ti GPUs to accelerate training. The learning model is implemented in Python 3.7.6 and Tensorflow 1.14.

Hyper-parameter Notation: In each communication round, $10\%$ of the whole clients are activated to participate in the training of that round, $\omega\%$ of which become the committee clients and the rest become the training clients. Each round $\alpha\%$ of the training clients become the aggregation clients, whose local gradients are accepted to use for the constructing of the global gradient. $\epsilon$ presents the percentage of malicious clients.

In this experiment we set $\alpha=40$, $\omega=40$ and $\epsilon=0$ to simulate a non-attack FL scene. The $\eta$ in FEMNIST and Sentiment140 is $0.001$ and $0.005$ respectively. In this experiment, we compared typical FL and CMFL under these two selection strategies:

• •

  Typical FL. Typical FL uses all the local gradients uploaded by the training client for the construction of global gradients.

• •

  CMFL with Selection Strategy I. The original intention of this selection strategy is to resist Byzantine attacks. The committee accepts the local gradients uploaded by the high-score training clients while rejects those uploaded by low-score training clients.

• •

  CMFL with Selection Strategy II. This selection strategy is opposite to selection strategy I, accepting the local gradients uploaded by the low-score clients while rejects those uploaded by high-score training clients.

We show the result in Figure 2. Note that the performance of CMFL under selection strategy II is better than that of CMFL under selection strategy I and typical FL. The CMFL under the selection strategy I has the slowest model convergence rate and the worst model accuracy among the three. The result shows that it is a more appropriate design for the committee to accept the local gradients uploaded by the low-score clients when in non-attack FL scene, which can not only enhance the global model performance but also accelerate the convergence of the global model. This is because over the Non-IID dataset, the CMFL under the selection strategy I makes it difficult for a few clients which are quite different from other clients to participate in the aggregation process, resulting in the training of the global model only using part of the data instead of all of the data. Figure 3 proves it. We record the aggregation times of each client and plot them as a curve graph. From the curve, we can see that the aggregation times of client under FL conform to a Gaussian distribution, causing FL selects the aggregation randomly. Besides, we found that a high percentage of clients never participate in the aggregation process when performing CMFL under selection strategy I. That is why CMFL under selection strategy I achieves worse performance than typical FL and CMFL under selection strategy II. It is difficult for the global model to learn comprehensive knowledge while using this kind of training algorithm. However, the CMFL under the selection strategy II significantly reduces the number of such clients. In this way more clients can participate in the aggregation process, making the global model learn a more comprehensive knowledge. Figure 3 and 3 explain why CMFL under selection strategy II achieves a better performance than typical FL. We record the testing accuracy of each client after training. Figure 3 shows that CMFL under selection strategy II has least clients with low accuracy, while has most clients with high accuracy. That is because those clients with low accuracy have more opportunities to participate in the aggregation process, which is proved by Figure 3. Figure 3 shows that the aggregation times of clients with low accuracy in CMFL under selection strategy II are much more than other training methods. By reducing the proportion of clients with low accuracy, CMFL under selection strategy II achieves a better performance than typical FL.

Nevertheless, it is not advisable to choose strategy II when considering the Byzantine attack. According to the scoring system we designed, the malicious client will get a lower score than the honest client. Taking the selection strategy II means that Byzantine attackers can easily attack the global model by uploading malicious local gradients. In this scenario, choosing strategy I is a more appropriate choice by avoiding the local gradients with low scores to participate in the aggregation, which is likely to be malicious gradients. Although the effect of CMFL under strategy I in the non-attack FL scenario is slightly weaker than that of the typical FL, it can make the training process more robust in a wider range of realistic scenarios.

In this experiment we set $\alpha=40$, $\omega=40$ and $\epsilon=10$, and we test the Byzantine resilience of CMFL under two selection strategies by comparing it with the following Byzantine-tolerant algorithms, which are all aim to design a robust gradient aggregation method.

• •

  Median[12]: This aggregation method first sorts the gradients and selects the median to be the global gradient.

• •

  Trimmed Mean[12]: This aggregation method first sorts the local gradients, then removes the maximum value of $\beta\%$ and the minimum value of $\beta\%$, and finally aggregates the remaining local gradients to construct the global gradient, where $\beta\in[0,50)$.

• •

  Krum[4]: This aggregation method assumes that the directions of the local gradients uploaded are relatively similar and sort these local gradients according to their similarity. The original Krum algorithm only selects the first local gradient after sorting as the global gradient.

• •

  Multi-Krum[4]: The Multi-Krum algorithm is an improved version of the original Krum, which selects the top $\alpha\%$ of the sorted local gradients to construct the global gradient. Compared with the original Krum, Multi-Krum reduces the fluctuation of the global model effect caused by random factors, making the training process more stable. Since Multi-Krum uses more local gradients to construct the global gradients, it can make the global model converge faster than the original Krum.

We compare these Byzantine-tolerant algorithms with CMFL considering three different types of attacks: gradient scaling attack, same-value gaussian attack, and back-gradient attack, where the attackers are all aim to compromise some clients and upload the malicious gradients.

• •

  Gradient Scaling Attack: The malicious clients multiply each element in the local gradient by a random value $\lambda\in[a,1)$, where $a$ is a defined constant which indicates the magnitude of the attack. In this experiment we set $a=0.5$.

• •

  Same-value Attack[55]: The malicious clients replace the local gradient with a vector of the same size whose elements are all 0.

• •

  Back-gradient Attack[56]: The malicious clients replace the local gradient with a vector of the same size in the opposite direction.

We show the performance of CMFL compared with other Byzantine-tolerant algorithms under various attacks in Figure 4-6. We analyzed the performance of each Byzantine-tolerant algorithm.

• •

  CMFL-selection strategy I always reaches the fastest global convergence speed and highest accuracy among these five algorithms. Note that in the model accuracy curve, the CMFL curve fluctuates more obviously than Median and Multi-Krum, which is the normal curve fluctuation caused by the replacement of committee members. Nevertheless, its overall accuracy rate is still higher than the other algorithms.

• •

  CMFL-selection strategy II achieves as poor performance as typical FL. Obviously, the malicious clients can easily participate in the aggregation process.

• •

  Median[12] has maintained a relatively stable performance under a variety of attacks. It only selects the median of the local gradients as the global gradient, which is regarded as a relatively conservative approach. Although Median can effectively resist Byzantine attacks, it is difficult to achieve excellent model performance.

• •

  Trimmed Mean[12] has an unstable performance under different attacks. Trimmed Mean performs well under gradient scaling attack but still not as good as Median and Multi-Krum, and performed extremely badly under same-value attack and back-gradient attack. This Byzantine-tolerant algorithm cannot effectively resist the Byzantine attacks.

• •

  Krum[4] performs poorly under all three attacks, and there were sharp fluctuations in the training process. Krum only selects the first local gradient after sorting as the global gradient each time. This aggregation method that does not consider the overall results in severe fluctuations of the global model.

• •

  Multi-Krum[4] has the second-best model performance overall. Multi-Krum improves the original Krum and eliminates the jitter generated during the training process.

Note that the CMFL is the only framework without involving a central server, which naturally achieves robustness against the influence of a malicious server. Other Byzantine-tolerant algorithms cannot achieve the same robustness owing to their naturally centralized architecture.

In this experiment, we consider the effect of hyper-parameter by varying the hyper-parameter $\alpha$, $\omega$ and $\epsilon$. Specifically, we consider the back-gradient attack and designed three sets of sub-experiments. In each set of sub-experiments, we fixed one of the hyper-parameters and changed the other two hyper-parameters to analyze their impacts on the model performance.

• •

  Sub-experiment I. Fixed $\alpha=40$ and vary $\omega,\epsilon$ to $\{10,20,30,40,50\}$.

• •

  Sub-experiment II. Fixed $\omega=40$ and vary $\alpha,\epsilon$ to $\{10,20,30,40,50\}$.

• •

  Sub-experiment III. Fixed $\epsilon=10$ and vary $\alpha,\omega$ to $\{10,20,30,40,50\}$.

We show the results in Figure 7 and the analysis as follows:

• •

  Appropriately increasing the number of committee members can enhance the performance of the global model. The results show that within a suitable parameter value range (e.g., $\alpha=10,\epsilon\leq 30,\omega\leq 30$), more committee members lead to better global model performance. This is mainly since an appropriate increase in the number of committee members can enhance the robustness of the committee to a certain extent, in the meanwhile avoid the existence of ”dictators” and prevent the training process from being controlled by a small number of clients, which makes it difficult for some other clients to participate in the aggregation process.

• •

  Appropriately increasing the proportion of the aggregation clients can enhance the performance of the global model. Within a suitable parameter value range, a higher proportion of the aggregation clients leads to better global model performance. Because in the normal operation of the committee, sorting the local gradients uploaded by the training client according to their scores makes the honest local gradients come first and the malicious gradients second. In this ideal situation, we control the proportion of aggregated clients to be less than a threshold $\chi$ to prevent malicious gradients from participating in the aggregation process. Under the limit of $\alpha\in(0,\chi]$, the increase of $\alpha$ means that more honest local gradients are selected to construct the global gradient in each round, which can achieve better global model performance.

• •

  Excessive $\alpha$, $\omega$, and $\epsilon$ will lead to a cliff-like decline in the performance of the global model. When we increase the proportion of committee members, it means that both malicious clients and honest clients have a greater probability of being elected as committee members. This increases the probability of malicious clients mixing into committee members and damage the committee’s scoring system, making it difficult to assign the correct score to the training clients. In the worst situation, once the proportion of the malicious clients among committee members is relatively large, the scores of the malicious clients will be higher than those of the honest clients, resulting in the malicious local gradient uploaded by the malicious clients being used to construct the global gradient. Hence, the performance of the global model will be devastatingly damaged. When we increase the proportion of the aggregation clients, it also means increasing the probability of malicious local gradients participating in the aggregation procedure. When the proportion of aggregation clients exceeds the threshold that the system can tolerate, the malicious local gradients are used to construct the global gradient, which causes irreparable damage to the performance of the global model. By the same token, if there are too many malicious clients, the malicious local gradients uploaded by them are more likely to participate in the aggregation process. As long as a malicious local gradient is aggregated, the performance of the global model will be greatly reduced.

Implementation Detail. In this experiment, we simulate the data transferring by calculating the transmission time using $T_{transmission}=s/r$, where $s$ represents the data and $r$ represents the transmission rate. We let the program process wait for $T_{transmission}$ second when it needs to transmit data for simulating the data transferring in a real scene. The maximum number of communication rounds is 600.

Hyper-parameter Setting. In order to exclude the influence of irrelevant variables, we conduct this evaluation without considering the Byzantine attacks. The number of committee clients and training clients is set as 43 and 65. And the aggregation rate $\alpha$ is set as $40\%$. We set different maximum transmission rates for one client: $1Mps$, $10Mps$, and $100Mps$, and under each transmission rate, we evaluate the performance of CMFL using wall-clock time, which includes computation time and communication time.

Comparative Method. We compare the efficiency of CMFL with three algorithms: typical FL, BrainTorrent[57], and GossipFL[58].

Figure 8 shows the result. Noted that the size of AlexNet ($\approx 200$M) is much bigger than that of LSTM ($\approx 16$k), so the communication time over FEMNIST dataset is much longer than that over the Sentiment140 dataset. The overall performance of CMFL is better than the other two decentralized FL frameworks but worse than typical FL. Typical FL has an overall better performance than CMFL because the clients do not have to communicate with each other. It is a centralized framework so the clients just send their local gradients to a server for aggregation. Never can the other three decentralized frameworks do that because the client has to broadcast their local gradients to other clients for reaching consensus, in which case the transmission of the local gradients occurs communication overhead. With the increasing transmission rate, the advantage of typical FL becomes smaller. The performance of the typical FL is taken over by CMFL when the transmission is up to $100Mps$ over the Sentiment140 dataset. CMFL always has a better performance than the other two decentralized frameworks, because it utilized the committee system to reduce the communication overhead, which is proved in Figure 9. The clients do not need to broadcast their local gradients to other clients but send their local gradients to committee clients. The committee clients can reach a consensus with lower communication overhead by performing CCP, while the other two decentralized frameworks must cost higher communication overhead. Considering the malicious server in a real scenario, CMFL can achieve both robustness and efficiency.

In this experiment we set $\alpha=40,\omega=30$ and vary $\epsilon$ to $\{10,20,30,40,50\}$. By recording the number of malicious training clients, malicious committee clients and malicious aggregation clients during the training process, we analyze the influence of committee members on the performance of the global model.

We show the result in Figure 10. Since in each round the training clients are randomly selected from the non-committee clients, when $\epsilon$ increases, the number of malicious training clients also increases. And with the increase of $\epsilon$, some malicious clients will inevitably be mixed into the committee members. Nevertheless, a small number of miscellaneous malicious clients cannot destroy the entire committee’s scoring system and influence the committee’s judgment, which instead improve the performance of the global model. This is mainly because the scoring system does not lose the ability to distinguish between the malicious clients and the honest clients since those malicious clients still receive a score much lower than that of honest clients. But for the honest clients, they get almost the same score due to the extreme scores assigned by the malicious committee clients. In this case, our method is equivalent to the typical FL method without considering the Byzantine attack. The typical FL achieves better performance than CMFL in such a setting, which has shown in Experiment 6.2. However, if there are too many mixed malicious clients, the committee’s scoring system will be destroyed and the committee members will lose the ability to eliminate malicious local gradients, resulting in a sharp drop in the performance of the global model.