ScaleSFL:
A Sharding Solution for Blockchain-Based Federated Learning

Privacy-preserving machine learning combines the use of federated learning with differential privacy (ref:privacy-preserving-deep-learning, ; ref:differential-privacy-sgd, ; ref:cit21, ). Federated learning allows sensitive data from edge devices (e.g., smartphones, tablets, IoT devices) to remain on-device while training is done locally (distributed training on edge devices) (ref:federated-learning, ). In the original FL environment, a centralized server manages the training process. The server broadcasts the current model during each round to a subset of participating nodes to activate local training (requesting a model update). After all model update submission responses are received, the server aggregates the local models into a global model and prepares for the next round (ref:federated-learning, ; ref:committee-consensus, ). While FL offers many privacy benefits, differential privacy in conjunction provides stronger privacy guarantees (ref:federated-learning, ) by mitigating data reconstruct-ability via random model update noise.

McMahan et al. (ref:federated-learning, ) proposed the Federated Averaging algorithm (FedAvg). Their algorithm is a generalization of federated stochastic gradient descent (FedSGD) (ref:differential-privacy-sgd, ) allowing each node to perform multiple batch updates while exchanging the models’ parameters, with which we based our FL model training approach. Experiments demonstrate that FL can be used for training a variety of high-quality machine learning models at reasonable communication cost, exemplifying high practicality (ref:federated-learning, ). Therefore, we choose to apply differential privacy and use an adapted version of FedAvg for our current work.

Liu et al. (ref:hierfavg, ) introduced HierFavg, a hierarchical scaling solution introducing multiple edge servers, each performing partial model aggregation, combining the benefits of both centralized cloud and edge computing. While centralized servers benefit from the vast reach of data collection, the communication overhead and latency of downloading and evaluating such updates limit this approach. Instead, model updates are sent to local edge servers to benefit from fast and efficient model updates, reducing the number of local iterations and total runtime. These local updates are then sent to a cloud server allowing for a sampling of the global data distribution to be considered and outperforming the local edge server performance. Combining these approaches is nontrivial, and the authors performed convergence analysis describing qualitative guidelines in applying their approach. While our approach is primarily motivated by increasing the throughput of model update metadata to blockchain ledgers, the composition of shards can benefit from the strategies found in hierarchical solutions.

Blockchain technology can be utilized to decentralize the aforementioned centralized server in federated learning. The aggregation and facilitation logic previously done by the central server can be replaced by implementing smart contracts (ref:committee-consensus, ).

In addition to Ramanan and Nakayama’s blockchain-based aggregator free federated learning (BAFFLE) (ref:baffle, ) and Li et al.’s blockchain-based federated learning framework with committee consensus (ref:committee-consensus, ) referenced earlier, numerous strategies for integrating blockchain technology with federated learning to achieve decentralization have been considered (SatybaldyN20, ). However, the approaches proposed by previous works do not directly focus on the performance of their chosen underlying consensus mechanism with which their proposed frameworks are built upon.

A notable issue when aggregating model updates from untrusted and potentially adversarial clients are model poisoning attacks. In order to disrupt or bias the training process, and adversary may submit biased, or corrupted model updates. Because model updates must be verified prior to being accepted to the ledger, we consider each defence mechanism applied to each endorsing peer rather than a single centralized server. One possibility to make the training more robust is to apply norm constraints to each model update, such as clipping model updates and applying random noise to the aggregated model to prevent overfitting (ref:fl-survey, ). However, further mechanisms are required to counter adversarial examples and backdoors (ref:fl-backdoor, ). A subset of model poisoning is data poisoning attacks. These occur when the clients’ dataset has been altered, such as a targeted attack by adversarial manipulation of edge devices, or untargeted such as a faulty data collection process, resulting in corrupted samples (e.g., incorrect labels). Here, we focus on readily applicable works related to the general detection of poisoned model updates.

Barreno et al. (ref:roni, ) proposed a technique to determine which samples from a training set are malicious. To do so, they present the reject on negative influence (RONI) defence. This method measures the effect each training sample has on the accuracy of the model. This defence strategy is appropriately modified to suit the needs of a FL setting. Instead, we measure the influence of each model update on the accuracy of the global model. This can be done by utilizing a held-out testing set, potentially unique to each endorsing peer. This defence is not suitable for general FL applications due to the nature of client data being non-IID and incorrectly rejected due to misrepresentation by the test set. Additionally, malicious updates may go unnoticed due depending on the threshold of acceptance.

Blanchard et al. (ref:krum, ) introduced a method designed to counter adversarial updates in a FL setting. Multi-Krum is a byzantine-resilient defence strategy that selects $n$ vectors furthest from the calculated mean, removing them from the aggregated gradient update. These distances are calculated based on the euclidean distance between gradient vectors. This strategy is effective for attacks with up to 33% compromised/adversarial clients. However, the calculated mean can be influenced by the Sybil updates and may fail for an attacker with more significant influence over the network.

Fung et al. (ref:fools-gold, ) proposed a novel defence strategy to discriminate between honest clients and Sybils based on the diversity of their respective gradient updates. The intuition behind this defence is to identify Sybils based on a shared objective, and thus their models’ updates are likely to have low variance. This works by using a cosine similarity, looking for similarity in terms of indicative features (i.e., those that contribute to the correctness of the update). This approach can be further augmented with other defence methods such as Multi-Krum to determine the poisoned updates.

Wan and Chen (ref:attack-adaptive-aggregation, ) introduced an attack-adaptive defence utilizing an attention-based neural network to learn plausible attacks. Instead of using similarity-based mechanisms, this introduces tailored defence strategies to prevent backdoor attacks.

Ma et al. (ref:dp-data-poisoning, ) explored differential privacy as a data poisoning defence based on creating a robust learning algorithm. While this defence is ideal for systems already using differential privacy, it enforces the use of noise in the learning algorithm, hurting performance. Additionally, this assumes a small subset of malicious clients, failing to defend against larger coordinated attacks.

Related attacks, such as model update plagiarism (ref:fl-meets-bc, ), may additionally occur, leading to a misleading distribution of data amongst clients. A malicious user may attempt to submit the same model update multiple times through several clients, known as lazy clients. They may then attempt to reap the rewards of contribution for each of these submissions. These can be detected by using a PN-sequence (ref:fl-meets-bc, ; ref:blade-fl-analysis, ), whereby the initial model update is submitted with random noise applied to the update and is then further verified by other clients using their own PN-sequence, checking for correlation.

The underlying problem of federated learning is to produce a global model from client updates trained on local privacy-sensitive datasets. We can define each client $\mathcal{C}_{k}$ to have their own set of data $D_{k}\subseteq D,\forall k,0\leq i\leq K$. Using this, we can define the global objective to optimize as a modification of the FedAvg algorithm (ref:federated-learning, ) as

where the batch loss for each client is defined as

and $b_{k,i}$ being a random subset from $D_{k}$. Each client updates their local objective with respect to their dataset $D_{k}$ by

where $\eta_{k}$ is the clients learning rate. The clients objective then becomes

and so the shards objective will become

This can be defined as weight updates by

This happens in parallel across all shards, where the global objective will become the sum of the results produced across all shards

We should note that this described approach follows the original FedAvg algorithm and can be modified in terms of fairness (ref:q-fed-avg, ) among other averaging strategies.

Each task deployed on the network can run an independent consensus mechanism. This allows for various task-related workloads to be accommodated by adapting the consensus algorithm. For example, when training large models where the performance of a single node may be limited, mechanisms such as PBFT (ref:pbft, ) can be used. However, in shards with a fewer number of clients, we can make use of Raft consensus (ref:raft-consensus, ). This may benefit settings where model complexity is lower, and so a peer may be able to handle the throughput bottleneck of Raft consensus, determined by node performance.

Within the chosen consensus mechanism, we apply a pluggable policy to determine the acceptance of model writes. This policy can be substituted to handle alternative schemes (Section 2.3). These schemes handle the identification of malicious clients by finding poisoned model updates or detecting Sybil attacks. It should be noted that while these plugins are applied at a task level to maintain the network’s security, they can be upgraded with the smart contract that they are governing.

Another quality to consider is the detection of lazy clients. Detection of these clients may use a PN-sequence, which is applied to the model updates. This technique could then be applied within the shard level consensus to verify the clients’ PN-sequence.

Because each shard contains a subset of the total participants for each task, the algorithmic complexity of consensus is significantly reduced to $\frac{\mathcal{C}\times\mathcal{P}_{E}}{\mathcal{S}^{2}}$. As a result, the network overhead needed to communicate model updates is also significantly reduced, which is the primary bottleneck in peer-to-peer federated learning.

The mainchain is responsible for coordinating all verified aggregated model updates from each shard. This chain contains all participants across all shards; however, the activity on this chain is limited to shard level aggregation results and task proposals. The consensus for this follows the same pluggable consensus policy as the Shard level consensus (Section 3.2); however, the submitting peers are limited only to the subset of endorsing peers on the shard chains. In this way, we limit the endorsing peers to those in possession of an aggregated model and are able to verify its authenticity.

All accepted model updates to the mainchain will have been endorsed by each shard endorsing peers, which can safely be distributed and aggregated. If multiple models from a single shard have been accepted by disagreements between a shards endorsing peers, the model with more endorsements will win. So, if the endorsing peers submit the same model (model hashes are identical), the endorsing peers can safely evaluate the model update once. Figure 2 shows this process. The final step here is to post the final global model of the mainchain. This model represents the result after a single round of FL, and each of the shards will start the next round with these parameters.

We note that a new round may begin earlier within a shard by assuming the model aggregated by a shard endorsing peers is valid before it is accepted to the mainchain. If this model is not accepted, then the round within that shard must be restarted. This should only happen if the majority of endorsing peers within a shard are compromised, and as such, the disruption will only affect the compromised shard.

This section discusses the entire workflow of our proposed framework, discussing the steps required to produce a trained model.

Firstly, we define the participants’ category in our system. We can break this into three categories: clients, peers, and endorsing peers, and described as follows.

1. (1)

   Clients: In a traditional FL setting, all participants are considered to be clients, where they are comprised of typical IoT devices such as smartphones, laptops, appliances, and vehicles, among others. Clients hold the private datasets that will be used for training models and are they are responsible for producing model updates. We consider the same assumption here, where clients do not validate models but rely on more powerful devices to ensure the system’s security. This is the lightest class of participants in terms of resource consumption.

2. (2)

   Peers: Peers are responsible for holding copies of the ledger. From this pool of peers, a committee can be elected to reduce the evaluation overhead. This helps take the load off of endorsing peers and allows peers to offer API services for clients.

3. (3)

   Endorsing peers: Members of the committee are responsible for evaluating model updates during consensus. They hold their own private dataset to endorse other model updates and can submit their own model updates, although they are not required to do so. Furthermore, the policy for evaluation of model weights can be modified, and these peers additionally must check for valid authentication of the write-set. Finally, committee members must participate in the mainchain consensus to determine which shard level updates will be accepted.

These participants will then participate in one or more tasks, as managed by the mainchain smart contract. Starting from a fresh network, we will describe the following process.

For evaluation, we employ Hyperledger Caliper (ref:hyperledger-caliper, ), an open source benchmarking tool allowing for various workloads to be conducted for our system under test (SUT). These tests are performed by an independent process that generates transactions to be sent to the SUT by a specified configuration, monitoring for responses to determine metrics such as latency, throughput and success rate. We conduct tests with varying numbers of workers, transactions counts, and transactions per second (TPS) to evaluate the limits of the system.

We run several workloads to test the effects of sharding on transaction throughput and model performance. The benchmark tasks include:

• •

  Update creation throughput: We first run an experiment to test the throughput of model creation. To do this, we generate a number of models updates, make the parameters available locally, and have the endorsing peers evaluate them during consensus. In this way, we can measure how sharding affects the performance of the consensus mechanism by parallelizing the workload among shards. We aim to verify our initial claim that the global computations required with become $\frac{\mathcal{P}\times\mathcal{C}}{\mathcal{S}}$.

• •

  Model performance: We evaluate the performance of the models across two scales: (1) the global number of epochs computed and (2) the number of gradients, which provide more accurate measures of performance for decentralized training (ref:bfl-shard, ). While we are primarily concerned with the performance of integrating existing FL solutions into the blockchain, sharding should allow a larger fraction of clients to be fit, allowing for model performance to be increased.

For experimentation, we use three datasets, namely the MNIST dataset (ref:mnist, ) consisting of 60,000 28x28 images of handwritten digits, and the CIFAR-10 dataset (ref:cifar10, ) which consists of a dataset of 60,000 32$\times$32 images separated into 10 classes with 6,000 images per class. We use these to represent IID data, where these classes of data are split evenly between clients.

For non-IID scenarios, we use the LEAF dataset (ref:leaf-dataset, ) which can be used to test model performance by partitioning the data between peers in a reproducible way. In this setting, the classes of data may not be evenly split. For example, in the case of the Federated Extended MNIST (FEMNIST) dataset, characters were split by the writer of the digit or character. This can be a strong benchmark to model real-world distributions of data.

These datasets are used as a common baseline for comparison and evaluation against previous solutions (ref:federated-learning, ; ref:bfl-async, ; ref:bfl-shard, ), showing the addition of a sharding solution maintains competitive results on model performance. While a non-IID assignment of data across shards may influence the generated global model during the global aggregation stage, the selection of mainchain defence mechanisms plays the largest role in model convergence. Thus, the results generated by the chosen datasets demonstrate the applicability of sharding solutions to existing Blockchain-based methods.

We created multiple workloads for the Update Creation Throughput benchmark to explore the effectiveness of our sharding solution. These workloads are conducted without malicious updates using the MNIST (ref:mnist, ) dataset to evaluate the performance of model creation transactions. We set the timeout period for each transaction to 30 seconds, allowing us to consider the number of failed transactions as stale (i.e., not malicious). In this experiment, each of the clients evaluated the update against their entire local dataset, in which we have used the entire test split of the MNIST dataset for each client.

We first measure the throughput with respect to the number of shards, as seen in Figure 4. This workload uses two Caliper workers, evaluated over 200 transactions, where we can see the throughput for each independently operating shard which allows the global throughput to scale linearly. This directly reflects the time for the evaluation of a model that is the primary bottleneck. We note here that the sent TPS for each number of shards is set just above its throughput in order to saturate the system for the evaluation. In this workload, we used the same number of sent transactions for consistency; however, higher numbers of shards have shown improvements in throughput for increased transactions sent, while a lower number of shards may begin to fail earlier.

To test the maximum throughput achieved by our system, we measure the sent TPS against the system throughput and average latency. See Figure 5. This workload is run with 2 caliper workers over 200 transactions. As we increase the sent TPS, the system will reach a point where it becomes saturated and is unable to handle a higher TPS. We can see this by looking at the point where the average latency begins to increase in the figure, where the throughput is maximized. We conduct this workload by measuring sent TPS in increments of 3, starting from 3 TPS. We can see each additional shard increases the overall system throughput similarly in Figure 4 while demonstrating the maximum throughput limit.

To explore the limits of a usage surge, we measure the number of transactions sent by system throughput, average latency, and failure rate. In this benchmark, we are interested in the system’s behaviour under a workload with a higher sent TPS than throughput. This workload is run with 2 caliper workers and a sent TPS just above the maximum throughput as seen in the previous workload. See Figure 6. When the system cannot handle the number of transactions currently queued, the average latency will spike, and the number of failed requests will increase because the system begins timing out requests. The system will then go through a “flush” period where many long-running requests begin to fail, so the average latency of successful requests will drop. We note here that the average latency peaks at around 16 seconds due to the average between the maximum latency (the timeout limit) and the minimum latency, which corresponds to the time it takes to evaluate a model fully.

Figure 7 shows the throughput over this workload, where the throughput of the system decreases when it is no longer able to handle the number of incoming requests, as the overhead to process these additional requests limits the number of successful requests processed. The decrease in throughput corresponds to the increase in average latency previously observed.

We finally test how the system handles concurrent requests. To do so, we run multiple workloads, varying the number of caliper workers, and measuring the system throughput and average latency. This workload configuration sends 200 transactions, with the sent TPS equal to the previously-mentioned maximum throughput. This can be in Figure 8. We observe that the throughput of the system with respect to the number of workers is quite noisy, however, there is a general downward trend in the system throughput. Increasing the number of caliper workers here allows us to scale the workload generation, however since the endorsement workers evaluating the model operate sequentially in these experiments (limited to a single thread), The throughput is limited. Similarly, we can see an upward trend in average latency caused by each transaction’s time waiting for execution in the queue. We can see that the number of shards plays the largest factor here as workloads with more than 2 shards are tightly grouped with respect to average latency due to these workloads being able to operate in parallel across shards.

To measure model performance, we measure the effect of various training parameters. We use minibatch sizes of $B\in\{10,20\}$, and local epoch sizes of $E\in\{1,5,15\}$. We present results here from the non-IID MNIST dataset, however similar results hold for both CIFAR-10, and LEAF benchmarks. The effects of both minibatch size and local epoch sizes can be seen in Figure 9, and Table 2, where we compare our ScaleSFL solution and the traditional FedAvg algorithm. This comparison is performed under the assumption all clients are honest, and thus we are simply comparing the effect of sharding. Both training loss and accuracy are considered, and workflows are run for 15 global epochs, where the number of ScaleSFL shards is set to 8, with each shard sampling eight clients each round, making for a total of 64 clients. The FedAvg algorithm is comparably run with 64 clients, with both methods using a client learning rate of $\eta_{k}=1e-2$.

In Figure 9, we observe the convergence rate of ScaleSFL is faster than the FedAvg algorithm, generally reaching an accuracy of $0.98$ within the 15 global epochs considered. This is due to the parallelized nature of ScaleSFL, where each shard can simultaneously filter and produce viable updates with reasonable client sampling rates, combining them to produce a stronger global model at each step.