Latency Analysis of Consortium Blockchained Federated Learning

Quantities of data have been generated continuously and become the new type of fuel that promotes the development of production. But the data security has to be considered during training a model cooperatively among different participators. In this regard, federated learning (FL) architecture was proposed [1].

The original FL system needs a central sever to collect and distribute the model weights and gradient parameters (called the local model update). This centralized architecture may introduce some problems. Firstly, the global model that all participators receive depends on the single central server. If a failure happens on the server, each participator would get an inaccurate global model. Secondly, because all the model updates are stored in the sever. Once the server is attacked, the whole system would be collapsed.

In order to avoid the negative effects brought by the centralized architecture, the decentralized architecture was proposed by exploiting the blockchain instead of the server [5].

FL based on the blockchain has been used on Internet of Things (IoT) [2], Internet of Vehicular (IoV) [3], Mobile Edge Computing (MEC) [4] and so on. It supports not only these Devices-to-Devices (D2D) applications but also Businesses-to-Businesses (B2B) scenarios. Enterprises that own mass of data, such as banks, securities and hospitals, would like to discover the intrinsic value hidden in the data collaborating with others. In this paper, we present a FL based on blockchain for these B2B scenarios.

Considering the efficiency of the architecture, consortium blockchain should be used for the decentralized federated learning [6, 7]. Because only authorized peers can join the network and have access to the data stored in the distributed ledger on the consortium blockchain. The consensus protocol of the consortium blockchain is usually not PoW (Proof of Work), but consensus algorithms such as PBFT (Practical Byzantine Fault Tolerance) [8] and Raft [9], which are more efficient and suitable for the multi-center network.

The verification mechanism of the blockchain is often used to authenticate identities of peers. But the quality of models is especially important in the FL. Thus we introduce a model verification mechanism in order to ensure the quality of local model updates trained by participators.

The efficiency of the blockchained federated learning system is a key issue for practical application. Therefore, it is important to analyse the latency of the system. Most of the existing works explained the system delay by the ways of numerical simulation. These empirical analyses are too costly to obtain accurate results. Furthermore, the underlying networks for deploying permissioned blockchains have a great impact on analysis results, thus these results are not comparable and lack versatility [10]. It is imperative to analyse theoretical latency to provide a quantitative model. The main contributions of this paper are as follows:

• •

  A decentralized federated learning based on consortium blockchain called CBFL is proposed to train a classification model by logistic regression with a horizontally partitioned dataset in B2B scenarios.

• •

  We introduce a model verification mechanism for CBFL to validate the availability of the local model updates trained by participators.

• •

  The theoretic latency model for CBFL system is divided into three parts. Each part involves several subdivisions for fine-grained analysis.

• •

  Through the latency model, we get an optimal throughput configuration for PBFT so as to improve the efficiency in practical application.

Let $E=\{E_{i}\}_{i=1}^{N_{E}}$ be a set of enterprises collaborating with each other in CBFL. The enterprises manage two types of nodes: compute nodes $\{C_{i}\}_{i=1}^{N_{E}}$ and communication peers $\{P_{i}\}_{i=1}^{N_{E}}$. The compute nodes have enough computing power to train the models. And communication nodes are responsible for maintaining the blockchain.

The CBFL architecture is organized as two layers: the model update layer and the blockchain layer as shown in Fig. 1. In the model update layer, compute nodes train the local models using its own raw data samples locally and upload the local model updates to the corresponding communication peers in the blockchain layer. Each peer verifies all the local model updates gathered from other peers and operates the consensus algorithm to generate a new block. Finally, the local model updates recorded in the newest block are aggregated locally by each compute node. So all the participators achieve data collaboration without leaking the raw data.

We aim to build a latency analysis model to quantify the time consumption of the CBFL. Before building the latency model, some reasonable assumptions are made as follows:

• •

  The compute nodes and communication peers have stable and enough computing resources for model training and verification.

• •

  The communication peers have certain communication and storage capabilities to ensure $tx$s sharing. And peers are defined to dispose the received messages on a FIFO basis, while the processing time at each peer follows the exponential distribution with the mean $\mu$.

• •

  The arrival of new $tx$s follows the Poisson Process with the arrival rate $\lambda$.

Let $T_{0}^{\ell}$ be the total time during $\ell$-th cycle process at a fixed enterprise $E_{0}$ and

$$T_{0}^{\ell}=T_{update}+T_{commun}+T_{consensus},$$

where $T_{update}$, $T_{consensus}$ and $T_{commun}$ are model update, consensus and communication delays respectively.

1) Model update latency: The model update delays are generated by Step 1 and Step 6. Let $\delta_{d}$ be a single data sample size and $f_{c}$ be the clock speed. So the local model update latency in Step 1 is evaluated as $T_{local,0}^{\ell}=\delta_{d}N_{i}/f_{c}$ [5]. And the global model update latency $T_{global,0}^{\ell}$ in Step 6 can be given as $T_{global,0}^{\ell}=\delta_{m}N_{B}/f_{c}$ [5], where $\delta_{m}$ is the size of a local model update $tx$.The model update latency can be calculated by

$\displaystyle T_{update}=T_{local,0}^{\ell}+T_{global,0}^{\ell}.$

(4)

2) Consensus latency: The consensus delays are brought by Step 3 and Step 4. And the latency of the PBFT consensus is fully considered according to its three-phase work flow.

Let $N(\tau)$ be the number of arrived $tx$s within the max waiting time $\tau$. So the leader $L$ sends the pre-prepare message to other peers when the number of arrived $tx$s reaches $b$ according to the conditions above-mentioned in Step3, where $b=max\{N(\tau),N_{B}\}$. The collection, verification and batch processes of $tx$s at $L$ can be modeled by the $M/M/1$ queue. According to the Little’s law, The average waiting time of each $tx$ can be formulated as $\frac{1}{\mu-\lambda}$. Thus, the total latency of the pre-prepare phase can be given as

$\displaystyle T_{preprepare}=\frac{max\{N(\tau),N_{B}\}}{\mu-\lambda}.$

For an arbitrary fixed $P_{o}$, its process of receiving prepare messages is the Poisson process with the intensity $\lambda$. Thus, the time lag $t_{i}$ between two adjacent prepare messages follows the exponential distribution with mean $1/\lambda$. The average waiting time of $P_{o}$ can be denoted as

$\displaystyle T_{wait}=E[\sum_{i=1}^{2f}t_{i}]=\sum_{i=1}^{2f}E\left[t_{i}\right]=\frac{2f}{\lambda}.$

The total processing time in this phase is calculated as

$\displaystyle T_{process}=\frac{2f+1}{\mu},$

so the latency of prepare phase is

$$T_{prepare}=T_{wait}+T_{process}.$$

The latency of the commit phase is similar to the prepare delay.

The total latency of consensus phase is

$\displaystyle T_{consensus}=T_{preprepare}+T_{prepare}+T_{commit}.$

(5)

3) Communication latency: The communication delays are contributed by Step 2 and Step 5. The local model upload latency in Step 2 is computed as

$$T_{up,0}^{\ell}=\delta_{m}/\left[W_{up}\log_{2}\left(1+\gamma_{up}\right)\right],$$

where $W_{up}$ is the bandwidth between $C_{0}$ and $P_{0}$, $\gamma_{up}$ is the signal-to-noise ratio [5]. Similarly, global model download delay in Step 5 is calculated as

$$T_{dn,0}^{\ell}=\left(h+b\delta_{m}\right)/\left[W_{dn}\log_{2}\left(1+\gamma_{dn}\right)\right].$$

So the latency of communication can be calculated as

$\displaystyle T_{commun}=T_{up,0}^{\ell}+T_{dn,0}^{\ell}.$

(6)

The time consumption data was fitted with Exponential, Weibull, Gamma, Hypoexponential, LogNormal and Pareto distributions using MLE (Maximum Likelihood Estimation) technique in [14]. The best-fit model of $T_{prepare}$ is Weibull distribution ($shape=2.092$, $scale=0.8468$). For another, $T_{prepare}$ is the sum of independent identically exponential distributed random variables given in Section III. Thus it follows Gamma distribution. While the Gamma distribution is a special kind of Weibull distribution, our latency analyses of $T_{prepare}$ and $T_{commit}$ are suitable.

As the number of peers increases, the probability of failure peers occurrence will also increase. In Fig. 3 [14], the mean latency to consensus increases with the augment of the number of peers. This is similar to what our model (7) shows.

Fig. 4 shows the impact of the transactions arrival rate $\lambda$ on the blockchain average completion latency. The relationship between the latency and $\lambda$ is a approximate inverse proportional function as shown in Fig. 4-a, which is consistent with our latency model. In Fig. 4-b, we compare the latency results from the simulation [10] and the latency model. In order to ensure comparability, the same configurations in [10] are adopted. Let $N_{B}=100$, $N_{P}=4,f=1$ and the transactions arrival rate $\lambda$ starts from 50tps to 250tps in this experiment. The model predicts the experimental measurements with an error lower than 3.1%.

In the B2B scenarios, each enterprise can enhance the computing and communication equipment to improve model update and communication efficiency. So it is crucial to optimize CBFL with respect to latency, computing and storage requirements by improving the underlying networks, which can reduce the consistent delays according to our latency analysis model.

In conclusion, the consensus latency is a bottleneck of the whole system, especially the latency of waiting for ordering. According to (7), the latency of the system is proportional to the number of faulty nodes and inversely proportional to the system TPS. The throughput can be adjusted to reduce latency according to actual situation. In addition, faulty nodes can be reduced by establishing a reward system and a node selection mechanism.

The authors thank professors Debiao He of Wuhan University and Xiaohong Huang of Beijing University of Posts and telecommunications for their valuable suggestions to improve the the innovation of this paper.