Leader Confirmation Replication for Millisecond Consensus in Private Chains

The rapid development of private chain technology represented by Hyperledger has made geo-distributed blockchain systems widely used[29]. Different from the public chain, any node needs to go through a reinforced identification check before it can participate in the blockchain network[12]. Therefore, malicious behavior in the private chain will be easily detected and traced. It allows the private chain system to weaken the security design in an anonymous environment and adopts a consensus model that supports better TPS performance and scalability[30]. Therefore, the existing widely used private chain architectures use CFT protocols (e.g., Raft[11], Paxos[10]) instead of BFT protocols (e.g., PBFT, PoW, DAG), thereby improving the cluster scalability and the efficiency of the consistency[14, 31].

However, CFT protocols are designed for low latency environments. In high network latency, most geo-distributed CFT protocols use leader-based SMR to achieve consistency, which can avoid the performance loss caused by replication conflict in the SMR process (e.g., the livelock in Paxos[35]). Nonetheless, the overhead of the leader will cause serious performance degradation when the consistency model runs in high concurrency. Although it is a good idea theoretically to dynamically change the controller of the consensus process to the follower nodes, it requires a stable network environment between each node to obtain the real-time node status. Otherwise, the consistency model will run at the speed of the slowest node[25].

As we discussed in Section II.A, the CFT consistency model which is commonly used in private chains, meets leader overhead in geo-distributed environments. To address these issues, existing popular approaches reduce the leader overhead by adopting multiple leaders or reducing the frequency of the log replication times by batching operations. We summarize the consensus models mentioned in this section in TABLE 1.

Raft numbers the entries and copies them to the follower nodes in order[43]. Thus, Raft can use the $lastIndex$ of each node to quickly reach a consensus. However, private chains are widely used in supervised multi-subject data-sharing scenarios. The entries they generate contain only a single database operation on new keys. In IoT systems, the leader performs log replication more frequently since the operational non-transactional data are generated on a scale. In addition, the private chain systems need to be deployed in each organization’s datacenter to ensure that they can participate in the consensus[44]. However, we found that the network traffic and I/O overhead generated by the SMR model are very sensitive to network latency. The overhead on the leader will incur higher network overhead and TPS degradation. In addition, most non-transactional requests only insert new key values. The operation of verifying the input and output of the transactions will increase the leader’s computing load, resulting in a significant increase in the response time, and adversely affecting the QoS of the private chain system[45].

To better understand the impact of ms-level latency on the efficiency of consistency in geo-distributed systems, we performed experiments on a real-time cross-datacenter backup system for Kingsoft Cloud’s database, which runs with a 1.5-2 ms network latency, and a Hyperledger-based private chain system, which runs with a 3-7 ms latency. The purpose is to obtain the performance bottleneck of geo-distributed SMR that enables us to make realistic assumptions for the design of LCR-Raft.

In experiments, we send requests concurrently to Raft with numbers of users, and count its TPS and entries’ retransmission times under different network latencies, as shown in Fig. 3.

As the number of clients increases, the TPS and the entries per replication of the cluster gradually rise and reach peak performance when the number of clients is less than 25. At this stage, the cluster does not reach its transaction processing capacity, and thus, a stable response time can be guaranteed. With the increase in the number of clients, we observed that the frequency of replication was significantly reduced and the response time increased, as shown in Fig. 4. This outcome occurs because the leader backlogs clients requests or SMR requests due to high network latency, which leads to worker thread starvation or even deadlocks on the leader[46]. Therefore, the limited processing capacity of the leader is a key performance bottleneck of leader-based consistency models in geo-distributed scenarios.

To express the network load feature of the SMR model more clearly, we took Raft as an example to analyze its log replication process. Ongaro provides the pipeline and batching approaches to replicate entries in the initial Raft paper[11, 47]. In practice, there are three main ways to implement it: serial-batching[48], pipeline[49, 32], and full parallel[50]. For LAN environments, high routing performance and a stable network are provided. Therefore, it is a better choice to use full parallel or pipeline methods since it can provide a lower response time. However, in geo-distributed systems, network resource are more expensive. In addition, the unstable network environment also makes designers tend to adopt a conservative replication strategy. Accordingly, pipeline strategies were carefully designed by [32] to improve the efficiency of the log replication in geo-distributed scenarios. In an ideal situation, each entry is transmitted k(n-1) times by the leader. However, estimating the round-trip time (RTT) is a key problem for pipeline-based approaches[47] and is not an easy task in WAN environments[51]. For verification, we counted the entry retransmission of the pipeline-based Raft under different network latencies in Fig. 5.

Therefore, it is one of our research motivations to reduce the leader’s message transmission cost in the WAN environment.

The numbering method of future entry is similar to the commonly used creasing numbering in the SMR protocol, to make it more applicable to data consistency models. The difference is that the index of two entries that are continuously appended to the future-log can be discontinuous. In addition, the replication process of future-log is led by followers. We call the follower leading a certain future entry replication a data-leader and other nodes are data-followers. When a follower with id $S_{i}$ receives non-transactional data $D_{nt}$, the follower will become the data-leader, allocate a future log index $o$ to $D_{nt}$ and assemble it into a new future entry $FE_{\lambda}$. The index allocation method is shown in Equation (1).

where $lastIndex_{i}$ is the max index of future entries that $S_{i}$ saved, and $G_{i}$ is the current $generation$ of $S_{i}$ ($generation$ is an increasing integer as the number of nodes changes and $generation>=max(S_{i})$, which we introduce in detail in Section V).

Theorem 1. The index of future entry $FE_{\lambda}$ generated by $S_{i}$ is globally unique.

Proof. For $FE_{\lambda}$, we can prove that

because

,

and

Therefore, for servers $S_{i}$ and $S_{j}$, $G_{i}=G_{j}$ , and the index of future entry that $S_{i}$ generates must be different from another server $S_{j}$.

When $G_{i}>G_{j}$, $S_{i}$ will reject the copy request from $S_{j}$. For $S_{j}$ with an obsoleted $G_{j}$, when $S_{j}$ receives a message from a node with a newer $generation$ $G_{i}$, $S_{j}$ can find all of the future entries that it created according to Equation (1). Then, $S_{j}$ reallocates a nonconflicting index for these future entries with $G_{i}$ (we will detail how servers address the $generation$ change in Section V.)

After $FE_{\lambda}$ is written on a future-log and committed by the data-leader, it copies $FE_{\lambda}$ to all nodes, as shown in Fig. 8.

The node will reject the future entry replication request from the nodes with a smaller $term$ or $generation$ and return the newer information that it knows to them. It ensures that nodes can reach a consensus on the same future entries if and only if the nodes have the same $term$ and $generation$. The future entries generated from a node with an obsoleted $term$ or $generation$ will not be applied because they cannot be accepted by quorum. It is very important because the signal entry must correspond to the same future entry on all nodes.

However, the data-leader’s log replication process could have two problems: 1) The leader receives future entry $FE_{\lambda}$, but parts of the nodes do not receive the $FE_{\lambda}$. 2) The leader did not receive the $FE_{\lambda}$ from the data-leader.

For the first case, leader $S_{i}$ does not know which followers have received the $FE_{\lambda}$. Therefore, the leader will use signal entry to replace $FE_{\lambda}$. In particular, if the $nextIndex_{j}+1$ of follower $S_{j}$ is a future entry, $FE_{j}$ will not be replaced by signal entry. As shown in Fig. 9, S5 did not obtain future entry with $FE_{7}$, and thus, it can only commit a normal entry with $E_{6}$ and set $lastIndex_{5}$ to 6 in the return value. For a follower $S_{k}$ whose $nextIndex_{k}+1$ indexed entry is a future entry, the leader believes that it has not received the future entry $FE_{nextIndex_{k}+1}$. Therefore, when the leader is packaging entries for replication, it will assemble the entry as a normal entry. Followers will trust the correctness of the leader and commit the entries.

For the second case, i.e., when follower $S_{i}$ with $G_{i}$ finds that the normal entry $E_{\lambda}$ sent by the leader conflicts with $FE_{\lambda}$, the follower trusts the leader. Then, it will check whether the data-leader of $FE_{\lambda}$ is itself according to Equation (6).

If the result is $true$, then, $S_{i}$ is the data-leader of $FE_{\lambda}$, and $S_{i}$ deletes $FE_{\lambda}$ from the future-log, reallocates the future entry index according to Equation (1), and rewrite it into the future-log. If it is $false$, it will discard $FE_{\lambda}$. Therefore, regardless of whether the result is $true$ or $false$, $FE_{\lambda}$ will be written by $E_{\lambda}$, as shown in Fig. 10.

We designed a window for LCR to manage the writing process of future entries. A window is a continuous range of index, and it has two states: closed and open. When the window is opened, the node can act as a data-leader to write future entries with an index within its window. When the window is closed, the data-leader will no longer be able to generate future entries with an index in this range. However, for node $S_{i}$, the state of the window does not reject the entries received from other data-leaders. The condition for the window to close is that the $lastIndex$ of the normal-log (the log that replicated in the sequenced index in SMR) is greater than or equal to the starting index of the window. The window can avoid the conflict described in Section III.B to a certain extent because it can ensure that the allocated index can keep at a distance from the committed index of the leader, as shown in Fig. 11.

In addition, we have added a $generation$ property to the window to ensure that there will be no conflict between future entries when the membership changes. We will introduce it in Section V.

The $generation$ of a node can change under the following three conditions: 1) The node receives the membership changing configuration from the leader. 2) The node learns a newer $generation$ from the replication response. 3) The node learns the new $generation$ from the replication request. When the $generation$ changes, it will immediately stop sending future entries replication requests and re-allocate the index of the future entries that have not been applied to the state machine in such a way that it can be correctly indexed by all nodes. In addition, to facilitate index calculation between nodes, we can use the number of nodes in the cluster as the $generation$. The $generation$’s change rules are flexible. For example, for clusters that frequently change membership, we can take the surplus of $generation$ to reflect the reduction in the number of nodes.

Theorem 2. The increase in $generation$ will not cause conflicts between future-logs and normal-logs.

Proof. For node $S_{i}$ with $generation$ $G_{i}$, future entry $FE_{\lambda}$ and $\lambda\%G_{i}=S_{i}$, when the $generation$ of $S_{i}$ becomes $G_{i}^{\prime}$, $G_{i}^{\prime}>G_{i}$, the new index $\lambda^{\prime}$ of $FE_{\lambda}$ is

Based on Equation (1), $\lambda^{\prime}$ will not duplicate with other future entries in $G_{i}^{\prime}$. In addition, suppose that when the $generation$ increases, the leader’s $lastIndex$ of the normal-log is $\omega$. Then, $\lambda>\omega$ because if $\lambda\leq\omega$, $FE_{\lambda}$ must have been committed in the normal-log according to Section IV.B. Here, we prove that $\lambda^{\prime}\geq\lambda$ and thus prove $\lambda^{\prime}>\omega$. From Equation (2), we have the following:

$\lambda^{\prime}$ can be presented as

Since $G_{i}$ and $G_{i}^{\prime}$ are integers, we have $G_{i}^{\prime}\geq G_{i}+1$.

Therefore, the increase in $generation$ will not cause $FE_{\lambda}$ to conflict with normal entries because $\lambda$ is larger than any other normal entries. If $FE_{\lambda}$ can be applied, then $FE_{\lambda}$ must have been successfully replicated to half of the nodes that contain the leader. This circumstance means that $FE_{\lambda}$ does not conflict with the normal entries of the leader and thus will not conflict with the normal entries of other nodes, because $\omega$ must be greater than or equal to the $lastIndex$ of the normal-log of all other nodes.

As shown in Section IV, the index allocation method of the future entry is related to the number of nodes. future-log windows with different $generations$ cannot keep the same content in future-logs. Therefore, when the node is notified of the $generation$ changing, it needs to re-allocate and re-replicate the future entries. In detail, when follower $S_{i}$ obtains the message of $generation$ update, it will immediately record all future entries whose index is larger than the $lastIndex$ of the normal-log, and clear the windows with the old $generation$. At the same time, it creates a new window with updated $generation$, reallocates new indexes for these future entries, and starts to replicate the future entries with the new $generation$. Since the $lastIndex$ of the normal-log on the follower must be less than or equal to the value on the leader, the follower will not conflict with the entries that have been committed in the normal-log, as shown in Fig. 13.

S1, S2, and S3 store some future entries in two windows: index range 800-899 and $generation=3$, index range 900-999 and $generation=3$. When the cluster size changes from 3 to 5, $generation$ is also updated to 5. S2 and S3 need to recreate the window with $generation=5$. In the window where $generation=3$, the nodes will reallocate the index of future entries in it to the new window according to Equation (1) .

Theorem 3. After the generation is updated, the re-replication process will not cause inconsistencies in the future-log.

Proof. In node $S_{i}$, for an unapplied future entry $FE_{\lambda}$ with generation $G_{i}$, $\lambda\%G_{i}=S_{i}$, and $lastIndex_{i}$ of the normal-log is $\omega$, $\omega<\lambda$. When $generation$ is updated to $G_{i}^{\prime}$, the new index of $FE_{\lambda}$ is $\lambda^{\prime}=\left\lfloor\frac{\lambda}{G_{i}}\right\rfloor*G_{i}^{\prime}+S_{i}$. In this case, node $S_{i}$ will replicate $FE_{\lambda}^{\prime}$ to all other nodes with generation $G_{i}^{\prime}$. If for node $S_{j}$, its $lastIndex_{j}$ of the normal-log is greater than or equal to $\lambda^{\prime}$, it will ignore $FE_{\lambda}^{\prime}$, which also indicates that the leader’s $lastIndex$ of the normal-log is greater than $\lambda^{\prime}$. Therefore, $FE_{\lambda}^{\prime}$ cannot be applied since it will not be committed by quorum. Node $S_{i}$ needs to re-allocate the index of $FE_{\lambda}^{\prime}$ according to the re-replication strategy in Section IV.B. If for node $S_{j}$, its $lastIndex$ of normal-log is less than $\lambda^{\prime}$, it will accept $FE_{\lambda}^{\prime}$ on the premise of confirming that its $G_{j}=G_{i}^{\prime}$. According to Theorem 1, the acceptance of $FE_{\lambda}$ will not cause conflict.

In order to verify the performance improvement effect of LCR-Raft, we implemented it in the open-source Raft consistency model[50]. Among them, the RPC protocol is implemented using brpc[52], and RocksDB[53] 5.1.4 is used to store state machine data. All of the instances are deployed on 3, 5, 7, and 9 servers (according to the number of nodes in the different experimental configurations) with a Xeon E5-2620 v4 CPU, 64 GB of RAM, 4 TB, and 7200 RPM SATA disk. The system runs CentOS Linux release 7.5.1804, 64 bit, with Linux kernel Version 3.10.0 The switch uses H3C Mini S8G-U, with 16 Gbps switching capacity and 12 Mpps forwarding performance.

Our client runs on 2 servers with 6 core 12 thread AMD 3600 CPU and 32 GB memory, and the default number of client jobs is 40. The non-transactional data are the real-time sensor data generated from our blockchain water quality monitoring project, while the transactional data are value transferring operations. The default transactional data and non-transactional data size are 80 bytes. The base configuration of Raft, LCR, and multi-Raft is detailed in TABLE IV.

It is worthwhile to note that to compare LCR-Raft and other consistency models fairly, we use pipelines to implement SMR, i.e., each log contains only one transaction, and the consensus node immediately invokes remote replication when it generates future entries. The reason is that our work focuses on the optimization of SMR. Although the use of batch messages could improve TPS tens to hundreds of times, its effect is related to the number of clients, the request frequency, the batch size, and the dynamically adjusted request queue, which makes it unable to truly reflect the contributions of our work.

LCR decouples the replication process of future-log and normal-log. Therefore, for transactional data, the client can still use a normal-log to submit the transactional data. For non-transactional data, using a future-log can significantly reduce the leader’s overhead and the response time of normal entry requests (because normal entries will be processed immediately by the leader). Therefore, in this section, we analyze and test the latency of applying transactional and non-transactional data, and compare it with Raft to verify its optimization effect.

For non-transactional data $D_{nt}$, the latency $\Delta_{nt}$ is

where the $\Delta_{c}$ is the network latency between the cluster and client, $\Delta_{n}$ is the network latency between consensus nodes in the cluster, $\eta_{l}$ and $\eta_{f}$ are the time periods from entry packaging to replication request sending on the leader and followers, and $\eta_{w}$ is the time period during which the data-leader wait for a signal entry to apply the data. Here, $\iota$ is composed of the disk operation latency, such as log writing and state machine applying.

For transactional data $D_{t}$, the latency $\Delta_{t}$ is

In $\Delta_{nt}$ and $\Delta_{t}$, $2\Delta_{c}$ is generated from the network latency between client the and nodes. As shown in Figure. 6, it needs $2\Delta_{n}$ for $D_{nt}$ and $4\Delta_{n}$ for $D_{t}$ to respond to the clients.

For $\Delta_{n}$, since the clients do not know which node is the leader, both transactional data and non-transactional data can be sent to any node. When transactional data are sent to a follower, the follower redirects the request to the leader. In this case, an additional RTT is generated from the redirection. Since this case occurs at a $\frac{N-1}{N}$ probability, the expected latency is $\frac{4N-2}{N}\Delta_{n}\approx 4\Delta_{n}$. The non-transactional data need $2\Delta_{n}$ to respond to clients since they will not be redirected.

In addition to the $\Delta_{n}$, the differences in latency for $\Delta_{nt}$ and $\Delta_{t}$ are $\eta_{w}$, $\eta_{l}$, and $\eta_{f}$ since $\Delta_{c}$ and $\iota$ are fixed latencies. From the experiments in Figure. 4, it can be found that the additional latency is generated from $\eta_{l}$ (tens-ms) because all other conditions are the same besides the request frequency from the clients. The limited worker threads and high network latency make $\eta_{l}$ higher as the client increases. Similar to $\eta_{l}$, $\eta_{f}$ increases because the high request frequency of non-transactional data makes followers run under high loads. Here, $\eta_{w}$ could be higher than 1 RTT since the leader might not send signal entries to the data-leader immediately when it receives future entries. Besides, the overhead on the leader could amplify $\eta_{w}$. However, $\eta_{l}$ will decrease significantly because the proportion of normal entries decreases. It allows the leader to reduce the number and frequency of normal entry replication operations. Therefore, the response time of the transactional data in LCR is lower than in Raft.

We tested the effect of LCR and Raft on TPS and the response time in a 5-node, 5 ms latency environment, which is the common network latency between distributed blockchain systems. The proportion of non-transactional (LCR future entries) data was 25%. The result is shown in Fig. 15.

In terms of the TPS, LCR outperforms approximately 30% when the client count is larger than 20. For the response time, as the number of clients and the entry size increase, resource starvation becomes more serious. This circumstance makes the effect of the LCR on reducing the concentrated load more obvious. LCR can provide a 40%-60% response time decrease for transactional data (LCR normal entries), which matches the observation in Fig. 3.

To show the availability of LCR, we test the node failures in the cluster. In this experiment, LCR, Raft, and multi-Raft run in a 5-node, 5 ms network latency environment. We randomly stop a non-leader node (an arbitrary node for multi-Raft) at the 10th second and reconnect it to the cluster at the 16th second. In the 25th second, we stop the leader node (an arbitrary node for multi-Raft) and no longer recover it. The result is shown in Fig. 14.

LCR: When the data leader is offline, the future entries that it generates might have not been replicated to all nodes. In this case, the followers that did not receive these future entries will ask the leader to re-replicate these entries, since the leader uses signals to replicate them (as described in Section IV.B). This circumstance will destroy the performance improvement brought by batching. However the performance can be quickly restored when these entries are handled. Nonetheless, the TPS will decrease slightly before the node recovers since the downtime of the followers reduces the frequency of future-entry generation, which reduces the optimization effect of LCR. During the 10th second to the 16th second, other followers can still act as data-leaders to replicate future entries. Therefore, the performance is not adversely affected (from approximately 1700 TPS to 1600 TPS). When the leader node fails (at the 25th second), it will trigger re-election, which makes the service unavailable during this period. When the new leader is elected (at the 31st second), the TPS performance can recover.

Multi-Raft: In multi-Raft, every node is the leader of shards. For a shard $C_{i}$ whose leader is $S_{i}$, failure on $S_{i}$ will make $C_{i}$ unavailable because of re-election. Although the TPS does not drop to 0 (from approximately 1200 TPS to 800 TPS), the remaining TPS is contributed by the active shards. Therefore, the performance of multi-Raft is more sensitive to node failures, since when any node fails, performance fluctuations will occur. However, in LCR and Raft, a non-leader node failure will not lead to serious performance loss. In addition, the availability problem still exists in multi-Raft. Since the probability of each node being offline is the same, the total availability performance is the same as that of Raft.

Raft: Raft suffers the least performance loss when facing a failure of the follower node. However, due to the high load on the leader, its performance under a 5 ms network latency is only approximately 60% (approximately 1200 TPS to 1700 TPS) of the LCR.

As an important performance indicator of the SMR protocol, we tested the TPS of LCR. In addition, we conducted experiments on the CPU load and network traffic to verify its effectiveness in reducing the load of the leader.

LCR can generate additional resource costs according to the following three points. 1) Network traffic: This cost arises because the follower nodes undertake part of the log replication work. As a result, the network traffic is increased in the followers. 2) Memory consumption: the leader needs memory to cache the relational mapping to ensure the writing efficiency of the future-log. 3) Computational resource consumption: future-log replication requires index calculation and verification. We calculated statistics on these costs to measure whether the extra cost generated by LCR is acceptable.

First, we measure the network traffic. Since additional network traffic is generated on the follower nodes, we will not compare the leader’s network traffic. Here, we count the extra network traffic with different proportions of future entries and compare it with the traffic consumption of multi-Raft. The traffic consumption of multi-Raft is the average of all of the nodes. Similar to Section VI.C, we still compare the traffic consumption under the same TPS. The experiment results are shown in Fig. 19. The additional network traffic generated by LCR is much less than that generated by multi-Raft.

The consumption of the network traffic generated by LCR is much less than that of multi-Raft traffic. At the same time, we found that at the same TPS, the network consumption of the followers will decrease as the cluster size increases. This finding is in line with our expectations because each follower needs to replicate fewer future entries.

In the consistency protocol, the entries must be written to the segmented log and memory at the same time before it is applied to the state machine to ensure that the received log will not be lost. Therefore, future entries need to consume additional memory. We tested the LCR on the extra main memory consumption. The experiments were run in an environment in which the number of consensus nodes is 5, 7, and 9, the average network latency is 5 ms, and the average size of key-value pairs is 80 bytes. The experiment results are shown in Fig. 20.

The experiments show that the consumption of LCR on the main memory is acceptable (KB level) and that the proportion of future entries is positively related to the additional consumption of the main memory. When the number of future entries is too large, the growth rate of its index will far exceed that of normal entries. However, applying the future entries to the state machine must wait for the normal entries whose index is greater than it to be applied. In extreme cases, such as when the proportion of future entries reaches 99%, too many future entries will make the growth rate of the normal entries index unable to catch up with the future entries index. This circumstance will cause future entries to wait a very long time before it can be applied. To address this problem, we designed a stepping mechanism to allow the leader to confirm future-entries in advance.

For computational resources, LCR only needs extremely low complexity such as list and hash map lookup. Although we calculated statistics on leader resource consumption in Section VI.C, our model did not cause a large overhead on the leader node. However, our model will generate an additional CPU load on the follower nodes. We calculate statistics on the CPU load of followers on Raft, multi-Raft, and LCR under different TPSs in a 9-node cluster in Fig. 21.

The results show that the CPU load of LCR will not exceed the leader in Raft. This finding is acceptable since each node should have similar processing capabilities to be elected as the leader. Therefore, the CPU load of LCR will not cause performance degradation.

LCR is an SMR model based on leader confirmation, but it performs better than semi-synchronous in security. LCR did not decentralize the power of the leader. There is a large number of existing methods that use multi-leader to reduce the load on the leader, such as multi-Raft[32] and Hash Raft[36], etc. However, regardless of the implementation, it will face the following two problems: 1) When any node becomes unavailable, the cluster must invoke re-election. 2) Each pair of nodes must synchronize their online status through heartbeats. The leader confirmation method solves these two problems well. First, when the follower node is unavailable, it cannot copy and receive the log from followers and the leader. This circumstance does not require the re-election process. At the same time, compared with multi-Raft, it does not require heartbeats to maintain the online status of each node. Because it does not need the followers to know whether other nodes are online, the follower only needs to know whether the future-logs are accepted by half nodes that contain the leader. Therefore, it does not need to reuse the connection to reduce the network load between nodes, which greatly reduces the complexity of the protocol. In addition, LCR does not guarantee that every future-log has been written into the state machine when it returns a success message to the client. However it will ensure that data will be written to the state machine at some point in the future, which also makes it applicable to more complex network environments.

LCR is an SMR model based on leader confirmation, but it performs better than semi-synchronous in terms of consistency. LCR did not decentralize the power of the leader. There is a large number of existing methods that use multi-leader strategies to reduce the concentrated load[32, 36]. However, regardless of the implementation, it will face the following two problems: 1) When any node becomes unavailable, the cluster must invoke re-election. 2) Each pair of nodes needs to synchronize their online status through heartbeats. LCR solves these two problems well. For the follower node failure cases, the node cannot copy or receive the log from followers and the leader. It does not require re-election. At the same time, compared with multi-Raft, LCR does not require heartbeats to maintain the online status of each node. Because followers do not need to know whether other nodes are online, the only thing they care about is whether the future-entries are confirmed by the leader, which greatly reduces the complexity of the protocol.

The improvement brought by the LCR does not cause additional costs. As described in Section VI.D, the overhead does not cause consumption that exceeds the original system capacity. Because the log replication process of the SMR protocol is extremely concise, the network, CPU, and other overhead are very sensitive to consensus protocols. As we described in Section II, the increase in the latency will make the TPS drop significantly. To reduce the network latency between geo-distributed nodes, it might be necessary to establish a dedicated network, which is very difficult in terms of feasibility. Although the cost of upgrading the CPU and router is lower, it can also improve the optimization effect brought by the LCR. Therefore, the LCR can trade at a lower cost for the improvement of transactional data consistency efficiency and TPS. In addition, as described in Section VI.C, the LCR can reduce substantially the network traffic. For cluster deployment in WAN, network bandwidth or traffic must be purchased from the Internet Service Provider (ISP), and the cost is positively related to the bandwidth or traffic. Therefore, the LCR can effectively save approximately 20%-30% of the network construction costs.