SeqBalance: Congestion-Aware Load Balancing with no Reordering for RoCE

Unlike traditional TCP software-based transmission, RDMA represents a hardware-based approach that implements transmission functionality within network card hardware, offering kernel bypass and zero-copy interfaces for applications. Consequently, in contrast to TCP, RDMA attains superior throughput, reduced latency, and low CPU overhead, leveraging its kernel bypass and hardware forwarding advantages.

However, RDMA has its unique order-preserving mechanism that adapts to lossless network environment. The order-preserving of RDMA is based on the granularity of queue pair (QP), which means that RDMA performs ordered transmission at the granularity of an entire flow. In the Base Transport Header (BTH) of RDMA packets, there is a field called Packet Sequence Number (PSN). This field is specific to each QP connection. In each pair of QP connections, there is an independent PSN. When the packet passes through the IB transport layer (the layer is solidified in RNIC), the PSN code in the packet header will continue to accumulate. At the same time, when the packet is transmitted to the destination end RNIC, the PSN code is identified at the IB transport layer. If the PSN is not received in order (i.e., packets arrive out of order), it will immediately initiate packet loss retransmission, which seriously affects transmission performance.

To illustrate this issue, we conducted experiments involving two sets of fixed-size flows, deliberately introducing a delay in the transmission of one packet, leading to the out-of-order arrival of a packet. The FCT sizes for both small flows (64KB) and large flows (10MB) are presented in Table I. Notably, the out-of-order arrival of a data packet results in a minimum threefold increase in FCT. Consequently, load balancing schemes for RDMA should prioritize avoiding out-of-order packet arrivals to ensure optimal transmission performance.

AI training requires high transmission rates, therefore, RDMA technology is often used as the communication method in AI training [23]. However, under the traffic load of AI, the load balancing effect of ECMP is not good. This is caused by the traffic characteristics of AI training. The flow generated by AI and storage workloads is often small, but very large. Therefore, the randomness is very low, and the load balancing effect using ECMP is poor [12].

From the perspective of entropy theory, entropy in the field of networks can be defined as a measure of the randomness of data in network packets or flows. ECMP relies on high entropy to evenly distribute traffic across multiple links. The traffic model of traditional cloud network services generates thoughts of flows randomly connecting to various clients, providing a high entropy. However, in the traffic mode of AI, the number of flows is often small, while the size of flows is large. The low entropy traffic pattern of AI training is also known as the elephant traffic distribution. In the elephant flow scenario, the impact of hash conflicts is particularly significant, with only 60% of the maximum throughput under ECMP [12].

Therefore, in the AI training where RDMA transmission is most commonly used, load balancing algorithms are crucial.

In SeqBalance, we make several design choices to build a load balancing framework that works with RDMA. We now discuss our design decisions.
End-host vs In-Network Load Balancing Decision. Determining where to implement load balancing stands as the primary consideration, specifically whether on the end-host or within the network. The nature of RDMA transfers, bypassing the kernel, precludes load balancing implementation in the transfer stack. Thus, if end-host load balancing is sought, it must be confined to operations on RNIC. However, it is difficult for end-host RNICs to perceive congestion: the ECN and RTT information measured by Programmable Congestion Control (PCC)[24] is only used for congestion control of the RNIC and cannot reach the upper layer for processing. MP-RDMA[19] achieves end-host multipath RDMA transmission through custom designed FPGA hardware, thereby implementing load balancing. However, the prevalent use of commercial RNICs, such as Mellanox CX-6 [21], in most data centers poses compatibility challenges with custom-designed RNICs like MP-RDMA. Moreover, the high construction cost in practical scenarios discourages their deployment. Consequently, our approach involves implementing load balancing decisions within the network. This strategic choice ensures SeqBalance’s compatibility with RNICs commonly found in existing data centers, facilitating straightforward deployment. This also corresponds to our Challenge #3.

Load Balancing Granularity. As mentioned earlier, ECMP has low efficiency in AI scenarios where RDMA transmission is commonly used nowadays. Therefore, choosing the granularity of load balancing is important, which corresponds to our Challenge #1.

One such alternative is the per-packet load balancing scheme [13, 14]. Despite the careful design of schemes like DRILL and Presto to minimize the generation of out-of-order packets, complete avoidance remains elusive, leading to subpar RDMA performance [25]. Consequently, per-packet rerouting schemes are deemed unsuitable for our considerations.

An alternative approach is the flowlet-based load balancing scheme [6, 17]. Flowlet distinguishes bursts of packets based on the inactive time interval. The inactive time interval exceeds the maximum delay difference between paths, allowing two bursts to traverse distinct paths without reordering packets. However, the characteristics of RDMA traffic render flowlet-based methods less applicable. This conclusion is drawn from experimental testing and observation. Fig. 1 depicts flowlet sizes for both TCP and RDMA traffic using varying inactive time thresholds. Even under a 10us inactive time threshold, RDMA flowlets are consistently larger than 10MB, and under a 100us threshold, flowlet sizes approach 1GB. In contrast, with a 100us threshold for TCP, flowlet sizes are predominantly below 10MB. Essentially, RDMA encounters greater challenges in identifying flowlets compared to TCP under the same inactive time threshold, aligning with findings in [19]. In practical scenarios, the inactive time threshold typically ranges between 100-500us [6]. This underscores the operational difficulty of identifying flowlets in RDMA traffic. The outcome is attributed to RDMA’s lack of TCP-like burst transfers and its sustained high-rate traffic. Consequently, flowlet-based schemes are excluded from our considerations.

The determination of suitable load balancing granularity in RDMA prompts the question: what is the most effective choice? Drawing inspiration from the Justitia [26], we opt for sub-flow as the load balancing granularity, striking a balance between per-packet and per-flow granularity. In this approach, the flow is partitioned into $N$ sub-flows, each hashed to distinct paths for achieving load balancing. See $\S$III for detailed design.
Single-Path vs Multi-Path Transmission. RDMA operates on a point-to-point transmission model. Upon establishing a QP connection, RDMA flow follows a predetermined path. In the case of substantial data transfers, RDMA exclusively employs single-path transfers. Consequently, if the established path encounters a failure, the RDMA transport lacks the inherent capability to automatically reroute traffic to an alternative path for transmission.

Distributed deep learning exemplifies a common scenario demanding extensive data transmission within data centers. For instance, Facebook employs Gloo [27], a collective communication library, for model synchronization. The integration of RDMA transmission is a prevalent approach adopted to enhance the efficiency of large data transmission. In contemporary data center networks, the leaf-spine topology stands out as a widely utilized configuration, providing end-to-end path diversity between any server racks. Maestro[7] conducted experiments revealing that the application of distributed deep learning relying on point-to-point RDMA transmission falls short in fully leveraging the network’s link resources. Meanwhile, the completion of LLM training is determined by the slowest GPU transmission, so imbalanced transmission can greatly affect training efficiency [12]. In order to increase entropy, introduce more randomness, and improve load balancing efficiency, we choose a multipath transmission scheme, which also corresponds to the sub-flow granularity we have chosen. MP-RDMA [19] stands out as a notable multi-path load balancing scheme tailored for RDMA. However, a notable drawback is its dependency on a custom RNIC implementation using FPGA, rendering it incompatible with widely deployed commercial RNICs. Consequently, this restricts its applicability in contemporary, mature, large-scale data centers.

Therefore, in order to address Challenge #2 (avoiding packet out of order delivery) and Challenge #3, we design SeqBalance, a multi-path load balancing scheme designed to seamlessly integrate with commercial RNICs, to solve the problem of sequential delivery during multipath transmission and achieve non out-of-order multipath transmission.
Per-Hop vs End-to-End Multipathing. The final design consideration revolves around whether load balancing should be based on a per-hop or end-to-end approach. In the case of SeqBalance, we opt to implement load balancing on an end-to-end basis within the network. This involves executing routing decisions on the ToR switch ($\S$III.B) and storing path information within the switch. Storing path information in this manner facilitates end-to-end path selection throughout the network. Notably, the stored information pertains only to paths experiencing congestion, ensuring minimal memory usage and allowing the memory concerns of the switch to be negligible. In contrast to hop-by-hop multipath, end-to-end multipath offers superior traffic distribution across multiple paths, particularly in asymmetric topologies [28].

The key idea of SeqBalance is to avoid out-of-order delivery of packets as fine-grained as possible. Fig. 2 shows the system diagram of SeqBalance. Most of the function is located at the ToR switch. Load balancing decisions are primarily executed at the source ToR switch, relying on information stored in the Congestion Table. This table accumulates entries from the Congestion Packet transmitted by the destination ToR switch. Notably, the path stored in the Congestion Table restricts the entry of new flows.At the end-host, integration of the SeqBalance Shaper function module into the RDMA driver of RNIC achieves fine-grained flow control without necessitating other modifications.

The source ToR switch formulates load balancing decisions by analyzing the initial packet of each sub-flow. Path selection for subsequent packets is contingent upon their five-tuple, ensuring the continuity of using the same uplink. The destination ToR switch bears the responsibility of informing the source ToR switch about congested paths, designating them as bad paths, and updating the Congestion Table accordingly. Consequently, all load balancing decisions are made in-network without the need for end-host involvement.

Ideally, achieving per-packet level routing, as demonstrated by DRILL [14], would maximize network utilization. However, adopting per-packet level routing introduces a substantial risk of out-of-order delivery for packets, significantly impacting the performance of RDMA transmission. Simultaneously, the flowlet-based routing mechanism proves unsuitable for RDMA transmission due to challenges in identifying flowlets within RDMA traffic. So how can we design a load balancing mechanism that is both fine-grained and can avoid out-of-order delivery?

Our routing mechanism only involves source ToR switches and destination ToR switches, ensuring a straightforward deployment. Upon the arrival of the initial packet of a sub-flow (refer to $\S$III.C for sub-flow division details) at the source ToR switch, it undergoes hashing based on its five-tuple, determining a specific path. If the chosen path lacks congestion markings, all ensuing packets of the sub-flow traverse through this designated path. Conversely, if the path bears congestion markings, the hashing process is reiterated until an uncongested path is selected. It is noteworthy that only the first packet of the sub-flow undergoes hashing, with subsequent packets adhering to the established transmission path. This is because RDMA implements the go-back-N mechanism. Once the transmission path is changed in the sub-flow and the subsequent packet arrives first, retransmission will be enabled, which will greatly affect the performance. Below are the functions performed by the source ToR switch and the destination ToR switch in the routing decision.
Source ToR Switch. The source ToR switch receives the Congestion Packet from the destination ToR switch. Upon receipt of a Congestion Packet, the source ToR switch designates the associated path as inactive for a fixed duration, denoted as $\phi$. Subsequent receptions of a Congestion Packet marking the same path refresh the duration, restarting the timing from $\phi$. The path retains its inactive status until no Congestion Packet is received within the specified time $\phi$. When a sub-flow is hashed to a path marked with the inactive tag, it undergoes double hashing, and the path exclusively transmits sub-flows that are yet to be completed. The determination of the $\phi$ value is discussed in detail in $\S$IV.A.
Destination ToR Switch. The destination ToR switch receives packets from the path. Upon receiving a packet marked with Explicit Congestion Notification (ECN), indicating congestion flagged by the Spine switch during transmission, the destination ToR switch initiates the transmission of a Congestion Packet to the source ToR switch, conveying information about the path.

Given the constraints of RDMA transmission, where per-packet-level routing (prone to out-of-order delivery) and flowlet-based approaches are challenging to implement due to RDMA traffic characteristics, the question arises: How can fine-grained load balancing be accomplished?

To address this, we introduce SeqBalance Shaper, a functional module designed for flow splitting. Fig. 3 illustrates the schematic diagram of this module. SeqBalance Shaper is capable of subdividing a flow into $N$ sub-flows for transmission. In RDMA, WQE is considered a ”task description” containing detailed task information (e.g., sending data with a length of 10 bits at address 0x12345678 to the opposite node). Consequently, all packets associated with a WQE are transmitted as a unified flow. Our approach involves achieving flow splitting by segmenting the WQE. We achieve this by dividing the flow into $N$ sub-flows through the subdivision of the WQE into $N$ sub-WQEs.

Nevertheless, RDMA’s order-preserving transmission relies on PSN bits, and a shared QP connection utilizes the same set of PSN. In other words, even if the WQE is fragmented into $N$ sub-WQEs and transmitted through the same QP, when these sub-flows traverse the source ToR switch and are subsequently hashed to different paths for transmission, the issue of out-of-order delivery arises. The cause of this problem lies in the order-preserving transmission within the same QP. To address this, we mitigate the out-of-order transmission problem by directing sub-WQEs through distinct QPs. At the same time, this design also corresponds to the business traffic of AI training. The number of streams trained by AI is small and the entropy is low. At this time, appropriately splitting WQE into different QP transmissions will increase the entropy multiple times, greatly improving the randomness of AI training traffic and thus improving load balancing efficiency. However, QP represents a scarce resource in RNICs [20], prompting a discussion on the trade-off involving the segmentation of the number of sub-WQEs in $\S$III.D.

We implement SeqBalance’s load balancing decision on an Intel Tofino [22] programmable switch in $\sim$1000 lines of P4_16[29] code. And we implement SeqBalance Shaper in the RDMA driver with $\sim$1200 lines of C code.
Congestion Packet. The destination ToR switch communicates path congestion to the source ToR switch through the transmission of a Congestion Packet, temporarily restricting new flows from traversing the congested path. We designate 10 bits from the reserved field in the RDMA BTH as the PathTag field. This 10-bit PathTag field accommodates representation for 1023 paths within the topology, a sufficient number for conventional network topologies. For larger-scale network topologies, scalability is achievable by expanding the PathTag field’s bit count. Upon the passage of a packet carrying an ECN tag through the destination ToR switch, the switch mirrors the packet, adjusts the PathTag field to include path information, and subsequently transmits it back to the source ToR switch to signal path congestion. The source ToR switch deciphers the path congestion information from the packet header, marking it as inactive for a duration of $\phi$.
Completion Queue Element (CQE). In RDMA, CQE corresponds to WQE and can be perceived as a ”task report” returned to the software following hardware task completion. Fig. 4 illustrates the original RDMA’s CQE generation scheme. Upon task completion at the receiver end, an ACK is transmitted to the sender end, indicating task fulfillment. Subsequently, upon ACK reception, the sender generates a CQE to notify the application of task completion. However, SeqBalance Shaper introduces segmentation of WQE, leading to concerns about CQE generation. The upper-layer application dispatches the complete WQE without awareness of SeqBalance Shaper’s WQE segmentation, necessitating a comprehensive CQE. Consequently, addressing the challenge of generating a complete CQE after WQE segmentation becomes crucial.

Currently, two viable solutions exist: the first approach resembles the initial CQE generation scheme of RDMA, involving the transmission of a comprehensive ACK after receiving all sub-WQEs at the receiver, resulting in the generation of a complete CQE. The second solution entails each sub-WQE independently generating its ACK, and upon receiving all ACKs, the sender produces a complete CQE to inform the upper-layer application of task completion.

We have opted for the second solution to implement CQE. This choice is informed by the limitation of the first solution, where the receiver lacks awareness of WQE segmentation. In the first solution, the receiver is only tasked with performing operations upon the arrival of the WQE, without knowledge of whether the WQE is a standalone or a sub-WQE of a complete WQE. This lack of information makes the implementation of this scheme challenging. Conversely, the second solution allows us to ascertain the completion of all sub-WQEs at the sender. In other words, upon receiving ACKs for all sub-WQEs, the sender can then dispatch CQEs to the upper-layer application. Fig. 5 illustrates our CQE generation scheme, where each segmented WQE independently generates its ACK. The complete CQE is generated only after receiving all ACKs, signaling completion to the upper-layer application.

So how can we ensure that the ACKs of all sub-WQEs have been received? To address this, we have chosen to employ the Bitmap data structure for implementation. Bitmaps are utilized to keep track of received ACKs for sub-WQEs. In contrast to the receiver’s lack of information about WQE details, the sender possesses knowledge of the number of sub-WQEs the WQE is divided into, the specific QP each sub-WQE is transmitted through, and related information. Consequently, employing the Bitmap data structure allows for a straightforward realization of monitoring ACKs for all sub-WQEs. Subsequently, upon receipt of all ACKs for sub-WQEs, a CQE is transmitted to notify the upper layer that the task has been completed.
SeqBalance Shaper. We have integrated SeqBalance Shaper into the RDMA driver. As detailed in $\S$III.C, SeqBalance Shaper segments the WQE, thereby dividing the flow into sub-flows to achieve fine-grained load balancing. To prevent out-of-order packet delivery, SeqBalance Shaper allocates the segmented sub-WQEs to different QPs for transmission. However, exceeding a certain threshold in the number of QPs can lead to frequent context switching, resulting in cache misses and subsequent performance degradation (refer to $\S$V for an in-depth discussion). Therefore, achieving a balance between the number of divided sub-WQEs (i.e., the number of QPs), denoted as $N$, and load balancing performance is crucial. For businesses like AI training with fewer flows, increasing the number of $N$ can improve the performance of load balancing; If it is a business with a large number of flows, the number of $N$ can be reduced (in extreme cases, it can be taken as 1) to ensure the scalability of QP.

In Fig. 6, we illustrate the variation in FCT with $N$, ranging from 2 to 6. It is evident that as $N$ increases, both the average FCT and $99^{th}$ percentile FCT decrease. Specifically, when $N$ = 4, compared to $N$ = 2, the average FCT and $99^{th}$ percentile FCT show reductions of 25.4% and 27.6%, respectively. With $N$ = 6, compared to $N$ = 4, the reductions are 15.1% and 14.9%, respectively, but this comes at the cost of a 50% increase in QP resource consumption. Fig. 7 focuses on load balancing efficiency for different $N$, presenting the Cumulative Distribution Function (CDF) of throughput imbalance across the 8 uplinks for each ToR switch. Throughput imbalance[6] is defined as the maximum throughput (among the 8 uplinks) minus the minimum, divided by the average. We observe that when $N$ >4, the improvement in load balancing efficiency diminishes. Therefore, to strike a balance between $N$ performance and load balancing, we opt for a compromise value of $N$ = 4 in subsequent large-scale simulations and testbed experiments.

We first introduce the setup of the testbed experiments.
Network topology. By default, we employ the leaf-spine topology commonly observed in data center networks. The testbed topology comprises 2 leaf switches and 4 spine switches, as illustrated in Fig. 8 (a). Specifically, we utilize Tofino switches for the two leaf switches, while the four spine switches employ commodity Ethernet switches. Notably, each leaf switch is interconnected with 3 servers, and each server is equipped with a CX6 RNIC [21]. All network links are configured with a bandwidth of 40Gbps.
Schemes compared. We conduct a comparative analysis between SeqBalance and ECMP. In the case of SeqBalance, the parameter $\phi$ is configured based on the ECN marking threshold, where $\phi$ corresponds to the time required to refresh the threshold for the link rate. Considering our testbed’s 40Gbps link, and following the ECN threshold recommendation of 160KB from [31], we set the value of $\phi$ to 32$us$.
Traffic loads. We employ Perftest[32] to establish enduring connections among three client-server pairs. For bandwidth-intensive applications, we utilize ib_write_bw. This application employs 4 QPs, with each QP maintaining 128 outstanding 2MB WRITE requests to fully utilize the link bandwidth.
Flow congestion control. SeqBalance is a load balancing scheme for RoCE. For the traffic congestion control (CC) scheme of RNIC, we use DCQCN [4]. For the parameter configuration of DCQCN, we refer to the observations in [31] and use the following parameters $(K_{min},K_{max},P_{max})$ = (160KB, 520KB, 0.2). Additionally, all switches are configured with PFC. The retransmission strategy implemented on the RNIC is Go-Back-N.
Performance metrics. We use port rate as the main performance metrics. For port rate, we measure the ingress rate on the switches.
Load balancing efficiency. To assess load balancing efficiency, we gauge the port rates of switches along the four paths to depict the load distribution. We activate a server every 150$s$ to initiate traffic. In Fig. 10, the alterations in port rates for both schemes are illustrated over a 450$s$ period. The maximum load can reach 75% when all three servers are operational. Fig. 10(a) details the path rates under ECMP. In the initial 150$s$, only one server injects traffic, involving three paths. From 150$s$ to 300$s$, as the second server contributes traffic, the bandwidth of the third path saturates. Subsequently, between 300$s$ and 450$s$, with the third server adding traffic, the total rate across the three links only reaches 101.4Gbps. Hash collisions lead to load imbalance, triggering a decrease in CC rates. In contrast, SeqBalance achieves a relatively balanced load through finer-grained multi-path transmission and an in-network load balancing strategy. This effective approach avoids the rate decrease associated with CC activation.
Bandwidth overhead of Congestion Packet. In this subsection, we examine the bandwidth overhead incurred by Congestion Packets. SeqBalance dispatches Congestion Packets to the source ToR switch through the destination ToR switch to convey path congestion information. Bandwidth overhead is assessed by tracking the total count of Congestion Packets at the source ToR switch within a specific timeframe. Table II presents the bandwidth overhead of Congestion Packets for three different loads. Observations reveal that at a 25% load, no Congestion Packets are generated; at a 50% load, the bandwidth consumed by Congestion Packets is merely 4Kbps, which is nearly negligible; and at a 75% load, the bandwidth utilized by Congestion Packets is 0.05Gbps. This indicates that Congestion Packets occupy only a small fraction of the bandwidth compared to data transmission. Consequently, the presence of Congestion Packets does not impede normal data transmission.

Next, we evaluate SeqBalance in large-scale NS3 simulation.
Network topology. In line with the testbed experiments, our simulation adopts a leaf-spine topology. The simulated topology comprises 8 leaf switches and 12 spine switches, with each leaf switch connected to 16 servers, totaling 128 servers. All links operate at a rate of 100Gbps, and the delay is set to 1$us$.
Schemes compared. We conduct a comparative analysis involving SeqBalance, ECMP, LetFlow[17], CONGA [6] and DRILL[14]—state-of-the-art load balancing schemes for data center networks. Regarding the flowlet parameters of CONGA, we opt for a 100$us$ inactive time gap threshold. In the case of SeqBalance, given the variability in link rates and ECN thresholds, we configure the value of $\phi$ to be 32$us$.
Traffic loads. We employ the ALiCloud Storage and Web Search [6] workloads, with the load distribution depicted in Fig. 9. ALiCloud Storage traffic is predominantly concentrated in the 1KB-100KB range, primarily comprising small flows. In contrast, Web Search traffic exhibits a mix of small flows ($\textless$100KB) and large flows (>10MB). During flow generation, we randomly designate two out of the 128 servers as the client and server to initiate a flow, selecting the flow size based on the load distribution.
Flow congestion control. We employ the DCQCN as the CC scheme in NS3 simulations. Given the simulation’s link rate of 100Gbps, we set the parameters $(K_{min},K_{max},P_{max})$ to (400KB, 1600KB, 0.2), following the recommendations in [31]. Additionally, switches are configured with PFC.
Performance metrics. We employ FCT slowdown and throughput imbalance as the primary performance metrics. FCT slowdown represents the normalization of actual FCT to the optimal FCT achievable in an idle network.
Impact on FCT. We conduct simulations across traffic loads ranging from 30% to 80%. In Fig. 12, we present the results for two distinct workloads. Parts (a) and (c) display the average FCT for each scheme, while parts (b) and (d) depict the $99^{th}$-percentile FCTs for each scheme. All values are normalized to those achieved by ECMP.

We observe that under the AliCloud Storage workload, compared to other algorithms, SeqBalance reduced the average FCT by 16% and the $99^{th}$ FCT by 23.6%. Under WebSearch load, SeqBalance reduced the average FCT by 13.5% and the $99^{th}$ FCT by 27.6%. This is because ECMP is flow granularity, resulting in the worst performance due to hash conflicts. For CONGA and LetFlow, the performance is not as good as that used in TCP because it is almost difficult to detect flowlets in RDMA. DRILL triggers retransmission due to packet granularity, so FCT is much higher than the other four algorithms. At the same time, some data is not displayed in the graph due to being too high. SeqBalance achieves better load balancing performance through flow splitting and congestion aware routing.
Load balancing efficiency. In this subsection, we delve into the load balancing efficiency of SeqBalance. Fig. 13 illustrates the CDF of throughput imbalance across the uplinks for each ToR switch, specifically at the 80% load level for both workloads. We employ sampling at ToR switch ports every 100 $us$ to calculate throughput. The results reveal that, for both workloads, SeqBalance significantly outperforms ECMP and CONGA, except for DRILL. However, DRILL, relying on per-packet load balancing, resorts to packet spraying, inducing out-of-order packets and a sharp increase in FCT.
Impact on FCT with 3-tier topology. The preceding simulations were conducted on a 2-tier topology. In this subsection, we assess the performance of SeqBalance within a 3-tier topology. A 3-tier topology involves more extended end-to-end paths between servers, leading to prolonged response times. The 3-tier topology chosen is FatTree[33], comprising 16 core switches, 20 Agg switches, and 20 ToR switches. Each ToR switch is linked to 16 servers, totaling 320 servers. The link speed connecting the Agg switch and the ToR switch is 400Gbps, while all other links operate at 100Gbps. Consistent with [31], all link delays are set to 1 $us$. Other configurations mirror those employed in the 2-tier topology.

We excluded CONGA from consideration in the 3-tier topology due to its design tailored for 2-tier topology [6]. Our comparative analysis involves SeqBalance, ECMP, and LetFlow. Fig. 14 presents the FCT results for two workloads. It can be seen that in a larger scale 3-layer topology, SeqBalance also maintains its performance advantage. Under the ALiCloud Storage workload, SeqBalance reduced the average FCT by 18.7% and the $99^{th}$ FCT by 27.1%. Under WebSearch load, SeqBalance reduced the average FCT by 13.1% and the $99^{th}$ FCT by 33.2%.