SIRD: A Sender-Informed, Receiver-Driven Datacenter Transport Protocol

Congestion can occur in Host-to-ToR uplinks and the network core, which are shared among receivers, and ToR-to-host downlinks, which are exclusive to a single receiver.

RD schemes excel in managing incast traffic because each receiver explicitly controls the arrival rate of data (Montazeri et al., 2018; Gao et al., 2015; Bai et al., 2015; Handley et al., 2017; Cho et al., 2017; Cai et al., 2022). This level of control also allows RD schemes to precisely dictate which message should be prioritized in downlinks, and can even factor-in a server’s application requirements (Ousterhout et al., 2019). Recent work has delivered significant message latency gains by scheduling messages based on their remaining size (SRPT policy) (Hu et al., 2020; Montazeri et al., 2018; Gao et al., 2015). In contrast, SD schemes (Alizadeh et al., 2010, 2012, 2013; Liu et al., 2021; Kumar et al., 2020; Li et al., 2019; Mittal et al., 2015) treat downlinks as any other link. Senders must first detect downlink congestion and then independently adjust their rates/windows such that the level of congestion falls below an acceptable target.

When core congestion occurs, multiple flows with unrelated senders and receivers compete for bandwidth at the network core. Note that the core may have multiple tiers or be configured differently but is fundamentally shared infrastructure. Congestion at the core of a fabric is less common than at downlinks (Montazeri et al., 2018; Singh et al., 2015; Ghabashneh et al., 2022) but can still occur due to core network oversubscription. Oversubscription can be permanent to reduce cost (Singh et al., 2015) or transient due to component failures. Congestion at the core can also occur as a consequence of static ECMP IP routing decisions that cause multiple flows to saturate one core switch and one core-to-Tor downlink, while other core switches have idle capacity (Greenberg et al., 2009; Alizadeh et al., 2014).

Uplink congestion occurs due to fan-out of multiple flows to different receivers or due to bandwidth mismatch between the sender uplink and the receiver downlink. Because the resulting packet buffering is in hosts and not in the fabric, sender congestion is generally a less severe problem. For RD schemes, uplink congestion is known as the unresponsive sender problem and leads to degraded throughput as independent scheduling decisions of receivers may conflict, wasting downlink bandwidth. Whereas the term unresponsive has been used in the context of SRPT scheduling, where messages are transmitted to completion, we will use congested sender as a general description of uplink congestion.

RD protocols have proposed various mechanisms for overcoming the tension between independent receiver scheduling and the fact that some links are shared. One of the earliest designs, pHost (Gao et al., 2015), employs a timeout mechanism at receivers to detect unresponsive senders and direct credit to other senders. NDP (Handley et al., 2017) employs custom very shallow buffer switches and eagerly drops packets, using header trimming to recover quickly. NDP receivers handle uplink sharing by only crediting senders as long as they transmit data packets - whether dropped or not. pHost and NDP do not explicitly deal with core congestion and, further, the analysis by Montazeri et al. (Montazeri et al., 2018) showed that neither of the two achieves high overall link utilization. Homa (Montazeri et al., 2018) introduced controlled overcommitment in which each receiver can send credit to up to $k$ senders at a time. Homa achieves high utilization as it is statistically likely that at least one of the k senders will respond. However, it trades queuing for throughput by meaningfully increasing the amount of expected inbound traffic to each receiver. To let small messages bypass long network queues, Homa relies on Ethernet priorities which are normally used for application-level QoS guarantees (Arslan et al., 2023; Kumar et al., 2020; Hoefler et al., 2023). dcPIM (Cai et al., 2022) employs a semi-synchronous round-based matching algorithm where senders and receivers exchange messages to achieve a bipartite matching. This coordinates the sharing of sender uplinks but congestion at the core is only implicitly addressed by the protocol’s overall low link contention. The downside of this link-sharing approach is message latency. dcPIM delivers small messages quickly by excluding them from the matching process. However, messages larger than the bandwidth-delay-product (BDP) of the network must wait for several RTTs before starting transmission. ExpressPass (Cho et al., 2017) manages all links, exclusive and shared, via a hop-by-hop approach which configures switches to drop excess credit packets, which in turn rate limits data packets in the opposite direction. To reduce credit drops, ExpressPass uses recent credit drop rates as feedback to adjust the future sending rate, and in this way also improves utilization and fairness across multiple bottlenecks. Out of the designs discussed so far, ExpressPass is the only one that explicitly manages core congestion and can operate with highly oversubscribed topologies. ExpressPass’s hop-by-hop design helps it achieve near-zero queuing (Cho et al., 2017), but is more complex to deploy and maintain due to its switch configuration and path symmetry requirements, and suffers under small-flow-dominated workloads (see §5.3).

Congestion control protocols generally depend on buffering to offset coordination and control loop delays and, as a result, face a throughput-buffering trade-off. Maximum bandwidth utilization can trivially be achieved with high levels of in-network buffering, but at the cost of queuing-induced latency and expensive dropped packet retransmissions. Conversely, low buffering can lead to throughput loss for protocols that are slow in capturing newly available bandwidth.

High-speed packet buffering is handled by low latency and relatively small SRAM buffers in switch ASICs. Figure 1 shows that the size of these buffers is not increasing in size as fast as the bisection bandwidth of switch ASICs. For example, nVidia’s top-end Spectrum 4 ASIC (rightmost point in Figure 1) has a $160$MB buffer, which corresponds to $3.13$MB per Tbps of bisection bandwidth (Nvidia, 2022). The previous $12.8$Tbps and $6.4$Tbps top-end Spectrum ASICs were equipped with $5$MB and $6.6$MB per Tbps respectively (Mellanox, 2020, 2017). Unfortunately, future scaling of SRAM densities appears unlikely given CMOS process limitations (Chang et al., 2022; David Schor, 2022). In parallel, datacenter round-trip times (RTT) are not falling as they are dominated by host software processing, PCIe latency, and ASIC serialization latencies. Consequently, CC protocols must handle higher BDPs with less switch buffer space at their disposal.

To better absorb instantaneous bursts of traffic, switch packet buffers are generally shared among egress ports. Some ASICs advertise fully-shareable buffers while others implement separate pools or statically apportion some of the space to each port (Ghabashneh et al., 2022). On top of the physical implementation, various buffer-sharing algorithms dynamically limit each port’s maximum allocation to avoid unfairness (Alizadeh et al., 2010). However, if a large part of the overall buffer is occupied, the per-port cap becomes more equitable, limiting burst absorbability (Ghabashneh et al., 2022).

Figure 2 provides some insight into whether existing congestion control schemes are future-proof given these hardware trends by measuring Homa’s buffer occupancy distribution as a function of load for the Websearch workload (Alizadeh et al., 2013). Hosts generate flows following the Poisson distribution but traffic patterns are often more bursty (Zhang et al., 2017) and thus challenging (full methodology in §5.1). To achieve best-in-class message latency and link utilization Homa uses Ethernet priorities and overcommitment of downlinks, the latter drastically increasing queuing in the fabric. Figure 2 compares the buffer occupancy distribution (assuming infinite buffers in the experiment) with the actual resources of two leading ASICs, adjusted to the port radix of the simulation. The simulated ToR switches have $16$ $100$Gbps downlinks to servers (Alizadeh et al., 2013) and $4$ $400Gbps$ uplinks for a total bisection bandwidth of $3.2$Tbps. Using the Spectrum 4 buffer-bandwidth ratio of $3.13$MB per Tbps corresponds to $0.31$MB of buffer per $100Gbps$ port (Figure 2 left) and a $10$MB total ToR buffer (right). In practice, the buffer is neither partitioned nor fully shared; however, between the two extremes, Homa operates with more than the per-port buffer and is close to overflowing a fully shared buffer. In the latter case, the presence of persistent queuing also limits the flexibility of the buffer-sharing algorithm.

SIRD defines three main packet types:

1. (1)

   DATA: a packet that contains part of a message payload. There are two types of DATA packets: scheduled, that are sent after receiving credit, and unscheduled.

2. (2)

   CREDITREQ: a control packet sent by a sender to a receiver to indicate that the former wants to send scheduled data, either an entire message if $msg\_size>UnschT$, or else just the scheduled portion of a message smaller than $UnschT$.

3. (3)

   CREDIT: a control packet sent by the receiver to the sender to schedule the transmission of one DATA packet.

Generally, the flow of packets consists of credit flowing from receivers to senders and data flowing in the opposite direction. Credit leaves the receiver in a CREDIT packet, is used by the sender, and returns in a scheduled DATA packet.

Each SIRD receiver maintains two types of credit buckets that limit the amount of credit it can distribute: a global credit bucket and per-sender credit buckets.

The global credit bucket $B$ controls overcommitment by capping the total number of outstanding credits that a receiver can issue. Configuring $B\geq BDP$ is necessary to ensure maximum throughput. Further, $B$ bounds the queuing length from scheduled messages to $B-BDP$ bytes at the ToR’s downlink. Our experiments show that a value of $B$ as low as $1.5\times BDP$ is sufficient to ensure high link utilization in challenging workloads.

Each receiver also maintains a credit bucket per sender machine it communicates with. The per-sender credit bucket caps the number of outstanding credits the receiver can issue to a sender. Informed overcommitment is implemented by adjusting the size of the per-sender credit bucket according to the level of congestion at the core and the sender (max $1\times BDP$). Reducing the bucket size means that less of the receiver’s total credit can be allocated to a congested source or path, and thus, more is available for other senders.

Unlike protocols that manage congestion on a per-flow basis, SIRD does so per sender. In SIRD, all flows will face the same level of congestion since it performs packet spraying. Thus, SIRD receivers aggregate congestion signals from all flows and can react faster. From an implementation perspective, receivers need to keep less state, which is beneficial especially for a hardware implementation.

The informed overcommitment control loop uses two input signals that communicate the extent of congestion in the network and at senders. The signals are carried in DATA packets and are the ECN bit in the IP header set by the network and a bit in the SIRD header set by the sender. Each receiver runs two separate AIMD (additive-increase, multiplicative decrease) control loops and uses the most conservative of the two (similar to Swift (Kumar et al., 2020)) to adjust per-sender credit bucket sizes. Each control loop is configured with its own marking threshold ($NThr$ and $SThr$). $NThr$ should be set according to DCTCP best practices (Alizadeh et al., 2010) to limit queuing in the network core. Note that $NThr$ is much higher than the allowed persistent queueing at ToR switches as controlled by $B$. Thus, ToR switches never have to mark ECN.

$SThr$ should be set to limit the amount of credit a sender can accumulate, thus allowing receivers to efficiently distribute their limited aggregate credit. Intuitively, $SThr$ determines the level of accumulated credit the control loop is targeting when a sender is congested, i.e., receiving credit faster than it can use it. Setting it too high means each sender can accumulate substantial amounts of credit, and consequently, B would need to be configured higher to increase aggregate credit availability. Conversely, a very low value of SThr does not allow the control loop any slack when converging to a stable state and can cause throughput loss.

To understand the relationship between $SThr$ and $B$, we analytically examine the congested-sender scenario from the perspective of a receiver R to find: how much total credit (B) R needs when receiving from $k$ congested senders to still have enough available to be able to saturate its downlink. Assuming each congested sender sends to $f$ receivers in total, the available uplink bandwidth for R is $BW/f$. Assuming uniform link speeds, we are interested in the case where senders are the bottleneck and cannot saturate R, or:

In this case, each of the $k$ senders accumulates up to $SThr$ credit in stable state, or $SThr/f$ from each receiver assuming an equal split (see §4.5). Therefore, R’s B must be large enough to allow $1xBDP$ of credit to be in flight (for any new senders) despite accumulation at congested senders, or:

If $k_{max}$ is the number of congested senders that maximizes the sum term, then $f_{max}=k_{max}+1$ (denominator) and the maximum value of the term is:

It follows that, in steady state, R can account for any number of congested senders as long as $B\geq BDP+SThr$. Under dynamic traffic patterns (see §5.5) higher values of B can help increase overall utilization. Further, policies other than fair sharing can loosen this property. For example, if senders prioritize some receivers, then, in the worst case where R is de-prioritized by all k senders, Equation 2 loses $f$ from the denominator and B depends on the number of worst-case congested senders.

Algorithm 1 describes the actions of a SIRD receiver. Assuming credit is available in the global bucket, the receiver tries to allocate it to an active sender (ln. 9). It first, selects one of the senders with available credit in the per-sender bucket (ln.10) based on policy (ln.11). The receiver sends a CREDIT packet (ln. 13) and reduces the available credit both in the global bucket, the per-sender bucket, and the remaining required credit for that message (ln. 14). Whenever a DATA packet arrives (ln. 2), if it is scheduled, the receiver replenishes credit in the global bucket (ln. 4) and the per-sender bucket (ln. 5). Then it executes the two independent AIMD control loops to adapt to a congested sender (ln. 6) and a congested core network (ln. 7), respectively. The receiver sets the size of the per-sender bucket as the minimum of the two values determined by the control loops (ln. 10).

Algorithm 2 describes the implementation of the sender-side algorithm for scheduled DATA packets. A host can send data to a receiver as long as it has available credit for said receiver. Congested senders mark the congested sender notification (sird.csn) bit if the total amount of accumulated credit exceeds $SThr$ (ln. 8). The sender can send unscheduled DATA packets at any point in time. SIRD senders naturally handle scenarios where a meaningful portion of uplink bandwidth is consumed by unscheduled packets because they limit credit accumulation and thus adapt the transmission rate of scheduled packets to the leftover bandwidth.

SIRD can be configured to schedule for fairness e.g., by crediting messages in a round-robin manner, or for latency minimization, e.g., by crediting smaller messages first (SRPT), or to accommodate different tenant classes. SIRD implements policies at the receiver (ln .11), which is the primary enforcer, and at the sender (ln. 7). By minimizing queuing in the fabric, SIRD does not need to enforce policies there, simplifying the design. Regardless of which policy is configured at senders, SIRD allocates part of the uplink bandwidth fairly across active receivers, as to ensure a regular flow of congestion information between sender-receiver pairs.

SIRD receivers pace credit transmission to match the downlink’s capacity for data packets. Pacing helps further reduce downlink queuing but is not needed for correctness.

Whenever possible in the datacenter, CREDITREQ and CREDIT packets are sent at a higher network priority to further reduce RTT jitter. The unscheduled prefixes of messages $<UnschT$ also use it. Using one higher priority level for control and prefix packets has minimal impact on the overall traffic and should not affect other operations. SIRD does not require priorities to deliver high performance (§5.5).

Packet loss in SIRD will be very rare by design, but the protocol must nevertheless operate correctly in its presence. Ethernet packets may be dropped for reasons other than congestion, e.g., CRC errors or packet drops due to host, switch, or link-level restarts. SIRD employs Homa’s (Montazeri et al., 2018) retransmission design which uses timeouts at the receiver and the sender to detect packet loss.

We implement SIRD on ns-2 (Altman and Jimenez, 2012), reuse the original ns-2 DCTCP implementation (using the same parameters as (Alizadeh et al., 2013), scaled to 100Gbps) and the original ns-2 ExpressPass implementation (Cho et al., 2018), and port the published Homa simulator (Montazeri, 2019) to ns-2 using the same parameters as in (Montazeri et al., 2018) scaled to 100Gbps. The published Homa simulator does not implement the incast optimization (Montazeri et al., 2018), which further relies on two-way messages. We use a Swift simulator, kindly made public (Abdous et al., 2021), and configure its delay parameters to achieve similar throughput to DCTCP, respecting the guidelines (Kumar et al., 2020). Finally, we use the published dcPIM simulator (Cai, 2023).

Table 2 lists protocol parameter values. DCTCP and Swift use pools of pre-established connections (40 for each host pair). SIRD approximates SRPT scheduling like Homa and dcPIM. SIRD, Homa, and dcPIM use packet spraying, DCTCP and Swift use ECMP. Homa uses 8 network priority levels, dcPIM 3, and SIRD 2 (§4.6). SIRD does not require priorities to operate well; we explore this in §5.5. The initial window of DCTCP and Swift is configured at $BDP$ like in (Alizadeh et al., 2013).

We simulate these 3 workloads on 3 traffic configurations, for a total of 9 points of comparison. The configurations are: (1) Balanced: The default configuration described above. We vary the applied load, which does not include protocol-dependent header overheads, from 25% to 95% of link capacity. (2) Core: Same as (1) but ToR-Spine links are 200Gbps (2-to-1 oversubscription). Due to uniform message target selection, $128/144\approx 89\%$ of messages travel via spines, turning the core into the bottleneck. We consequently reduce the load applied by hosts by $\times 0.89*2$ to reflect the network’s reduced capacity. This is not meant to reflect a permanent load distribution, but we hypothesize that it is possible transiently. (3) Incast: We use the methodology of (Li et al., 2019; Cai et al., 2022) and combine background traffic with overlay incast traffic: 30 random senders periodically send a 500KB message to a random receiver. Incast traffic represents 7% of the total load.

We report the goodput (rate of received application payload), total buffering in switches, and message slowdown, defined as the ratio between the measured and the minimum possible latency for each message. In the incast configuration, we exclude incast messages from slowdown results.

Figure 4 shows how protocols navigate the trade-offs between throughput, buffering, and latency by plotting their relative performance across all 9 workload-configuration combinations. The best-performing protocol on each of the 9 scenarios gets a score of $1.0$ (per metric) and the others are normalized to it, so that goodput is always $\leq 1.0$ whereas queuing and slowdown are always $\geq 1.0$. We report the highest achieved goodput and the highest peak queuing over all the levels of applied load. We report slowdown at 50% applied load which is a level most protocols can deliver in all scenarios. The following figures and calculations do not include cases where a protocol cannot satisfy a specific load level, or cannot stop network buffers from growing infinitely.

Overall, SIRD is the only protocol that consistently achieves near-ideal scores across all metrics. Specifically, SIRD causes $12\times$ less peak network queuing than Homa and achieves similar latency and goodput performance. SIRD outperforms dcPIM in message slowdown, and peak goodput and queuing by $46\%$, $9\%$, and $43\%$ respectively. ExpressPass causes practically zero queuing thanks to its hop-by-hop design, and 88% less than SIRD, but SIRD delivers $10\times$ lower message slowdown and $26\%$ more goodput. SIRD never causes more than 0.8MB and 2.3MB in ToR queuing on receiver- and core-bottleneck scenarios respectively, which maps to 8% and 23% of the packet buffer size assuming 3.13 MB/Tbps (§2.2).

We now zoom in on how each protocol manages congestion across different levels of applied stress. Figure 5 plots maximum buffering across ToR switches as a function of achieved goodput for balanced (top), core (middle), and incast (bottom) configurations. The reported goodput is the mean across all 144 hosts and reflects the rate of message delivery to applications. ToR queuing covers both downlinks and links to aggregation switches (core).

Overall, SIRD consistently achieves high goodput while causing minimal buffering across load levels, even under high levels of stress. Through informed overcommitment, SIRD makes effective use of a limited amount of credit and achieves high goodput while reducing the need for packet buffer space by up to $20\times$ compared to Homa’s controlled overcommitment. Further, SIRD displays a more predictable congestion response compared to dcPIM and ExpressPass. As expected, based on 4(a), DCTCP and Swift cause meaningful buffering without achieving exceptional goodput. Mean queuing is qualitatively similar (appendix Figure 12).

dcPIM and ExpressPass sometimes compare favorably to SIRD but only when the workload mean message size is not small. They both struggle in terms of achieved goodput and buffering in WKa for all three configurations (left column). In WKa, $\approx 99\%$ of messages are smaller than $1\times BDP$ and are responsible for $\approx 40\%$ of the traffic. ExpressPass’s behavior is known and discussed in (Cho et al., 2017): more credit than needed may be sent for a small message which may then compete for bandwidth with productive credit of a large message. For dcPIM, the likely reason as discussed in (John K. Ousterhout, 2022b) is that to send a small message, a sender has to preempt the transmission of a larger message which can cause the receiver of the latter to remain partially idle. SIRD’s behavior is consistent across workload message sizes thanks to slight downlink overcommitment which absorbs small discontinuities in large message transmission.

In the core configuration (Figure 5 - middle row), where the core is the bottleneck, we observe that SIRD’s reactive congestion management, despite sharing the same ECN-based mechanism as DCTCP, achieves steadier behavior because it limits the number of bytes in the network. Homa grows ToR uplink queues aggressively as it does not explicitly manage core queuing which is only limited by the aggregate overcommitment level of receivers. The same applies to dcPIM but because it does not overcommit, queuing is much lower.

Last, the incast configuration (bottom row) illustrates the relative advantage of RD schemes over DCTCP and Swift.

Figure 6 shows the median and $99^{th}$ percentile latency slowdown for different message size ranges across configurations at 50% applied load (except WKb results which fall between the other two and can be found in appendix Figure 11). Across each workload as a whole (rightmost bar cluster ”all”), SIRD is generally on par with Homa, and often outperforms dcPIM in terms of tail latency. For small, latency-sensitive messages $<BDP$ (100KB - groups A and B), SIRD, dcPIM, and Homa offer close to hardware latency. Both in aggregate and for small messages, DCTCP and Swift perform an order of magnitude worse at the tail because they cause meaningful buffering without having a bypass mechanism like Homa.

For messages larger than $1\times BDP$ (groups C and D), SIRD comes closest to near-optimal Homa and strongly outperforms the other protocols. SIRD achieves up to $4\times$ lower latency than dcPIM in this size range because it does not wait for multi-RTT handshakes before sending a message as discussed in §3. Note that 6(b) is comparable to Figure 3d in (Cai et al., 2022). Similarly, SIRD outperforms ExpressPass on latency because it does not take multiple RTTs to capture the full link bandwidth and it implements SRPT.

The differences in latency for protocols other than dcPIM and ExpressPass, which wait before ramping up transmission, can be explained through their effective scheduling policies. That is, assuming that a protocol manages to deliver enough messages to achieve 50% goodput, lowering latency is a matter of appropriately ordering message transmission (when there are no conflicting application concerns). For example, perhaps surprisingly, DCTCP achieves equivalent or better latency than Swift in all but the incast scenarios. We argue that this is because Swift is better at fairly sharing bandwidth between messages thanks to its tighter control loop which converges faster. Latency-wise, fair sharing is inferior to FIFO or SRPT since it delays the completion of individual messages in favor of equitable progress. Along the same lines, SIRD cannot always match the latency of Homa because it approximates SRPT less faithfully as: (1) unlike Homa, a portion of the sender uplink is fair-shared (50% in this case). Note that Homa’s Linux implementation does the same to avoid starvation (Ousterhout, 2021). (2) To avoid credit accumulation in congested senders, informed overcommitment adjusts per-sender credit bucket sizes equitably.

In summary, SIRD delivers messages with low latency, outperforms dcPIM, ExpressPass, DCTCP, and Swift, and, without relying on Ethernet priorities, is competitive with Homa, which nearly optimally approximates SRPT (Montazeri et al., 2018).

In this section, we explore SIRD’s sensitivity to its key parameters: $B$, $SThr$, and $UnschT$, as well as to the availability of Ethernet priorities. SIRD’s sensitivity to NThr is the same as DCTCP’s to $K$ (Alizadeh et al., 2010).

We observe that the presence of informed overcommitment increases goodput by $\sim$$25\%$, confirming that the introduced sender-informed mechanism is necessary to achieve high throughput with a limited amount of credit, and thus low queuing. When the mechanism is disabled ($SThr=\inf$), there is stranded and unused credit at congested senders, which prevents receivers from achieving the full rate. With the mechanism enabled, the curves asymptotically converge to the same maximum goodput of 90Gbps. Reaching the plateau with a lower value of $B$ also demands lowering $SThr$ to reduce credit stranded at senders (Equation 2). Queuing increases with B similar to Figure 3 and remains stable when varying SThr. Finally, we observed that the median and $99^{th}$ percentile workload slowdown for $SThr=0.5$ range within $[1.25,1.31]$ and $[3.53,5.76]$ respectively, positively correlated to B. We selected $B=1.5\times BDP$ rather than $B=2.0\times BDP$ for our experiments because it halves expected buffering from $1\times$ to $0.5\times BDP$ ($=B-BDP$). We do not configure $SThr$ lower than $0.5\times BDP$ as we deem it unrealistic to implement in a real system because it may cause unwanted marking due to batch credit arrivals or processing loop variability.

The default value of $UnschT=1\times BDP$ is a satisfying compromise as large values do not yield latency benefits, yet can unnecessarily expose the fabric to coordinated traffic bursts (e.g., incast). We confirm this by running the same workload under the incast configuration which has a high concentration of $5\times BDP$ message bursts. We observe the following performance degradation when comparing $UnschT=4\times BDP$ to $UnschT=16\times BDP$: Overall $99^{th}$ percentile slowdown increases by 34% while maximum and mean ToR queuing increases by $5.7\times$ and $1.8\times$ respectively. These results justify our decision to introduce a size threshold above which messages are entirely scheduled.