FaiRTT: An Empirical Approach for Enhanced RTT Fairness and Bottleneck Throughput in BBR

In contrast to the initial promising results reported by Google on the effectiveness of BBRv1 (BBR-version 1), subsequent studies uncovered several issues in its behavior. Scholz et al. [5] identified the creation of long-standing queues during the startup phase, suppressing existing flows. Hock et al. [6] observed bandwidth discrepancies between elephant and mice flows sharing a bottleneck link due to the higher BDP of elephant flows. Scherrer et al. [7] found that in the presence of multiple flows managed by BBRv1, overestimated delivery rates lead to standing queues exceeding 1.5 times BDP, causing packet loss and unfair bandwidth shared among flows, particularly for small buffer sizes. To address the above concerns, Google introduced BBRv2 (BBR-version 2) [11] in 2018, incorporating explicit congestion notification (ECN) and packet loss rate. Zhang et al. [12] while evaluating various BBR versions, highlighted BBRv2’s improvements in flow fairness and enhanced coexistence with CUBIC and Reno. However, suboptimal channel utilization was noted with a 5% loss rate. Song et al. [13] reported fairness enhancements and reduced re-transmissions with BBRv2, emphasizing its effectiveness in limited buffer scenarios but noting challenges in fair bandwidth convergence with two flows entering a bottleneck link at different times. Nandagiri et al. [14] conducted an experimental evaluation comparing BBRv1 and BBRv2, revealing BBRv2’s ability to overcome inflight capacity limitations in networks with small buffers. However, in networks with large buffers, long RTT flows still consume more bandwidth.

Ma et al. [15] introduced the BBQ algorithm to enhance RTT fairness by minimizing probing periods for flows with extended round-trip times. While effective, BBQ’s performance degrades with an increasing number of competing flows. Kim et al. proposed solutions in [16] and [17] to limit BBRv1’s inflight capacity during the ProbeBW phase, addressing bias towards long RTT flows and preventing excessive data transmission. However, it causes a decrement in total throughput due to limiting inflight capacity to 1 BDP to prevent congestion. In [18], the authors presented the Delay-Aware BBR (DA-BBR) algorithm to improve RTT fairness and throughput using the relation of each flow’s RTT with RTprop. However, this algorithm was noted for slow and unstable convergence. Njogu et al. [8] introduced BBR-With Enhanced Fairness (BBR-EFRA), dynamically adjusting congestion window (CWND) based on buffer queue status for equitable competition among different RTT flows. Pan et al. [9] addressed the fairness and excessive re-transmissions in BBRv2 with the introduction of BBRv2+, utilizing flow-aware explicit congestion notification. However, these algorithms rely on accurate real-time queue estimation, which is challenging in dynamic network conditions.

Addressing these issues remains crucial for achieving equitable bandwidth allocation among different RTT flows without compromising on the throughput through a stable convergence. With the continuous updating of the BBR algorithm, BBRv2 can solve some fairness problems and limitations of BBRv1. BBRv3 (BBR-version 3) [19] has recently been released, aiming to rectify bugs and optimize performance parameters observed in BBRv2. While BBRv2 and BBRv3 present updates and improvements, experiments reveal persistent RTT unfairness. Thus, continued research is essential to enhance RTT fairness and intra-protocol fairness, providing potential refinements for the final version of BBR.

Given a network, the main objective of BBR is to maximize the utilization of the bottleneck link with a minimal delay. BBR estimates the bandwidth of the bottleneck link as the maximum observed delivery rate and the propagation delay as the minimum observed RTT over periodic intervals. The bandwidth of the bottleneck link is denoted as BtlBw, and the propagation delay is denoted as RTprop. However, sending more traffic (probing more bandwidth) to estimate BtlBw may cause congestion at the bottleneck link, thus increasing RTprop and vice-versa. BBR, therefore, estimates these values separately in its different execution phases.

In Fig. 1, we present the flow diagram of BBRv2. We provide the meaning of mathematical notations used in the BBRv2 algorithm in Table I. The BBRv2 algorithm is composed of four phases: Startup (BBR_START_UP), Drain (BBR_DRAIN), Probe Bandwidth (BBR_PROBE_BW), and Probe RTT (BBR_PROBE_RTT). The first phase (BBR_START_UP) adapts the exponential Startup behavior from CUBIC by doubling the sending rate with each RTT. The pacing_gain is set to $2.89$. If the increase in the sending rate does not exceed 25% for three consecutive attempts or if packet loss or an ECN-marked rate surpasses a predefined threshold (e.g., 2%), BBRv2 assumes to have reached the bottleneck bandwidth. Since this observation is delayed by one RTT, a queue was already created at the bottleneck. In the second phase (BBR_DRAIN), BBRv2 tries to drain the queue by temporarily reducing the pacing_gain to approximately $0.34$. Afterwards, BBRv2 enters the third phase (BBR_PROBE_BW), in which it probes for more available bandwidth. BBRv2 specifies exit conditions for each stage of the third phase (ProbeBW Refill, ProbeBW Up, ProbeBW Down, and ProbeBW Cruise). BBRv2 initiates bandwidth probing by linearly augmenting the amount of inflight data over one $RTprop$ in the ProbeBW Refill stage, effectively filling the network pipe. Subsequently, BBRv2 engages in the ProbeBW Up phase, during which it increases the inflight data swiftly to probe for more bandwidth. This increase of inflight data ceases when the volume reaches 1.25 times the BDP, or when packet loss or the ECN rate surpasses a predetermined threshold. BBRv2 establishes inflight_hi when confronted with a higher loss/ECN rate than the threshold, ensuring that it does not surpass the operational point where excessive packet loss may occur. Upon concluding the ProbeBW Up phase, BBRv2 transitions to the ProbeBW Down phase to offset the queue. It exits this phase once the volume of inflight data becomes less than BDP and maintains a constant delivery rate in ProbeBW Cruise until the initiation of the next bandwidth probing cycle. If BBRv2 experiences packet loss in the ProbeBW Cruise, it uses inflight_lo to cope with the temporary packet loss. BBRv2 continuously samples the bandwidth for BDP estimation using a time window basis. At the time window $t$, $\texttt{BtlBw}_{t}$ is defined as follows:

where $maxBw_{t}$ is the maximum delivery rate captured from the socket within time window $t$, normally set to two times the probe bandwidth phase. It is to be noted that $\texttt{BtlBw}_{t}$ is measured only for the applications with no rate limit, indicated by the flag RATE_LIM_APP = FALSE.

After not measuring a new RTprop value for 5 seconds, BBRv2 stops probing for bandwidth and enters the last phase (Probe RTT). During this phase, the bandwidth is reduced to half of BDP to drain any possible queue and get a real estimation of the RTT. This phase is kept for 200 ms, controlled by the flag PROBE_INT_EXP. If a new minimum value is measured, $\texttt{RTprop}_{t}$ is updated as follows:

where $lastRTT_{t}$ is captured from the network whenever an ACK packet is received.

The BBRv2 algorithm estimates the amount of data to be transmitted based on the current BDP. The BDP is in turn calculated by the product of the bottleneck bandwidth at time window $t$ and the estimated RTT of the flows propagating in that time window, $\texttt{RTprop}_{t}$.

When multiple flows with diverse RTT coexist over a shared bottleneck link, BBR tends to transmit excess data, approximately $1.5$ times the intended amount [5]. Consequently, BBR deviates from its typical operating point, leading to a total transmission rate exceeding the available bandwidth of the bottleneck link. This results in a persistent queue on the bottleneck link which is shared among all flows — predominantly favoring the elephant flows, causing bandwidth unfairness.

Elephant flows, with higher BDP values, transmit more packets and utilize more bottleneck buffers. Our experimental analysis reveals that the BDP value of mice flows becomes smaller than that of elephant flows. Consequently, mice flows inject fewer packets into the pipeline, reducing the delivery rate. The decrease in delivery rates amplifies the dominance of elephant flows, leading to increased throughput imbalance, higher packet loss, re-transmission rates, and elevated latency. In BBRv1, mice flows were found to be constrained by the congestion window, thereby reducing their transmission rates. BBRv2 aimed to mitigate this issue by using ECN and setting of inflight upper and lower bound. However, in  [13], the authors reported that BBRv2 still experienced significant issues with coexistence between different RTT flows particularly with large buffers. Furthermore, in [20], the authors revealed that BBRv3 continues to exhibit RTT unfairness. Malicious actors can exploit this vulnerability, deliberately increasing delay to gain a disproportionately higher bandwidth share.

To address the inherent design issue of BBR, we develop Algorithm 1, which dynamically captures the changes in RTT and adaptively adjusts inflight for each BBR flow request. The proposed algorithm aims to improve RTT fairness while maintaining the overall throughput of the network. We leverage the calculated RTT from each acknowledgment packet (lastRTT) and minimum RTT (minRTT) to calculate an adjustment coefficient for curtailing the amount of inflight data for each flow. The inflight is defined as follows:

where $cwnd\_gain$ is fixed at $0.5$ for the BBR_PROBE_RTT phase and at $2$ for all other phases of BBRv2. While all BBR flows exceed their intended inflight data transmission, the magnitude differs, particularly with elephant flows transmitting significantly larger volumes. To account for this, we examine the relationship between lastRTT and minRTT, introducing an adjustment coefficient with a value below $1$. This coefficient will approach $1$ for mice flows, while for elephant flows it would have a smaller value. The BDP calculation incorporates the adjustment coefficient, dynamically adjusting inflight data based on RTT. The adjustment coefficient is multiplied with the BDP only when the value of lastRTT is more than the RTT fairness threshold $\alpha_{t}$, calculated as:

where

and

where $|\mathbb{R}\texttt{x}|$ and $|\mathbb{T}\texttt{x}|$ are the size of the receiving window and sending window , respectively. $Wfcount_{t}$ is the estimated number of BBR flows in the time window $t$ calculated by counting the number of unique $minRTT$’s in that time window, and $\beta$ is a constant fixed at 0.8 to balance the estimate. Further, we introduce a discount factor $\gamma$ of $0.99$ to stabilize BDP and promote equitable bandwidth-sharing. Elephant flows exhibit a lower adjustment coefficient than mice flows, resulting in a more substantial reduction in elephant flow’s BDP compared to that of mice flows. Consequently, by transmitting less data than the original BBR, elephant flows experience a significant reduction in buffer occupancy, thereby enhancing RTT fairness.

We used a dumbbell topology setup in the NS-3 simulation framework, featuring multiple flows sharing a common bottleneck link, as illustrated in Fig. 2. We applied the default active queue management as the Drop-Tail policy and every packet size as 1 kB. Each simulation lasted 120 seconds and was repeated 5 times with different random seeds for the error rate in packet delivery. The obtained results were plotted with 95% confidence interval.