ECN based Congestion Control for a Software Defined Network

The DECbit mechanism [10] is one of the first works that used an explicit congestion control protocol to signal the end nodes about congestion at a router. An router, when it is likely to experience congestion, reacts by marking a bit at the packet header for some of the packets. This bit was then sent by the receiver to the sender for taking appropriate action.

The Random Early Detection (RED) mechanism [8] attempts to maintain an average queue length at the routers. Using threshold values for time-averaged queue lengths, a router drops packets with a probability that increases with the size of the queue. This packet drop results in TCP sources reducing their congestion window and hence transmission rate.

In the Explicit Congestion Notification (ECN) mechanism [2], packets will be marked with Congestion Encountered (CE) bits instead of being dropped. The end-node TCP protocol is modified so that the receiver echoes the CE bits to the sender. When the sender receives such TCP segments, it reacts by reducing the congestion window. TCP’s performance can be increased significantly using RED/ECN by setting the parameters with caution. The DCTCP protocol [5] is a variant of TCP optimized for data center networks. DCTCP uses Explicit Congestion Notification (ECN) at the switches and runs its variant of TCP at the end nodes. In [6], another ECN-based scheme that does not require end-host TCP modification is presented for data center networks.

In all these approaches, the decision to drop or mark packets during congestion is taken in a distributed manner by each router. In an SDN-enabled network, it is possible to take advantage of the (logically) centralized control plane to set the bits based on the global view. This paper presents an attempt to explore the advantages of pushing this decision to the SDN controller rather than doing it at the routers.

OpenTCP [1] is a congestion control framework proposed for SDN based networks. The OpenTCP software at the controller collects information about the underlying network such as link utilization values. OpenTCP aims at reducing the Flow Completion Times (FCTs) by updating initial congestion window and retransmission timeout interval for TCP flows. OpenTCP sends Congestion Update Epistles (CUE) packets to the end nodes consisting of information about the suggested changes to their TCP state. OpenTCP’s kernel module running in the TCP stack of the end node updates their TCP variant’s state using the information in the CUE packets. Thus, OpenTCP requires the end node TCP protocol to be changed.

The approach proposed in this paper is SDN-based and using the ECN mechanism, but does not require any changes to the end nodes’ TCP protocol. This is advantageous since changing end nodes may not be possible if all the nodes are not under the same administrator, especially in multi-tenant networks and data centers providing cloud services.

The congestion detection component (Algorithm-A) periodically collects information about the network and its traffic and tries to recognize/predict congestion. SDN switches have the capability of collecting statistics. The application can probe the switch for these statistics and get a complete picture about the network and its present traffic conditions. Queue statistics and link utilization statistics will be used by Algorithm-A for detecting congestion. A simple metric to identify potential congestion is if the link utilization or average queue length is greater than a threshold. More sophisticated algorithms could consider global network state and predict congestion based on learning algorithms or other complex heuristics. In the evaluation section, we will present an example for Algorithm-A.

Once Algorithm-A predicts congestion in a link/switch, Algorithm-B decides on how to handle the congestion. The administrator can also set policies for congestion control. For instance, the policy can specify that certain type of high-priority flows should not be affected. It will compute different ways to avoid congestion, based on the global view and administrator policies. It will identify the set of end nodes whose congestion window can be reduced to avoid congestion while maximizing a global objective.

In traditional TCP, all the end nodes react to congestion by reducing their congestion window. In data center networks and other SDN applications, it is not necessary that all the flows should react to congestion. For instance, we would like the interactive traffic or short-term (i.e. mice) flows to be not modified but control the congestion windows of bulk (i.e. elephant) traffic. More over, when all the nodes react to congestion, they all back off at the same time (leading to under utilization) and then move forward simultaneous giving rise to a saw-tooth like bursty traffic at the switches.

For example, the following scheme can be used as Algorithm-B in data center networks for prioritizing interactive flows. Whenever congestion is detected in a link, the framework will obtain all statistics about the flows traversing that link. The algorithm can select the top $T\%$ of the the flows based on their bandwidth utilization and reduce their congestion windows. Generally, only about 10% of the flows are elephant flows in the network, but they utilize about 90% of the bandwidth. Penalizing the high utilization flows can ensure that interactive traffic is less affected by congestion.

The next step is to convey the congestion handling information to the end nodes so that they can react appropriately. The Algorithm-B, after computing the set nodes that have to reduce their congestion window, will send explicit flow rules to the switches with an additional action of marking the ECN bits apart from forwarding the packets to the right port. These new high priority rules sent by Algorithm-B are called the Congestion Control Flow Rules (CC flow rules). When new packets from these flows arrive, they get matched with these top priority CC flow rules and their ECN bits get set at the switch before forwarding.

The ECN bit processing is based on the end-nodes’ TCP implementation. As mentioned earlier, we do not require any changes in the TCP implementation. A receiver, upon receiving packets with marked ECN bits, echoes the information to the sender by marking the ECE bit in their ACK packet. The sender acts upon these marked ACKs and reduce their congestion window according to their native TCP variant.

The new ECN marking flows added are given higher priority in the OpenFlow tables at the switches, compared to the regular flows. This is to make sure they are not skipped. Also, these new CC flow entries are set with a very small rule timeout interval so that adequate number of packets are marked. If a new CC flow rule is added for a very long time, it might send signals of large congestion to the end node that would affect the throughput of that flow adversely. At the same time, the timeout interval should not be too low such that not enough packets from that flow actually get marked. Choosing this timeout value is a critical parameter. Once the timeout interval is over, these extra high priority flows get evicted and the flows start matching with the normal low priority flows rules that were already present in the switches.

Another critical parameter is the periodicity of the network probes. As mentioned earlier, Algorithm-A probes the network for network and traffic state. There should be enough time for the system to settle down after the congestion window changes are implemented by the source TCP and the network traffic stabilizes. The ECN packets have to reach the receiver and the receiver has to reply back with bits marked in the ACK packet. It clearly takes about 2 RTT to get communicated to the end nodes and it might take a couple of more RTTs for the network to become stable. Very large probing interval cannot detect sudden congestion and can be less efficient. Thus, the probing interval is a sensitive parameter.

In summary, the salient features and advantages of this framework include:

1. 1.

   Global View: This enables the congestion control scheme to obtain a more complete picture about the network state and hence make better congestion predictions that can achieve more globalized objectives. In conventional end-to-end systems, all the end nodes try to optimize their own local objective function and could end up in a Nash equilibrium solution.

2. 2.

   Prioritization in Congestion control: Flow priority can be considered in making congestion control decisions and in setting of flow table rules.

3. 3.

   Fairness: Fairness can be ensured even across TCP and UDP. Typically, UDP flows can hog the bandwidth placing TCP flows at disadvantage. With the proposed approach, we can monitor the bandwidth used by UDP and use special flow rules to restrict UDP’s uneven share of the bandwidth.

4. 4.

   No change to the end-nodes: The proposed approach does not require changes to the end-nodes’ protocol stack.

5. 5.

   No change to switches: This approach uses normal flow rules and the functionalities available with existing SDN switches to achieve congestion control.

6. 6.

   Easily pluggable CC algorithm: The Algorithms $A$ and $B$ can be changed according to different network and traffic needs. A data center network may need a different kind of algorithm than an enterprise network. Congestion control algorithm at the controller becomes a easily changeable mechanism.

The proposed framework has been implemented in an SDN emulator package called Mininet (version 2.2.1) [15]. The Floodlight SDN controller [16], Open vSwitch (version 2.3), OpenFlow 1.3 that supports setting of ECN bits through flow rules have been used. The congestion control framework has been implemented as an application in Floodlight. In all our experiments, the switch contains a single flow table. All the virtual end hosts created by Mininet in our experiments run TCP Cubic implementation that comes with Ubuntu 14.04 kernel. The ECN mechanism has been turned on at all the end nodes. Since Mininet is not able to handle high bandwidths accurately, 100Mbps links have been used everywhere unless stated otherwise.

The analysis is done for data center networks with the objective of achieving lower flow completion times (FCT) for interactive traffic when it co-exits with bulk traffic. As a proof of concept for our framework, we present sample congestion control algorithms, (Algorithm-A and Algorithm-B), and evaluate their performance.

The proposed Algorithm-A probes the switches every 2 seconds and collects port statistics information. From these, we compute the average link utilization of every link in that interval. If the link utilization is greater than 75% , we consider that congestion is likely to occur in that link and inform Algorithm-B about it. The handling algorithm, Algorithm-B, is defined as follows. Congestion control is imposed on the top $T$% of flows (based on bandwidth utilization) going through the congested link once the link utilization is greater than 75%. We linearly increase $T$ as link utilization increases and when link utilization reaches 100%, we set $T=50\%$ penalizing top 50% of the flows when utilization hits the maximum. Since we are not penalizing all the flows at the same time, this can prevent saw tooth like behavior of utilization. Additionally, we start penalizing more and more as we get closer to 100% utilization making sure we have enough bandwidth for new incoming short flows.

The proposed approach is compared with Linux’s TCP Cubic scheme, and approaches that use in-network elements in congestion control, namely RED [8] and ECN [2]. The implementation of RED and ECN is available in Mininet.

The objective of this experiment is to show that the proposed algorithm does not compromise on bulk flows and achieves good throughput on long-lived traffic.

The topology studied, denoted Topology 1, consists of one switch connected to four end nodes with 100 Mbps links each. Three nodes are senders ($S_{1},S_{2},S_{3}$) and one is a receiver ($R_{1}$). Three of the senders run iperf for two minutes to generate traffic to the receiver. The results are shown in Table I. As seen, the proposed approach gives 8% better total throughput than Cubic and comparable performance with the distributed ECN and RED schemes. The proposed method achieve better fairness than TCP Cubic and is on par with ECN and RED.

The next topology, called Topology 2, is a dumbbell topology with 2 switches and 3 nodes connected to each switch. Nodes connected to one of these switches are all senders, called $S_{1},S_{2},S_{3}$. The nodes connected to the other switch are receivers $R_{1},R_{2},R_{3}$. Three pairs ($S_{1},R_{1}$),($S_{2},R_{2}$),($S_{3},R_{3}$) are selected and long-lived flows are established between them for 2 minutes. All the links have 100 Mbps capacity. The throughput results are shown in Table II. As seen, the proposed approach achieves 14% better total throughput than TCP Cubic and is similar to that achieved by RED and ECN. The individual through-puts are also closer to the fair-share values.

In this set of experiments, we show that the proposed algorithm significantly improves the flow completion time of interactive flows in presence of bulk traffic. In the first experiment, we consider the single switch-four node Topology 1. Two long-lived flow between from $S_{1}$ and $S_{2}$ to the receiver $R_{1}$ are established. A 2 MB interactive flow is sent from $S_{3}$ to $R_{1}$. This experiment is repeated 30 times for each of the congestion control algorithm and the interactive flow’s mean flow completion time is computed, as shown below.

As seen, the proposed scheme obtains 30x improvement to Cubic in flow completion time and about 1.2x improvement to RED. ECN was close (but worse) to our scheme in this topology. We also found that our algorithm shows the least variance in flow completion time compared to the other schemes, based on the 30 iterations.

The Topology 2, discussed in the previous section, was next considered and flows established as above. The flow computation time results based on 30 iterations is shown below.

As seen, the proposed approach performs 28x better than TCP Cubic, 1.25x better than ECN, but on par with RED. It was observed that the proposed schemes produce had the least variance among the four.

To summarize, the proposed scheme outperforms TCP Cubic by a large factor. It works the best for both the scenarios, while ECN and RED failed to perform better than the proposed scheme in at least one of the cases.

We evaluate the proposed algorithm in multi-hop networks with multiple bottleneck links. The 7-switch topology is shown in Figure 2. In the first experiment, we establish long lived flows from $H_{1},H_{2},H_{4},H_{5}$ to $H_{7},H_{8},H_{10},H_{11}$ respectively for 2 minutes. The total throughput (in Mbps) is presented below. As seen, the proposed scheme achieves throughput better than TCP Cubic and comparable to ECN, but less than that of RED.

To check flow completion time for mice flows in this setting, We created long-lived flows from $H_{1},H_{2},H_{4},H_{5}$ to $H_{7},H_{8},H_{10},H_{11}$ respectively. We simultaneously sent 2MB data from $H_{3}$ to $H_{7}$ and from $H_{6}$ to $H_{8}$ and measured the flow completion times. This experiment was repeated 30 times. The average flow completion time (in seconds) achieved by each of the algorithms are shown below.

In multi-hop conditions, our scheme clearly outperforms ECN by a factor of 1.5x and RED by 1.7x without compromising much on throughput of bulk flows. Our scheme also outperforms TCP Cubic by 30x.