Recent Advancements In Distributed System Communications

At the beginning of the survey, but also at certain times during it, the supervisor Animesh Trivedi provided papers related to the topic. These papers were added to the body of the survey and were :

References on some papers led to discovering other papers relevant to the content. If those new papers fitted the selection criteria described below, they were added on the survey too.

Google Scholar was also used for discovering papers related to the survey topic, since it searches through various sources of research publications. The search results were filtered by the publishing date range 2010-2020.

Besides being relevant to the research questions, these were the other criteria of selecting or rejecting papers:

Publishing date between 2010 and 2020 : the reason for selecting this date range was that the survey focuses on recent advancements. Looking only to the latest publications though (2020) would exclude papers from previous years that are still relevant today. Therefore, a decade ending to the year of writing was considered as a suitable timeline for staying within the recent developments, without going too far in the past. This also allowed concentrating more material on certain technologies presented here and provided further understanding on how these technologies developed over time.

Focus on technology for servers : Only technologies and techniques applicable to servers and usable by distributed systems were added on the survey. Papers referring to similar solutions for switches and routers or other networking equipment, even the software counterparts (like software routers), were not included, as was explained in the introduction for the scope of the survey. The reason was to limit the scope of an otherwise very broad topic.

Sockets are used to establish connections with remote machines and to exchange data with them over the network. As explained in [12], [34] and [15] though, the socket implementations provided by Linux lack efficiency on multi-threaded management of connections, while they also suffer overheads by the file system and the use of system calls to the kernel. These issues incur time costs that increase latency and reduce throughput during data transfers.

Looking into the multi-threaded management of socket connections, there are two factors that degrade performance. The first is the contention of multiple threads on a single accept queue in a socket, that contains incoming remote connections and is protected with an exclusive lock, allowing only one thread to accept a connection at a time. This causes contention between kernel threads that enqueue incoming connections and application threads that remove them from the queue to manage them, which also leads to idle times for threads that wait to acquire the lock. Recent releases of the Linux kernel deal with this though, by allowing multiple sockets running on different threads to listen on the same socket address with the SO_REUSEPORT option, while the kernel takes care of load balancing connections on the different threads. The second factor is the lack of affinity to the same core for all activities related to a given connection, like between packet delivery and reading or writing data on the connection’s socket. There are technologies like Receive Side Scaling that can enable Network Card Interfaces (NICs) to load balance incoming connections to threads running on different CPU cores and keep sending incoming packets of a connection to the same core. That affinity though can be lost when an application thread residing on a different core accepts a connection or writes data to the socket. This lack of affinity causes inefficient cache usage when different CPU cores need to maintain the state of the same connection and perform different activities on it.

Performance issues can also occur by the association of sockets with the file system. For instance, allocating a new file descriptor for a socket can become expensive with a large number of connections. Specifically, Linux searches for the minimum available file descriptor number for the process, which incurs an undesired delay when the application manages many sockets. Additionally, this searching process uses a lock, since files are shared within the process, which can cause contention on a multithreaded environment. Another issue is that sockets are treated like file types by the Virtual File System, which associates them with data that is globally visible and subject to system-wide synchronizations.

Sockets also use system calls to the OS kernel that incur expensive context switches between user space and kernel space and also require data copies between those spaces. When an application uses sockets, system calls transfer the control from the user space application to the kernel, which interacts with the NIC in order to perform the networking operation requested by the application. This context switch from user space to kernel space introduces latency. Moreover, any data to be transmitted through a socket is copied from an application owned buffer to another buffer inside the kernel, associated with the socket. From there, the data is finally forwarded to the NIC for transmission over the network. This is illustrated in Figure 1. Reception follows the opposite path, from the NIC to the kernel and then back to the application, incurring a context switch and a data copy between kernel and user space, as with transmission. These context switches and data copies delay socket operations and decrease the throughput in request processing in distributed systems.

The OS kernel’s network stack’s processes incoming and outgoing packets and implements networking protocols like TCP and UDP. The network stack implementations on OS kernels though have been unable to process incoming packets at the full speed of network links for years. The problem lies in the time it takes to process a packet in the stack. When receiving packets from a high speed network link, like 100 Gbps, the kernel network stack has very limited available time to process the packets, in order to achieve a processing rate which is the same as the link rate and fully utilize the link capabilities. The same can be observed even with slower links when small packets are sent, because small packets can be transferred by NICs at high rates as well. The delays in packet processing put a limit on the throughput of OS network stacks that also limits the maximum achievable performance in distributed system communications.

These delays are attributed to the code of kernel network stacks, which incurs high computational complexity and memory footprints [21]. Network stack processing has a modular design with various stages, connected with function calls, queues and software interrupts, built for extensibility. This design though results in a large number of instructions per packet, while software interrupts exacerbate the processing delays. The TCP state machine is also implemented in a monolithic code block with many branches that does not leverage batching and prefetching efficiently, causing cache mispredictions and increasing the instruction cache memory footprint. Moreover, inefficient cache usage results from the use of large metadata structures that span multiple cache lines and can cause false sharing, while state might be shared by multiple cores requiring locks and causing cache coherence overheads. Additionally, expensive dynamic memory allocations for packet buffers and metadata per packet happen frequently during data transfers. All these issues with network stacks aggregate time costs that make it hard to achieve packet processing at link rates.

Building Remote Procedure Calls on top of TCP for reliable packet delivery can actually hurt the communications performance of RPCs [23]. One of the causes for TCP’s mismatch with the requirements of RPCs, is its design to perform better with large data transfers spanning multiple packets, while RPC messages are often small and can fit to a single packet. For such small RPC packets, that are considered either requests or responses and are semantically independent, TCP’s packet ordering is unnecessary and only adds extra delay. Furthermore, RPC responses are also acknowledgements themselves, so sending separate packet acknowledgements with TCP also adds unnecessary delays. Moreover, by establishing only one-to-one connections, TCP requires maintaining and managing a lot of connection state, which becomes harder as the number of connections increases, causing a scalability limitation. To avoid such a limitation multiple RPCs are often layered on top of the same TCP connection. This can also lead to head-of-line blocking between different RPCs, while scheduling options are limited.

After explaining the performance issues with traditional communication methods for distributed systems and their causes, the following sections continue with hardware and software-based approaches to overcoming them. Among those approaches there is a subset that completely bypass the kernel by either offloading networking operations to specialized hardware or creating their own network stack implementations as software run in the user space. The term zero-copy transfers is also mentioned a lot with kernel bypass techniques and here it means that there are no intermediate copies of transmitted or received data between user space and kernel space buffers, unlike with OS-provided sockets. Instead, applications and NICs share the message buffers and the time cost of intermediate copies in the kernel is eliminated. The RDMA technology presented in the next section is a hardware-based approach that can achieve very low latency and high throughput by implementing kernel bypass and zero copy transfers.

Remote Direct Memory Access (RDMA) is a feature that allows an application to read or write memory located on a remote machine, without requiring the active participation of an application in that remote machine during the data transfer. The RDMA technology though supports both the RDMA feature and also messaging semantics, in which a receiver in the remote side is notified about a new message and thus participates in the communication. In the rest of this text RDMA refers to the technology and not the feature. With the RDMA technology applications can specify memory regions that will be used during the communications, before the latter commence. Then RDMA-capable Network Card Interfaces (NICs) can use Direct Memory Access (DMA) to directly read or write data on these memory regions without involving the kernel. Additionally, applications can post transmission or reception requests directly to the local NIC, without passing through the kernel. After the NIC receives a request, it can use DMA to either fetch local data that will be transmitted or to store remote data locally, when it receives it from a remote NIC. DMA does not involve the kernel and thus does not incur context switches and additional data copies between kernel and user space. In this manner, the kernel is bypassed and zero-copy transfers are ensured, as the NIC is the one performing the networking operations, thus saving CPU cycles for other tasks.

With RDMA an application can choose two ways to communicate with another machine. First, as already mentioned, it can either read or write directly on a memory buffer in the remote machine without notifying the remote application. This is useful in cases where there is no need for the remote side to perform some task upon the reception of a message. Second, RDMA also supports sending messages for which the receiver is notified on arrival, which can be used when the receiver has to act upon the reception of a message, like in RPCs. Both options are available through RDMA’s communication primitives, which send requests directly from the user application to the NIC. Moreover, RDMA provides connection methods that can offload operations like reliable transfer and packet ordering on the networking hardware, allowing the CPU to perform other tasks. With kernel bypass, zero-copies and offloading certain networking operations to hardware, RDMA can achieve lower latency and higher throughput than traditional socket based networking.

Technologies that support RDMA are Infiniband, RoCE and iWARP. Infiniband uses specialized networking equipment that provides a high speed lossless network (packet errors are rare) and implements flow control mechanisms. RoCE and iWARP on the other hand, support RDMA semantics over Ethernet with the use of hardware adapters. One of the differences between RoCE and iWARP is that the latter is designed to provide RDMA over TCP/IP, by offloading TCP operations to the NIC. RoCE version 2 on the other hand uses UDP/IP and requires a lossless network to provide low latency. By offloading certain operations to the networking hardware, like reliable delivery, flow control and ordering, these three RDMA technologies save CPU cycles for other tasks. There are also ways to use RDMA entirely in software like Soft-RoCE and SoftiWARP, but they do not offload operations to hardware like the previous options and thus achieve less CPU efficiency and lower performance. Therefore, in production environments like datacenters, Infiniband, RoCE or iWARP are more suitable to achieve the best RDMA performance and this is the reason for categorizing RDMA to the hardware-based approaches of the examined literature.

RDMA has gained a lot of popularity over the years for its low latency of a few $\mu$sec and high throughput of millions of operations per second. RDMA’s high performance has already led to its adoption in a variety of distributed systems, like key-value stores, distributed consensus and transactions, High Performance Computing and Deep Learning systems. Some systems have been designed from scratch to work with RDMA, while others have been extended to support it, like in [25] for the RPC communications in the Hadoop ecosystem, in [41] for boosting the performance of the Memcached key-value store or in [4] for optimizing gRPC for tensor transfers in Tensorflow. To gain a better understanding of how RDMA is used in distributed systems, it is important to explain in more details the networking operations and connection methods that it supports.

In RDMA the operations that take place to accomplish data transfers are split into two types, those belonging to the control path and the rest belonging to the data path. The control path contains operations related to managing the resources for the communications, like pre-allocating message buffers and registering them to NIC to be used during data exchanges. Such operations are slow, but happen before commencing the communications and hence their overheads do not have to be repeated with each data transfer, as with sockets. This requires knowing beforehand how many resources to allocate, while with sockets memory allocations happen dynamically. On the other hand, the data path operations take care of transferring the data to its destination and copying it from a network card to memory or the opposite. The control path uses the kernel to manage resources while the data path is the one that uses Direct Memory Access to bypass the kernel and reduce CPU usage, as illustrated in Figure 2. This separation of paths that avoids repeating the time cost of the control path in every data transfer reduces latency and leverages the saved time to increase the rate of networking operations performed.

RDMA provides its own primitives to communicate with a remote machine, which are also called "verbs". There are two types of such network operations, the two-sided and the one-sided, which have also been called messaging and memory verbs. The messaging verbs are the SEND and RECV operations that send and receive data respectively. The are two-sided operations because both sender and receiver participate in the communication. The receiver has to post a RECV operation to the NIC prior to receiving messages and it is notified upon reception by the NIC. The memory verbs, or one-sided operations, perform data transfers that do not involve the remote side. There is a WRITE operation that can directly write data to a memory location on a remote machine and a READ operation that directly fetches data from a remote memory location. With one-sided verbs the NIC doesn’t notify any application thread on the remote side that a data transfer took place. One-sided verbs also include the remote atomic operations fetch-and-add and compare-and-swap, which can guarantee atomic access on remote memory locations, as long as this access happens between NICs on different machines.

Both SEND and WRITE can optionally transmit a small immediate value along with the data, which is sent to the remote side as a notification. If this option is used, then WRITE will involve the remote side to process this value. Although it seems like a minor detail, the immediate value has been used in constructing request-response or RPC protocols with one-sided operations.

RDMA also supports different types of connections between machines, which play an important role in the design of a distributed system. Before describing these connection types and their differences though, it’s necessary to introduce Queue Pairs (QPs). RDMA communications are established through Queue Pairs, which contain a Send and a Receive Queue. To perform an RDMA operation, an application has to submit a Work Request (WR) to a QP; the WR describes the RDMA operation to perform and which network buffer to use. Additionally, a submitted WR is stored as a Work Queue Element (WQE) in either the Send or Receive queue of the QP, depending on the type of RDMA operation. The RDMA NIC executes the operations described in the WQEs of QPs.

Having explained QPs, the different RDMA connection types can be introduced. These are Reliable Connection (RC), Unreliable Connection (UC) and Unreliable Datagram (UD). With Reliable Connection a QP is associated with only one other QP. Messages are delivered by the NIC reliably by using acknowledgements and packets are ordered. RC is similar to a TCP connection, but reliability and packet ordering are managed by the NIC. With Unreliable Connection a QP is again associated with only one other QP, but the connection is unreliable, which means packets are not ordered, may be lost and no acknowledgements are sent. Applications have to take care of retransmissions themselves when errors occur during transmissions. Finally, with Unreliable Datagram a QP can send and receive single packet messages to/from any other QP and multicast is also supported, which is not available in other connection types. UD is unreliable like UC and similar to UDP sockets.

Different types of connections support different networking operations and maximum message sizes. The following table is taken from Mellanox’s user manual [28] to show what can each type of connection support. The MTU entry in the last row stands for Maximum Transmission Unit.

Choosing the right connection method to fit a system’s needs requires comparing their features. As already shown, the three methods support different subsets of RDMA networking operations and offload different functionality to the NIC. Thereby, some of them can satisfy a system’s requirements better than others. Moreover, all three connection methods have benefits and pitfalls that are not obvious by their features alone and users of RDMA have to be aware of them before choosing a connection type.

A Reliable Connection (RC) is very useful when reliability and packet ordering guarantees are necessary, like with a TCP connection. By offloading these two operations to the NIC, RC allows the CPU to save cycles for other tasks, leading to better overlapping of computations with communications. This overlapping can be improved when running an RC connection over an Infiniband lossless network, which takes care of flow and congestion control with the networking equipment as well. Moreover, RC supports all networking operations, giving the system designers more freedom of choice in how to perform data transfers or message exchanges. A drawback of RC though is that, like TCP, it supports only one-to-one connections and managing this connection state with a large number of clients can lead to a performance degradation. This scalability issue though can be solved and techniques to do so will be presented in the following subsection. It would suffice to say for now that ensuring scalability requires taking certain steps during building the RDMA communications layer of a distributed system, which increases the complexity of the system design process.

An Unreliable Connection (UC) doesn’t provide reliability and ordering guarantees, reducing the processing time on the NIC and the network traffic by not sending packet acknowledgements. Hence it allows the NIC to spend more time in handling requests than dealing with these guarantees when they are not required. As explained in R2P2 [23], packet ordering is often redundant for Remote Procedure Calls, as requests and responses usually fit in a single packet and they are semantically independent. Furthermore, UC is also useful when sending acknowledgements for received packets is not necessary. More specifically, lossless networks lose packets rarely, so running UC over such a network will deliver the packets reliably most of the time, without the need for the NIC to send acknowledgements. Moreover, in an RPC protocol, the response can also act as an acknowledgement, thus the NIC does not need to send one. A drawback of UC though, is that although rare, any occurring errors and retransmissions have to be handled with software, which adds extra complexity on the software development process and involves the CPU when it happens. Additionally, UC supports only the two-sided operations and WRITE, hence systems cannot use UC to read remote data directly with READ or use the atomic RDMA operations. Finally, since UC supports one-to-one connections like RC, it suffers from the same scalability issue.

Unreliable Datagram (UD) bares similarities with UC but comes with its own advantages and disadvantages as well. Like UC, it doesn’t offer packet reliability and ordering and can be useful in cases where these guarantees are not necessary, as explained before. Additionally, its support for multicast makes it suitable for systems that require this capability. Moreover, UD can offer greater scalability than RC and UC, since a UD Queue Pair can be used to exchange messages with any other UD QP. There is no need for a server to have a separate QP for each different destination. Therefore, less connection state has to be maintained than using RC and UC connections. Furthermore, UD also makes it possible to batch requests for networking operations to the RDMA NIC regardless of the destination of the messages, while RC and UC support only batching such requests for the same destination [20]. One limitation with UD though is that it supports only two-sided operations, which always include the step of notifying the remote side. Another drawback is that like UC, it requires developing code to handle retransmissions when errors occur.

Choosing the best suited connection method is not restricted to picking one of the three, as combinations of them are also possible. For example, the HERD distributed key-value store [18] uses both UC and UD in their RPC protocol, in which multiple clients communicate with one server. The clients send their requests to the server with UC, while the server responds through a UD connection. UC allows the clients to write their requests directly to the server’s memory with one-sided WRITE and the server polls the memory location to detect new requests. Although this could also be implemented with RC, UC is preferred in this case because it involves less processing in the NIC, and reduces the network traffic by not sending packet delivery acknowledgements. Moreover, the scalability issue with UC is greatly mitigated by the fact that the server does not use it to reply to clients. Although there are multiple UC connections from clients to the server, the server only uses the UC connections to receive client requests. Thus, the server’s NIC manages less connection-related state than if UC was also used for sending responses. For client responses, the server uses UD to ensure better scalability than UC. For systems that have different needs in different parts, like HERD, such a combination can lead to more in-depth optimization and hence, higher performance.

As it should be obvious by now, there is no perfect connection type, as they all have their benefits and drawbacks. Distributed system designers have to decide on which one to use based on which features are desirable, but also which tradeoffs are tolerable. Different connection types can also be used in the various parts of a system, to further optimize the data exchanges, according to the needs of each part.

RDMA does not guarantee scalability by default and users have to be aware of what can cause scalability issues and what can be done to deal with them. Scalability problems occur from thrashing in the limited cache on the RDMA NICs that stores connection state, high memory consumption by allocating a message buffer for every connection, and allocating and managing multiple Completion Queues (CQs). Completion Queues store notifications about completed Work Requests submitted to QPs. Two-sided SEND/RECV uses CQs for notifications about new message receptions. The rest of this section presents solutions proposed for the scalability issues with RDMA communications, found in research literature.

After selecting a connection method and ensuring scalability, the third factor that affects the communications performance of a distributed system is choosing between one-sided RDMA operations and Remote Procedure Calls. Each of these choices offers benefits that the other can’t offer and as with connection methods, selecting the right one depends on the system’s needs. Moreover, this dilemma exists for both data transfers and remote locks over RDMA, and further explanation follows, regarding each of these two use cases.

The last of the design choices presented here for efficient RDMA usage, is whether to build a custom RPC protocol with the RDMA operations or use an existing RPC framework that uses RDMA. Although the latter option is more convenient, building a custom RPC protocol for a distributed system can enable optimizations specific to that system, which are not possible with a general purpose RPC framework. The rest of this section explores what does each of the two options offer in more detail, as well as how they affect the communications performance of a distributed system.

RDMA RPC frameworks can speed up the system design and development process. By using such a framework there is no need to go through all the design choices for efficient RDMA usage presented so far, or write code to implement RPCs over RDMA, since many features are provided "out of the box". For example, the ScaleRPC [5] and DaRPC [42] frameworks that offer RPC over RDMA, deal with the scalability issues of RDMA, as described in section 5.2. Additionally, the eRPC [17] framework offers RPCs over RDMA’s Unreliable Datagram connection, to avoid scalability issues with Reliable and Unreliable RDMA connections. Furthermore, the Mercury [40] RDMA RPC framework also provides a way to deal with bulk data transfers, eliminating the need from users of the framework to do so themselves. Mercury can split bulk data transfers to smaller chunks and send a chunk at a time, enabling the server to process the data as it flows and not having to pay the latency cost of waiting until the whole transfer is complete to start processing it. This can also deal with large messages that don’t fit into a memory buffer at the server as a whole. So for distributed systems that are under development, an RPC framework can greatly reduce the effort to achieve good performance with RDMA as the base for RPC.

Although RPC frameworks are convenient by providing features "out of the box", they are general purpose solutions, which lack the flexibility of making design choices that fit to a specific system’s needs. Systems designers that desire this flexibility to apply customized optimizations for their system will have to built their own RPC protocol, which comes with the tradeoff of increased complexity. For example, the designers will have to choose an RDMA connection method and deal with any scalability issues that this connection method incurs. Moreover, they will have to select with which RDMA operations to implement RPCs, since it’s possible to use either one-sided or two-sided operations. Regardless of the choice, to build a request-response protocol like RPC, there must be a way for the receiver to be notified about new messages.

There are different ways to detect new messages, depending on which RDMA operations are used for RPCs. With two-sided SEND/RECV, notifications about new messages can be retrieved from Completion Queues. With one-sided WRITE on the other hand, there are two possible options. The first is to use an immediate value, which is sent along with the data to the remote side and is passed as a notification to the receiver. The second option for message detection with WRITE is to use a polling mechanism. This technique can achieve less latency than the previous option, by avoiding the notification part, which requires a DMA transfer from the NIC to a Completion Queue in main memory. With polling, instead of waiting for a notification to be sent by the NIC, the receiver directly looks into the memory location where messages are written to detect new ones. Polling can cause high CPU usage though, so in [45] and [2], sleep functions are proposed in order to have the servers sleep after a period of inactivity. Furthermore, it also possible to combine two-sided and one-sided operations for an RPC protocol like HERD [18] does, which allows HERD’s server to handle multiple client connections in a scalable manner, as described in 5.1.

In conclusion, RDMA offers high performance communications but at the cost of complex design decisions. Since a high level abstraction like sockets is not provided by default, users of RDMA have to either build their custom protocols or use existing RDMA frameworks. While the former option offers more freedom of design choices and the ability to make customized optimizations according to system requirements, the latter option makes both design and development easier. Moreover, building a custom solution requires spending time to study research papers in order to learn about what has been tried before and how can certain design choices affect the performance of RDMA communications. This task would also involve learning low level optimizations, like how can networking requests to the NIC be shrinked or batched to maximize performance [19], thus requiring a higher level of expertise on the RDMA technology for developers than using simpler, socket-like APIs. Nevertheless, it is up to the users of RDMA to decide whether they wish a more convenient solution like an existing framework or a custom solution in order to achieve the maximum performance for their specific system, since both options are available.

Besides kernel bypassing techniques, there are also proposals to optimize the existing communication utilities provided by operating system kernels, by offering more efficient socket primitives. These solutions can improve the performance of communications without requiring specialized networking equipment. Moreover, they offer a more familiar and abstract interface to application developers, than using low level primitives, as with RDMA or any similar protocols implemented in hardware. Their main drawback though is that since the kernel is not bypassed, they do not solve all of the overheads that sockets and the kernel network stacks impose during data exchanges.

The purpose of optimized socket implementations proposed in literature is to deal with the performance overheads of sockets explained in section 3.1. As a reminder, the overheads examined here are induced by the internal socket mechanism that allows only one thread to accept a new connection at a time and the treatment of sockets as file system objects, thus inheriting file system related overheads. To solve the issue of only one thread accepting a connection, MegaPipe [12] and Affinity-Accept [34] propose the replacement of a socket’s internal synchronized accept queue that accepts new connections, with one accept queue for each thread that handles connections on that socket. Thus a socket can be shared among cores without the need for synchronization. A similar effect can be achieved by the Linux kernel though, by allowing multiple sockets run in different threads to listen on the same socket address through SO_REUSEPORT, while the kernel handles load balancing the connections to these sockets. Affinity-Accept also aims in maintaining the CPU core affinity for connections, by accepting a connection on the core that was already assigned by the NIC to process the packets for that connection. This method can lead in better performance, as data related to the connection remains on the same cache. Moreover, due to being treated like file system objects, sockets are associated with globally-visible file system data, which makes them susceptible to system-wide synchronization and also require a time-costly file descriptor allocation. To solve these problems as well, MegaPipe provides its own socket implementation that is not related to the file system.

Techniques that optimize sockets offer a performance increase for distributed systems communications, but only solve a small subset of the issues in 3. System call overheads, context switches and redundant copies between user space and kernel space, remain. Furthermore, the kernel’s network stack is still used, which means that packet processing remains slow as well. Therefore, such techniques can not offer the same performance optimizations as kernel bypass approaches.

Looking at a higher level than sockets, there has also been effort to improve the efficiency of RPC frameworks, with the goal to achieve low latency and high throughput for distributed system communications. Since TCP can hurt the performance of RPCs, as mentioned in 3.3, certain RPC frameworks switch to UDP based communications to build more efficient RPC protocols. Like optimizing sockets, optimizing RPC frameworks is not a kernel bypass approach, but can aid in increasing the performance of RPCs. Nevertheless, it will be shown that it’s possible to combine optimizations on RPCs with user space networking techniques, which bypass the OS kernel for networking operations, and achieve performance close to RDMA.

The mismatch of some TCP assumptions with RPC semantics has led to the creation of RPC frameworks that operate on UDP, like eRPC [17] and R2P2 [23], in order to meet the performance requirements of large scale applications. As explained in R2P2 [23], RPC requests and responses usually fit into a single packet and since they are semantically independent, there is no need for TCP to order these packets. Moreover, to amortize the cost of setting up TCP flows, RPCs are often layered on top of a persistent TCP connection, which leads to ordering packets even if they originate from different RPCs. This connection sharing can also lead to head-of-line blocking of independent RPC packets. UDP on the other hand does not enforce any packet ordering, allowing RPC semantics alone to distinguish the requests from the responses. Furthermore, unlike TCP, UDP does not setup and maintain connection information, leading to greater scalability than TCP with a large number of clients and servers, while eliminating the need to share persistent connections. R2P2 uses UDP to also enable the implementation of various load balancing policies for processing RPCs. More specifically, clients forward their requests to an intermediate router, which selects the server that will process the request and send a response, based on a load balancing policy. With UDP there is no need to setup a new point-to-point connection between a client and the server that is chosen to handle the RPC every time, thus saving time, memory and computing resources. Apart from these advantages though, a UDP based RPC would have to implement custom methods of handling packet losses, congestion and flow control, unlike TCP which provides this "out of the box".

The eRPC framework [17] has also proven that it’s possible to get close to RDMA performance for RPCs, with a general purpose RPC framework. Its goal is to achieve high performance RPC communications, even without specialized networking equipment. eRPC supports both UDP for lossy Ethernet networks and Unreliable Datagrams when Infiniband is available. Unreliable transports help eRPC solve the mismatch of TCP and the TCP-like Reliable Connections of RDMA with RPCs, regardless of networking infrastructure. eRPC also uses three additional optimizations, to be able to reach RDMA performance, even when Infiniband is not available. The optimizations are writing code optimized for the common case, restricting each flow to at most one Bandwidth-Delay Product (BDP) of outstanding data, and using user space networking techniques. In eRPC, the common case refers to the fact that usually the network is uncongested, the messages are small and RPC handlers are short. Therefore eRPC’s code is optimized to execute fast when these conditions hold. Additionally, restricting the flow of messages to BDP can allow eRPC to reach a throughput equal to the network link’s data transfer rate without dropping packets, since network switches have large enough packet buffers to support BDP tranfer rates without packet drops. The third and very important optimization is to use user space networking techniques, which actually bypass the OS kernel for networking operations without specialized hardware, but with software. More specifically, zero-copy transfers are achieved by dedicating DMA-capable buffers for messages, similarly to what RDMA does, which are owned and accessible directly by the applications in the user space. Thus, intermediate copies of these buffers are avoided, and both applications and NIC can access them without requiring the kernel’s involvement. The performance analysis in eRPC shows that all the above features allow it to perform almost as well as specialized RDMA solutions, even by being a general purpose RPC framework.

Therefore, besides optimized socket implementations, optimizing RPC frameworks is another option to improve the performance of distributed system communications through software. One advantage of these frameworks over the optimized sockets is the provisioning of an RPC API to developers, that they would otherwise have to build themselves. A drawback is that optimizing RPC frameworks by making RPC implementations more efficient does not solve the overheads with sockets and the kernel network stacks. Nevertheless, combining optimizations on RPC frameworks with kernel bypassing techniques like eRPC does, is a great alternative to relying on specialized networking equipment to enable the bypass. This software-based kernel bypass is not a new concept though, as it has been a research topic for years in the scope of user space networking, which will be presented in the following section.

User space networking means managing networking operations in the operating system’s user space, instead of the kernel space. The kernel’s network stack is bypassed and networking operations use a network stack running in the user space instead. Packets are directly stored and processed in buffers shared between the NIC and the applications, without intermediate data copies happening on the kernel (zero-copy transfers), achieving faster communications. Additionally, it is considered more convenient for developers to build and optimize network stacks in the user space than attempting the same with complex and large kernel code. Furthermore, the lack of tight coupling to specific networking equipment like RDMA, makes user space networking a more widely available option.

A very important feature of user space networking that affects the performance of communications its ability to separate networking operations to control and data planes. To avoid confusion with the common use of these terms in network switches, in this text these two planes refer to operations that take place in an operating system (OS) to transfer and receive messages. They are practically the same as the control and data "paths" for RDMA presented in 4.2. The control plane is comprised of operations related to the allocation and management of I/O resources, like pre-allocating message buffers for the communications or implementing security checks on access requests. The data plane refers to the actual data transfer operations. In a traditional OS, these two planes run both in the kernel space, but in user space networking, the data plane runs in the user space. Separating these planes allows the more time-consuming control plane operations to take place only when needed, instead of repeating them on every data transfer, reducing the time cost of communications. Moreover, if the data plane fails it can be restarted independently, whereas in a kernel network stack the control plane would fail as well, leading to a slower restart of communications.

Due to achieving low latency and high throughput comparable to hardware-based approaches for kernel bypass, user space networking is a topic that has caused a lot of interest in the past decade. During that time, this interest led to the development of various tools for packet processing, entire TCP network stacks run in the user space and even data plane operating systems, that combine user space networking with the OS kernel, to provide fast but also secure communications. The survey proceeds with providing more details on these user space networking implementations.