Impact of ROS 2 Node Composition in Robotic Systems

A distributed system like ROS 2 can be conceptualized as a graph of end-points, each representing a process or subsystem. The graph is built from these nodes which act as units of processing. These can be a hardware driver, data processing, localization, and more. Each node connects to other nodes to communicate data via ROS interfaces; namely topics, services, or actions.

There exist three major types of nodes. The most basic is the Node (without qualifiers) which is the base building block for all others. The next is the Lifecycle Node, which implements a state machine for managing the program deterministically for reliable execution. A node’s state may be made dependent on other nodes to create a cascading bringup whereas each node is only activated once its constituent dependencies are also activated.

Finally, the Composition Node allows for a program to be placed within processes either at compile or run-time. This contrasts with other nodes, which are created in their own processes. Composition reduces networking-related delays by using process-local optimizations and reduces overall resource consumption. This is explored in detail in Sec. IV. Note that these nodes are not mutually exclusive; a node can be a lifecycle-component to leverage both deterministic execution and networking optimization.

The Executor is the mechanism by which the order and timing of available tasks are coordinated. The executor allows the communication system’s execution model to be decoupled from the structure of the system graph. As network and timer events occur, they are marked in a structure called a waitset. The waitset is an array of entities, each indicating the status of the corresponding task. Multiple nodes or groups of callbacks can be serviced by an executor. When evaluated, an executor checks the waitset and services the next appropriate timer, topic, service, or action according to scheduling semantics, invoking relevant user-defined callbacks.

ROS 2’s C++ client library currently provides three default executors. The first is a Single-threaded executor, where all incoming events are serviced in order within a single thread. A disadvantage of this executor is that any user-implemented blocking callback will also block other pending tasks.

As an alternative, the Multi-threaded executor uses a pool of threads to execute many user callbacks in parallel. This can, however, cause callbacks to be starved if others use the shared resources greedily. To refine the parallelism, interfaces can be added to a callback group. Each callback group can be mutually exclusive (members should not be executed in parallel) or reentrant (members may be executed in parallel).

An executor’s overhead can be considerable when reconstructing the waitset every iteration. Thus, a third Static single-threaded executor is available. The static single-threaded executor does not reconstruct the list of event sources each iteration, but modifies an existing list as entities are added or removed.

In addition to the default set of executors, additional executors have been proposed: The Priority-Driven Chain-Aware Scheduler [13] improves end-to-end system latency, the rclc Executor [16] provides deterministic scheduling for resource limited platforms, and the events-based executor [17] eliminates overhead found in default implementations’ waitsets.

ROS 2 allows for improved performance through zero-copy communication using IPC and loaned messages optimizations. Both avoid the copying of data to and from the middleware layer, serialization and de-serialization, and interactions with the ROS interfaces.

IPC is used to maximize message passing performance of components in the same process, achieving similar performance to functional programming while retaining pub-sub semantics and modular architectures. When a subscription and publisher of a topic reside in the same process, the underlying middleware can be entirely short-circuited; messages are passed via smart pointers pushed into shared ring buffers [18]. IPC can be enabled through a node’s options, but it presently has some implementation limitations: it is only compatible with a subset of Quality of Service (QoS) settings[14] and enabling it with subscriptions from other processes reduces its benefits.

The loaned messages API borrows memory from the middleware to perform optimizations like using shared memory buffers to enable multi-process zero-copy communication. Using loaned messages requires the use of different publisher APIs and a relatively complex, vendor-specific configuration of the DDS layer and underlying Operating System to allocate the shared memory segment. Messages published using loaned message must also have a fixed size that is known at compile-time. If subscriptions from other machines exist, it is not recommended to enable loaned messages due to double delivery of messages. Tests using loaned messages exhibit stability issues such as frequent crashes and dead-locks, discussed in further detail in Sec. V. In contrast to IPC, these problems are due to the currently available rmw implementations. For these reasons, loaned messages support is considered experimental in ROS 2 as of the time of writing. rmw_iceoryx is an example rmw implementing shared-memory IPC with message loaning [15].

Manual composition is performed at compile time, with a user developed executable that explicitly instantiates components. Since decisions are made at compilation-time, this method provides the least flexibility, but the maximum opportunity to control resource allocation between the components. With manually-defined components in a process, the developer has the ability to explicitly control thread priorities and distribution. Any executor(s) can be used in manual composition and is specified in the program’s entry point.

In general, manual composition with a single executable is most optimal for encapsulating an entire system once all exploratory development is complete and the overall system architecture is static and stable. Typically, this is leveraged in highly resource constrained environments, whereas minor overhead for flexibility is undesirable.

Dynamic components are built as shared libraries and registered with a global index for bookkeeping. They are then available to be loaded into a component container, which instantiates the nodes. This can be done within a ROS 2 launch file or from a service call.

In contrast to manual composition, dynamic composition retains maximum run-time flexibility, allowing designers to distribute components across various processes at or even after launch. Distributing components allows for the grouping of related tasks in the same process, while retaining the ability to isolate components at will. All components may be loaded into a single process, like in manual composition, with little overhead.

Component containers provide a common executable for dynamically composing components, typically surrounding a particular executor. The containers provide services to add, introspect, or remove components via the component manager. A component container will search the component index and load the shared library module that contains a node of interest. If the default containers are insufficient, new component containers may be written without modifying ROS 2 itself.

There are 3 default component containers available:

• •

  A single-threaded executor container, where components share the executor

• •

  A shared multi-threaded executor container, where components share the executor

• •

  An isolated singled-threaded executor container, where each component receives its own executor

Best practices can be summarized as the following:

• •

  All nodes should be written as components

• •

  Prefer dynamic composition over manual composition when not in extreme resource constrained environments

• •

  With many independent components, prefer single-threaded executor(s) over one multi-threaded executor

• •

  Prefer one multi-threaded executor when component(s) are specifically designed to not interfere with each other

The first experiment evaluates the memory requirements of ROS 2 applications, comparing the two composition approaches with a standard multi-process system. Each was tested with a varying number of empty nodes to evaluate baseline memory requirements. The single-threaded executor was utilized for each test in this analysis. Note that the choice of executor does not impact memory utilization.

The results of this experiment are shown in Fig. 2 and indicate a significant improvement utilizing any of the composition approaches. The standard deviation between experimental runs was less than 1%.

Each ROS 2 process creates its own DDS network participant to communicate, which represents a considerable memory increase. The increase in memory in the multi-process system is non-linear: 2.1 MB increase on the second process that grows to 4.3 MB by the twentieth. This is due to the creation of a fully connected and distributed network graph needing to allocate resources for every discovered end-point, growing $O(N^{2})$. This happens during network discovery and it is irrespective of whether the nodes will ever communicate. The consequence is that a 20 node system with composition requires approximately the same memory as a 2 node multi-process system. It is common for robot systems to contain 20 nodes or more; each hardware driver, robotic algorithm, or subsystem are typically modeled as a node. The autonomous navigation system evaluated in Sec. VI-A alone contains 15 nodes, not counting the drivers, autonomy application, etc needed for a complete product. Succinctly, practically complex robotic systems must use composition to stem the quadratic growth of DDS overhead.

Note that dynamically composed systems are slightly more expensive due to the additional component manager node. They also require registration with an index and linking with the component library, both of which may be omitted for manual composition. However, experiments showed a negligible increase in RAM (less than 100 KB) for twenty nodes. Thus, it is recommended to register components to take advantage of flexible dynamic composition for prototyping and testing, even when somewhat resource constrained.

The next experiment evaluates the communication performance of composition. This is not only affected by how the components are organized, but also by design decisions such as the selection of executors and communication settings.

The first test measures the steady-state CPU usage and mean latency across different systems comprising two nodes: one with a publisher and another with a subscription. This experiment was run using both composition types and multi-process systems, each with a single-threaded executor. The comparison also includes a dynamically composed system with IPC and a multi-process system using loaned messages.

Each test is repeated using different message sizes. The publication frequency is set to 50 Hz to allow for appreciable differences without nearing limitations of the embedded platform for larger messages. Subscription callbacks do not perform any computation, thus all of the measurements can be attributed to communication.

The results are shown in Fig. 3 on a log-scale, showing reduction in both CPU and latency by more than half for every message size with either type of composition when compared to the standard multi-process approach. Without zero-copy, performance degrades proportional to the message size.

Multi-process communication without loaned messages has a latency exceeding 30% of the publication period (a key performance indicator) for messages above 1 MB. Enabling loaned messages shows a near constant performance regardless of the message size. This displays the highest overhead for small messages, but later demonstrates a major improvement for the larger messages - although overall remaining less efficient than IPC. The loaned messages optimization showed problems when using messages larger than 2 Mb, which caused the application to repeatedly crash. For this reason, this is not available in Fig. 3.

Each of the composition methods without IPC have remarkably similar performance; nevertheless the manual approach is slightly better due to the absence of the component manager. Although this node does not process work at steady-state, the single-threaded executor is required to check it, adding a minor amount of overhead. Indeed, the performance difference between the composition methods is only perceptible for the smallest message sizes where the manager contributes a larger proportion of the work completed ²²2Later experiments show the ‘worst-case’ (dynamic) due to the very small performance differences, adding clarity to later figures. Manual composition experiments were performed and exhibited analogous behavior..

IPC in dynamic composition displayed a constant latency of 40 $\mu$s throughout all of the experiments, without scaling by message size. CPU is reduced by almost an order of magnitude for larger messages; though it scales by message size due to the time it takes to populate the messages themselves. Composition and IPC allow ROS 2 systems to achieve negligible latency, similar to monolithic applications. Without Composition with IPC, ROS 2 could not support extreme resource constrained systems.

The next experiment computes the maximum amount of data that can be transferred between two entities. The system is made up of a publisher and a subscription node, where we measure the goodput by publishing continuously without pause using a single-threaded executor. This metric measures only the amount of useful data received by the subscription callback, excluding additional payloads used by the rmw for the transmission. The results for variable message sizes are shown in Fig. 4, where single-process solutions can reach a far higher maximum rate.

There is consistent overhead independent of data size from the executor and data handling deep within ROS 2. For this reason, smaller messages have a relatively low goodput as these fixed costs reduce the maximum rate at which a subscription can receive data.

Multi-process without loaned messages reaches its peak for messages slightly smaller than 65 KB, which corresponds to the maximum UDP datagram length [21]. When the payload exceeds that, the DDS middleware fragments the message into multiple packets for transport, with a negative impact on performance. Although the multi-process approach transmits the least amount of data, it uses the most CPU, slightly above 200%. By comparison, a single-process approach used 82%.

Using loaned messages with multi-process communication resulted in deadlocks and other stability issues when the publication frequency reached 500 Hz. This required artificially limiting the maximum rate to 450 Hz to obtain valid data and is why loaned messages underperformed when publishing small payloads. The 1000 KB and 5000 KB tests did not reach the 450 Hz cap, with maximum frequencies 430 Hz and 110 Hz respectively. This allowed appreciable performance improvements to be seen while using loaned messages in the multi-process scenario at these message sizes - though still far slower than IPC.

Enabling IPC yields again the best results. The goodput increases with message size, peaking due to the publisher’s message allocation speed limit.

Our analysis shows that composition with IPC is a requirement for applications that must produce data at very high frequencies, such as above 1 KHz. Many SLAM and Visual Odometry systems require images at 30+ Hz: using a VGA three channel image, multi-process cannot communicate an uncompressed feed at 20 Hz, while composition and IPC can stream at 85 Hz and 415 Hz, respectively. Multi-process solutions with loaned messages can achieve the required frame-rate, but with a higher CPU usage, added complexity and, as of today, with the risk of stability issues.

The component containers presented in Sec. IV-C play a fundamental role in the performance of a system via different executor patterns. In this last experiment, we compared the container’s performance using a simple dynamic composition application. Each system comprises one publisher node, publishing messages of 500 KB at 50 Hz with a variable number of subscriptions. Different from the previous tests, computations of 500 $\mu$s were added to each subscription callback to emulate data-processing (Fig. 5).

The single-threaded pattern requires the least CPU because the pipeline needs to interact with the rmw only once per iteration, regardless of the number of subscriptions. However, it also displays the highest latency since callbacks cannot execute in parallel. Interestingly, when removing work from the callbacks, this approach actually becomes the fastest since the sequential execution of callbacks is outweighed by the reduced rmw overhead.

The isolated and multi-threaded containers have similar performance, but with worthwhile differences. The multi-threaded container spawns the maximum number of concurrent threads supported by the architecture, which are not tied to any specific interface. With only one subscription, parallelization is not possible, so the performance of the multi- and single-threaded containers are essentially identical.

The performance of the isolated and multi-threaded containers are similar as the number of subscriptions increases, with parallelization allowing to achieve lower latency than purely sequential execution. The isolated container however yields better performance after the number of subscriptions exceeds the number of threads in the multi-threaded container (e.g. number of cores). Additionally, the isolated container achieves the best performance when ROS 2’s timers are used for periodic tasks. Timers must be checked at every iteration of the executor, so by using dedicated executors for each node, the overhead of a timer applies only to the executor which contains it.

This experiment highlights that executors behave differently depending on the topology of the system and the constituent node’s particular implementations. In general, the isolated container provides a reasonable default when composing systems of non-trivial nodes. Application developers are encouraged to explore the different modes and develop their own containers with more advanced configurations, such as specifying different types of executors for the different nodes or thread priorities and CPU affinities.

The Nav2 project is used as the autonomous robotics system for analysis [3]. It consists of a configurable and extensible behavior-tree navigator with several modular servers to accomplish tasks such as planning, control, recovery, path smoothing, route following, and more. Each of these servers is modeled as an independent component-lifecycle node that can be placed dynamically throughout the computation network. Nav2 is an ideal candidate for evaluation, with 15 nodes following the best practices in ROS 2 style, API, and design.

This experiment captures the same computation and memory metrics on the same hardware platform as presented in Sec. V. The navigation system is tested in the following configurations: multi-process, manual composition, and dynamic composition. Both composition methods used the isolated single-threaded executor.

In these experiments, the navigation system was executed with a simulated differential-drive robot in Gazebo and measurements were taken during steady-state navigation between two arbitrary goal points on a map generated using SLAM Toolbox with sensor processing for dynamic obstacles [22, 23, 24]. It uses the NavFn global path planner alongside the DWB local trajectory planner processing inputs from an RGB-D sensor and a 2D planar laser scanner to represent a sandbox environment. The experimental setup and its configuration are identical to that which would be used in practical service robotics applications, which can be found in the nav2_bringup package.

The testing scripts and simulator were run on a separate host computer to remove the impact of shared library components on PSS memory metrics, with setup in Fig. 6. The results of this experiment are shown in Tab. I representing the average of 10 trials.

Manual and dynamic composition on the embedded ARM processor both saved an astounding 28% CPU and 33% RAM over multi-process. This is a resounding improvement in performance displayed by a full system containing advanced algorithms by composing them into a single process. While both component containers consumed relatively similar memory, the multi-threaded container consumed far more CPU - on par with full-blown multi-process communication. This is due to the use of more nodes than CPU cores.

A complementary experiment was run using an x86 platform (Intel i5-8300H) to highlight any differences due to compute architecture. In that experiment, composition saved 25% CPU and 44% RAM over a multi-process system architecture - displaying equivalent trends on both major hardware platforms.

These are significant improvements in performance due to only changes in the networking characteristics of an entire autonomous mobile robot navigation system. While continuing to plan, control, and perceive the environment, a robot has more resources to compute additional tasks such as its autonomy system or deploy more advanced perception techniques with the liberated memory and CPU time. Optionally, the freed memory resources could be used to store increasingly larger maps (40 MB can store an additional 100,000+ $m^{2}$) and freed compute time to plan longer global paths - enabling high quality navigation in larger scale spaces. Over 10% of an x86 core and 40% of an ARM core are made available again for additional tasks which would not have otherwise been possible. Since navigation is only one system on a robot, it is essential to optimize its performance to leave room for other necessary subsystems (scheduler, external communication, error checking, payload application, data management, etc).

The iRobot Create® 3 is a robotic platform designed to give developers access to a robust mobile base through standard ROS 2 APIs. It is equipped with a variety of sensors and actuators: encoders, optical flow and IMU for pose estimation, bumpers, cliffs and IR for obstacle detection, and wheels, LED lights and audio speakers controllable by the user.

The robot produces 70 KBps of raw sensor data and processed information, while also running a 50 Hz motion control loop and an obstacle detection pipeline. Users can either publish individual actuation commands or send action goals to execute autonomous behaviors such as pose regulation, wall-following, and docking.

The single-board computer that is present on the robot imposed severe resource constraints to the design of the application, due to a processor with limited CPU and less than 60 MB of RAM. The robot runs a single-process, manually composed, ROS 2 application which takes advantage of IPC and an optimized executor [17]. This ROS 2 system is made of approximately 10 nodes, with more than 30 topics, 10 services and 10 action servers, in addition to the automatically created entities. ROS 2 is used both for the internal implementation of algorithms and drivers, as well as for interfacing with the user.

The iRobot Create® 3 application is normally controlled by an external navigation software, such as Nav2 described in Sec. VI-A, and this results in the use of approximately 60% CPU and 32 MB RAM. More performance intensive operations such as remotely subscribing to all the published topics (e.g. for visualization purposes or to record a log), can increase the CPU usage to 80%. The robot also supports multiple rmw, and with some of them the RAM usage grows up to 40 MB.

By taking into account the current resource usage and the results presented in the previous experiments, it is evident a multi-process solution wouldn’t have been possible. Indeed, in Fig. 2 it is shown how a 10 nodes multi-process system required roughly 3 times the RAM of a single-process, composed, one - far exceeding the embedded compute platform’s resources. The run-time overhead of a multi-process configuration would have also caused consistent thread starvation and a CPU load average [25] above the number of available cores, making it impossible to achieve desired rates and smooth robot actions. Without access to ROS 2 composition, the software architecture of this robot would have been vastly different, with a non-ROS monolithic application and a single ROS node acting as the bridge with the outside world.

When creating consumer products there is often a trade-off between cost and value. The use of ROS does not directly add value to iRobot’s customers; its main benefit is during development. Thanks to ROS 2 composition, it is possible to deploy ROS-based applications on low-cost embedded platforms, as exemplified by the iRobot Create® 3.