BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robots

Standardized benchmarks are not common in robotics research when compared to fields like computer vision. This can largely be attributed to how difficult standardizing a robotics test can be, requiring standardized hardware, software, and environments [1]. This has led to a culture of robotics testing via experimentation to prove a hypothesis rather than comparison and evaluation [6]. Corke et al. [6] postulates that this is a factor which limits the speed of progress in robotics research in comparison to similar fields which instead perform regular evaluation and comparison of techniques. Despite being uncommon, there are some typical approaches used when trying to create benchmarks for robotic systems.

The first approach is to use pre-recorded data and evaluate how well algorithms interpret that data for specific tasks. Some well-known examples of this are the KITTI [7], Cityscape [8] and Oxford RobotCar [2] datasets which enable tasks such as object tracking, visual odometry, SLAM, and semantic segmentation to be evaluated. While this approach drives research in data interpretation, it loses the active nature of robotics wherein observations inform actions to solve problems.

Another approach to standardize robotics testing is to provide a consistent environment for robotic testing while leaving other variables of robot design open. This enables both active interaction with the environment, and different hardware and software solutions to be tested and compared. This approach is seen by competitions like RoboCup@Home [9], the DARPA robotics challenge [3], and the Amazon picking challenge [10]. While enabling system comparison, we see three main limitations to this approach. Firstly, these events are too infrequent to drive research in the same manner seen in computer vision and machine learning research. Secondly, while good for systems-level comparison, precise research outputs (algorithms, sensor design, etc.) cannot be easily compared. Finally, these competitions become monetarily restrictive with groups needing access to their own physical robotic platforms, large engineering investment, and significant transport funding.

An interesting approach being newly adopted, is to provide remote access to robot platforms. This is present in challenges like RoboThor [11], iGibson [12] and the Real Robot Challenge²²2https://real-robot-challenge.com/en. This avenue presents promise in giving users access to physical hardware without the costs of setup and maintenance of the hardware for most users. While not currently a highly scalable solution, using real robot platforms provides the most realistic real-world performance whilst enabling interaction. The issue of scalability can be lessened somewhat by combining remote access to real platforms with the use of high-fidelity simulations.

Robot simulation endeavours to provide tools that enable consistent robotics testing. We define two approaches to this which are utilised within the literature.

The first approach is the creation of fully simulated environments where the environments are hand-crafted and designed, typically through using a game engine. Some well known examples of this are AirSim [4] and CARLA [13] for outdoor environments, and AI2Thor [14], RoboThor [11] and Isaac³³3https://www.nvidia.com/en-us/deep-learning-ai/industries/robotics/ for indoor environments.

The second approach is the creation of simulated environments by utilizing real-world data. Generally this comes in the form of a full 3D environment that a simulated agent can explore freely, such as is seen by AIHabitat [15], Gibson [5], and iGibson [12]. Functionally these can act identically to those created in a games engine but generated using sets of depth and image data collected throughout the environment, which are stitched together to create the final simulated environment. Alternatively, real-world data can be used to provide precise real-world sensor readings from specific pre-defined poses within an environment. This is the approach of the active vision dataset [16], which, while not a simulator in the same sense of the others shown here, enables “traversal” between densely sampled poses to simulate movement while providing the precise visual data captured at that location.

There are distinct advantages and disadvantages when using either fully simulated or real-world data simulators. On a practical level, fully simulated environments are easily manipulated and adapted for new conditions (e.g. lighting variations, rearranging objects, etc.). This is more challenging for simulated environments created from manually observing real-world environments. However, manually collecting data typically provides more naturally messy environments with randomly cluttered surfaces. This seems more realistic when compared to the clean and spacious environments given by many fully simulated environments. However, the subject of visual realism between approaches is a hotly contested topic. Without fixed agent poses such as those used in the active vision dataset [16], using real data to create a virtual environment can lead to visual artefacts being introduced and realistic lighting reflections cannot be achieved as lighting is not actively calculated. While this is not an issue for fully simulated environments, current robotics simulators of this type don’t typically have particularly realistic textures (unlike when using real-world data) which can leave object appearances looking “flat”. You can see a simplistic comparison of simulator approaches in Figure 2. Regardless of the approach used, it is important to accept that a sim-to-real gap will always be present when using simulators.

The sim-to-real gap is perhaps best addressed when combined with real-world robot platforms. This is the approach used by RoboThor [11] and iGibson [12] that can directly examine the performance degradation in sim-to-real transfer. This enables rapid repeatable prototyping in high-fidelity simulation while still enabling direct analysis of real world performance. This is also a key component used in the design of our BenchBot system.

Using BenchBot begins by clearly declaring a problem that needs to be solved, a step synonymous with beginning research. The back end of the BenchBot system (including entire robot platforms, simulators, configuration, initialisation, networking, and interfacing) is started through a single script called benchbot_run. The script requires the user to select a target research task, robot platform, and operating environment from the pool of available options. Supported options are listed through helper flags, and the script validates whether the selected configuration is achievable (e.g. running a real robot in a simulated environment is not a valid configuration).

Options are declared to the BenchBot back end simply by creating a YAML file in the appropriate pool describing the configuration option, and providing any necessary data described by the configuration. For example, a simulated environment definition would contain the data for the environment simulation and a YAML file declaring an identifier, the type of environment (simulated or real), a start pose for the robot, a trajectory the robot may use to travel through the environment, etc. Robots are declared as a series of directed connections between robot platform and BenchBot API along with functions for translating data between the two endpoints, and tasks are declared as a list of available robot capabilities.

The next step in the research process is creating a solution to the research problem, which once again has an analogous step in the BenchBot process. The user creates a solution that uses the BenchBot Python API to interact with sensorimotor robot capabilities, obtain back end configuration details, and generate structured task results. A user solution is submitted to a running BenchBot back end using the benchbot_submit script, which supports both native (i.e. running a Python script locally) and containerised (i.e. building a Docker image from a Dockerfile) submission modes. Containerised research solutions can run independently on other systems, enabling easy access and verification of solutions by the research community. Shareable and repeatable research outcomes, a key contribution of the BenchBot system, are a significant driver of progress in the research community.

Design of the BenchBot API is inspired by the “observe and act” framing employed in areas of robotics like reinforcement learning and the OpenAI Gym ecosystem [19]. The API uses data in the task definition to provide a list of sensor observations and possible robot actions to the user. A solution can either combine these manually into a control loop, or declare an agent with the three capabilities required to complete a BenchBot task: choosing an action given a set of observations, knowing when the task is done, and saving results for the task. Breaking the entire process of solving a complex robotics task down into simply providing three functions is an embodiment of the directness in which research can be conducted with the BenchBot system.

Once results have been attained through the BenchBot system, the final step is to evaluate the performance of the solution given the generated results. A benchbot_eval script is provided to pass a collection of results to the underlying Python evaluation module. The evaluation module supports scoring results individually and producing a summary score for multiple results.

An appropriate evaluation method is selected from the pool of available methods based on the task identifier provided with the results. This flexibility recognises that different tasks will have different metrics that best capture their performance, while also consolidating metrics into single reusable implementations rather than each researcher creating their own implementation.

Although the BenchBot system makes generating a result simple, we recognise that a single result is rarely enough to gain a comprehensive understanding of an algorithm’s performance. BenchBot provides a final script benchbot_batch that generates a set of task results by sweeping over a set of different environment and robot combinations. The script, in combination with evaluation tool support for multiple results, allows a user to produce a comprehensive performance profile for their research contribution with a single command. A performance profile comprising of results from multiple varying simulated environments, multiple robot platforms, and real world results empowers researchers to glean more comprehensive and meaningful insights from their research.