Towards Reproducible Evaluations for Flying Drone Controllers in Virtual Environments

In recent years, quadcopter-style drones have been widely available due to their highly portable features, and users can control their drones with dedicated controllers. Drones can serve as a representative in our physical world, and users with augmented reality headsets, by leveraging the drone views, can teleport to such environments [1] for various industrial and commercial purposes [2][3]. Although the two-handed remote control transmitters (RCT) dominate the way of drone flights, one-handed controllers, e.g., commercial products: FT Aviator and research prototypes: DroneCTRL [4], allows users to enjoy higher mobility by reserving a spare hand for other tasks, e.g., holding a handrail in a train. Furthermore, researchers have explored alternative solutions to pursue more natural human-drone interaction [5] that promotes invisible interfaces inherited in our bodies, e.g., gaze-driven drone flights [6] and ring-form addendum [7]. However, the proliferation of flying drone controllers leads to difficult comparisons between research prototypes. Additionally, the performance of such controller prototypes is subject to flying tasks in real-world environments that include a collection of external factors, such as physical obstacles, and weather conditions (e.g., windy or rainy days), More notably, replicating research prototypes and flying tasks are usually time-consuming and expensive.

Therefore, we leverage virtual environments (Microsoft Airsim) to develop an evaluation tool that offers comprehensive profiles of controller prototypes. Our virtual environment enables interaction designers to create new tasks (e.g., flying routes, checkpoints, and obstacles), and hence records both the performance of users and flying drones. In our current tool, the performance refers to users’ task completion time, as well as drones’ velocity, acceleration, jerk, and flying trajectory. A newly designed controller can be evaluated by a task in which a flying drone reaches checkpoints in a high-resolution virtual world. It is worth noting that constraints of task design appear with traditional measurements in physical worlds (Figure 1), including short and simple task [7], manual measurements [8], room-size setup, short battery life and hence long recharging time [1]. In contrast, conducting evaluations in virtual environments can potentially get rid of the aforementioned constraints, hence facilitating the evaluation progress.

To illustrate the prominent features of our evaluation tool, we implemented a prototypical interface of a SOTA controller [7], driven by one-handed and one-thumb operations, and further compared the SOTA interface with a baseline that emulates a two-handed tangible controller. Our evaluation tool reveals new insights into both the two-handed and one-handed controllers, in which two-handed controller remains superior to the SOTA method in complicated scenarios. Interaction designers can utilize this design cue to rationalize the practical use cases in physical environments. To conclude, this paper primarily contributes to an evaluation tool for drone controllers with comprehensive metrics that offers comparable results between controller prototypes, facilitating result replicability and open-data research.

This section highlights the related work of aircraft control, simulation, and virtual environments. As one of the most widely accessible robotic crafts to the public, drones have attracted many researchers’ attention. Different controllers are developed to enhance human-drone-interaction scenarios, including the traditional hand movement controller [9], eye-gaze-based controller [10], voice-based-controller [11], etc. However, all of these controllers should be tested in a standardized environment, which takes great effort to deploy.

Experiments for robotic deployment in physical environments calls for resources of time and space. Thus, simulation platforms have been built, allowing easy-to-access, faster and low-cost evaluations, for instance, behavioral dynamics between human avatars and robotics [12], simulation platforms enable emulations of 3D-printed robots [13], an interactive robotic arm for housework [14], social robot navigation [15], etc. Lately, evaluation of commercially-available service robots moved to online platforms for scalable evaluations. On the other hand, the emerging AR/VR in recent years can serve as an alternative to simulation. The virtual environment can go beyond physical constraints. For instance, AR can help architects to understand the user perception of movable walls [16], while building such movable walls in real buildings are usually impractical. Similarly, mixed reality robots can induce people to empathize with bad incidents [17]. VR environments can deliver a testing ground of risky operations in which injury and casualty are not affordable, for instance, virtual driving and risky driving events [18], and virtual jumping between intervals to observe users’ locomotion [19].

We acknowledge that numerous open-source drone flight simulators exist, e.g., autonomous drones [20] and training modules for human operators [21]. However, a recent survey [22] highlights the difficulty of evaluating one flight controller, i.e., substantially high development time and testing efforts. Also, our work uniquely leverages virtual environments to resolve the lack of standardization of flight controller architectures and hence, drone controllers’ evaluation.

Our system aims to provide a ubiquitous and open-data environment for human-drone interaction research, with a focus on the designs of drone controllers. We implement the system with four significant modules in order to create a standardized environment for conducting the human-drone experiment systematically (Figure 2). As mentioned, one of the limitations of drone experiments is the difficulty of efficiently comparing results among the research community. To resolve this problem, we aggregate the data from Input, Pathway, and Environment modules into experiment logs for a reproducible and trackable experiment. Another challenge is the external factors of drone experiments, for instance, the drone models, drone sizes and speeds, and the physical environment: wind speed and weather. For manipulating the external contributors, we implement the Environment module and Drone state manager indirectly linked to the Airsim platform [23], based on the UNREAL engine. Therefore, we can standardize all the experimental factors and perform human-drone interaction experiments in an identified yet united environment.

To understand the efficacy of VRFlightSim for evaluating controllers with Human-Drone interaction, we implemented two common drone operations, namely drone pointing and crossing operations on the Microsoft Airsim platform (version: 1.5.0.) [23] with UNREAL Engine (version: 4.25.4) The pointing operation refers to a drone flying from one location to another, and both start and end points require the drone to take off and land, respectively [24]. The crossing operation refers to a drone flying through a door frame in mid-air [8], which can be regarded as a subset of the pointing operation.

This section first highlights the results of crossing operations and subsequently pointing operations with all the metrics. All the participants’ performance with two kinds of controllers under different operations are parsed by three high-level perspectives: completion time, drone states (Velocity, Acceleration, and Jerk), as well as trajectory areas. We depict all the statistical analyses (two-way ANOVA), in Tables I and II, for the crossing and pointing operations, respectively.

All the graph plots in this section follow the difficulty index (i.e., ID on the x-axis), inspired by Fitts’s Law [8]. The original definition of Fitts’s law states that the amount of completion time it takes for the device moving from the current position to the target is highly related to the width of the target$(W)$ and the distance between two points$(D)$. To be more precise, the completion time is proportional to $log(2D/W)$ (i.e., Fitts’s index of difficulty), which has been commonly used to determine whether a controller is easy to handle in the scenario of human-computer interaction.

We collect all the experiment results of crossing & pointing tasks, and draw the linear relationship of dependent variables (e.g., time) in each trial and review the performance in different difficulties. Note that the larger the proportional coefficient of the ID is, the more challenging users could handle the drone flight as the tasks’ difficulty grows.

In this paper, we designed and implemented an evaluation methodology with an open-source virtual environment (Microsoft Airsim), named VRFlightSim, to examine alternative input methods (i.e., new controllers) for drone flights. VRFlightSim aims to improve the ease of collecting drone performance data, such as completion time, flying trajectory, velocity, acceleration, and jerk, with flying diversified tasks, albeit in virtual environments.

Identifying a large number of potential candidates of input methods with real-life experiments of drone flights are often tedious and expensive, while our methodology lowered the hurdle by offering (1) a preliminary understanding of flight controllers, and (2) virtual drones and environments as a comparable yet open testing ground that can potentially accumulate data from various testers/experiments. We note that VRFlightSim can screen out the inappropriate candidates, and work complementary with the real-life experiments, as a preliminary step in the entire cycle of interaction design. More importantly, the interaction designers have to conduct real-life experiments to test the effect of physical form factors caused by the actual devices, e.g., moving from a smartphone’s touchscreen to a ring-form device [7].

We further demonstrated the necessity of VRFlightSim with two flying tasks, namely pointing and crossing. We followed the proposed control method in [7], and re-implemented two control methods for a drone. VRFlightSim reveals a new profile of the one-handed control method driven force-assisted input. As reported in [7], their tasks show fixed difficulty ID, e.g., moving towards a direction (e.g., upward) for a 2-meter distance. In contrast, virtual environments can quantitatively capture the difficulty of flying tasks. Accordingly, VRFlightSim facilitates the construction of predictive models [25].

Furthermore, as indicated by the Fitts’s Law, the task difficulty allows interaction designers to investigate the robustness of their proposed input methods in various scenarios, and further offers insights of mapping the controller designs with tasks. For instance, the intersection between two lines could indicate that a controller works better with certain tasks. As highlighted by the performance time during the pointing task, the user population with a higher preference to the one-handed controller (Figure 7(c)) reflects that the one-handed controller can outweigh the two-button controller until it encounters tasks over 3.50 difficulty. Interaction designers with these cues can define that the highly mobile controller, driven by one-handed input with force levels, is suitable for easier tasks. Furthermore, the users with a neutral preference for both methods (Figure 7(d)) show the intersection at the task difficulty index value of 3.00. As such, interaction designers can narrow down the interaction scenarios, e.g., short-distance drone flights to landing targets of middle to large sizes.

Human-drone interaction requires a common ground for evaluation. That is, the existing prototypes of drone controllers were experimented with under different conditions and external factors that led to difficult comparisons between studies. Re-building the testing environments or re-implementing the input methods are challenging. It is worthwhile to mention that a recent work [7] limits to simple flying tasks and lacks data collection channels for drone movements (e.g., trajectory, jerk, acceleration). Otherwise, setting up numerous markers in room-scale environments to track the trajectories for such data collection is costly and time-consuming [1], not to mention the efforts of building physical obstacles [8]. Thus, our paper calls for research efforts leveraging a common virtual platform of drone flight controls. Researchers can collaboratively employ our evaluation methodology¹¹1https://github.com/alpha-drone-control/VRFlightSim to compare diversified controllers under a shared testing ground.