Reactive Multi-Robot Navigation in Outdoor Environments Through Uncertainty-Aware Active Learning of Human Preference Landscape

Human preference $H$ is parameterized by a set of actions $A$ with $|A|=d$. Each action $a\in A$ represents one factor characterizing the multi-robot flocking process. In this work, multi-output Gaussian Processes (GPs) are used to model the underlying human preference function $f(\cdot):=[f^{(1)}(\cdot),f^{(2)}(\cdot),...,f^{(d)}(\cdot)]^{T}$, which doesn’t have strong assumptions about the form of $f(\cdot)$. Human preference feedbacks in trial $i$ are collected into a data set $Data_{i}=\{f_{i}^{t}+\epsilon_{i}^{t}|t=1,2,...,N_{p}^{i}\}$, where $N_{p}^{i}$ denotes the total number of collected human feedbacks, $\epsilon_{i}^{t}\in\mathbb{R}^{d}$ represents noises in human feedback. Defining $\hat{f}(\cdot)$ as the maximum posterior (MAP) estimation of $f(\cdot)$ given $Data_{i}$, we aim at using a Bayesian approach to iteratively update the parameters of $f(\cdot)$ to minimize the prediction error ($|f(\cdot)-\hat{f}(\cdot)|$).

After the problem statement, we are now ready to use multi-output Gaussian Processes [24] to learn human preference function $f(\cdot):\mathbf{X}\to\mathbb{R}^{d}$. The $p^{th}$ output of $f(\cdot)$ is denoted as $f^{(p)}(\cdot)$. And the distribution of the stacked outputs ($F(X)=\{f(x_{n})\}_{n=1}^{N}$) for all $N$ inputs $x_{n}$ is Gaussian distributed, where $X=\{x_{n}\}_{n=1}^{N}$ and the input $x_{n}\in\mathbf{X}\in\mathbb{R}^{D}$ is $D$-dimension combined features. Like single-output GPs, multi-output GPs are fully specified by their mean and kernel functions. The matrix-valued kernel obeys the same positive definiteness properties where the input space $\mathbf{X}$ is extended by the index of the output:

Then the distribution of $F(X)$ can be denoted as:

Then, a linear transformation $W\in\mathbb{R}^{d\times L}$ of $L$ independent functions $g_{l}(\cdot)$ is used to construct multi-output function $f(\cdot)$:

where $k_{l}(\cdot,\cdot^{\prime})$ is a squared exponential kernel.

Given that the human feedback for each output has a different noise level, a Gaussian process model where different noises are assumed for the data points is used:

In this paper, the covariance of noise ($\Sigma_{noise,n}\in\mathbb{R}^{d\times d}$) is represented as a diagonal matrix:

where $\alpha_{ij}$ and $\beta_{ij}$ are hyper-parameters related to human feedback noise. Then the distribution of $F(X)$ combined with noise is shown as follows:

where $\mathbf{K}_{noise}=diag(\Sigma_{noise,n})_{n=1}^{N}\in\mathbb{R}^{Nd\times Nd}$ is a diagonal matrix. With $d$-dimension outputs and $N$ samples, a $dN\times dN$ covariance matrix with $\Theta(d^{3}N^{3})$ computational complexity is needed for obtaining an accurate multi-output GP. Hence, a posterior approximation method (stochastic variational inference) is used. By introducing a set of inducing variables ($\mathbf{u}=\{f(x_{m})\}_{m=1}^{M}$), the computational complexity can be reduced to $\Theta(d^{3}M^{3})$. At last, the Adam optimizer can optimize the noise variance and the variational parameters separately. Capturing the essence of our theoretical groundwork, Procedure 1 streamlines the execution of the PLBA framework (shown in Fig. 2), from mission initiation to MRS safety assurance during ongoing operations. The algorithmic procedure outlined offers a succinct roadmap for dynamic decision-making, integrating path planning and real-time adaptive behavior modeling, which is pivotal for the MRS’s performance in diverse environments. As for the optimization details, please refer to [24, 25].

In this work, we extended the flocking method proposed in [26] by involving obstacle-free largest convex region searching and optimization-based MRS behavior planning for stable and safe robot behaviors. The desired MRS behavior adjusting velocity is calculated by:

where $i$ is the index of a drone and $v_{i}^{flock}$ denotes the flocking speed. $v_{i}^{fmt}$ represents MRS team formation maintaining term. $v_{i}^{rep}$ is short-range repulsion term and similarly $v_{i}^{att}$ denotes long-range attraction term. $v_{i}^{saf}$ keeps a safe distance between robots and obstacles. $v_{i}^{hei}$ maintains the robot at a preferred flying height. $v_{i}^{ali}$ is velocity alignment term. Then to prevent $v_{i}$ from largely exceeding the human preferred flying speed, an upper limit $h_{speed}$ is introduced:

Specifically, each type of speed can be parameterized by one or a few human preferences in the set $H=\{h_{inner},h_{height},h_{speed},h_{safety},h_{formation}\}$. $h_{inner}$ denotes the minimal distance between robots; $h_{height}$ denotes the flying height of robot system; $h_{speed}$ represents the flying speed of multi-robot system; $h_{safety}$ represents the minimal distance between robots and obstacles; and $h_{formation}$ denotes coordination pattern. Next, we determine the largest convex region of obstacle-free space using an iterative regional inflation method based on semi-definite programming [27]. Drones are then assigned to the vertices of this convex region to establish the formation. The flocking model helps the drones maintain their relative positions within the formation throughout the operation.

To maintain informative and helpful interactions with humans and learn a more representative preference model $f(\cdot)$ via human feedback (developed in Section A), Algorithm 1 was developed to decide the timing and manner of seeking human guidance. In Algorithm 1, an interactive learning framework was designed by utilizing active learning to inquire informative guidance from humans. The mean and covariance of human preference predictions are calculated during the learning process. A high covariance value means that the human preference model does not generalize well to the current environment; thus, more human guidance is needed to refine the preference model. With the guidance of low-variance human preference, which means good preference estimation, MRS can adjust robot behavior safely without aggressive or unexpected adjustments.

In Fig. 3, an ”MRS search and rescue in flood disaster” environment was designed based on the software AirSim [29] and Unreal Engine [30]. The environment (400 $m$ $\times$ 400 $m$) was designed to include various scenes (city, bridge, forest, park, etc.), which is enough to test PLBA’s effectiveness in supporting unstructured environment adaptation. Five kinds of flocking behaviors that can illustrate human preferences were considered: 1). The ”flying height” denotes the average flying altitude of the MRS; 2). ”Coverage area” is represented by the minimum distance between drones; 3). The ”safe distance to the obstacle” is the minimum distance between the drone and obstacles; 4). ”Flying speed” represents the average speed of MRS; 5). ”Team formation” is represented by the assigned Cartesian positions of all robots.

In this work, outdoor environments mainly consist of four typical scenarios – ”City”, ”Park”, ”Forest”, ”County”, and ”River”. According to obstacle density, these scenarios can be roughly classified into three categories ”Cluttered”, ”Structured”, and ”Open Space”. ”Open Space” represents a scenario where there are no or few physical obstacles (e.g., lake and park). In ”Open Space” environments, humans can operate MRS more aggressively to achieve fast mission progress. ”Cluttered” represents areas that are characterized by an excessive accumulation of objects or debris, which can make it challenging to move around or perform activities (e.g., forest). Therefore, humans should operate MRS more reserved to pay more attention to robot safety. ”Structured” refers to spaces that are designed and organized with well-defined boundaries and guidelines, which are helpful to promote safety and consistency outcomes (e.g., city). Fig. 4 shows expected robot behavior corresponding to three specific environment types.

To analyze the effectiveness of PLBA in adaptation to outdoor environments, a test procedure for MRS to continuously adapt to multiple environments was designed; two baseline methods (linear regression and neural network regression) were used as comparison groups. The adaptation speed and prediction errors of human preferences in unfamiliar environments were comparatively analyzed. In addition, the relationship between environmental similarity and human preference prediction error was also analyzed, illustrating that the spatial correlation between environmental features can be used for environmental adaptation.

Human User Study. With the pioneer user study involving 20 human volunteers, the mean and variance human preference values in various environments were calculated. Then, two kinds of human preferences – simple human preference only with mean values and complex human preference with mean and variance values were conducted to validate the effectiveness of PLBA in rapid adaptation to unstructured environments. The details of human preferences are listed in Fig. 5. The sample behavior illustrations for human preferences in different environments are shown in Fig. 4.

Environment analysis and learning Performance Validation. Under the guidance of volunteers, human preference over motion behaviors such as {”covering area”, ”flying height”, ”distance to an obstacle”, ”flying speed”, and ”team formation” } were collected to generate a data set with $1764$ samples; the data set is mainly sampled from a flood disaster site which can be mainly categorized into eight typical environments. Each category includes about 12.5% of the total data samples. Each sample consists of environment features and corresponding human preference guidance. The environment features are concatenated by environment visual features, MRS status, and the relationship between MRS and the environment. The distribution and similarity of various environments are illustrated in Fig. 6 (C) and (D). As shown in Fig. 6 (A), after five updates, the $L2$ losses between predicted human preference and real human preference generated by PLBA is less than 0.05; while the $L2$ losses obtained from baseline methods are greater than 1.00. THey show the proposed method PLBA has a faster convergence speed to customize MRS motions for unstructured environment adaptation.

Adaptation Effectiveness Validation. The relationship between the prediction errors and environment similarity was analyzed to validate the method’s effectiveness in facilitating adaptation of unstructured environments. The rationale is that when two environments are similar, the prediction errors of a model trained in an old environment but used in new environments will be minimal. Two examples of changes in human preference prediction errors from one environment to another new environment were illustrated. As shown in Fig. 6 (B), the human preference adaptation error will decrease with the increase of similarity between the previous and new environments for our proposed method and all the baseline methods. That means the spatial correlation between environments can be leveraged for fast unstructured environment adaptation. Besides, the adaptation error of PLBA is 0.5 less than that of baseline methods, which means our proposed method is more suitable for fast and accurate unstructured environment adaptation. Fig. 7 shows two samples of new environment adaptation. (A) The human preference prediction error for MRS to adapt from a narrow street to a park environment is around 0.5; (B) while the human preference prediction error for MRS to adapt from a river to a park environment is around 0.3. That means the model trained in a river environment can quickly adapt to the park environment because the part environment is more similar to the river environment than the narrow street environment, verified in Fig. 6 (D).