A Closed-Loop Perception, Decision-Making and Reasoning Mechanism for Human-Like Navigation

In recent years, autonomous navigation and flexible obstacle avoidance have attracted extensive attention. Conventional navigation approachesKhatib (1986) are usually implemented by global and local path planningFox et al. (1997). These approaches have trouble addressing navigation challenge in an unknown environment as they must be provided with the map in advance. In addition, many dynamic path planning and re-planning algorithmsStentz and others (1995); Koenig and Likhachev (2002); Qi et al. (2021) can also achieve path planning in unknown dynamic environments, but the high computational overhead of these algorithms constrains real-time performance in large-scale dynamic environments. Besides, most of them require fine-tuning of parameter in real applications.

With the breakthrough of deep learning, many researchers have deployed supervised learning and reinforcement learning to achieve automatic obstacle avoidanceChen et al. (2017). But collecting abundant samples for training is another annoying issue, as manual annotation requires tremendous tedious labor. To overcome this limitation, some researchers attempt to adopt imitation learningHo and Ermon (2016); Pfeiffer et al. (2018). Imitation algorithm designs a teacher-student learning paradigm to mimic the teacher’s action output and the intention behind it. Furthermore, some researches focus on modeling the environment. Ha and Schmidhuber (2018) adopted a variational autoencoder(VAE) to learn a compressed spatial and temporal representation of the environment. Although the above-mentioned studies have strikingly improved the performance of the navigation, most of them are based on learning an explicit mapping from perfect observations to deterministic actions. Such an assumption is impractical in a real scenario. These methods may have weak robustness and generalization, especially when confronted with novel scenarios.

Deep Reinforcement Learning(DRL) is an optimization algorithm based on Markov Decision Process(MDP). Its optimization goal is to maximize the expected accumulated discount rewardsSutton and Barto (2018); Mnih et al. (2013). Many researchers have applied DRL to traditional decision-making tasks and achieved remarkable results. Tsounis et al. (2020) applied DRL to robot control and realized autonomous walking of a quadruped robot. Long et al. (2018); Zhang et al. (2021) introduced DRL to solve path planning and obstacle avoidance in a dynamic environment. Some low-resource tasksHe et al. (2016a); Artetxe et al. (2017); Cao et al. (2020) can also be solved using dual DRL. These dual-DRL frameworks utilize the dual property of two tasks to enhance the model performance. However, DRL also has some inherent drawbacks that hinder its performance. One of the troubling challenges is commonly referred to as the “curse of dimensionality”, which hinders the collision avoidance performance in large-scale real scenarios. The other is the poor generalizability and robustness when encountering complex and dynamic scene.

In previous approaches, the model directly maps external observations into actions Long et al. (2018); Zhang et al. (2021). In fact, these observations also contain lots of irrelevant information and measurement noise. We use $o_{t}$ to denote the current observation, $a_{t}$ for current action, and $s_{t}$ for current latent state based on observation. Assuming that $s_{t}$ obeys a normal distribution $N_{t}(s)$ and it represents the valid features extracted from $o_{t}$. The perception, decision-making and reasoning models achieves the transition between $o_{t}$, $a_{t}$, $s_{t}$. The perception and reasoning are two sequence encoders based on observations and actions, while decision-making is a simple decoder depending on the current latent state. This process is similar to human navigation that the perception and reasoning are based on long-term memory, while decision making is an instant subconscious response.

Using DRL to directly train navigation policy from scratch is inefficient since DRL usually encounters a cold start problem. We design a VAE-Enhanced phase to learn from data. This process is similar to imitation learning, where three models learn the general rules from the data. The overview of this process is visualized in Figure 2. It contains two channels:

• •

  Prediction-Denoise-channel: decision-making (D) and perception model(P)

• •

  Reconstruction-VAE-channel: decision-making(D) and reasoning model(R)

After VAE-Enhanced Demonstration Learning, the model is able to learn general collision avoidance rules. However, the limited training data still confines the performance of the algorithm in highly dynamic scenarios. We introduce the RL-Enhanced Interaction Learning to enhance dynamic obstacle avoidance capability through two dual learning processes.

The perception and decision-making models explore the reasonable actions by interacting with the environment, while the reasoning model deduces the latent state distribution based on the action sequence. These two dual processes generate reward feedback for each other and collectively enhance their performance. The whole process is similar to a blind person carefully exploring the outdoor scene and always imagining the surrounding according to the movements.

As shown in Figure 3, the trained parameters from the first phase are used to initialize three models. A value network with random initialization and decision-making model form an Actor-Critic framework for proximal policy optimization(PPO)Schulman et al. (2017) update.

First, the current $o_{t}$ and historical observation sequences $o_{t-n:t-1}$ are concatenated together and fed into the perception model to calculate the mean $\mu_{t}^{P}$ of the latent state distribution. Since policy module in Actor-Critic involves a sampling process, we directly treat $\mu_{t}^{P}$ as the latent state $s_{t}$ without sampling. Then decision-making model(policy module) predicts the action $a_{t}^{pred}$ based on $s_{t}$. After that, the environment executes the action $a_{t}^{pred}$ and returns navigation-reward $R^{nav}_{t}$ . Simultaneously, the reasoning model infers the most likely distribution of latent state $N_{t}^{R}$ based on the current action $a_{t}^{pred}$ and the historical real actions $a_{t-n:t-1}^{real}$. The probability of $\mu_{t}^{P}$ in $N_{t}^{R}$ is treated as a similarity-reward $R^{sim}_{t}$ for evaluating the quality of the $a_{t}$. This process can be formulated as

As Figure 5, compared to other methods in few-shot scenes, our algorithm achieves comparable performance as others in most scenarios, except for Move-base. However, Move-base requires an environmental map in advance.

As shown in Table 1, the performance of all the methods in zero-shot scene is significantly degraded. Obviously, environmental distribution shift does have a dramatic impact, and our method is more robust and generalizable than others. In highly dynamic scenes like $S_{6}$-Obstacle change($SC_{6}$ for short), Move-base is almost impossible to avoid the obstacle (10% in $SC_{4}$ and 15% in $SC_{6}$), but our algorithm achieves the best results(56% and 83%). In complicated maze scenes, our algorithm lead over the runner-up (DM-RCA) by a large margin (9.6% in $SC_{1}$ and 2.3% in $SC_{2}$). Additionally, $SC_{8}$ simulates a complex traffic roundabout, and our algorithm is only inferior to PPO-Pretrained method by a small margin(74.5% compared to 79.75%).

MOE-VUCA performs the worst among all methods in zero-shot scenes, although it has the shortest Arriving step. We guess it suffers from poor generalization due to the parameter fusion trick it used. DM-RCA is a pretty simple but effective model that greatly outperforms other sophisticated models except ours. It reveals that more sophisticated networks may suffer from the challenge of poor generalization. In addition, we investigate the effect of pre-training using our collected data. For the comparison algorithms, the performance after pre-training with our collected data (denoted by *) does not show a significant improvement over training from scratch. We also observe that the Arriving step of ours is much longer than other algorithms. We suspect that the reasoning model encourages the agent to generate more conservative actions for higher similarity-reward $R_{t}^{sim}$, resulting in a longer navigation step. A detailed analysis is provided in Section 4.4.

We analyze the contribution of different components in our algorithm. We design three variants of the algorithm: (1)We remove the reasoning model and then train the algorithm using two-stage phases. (2)We replace the DRW-Loss with the mean square error when updating the reasoning model. (3)Without VAE-Enhanced Demonstration Learning phase. Table 2 illustrates that the reasoning model is essential to our algorithm. The introduction of DRW-Loss and VAE-Enhanced Demonstration Learning can facilitate the learning of the reasoning model.

We investigate the impact of the reasoning model on actions. The outputs of the decision-making model include Forward Velocity $v$ and Angular Velocity $w$. Firstly, we collect the $w$ and $v$ of several methods in the same scenario and analyze the corresponding behavior patterns. In Figure LABEL:fig:visual_distribution, the action($v$) distribution output by DM-RCA are clustered at a larger velocity value($1m/s$), forming a uni-modal distribution. In contrast, our actions form a bi-modal distribution($0m/s$ and $1m/s$). Thus our strategy is more conservative, preferring to brake rather than bypass when encountering an obstacle. Obviously, comparison policies are more dangerous than ours. It shows that the reasoning model makes the system safer by constraining the action output, albeit sacrificing navigation efficiency. Secondly, we analyze the relationship between the high-dimensional latent vectors $s^{R}$ and the action $a^{w}$ in Figure 6. It visualizes the distribution of latent states after dimension reduction. The 90-dim continuous latent states $s^{R}$ are calculated by reasoning model from continuous action sequences $a_{t-n:t}$. These latent vectors are clustered in different regions depending on the action before. It reveals that reasoning model maps similar actions to neighboring regions in latent space, and different actions are projected to distinct. It exhibits a clustering-like effect.

We deploy the algorithm on a wheel robot. As shown in Figure 7, the robot navigates in a crowded and complex building with moving staff. We randomly select goals and the robot navigates to the targets one by one. The robot is equipped with a 16-line 3D-LIDAR (Velodyne-16), a depth camera (Real-Sense D435) and an edge computing device (NVIDIA Jetson AGX Xavier with Ubuntu and ROS Melodic). All computational processes are performed onboard. Despite the considerable discrepancy between the training scenes and the real scenes, the robot successfully achieve navigation without human help. It means our closed-loop reasoning mechanism can successfully transfer from a simple simulator to a real robot, due to the good adaptability of our approach.