A Transferability Metric Using Scene Similarity and Local Map Observation for DRL Navigation

This paper focuses on navigation tasks in unknown environments without a global map. Given only the local perception of the robot, traditional path/motion planning algorithms such as artificial potential fields (APF)[12] and dynamic window approach[13] would likely fail due to the local optimum. In addition, their navigation performance is highly dependent on their selected parameters, while it is generally difficult to update the parameters for an unknown environment[3].

To overcome the limitations of the traditional methods, DRL algorithms have been applied to enhance navigational performances. Sensor data used in the DRL navigation of mobile robots includes but not limited to 2D LiDAR [3, 4, 5, 6, 10, 14, 15], RGB images [16], and depth images [1, 2, 8]. 2D LiDAR data is most commonly used due to the minor sim-to-real gap and extensively represented in the form of vectors [4, 5, 6, 10]. This representation results in a fixed sensory FoV and a fixed-size measurement as the observation.

To relax the constraint on 2D LiDAR observations and enhance the DRL navigation performance, researchers have investigated a number of alternative observation designs. For example, Leiva et al. [3] converted LiDAR data into 2D point clouds with variable sizes as observations and achieved improved navigational performances. Pfeifer et al. [17] and Yao et al. [18] converted 2D LiDAR data to occupancy maps to address more complex and dynamic environments showing great potential in robot navigation with sensor fusion.

This paper proposes a local map-based DRL navigation algorithm independent of a global map. The 2D LiDAR data, the agent position, and the destination position are all fused onto the local map as the observation. Convolutional neural networks (CNNs) are utilized to efficiently extract the features of the fused spatial data and learn a robust navigation policy. The local map-based navigation algorithm is independent of the use of the transferability metric.

To enhance the transferability of DRL, many studies have been conducted to bridge the gap between the training and test performances, including but not limited to domain randomization[19, 20, 21, 22], learning from demonstration[23], domain adaptation[24, 25, 26], and meta-RL[27].

For instance, Chaffre et al.[22] designed a DRL policy with incremental environment complexity to reduce the need for additional training in the real world. Hieu et al.[23] developed transfer learning with demonstrations to accelerate the training process when an autonomous vehicle entered a new environment. Bharadhwaj et al.[26] performed adversarial domain adaptation to learn an encoder that generated the same distribution of latent states over real images as the simulated images and then fine-tuned the learnt policy in real environments. Luo et al.[27] proposed meta-learning on the latent space algorithm for rapid adaptation of DRL-based visual navigation to new observations. The studies above mainly focused on the design of DRL algorithms to increase the success rate when transferred from a training scene to a test scene, but did not look into the quantification or the measure of the DRL transferability.

Chebotar et al.[19], on the other hand, quantified the DRL transferability by introducing a discrepancy function evaluated by computing the difference between the simulated and real-world robotic arm trajectories using weighted $l_{1}$ and $l_{2}$ norms. This discrepancy measure calculates the difference given the motion state inputs, however, cannot be directly applied to calculate the difference of two navigation scenes.

This paper focuses on finding an appropriate metric to quantify or measure the transferability of DRL navigation algorithms but without looking into how to re-train or update the algorithms in the new environment. This paper proposes a new transferability metric for robot navigation using the improved image template matching algorithm to measure the similarity between the training scenes and the real test scenes. Two straightforward and easy-to-implement performance indicators are designed including the global and local scene similarities. The global scene similarity is used to evaluate the transferability of different DRL navigation algorithms. The local scene similarity indicates the safety of the local map-based DRL navigation algorithm when deployed in a new environment without a global map.

Motion planning is considered as a Markov decision process (MDP). The MDP is represented by the tuple ${(\mathcal{S,A,T,}r,\gamma)}$, where ${\mathcal{S}=\{s\}}$ is the state space, ${\mathcal{A}=\{a\}}$ is the action space, $\mathcal{T}$ is the state transition function indicating the probability distribution of the next state when performing action $a$ in the current state $s$, $r$ is the reward function, and ${\gamma\in[0,1]}$ is the discount factor. At each discrete time step, the agent selects and takes an action based on the current state. The agent then receives a reward and transitions to the next state. The objective of the agent is to learn an optimal policy ${\pi^{*}(a\mid s)}$ that maximizes the expected return.

In this paper, we use DQN[28] and TD3 [29] as example DRL algorithms to learn navigation with discrete and continuous actions, respectively. Both algorithms are sample efficient with the replay buffer to collect experience during training and have been proved with high navigation performance [1, 15]. The implementation of DRL navigation is specified as follows. The observation or the input of DRL is the local map $\bm{M}$, which will be discussed in detail in Section IV-B.

Action space: A differential drive robot is selected as the testing platform in this study. The action space of the differential drive robot includes a translational velocity $v_{t}$ and a rotational velocity $\omega_{t}$, i.e. ${\{\bm{a_{t}}:=[v_{t},\omega_{t}]\}}$. Specifically, for DQN, three discrete actions are considered, i.e., ${v=0.15\text{ m/s}}$, ${\omega\in\{-1.0,0.0,1.0\}\text{ rad/s}}$. For TD3, continuous actions are considered, i.e., ${v\in[0.,0.25]\text{ m/s}}$, ${\omega\in[-1.0,1.0]\text{ rad/s}}$.

Reward design: The objective of the agent is to use the oncoming local map observations to reach the destination point as fast as possible without colliding with obstacles. The reward at time $t$, denoted by $r^{t}$, is designed as

Termination conditions: During training, an episode will be terminated if one of the following conditions is satisfied. 1) The agent reaches the goal. 2) The agent collides with an obstacle. 3) The number of the time steps exceeds the maximum time step $\rm T_{max}$. For DQN, $\rm T_{max}$ is set to 10,000. For TD3, $\rm T_{max}$ is set to 2,000.

The 2D LiDAR data, the agent position and the goal position are all spatial information. They are all fused onto the local map as the observation or the input of the DRL algorithm. Using the local map as the observation allows for learning a more robust DRL navigation policy and interchanging LiDAR sensors with different angular resolutions and FoVs at the deployment time.

The local map is denoted by $\bm{M}=\left[M_{\rm o},M_{\rm p},M_{\rm g}\right]\in\mathbb{R}^{3\times H\times W}$. Here, $H$ and $W$ represent the height and width of the map, respectively. $M_{\rm o}$ is the obstacle map converted from 2D LiDAR data. $M_{\rm p}$ is the position map, indicating the agent’s current position. $M_{\rm g}$ is the goal map, indicating the distance and the orientational angle of the destination point from the agent. $M_{\rm o}$, $M_{\rm p}$ and $M_{\rm g}$ are merged together to fuse all the data and generate the local map.

The process of generating the local map is illustrated in Fig. 1. The obstacle map $M_{\rm o}$ is converted from the LiDAR data. The LiDAR data $\{(\rho_{t},\theta_{t})\}$ received at time $t$ typically consists of the distance $\rho_{t}$ and the orientation angle $\theta_{t}$ of an obstacle with respect to the agent expressed in the polar coordinates. The LiDAR polar coordinates $(\rho_{\rm A},\theta_{\rm A})$ is convertible to the image coordinates $(x_{\rm I},y_{\rm I})$ by

The position map $M_{\rm p}$ encodes the agent’s shape and current position. In this paper, a circle with a radius of the same pixel as the agent is used to represent the agent’s shape. The circle in $M_{\rm p}$ is set at the center of the local map at any time step helping the network recognize the danger of collisions with nearby obstacles and the navigational orientation towards the goal destination.

The goal map $M_{\rm g}$ uses the circle to represent the locations of the destination. There are cases where the target point is outside the boundary of the map $M_{\rm g}$. To ensure $M_{\rm g}$ is non-empty, the distance of the goal with respect to the agent at time $t$, denoted by $d_{t}$, is clipped to $d_{t}^{\prime}$ by

The local map fuses all information to take advantage of the powerful feature extraction capability of CNNs. The network architectures of DQN and the actor-critic network of TD3 are shown in Fig. 2. The three convolutional layers of DQN and the TD3 actor network efficiently extract the features of the input local map $\bm{M}$. The feature map is then processed by the global max pooling layer to extract global features. The global features are flattened and fed into the fully connected layers. The output of DQN is the $Q$ value of the three discrete actions. The output of the TD3 actor network is the predicted continuous actions $[v,\omega]$ using the activation function of Tanh.

The critic network of TD3 requires additional input, i.e., the agent’s action. The input of the critic is either formulated as a numerical vector that concatenates the agent’s action and the extracted feature vector of CNNs output $[v,\omega]$ or a fused local map that combines the local map with the action map designed as ${\bm{M}_{a}=[M_{\rm v},M_{\rm\omega}]}$ by replacing non-zero values of $M_{\rm p}$ with action values. We find that fusing $\bm{M}_{a}$ with the local map $\bm{M}$ as the input of the TD3 critic network achieves better navigation performance. The output of the critic network is the state-action estimate value.

The image template matching is an image processing method to search and find the maximum match between a template image and another larger image. Sliding the template image over the image of interest, the algorithm calculates the similarity between the sub-images and the template image and outputs the position with the maximum similarity as the matching result. The similarity is calculated as the normalized correlation coefficient $R_{I,T}(x,y)$ by

The template matching above is suitable for the case when the template and the target object in the image have the same orientation. Matching often fails when there is an orientational difference between the target and the template. Therefore, the template image needs to be rotated to generate a series of templates with different rotation angles $\phi$. The best match score is then updated by

Currently, most DRL navigation algorithms achieve good navigation performance when the test scene is sufficiently similar to the training scene. However, some abnormal behaviors of DRL agents are often observed when tested in dissimilar scenes [10]. We propose a global scene similarity performance indicator based on the image template matching algorithm described in Section V-A. This indicator quantifies the similarity between the training scene and the test scene to evaluate the transferability of DRL navigation algorithms in both similar and dissimilar scenes.

The calculation of the global scene similarity metric is illustrated in Fig. 3. The image to be matched is the global map of the training scene denoted by ${M}_{{\rm train}}$. The template is the sub-images $T_{ij}$ of the global map of the test scene ${M}_{{\rm test}}$ generated through a sliding window. The global scene similarity score $SS_{\rm global}$ between ${M}_{{\rm train}}$ and ${M}_{{\rm test}}$ is calculated as the averaged best match scores by

The local scene similarity is proposed to quantify the transferability of the local map-based navigation algorithm when deployed in a new environment without the global map. Without the global map of the test scene, only the local obstacle maps collected along the agent’s trajectory are available. The metric calculates the similarity between the actual observations in the test scene and the observations in the training scene. Since the global map of the training scene is available and easily obtained, the proposed local scene similarity utilizes the global obstacle map for the convenience of calculation instead of the local map.

The calculation of the local scene similarity metric is based on the image template matching method described in Section V-A. The image to be matched is the global obstacle map of the training scene denoted by $M_{\rm o}^{\text{global}}$. The template is the local obstacle maps converted from the LiDAR measurements in the test scene. The image template matching method finds the averaged best match scores between the local obstacle maps collected in the test scene and the global obstacle map of the training scene.

We find that the direct use of template matching causes the following two problems. First, as shown in Fig. 4(a), the center of the best match sub-image sometimes lies within the obstacle region in the training scene, and the agent cannot reach such a location during the training process. Second, Fig. 4(b) shows all the trajectories of the agent stored in the replay buffer during training with the darkness in red indicating the total number of arrivals. We observe a significant variance among different locations regarding how many times an observation is used in the training. Therefore, it is unfair that different positions in the global obstacle map carry the same level of importance.

To solve the aforementioned problems, the template matching method is updated by introducing a weight matrix in the calculation of the match score. The weight is calculated based on how many times the agent reaches a specific position on the global map during training, which indicates the importance of the match score at the corresponding position. The weight at position $(x,y)$ in $M_{\rm o}^{\text{global}}$ is calculated by

The scene similarity score $SS$ is formulated as follows. First, the agent collects $N$ local obstacle maps ${\{M_{{\rm o}i},i\in[0,N)\}}$ from the test scene. Then, the averaged best match score between ${\{M_{{\rm o}i},i\in[0,N)\}}$ and the global obstacle map ${M_{\rm o}^{\text{global}}}$ of the training scene is calculated by

Considering that the similarity score of the training scene itself $SS_{\rm train}$ varies, to debias the influence of the training scene, the local scene similarity $SS_{\rm local}$ is therefore defined as the relative scene similarity score or the difference between the similarity score calculated in the test scene $SS_{\rm test}$ and the similarity score calculated in the training scene $SS_{\rm train}$, i.e.,

The configuration of the local map and the hyper parameters of DQN and TD3 are listed in Table I. DQN and TD3 are implemented in PyTorch and trained using the Adam optimizer. The generation of the local maps and the calculation of the scene similarity metric use the OpenCV library.

The training and the testing in simulation are carried out in the PyBullet simulator on a computer with an i7-12700F CPU and an NVIDIA GeForce RTX 3070 GPU. The agent is a differential drive robot equipped with a laser range finder that has a max range of 3.5m, a $360^{\circ}$ FoV and 360 measurement readings. A total of 12 scenes with a 10m$\times$10m two-dimensional space are constructed in simulation as shown in Fig. 5. The training of DQN and TD3 takes approximately 18 hours and 3 hours for each agent in scene (a), respectively.

The simulation environment is set up as described in Section VI-A. 17 different agents are first trained using DQN in the same training scene (Fig. 5(a)) with the same hyper parameters listed in Table I but different DQN initial weights and exploration processes. Following that, the agents are deployed in 12 test scenes that include the training scene itself as shown in Fig. 5. The comparison results between the calculated local scene similarity scores $SS_{\rm local}$ and the navigation success rates are analyzed and presented to support the validation of the effectiveness of the proposed scene similarity metric in quantifying the transferability of the navigation algorithm.

Simulation results show that the local map-based DQN navigation algorithm successfully steers the robot to the destination in the 12 test scenes with various success rates. In each scene, 100 destination points are randomly generated to calculate the success rate.

Figure 6(a) shows the scatter plot between the navigation success rate and the local scene similarity metric over all 17 agents represented by 17 different colors. We observe that with a decreased local scene similarity score $SS_{\rm local}$, the navigation success rate in general has a wider distribution or a larger variation along with a decreased mean value. Figure 6(c) classifies all the test data in Fig. 6(a) according to $SS_{\rm local}$ into nine equal intervals ranging from 0.04 to -0.26, showing the mean and standard deviation of the success rate for each interval. We observe a similar trend that an increased uncertainty in the success rate is correlated with a decreased scene similarity.

Figure 6(b) presents the simulation results according to the scene, using 12 different colors to represent the 12 test scenes. Figure 6(d) shows the mean and the standard deviation of the success rate and $SS_{\rm local}$ for each scene. The scene alphabet labels are consistent with that in Fig. 5. Generally speaking, as the local scene similarity score decreases, the mean of the success rate shows a decreasing trend, and the variance shows an increasing trend.

It is noteworthy that scene j has a lower similarity than scene e, but the navigation success rate is approximately equal. We conjecture that the navigation success rate is not only related to the similarity between the training and test scenes but also to the complexity of the scene itself. Although scene e has a high similarity, it contains narrower passages, and thus the agent is more likely to collide during navigation. In contrast, scene j is less similar to the training scene, but with more open space, thus making the navigation task easier.

A TurtleBot4 wheeled robot with a maximum linear velocity of 0.31 m/s and a maximum angular velocity of 1.90 rad/s is used to experimentally test the proposed navigation algorithm and the local scene similarity metric. It is equipped with a RPLIDAR A1M8 LiDAR featuring a maximum measuring range of 12 m, a FoV of $360^{\circ}$ and an angular resolution $\leq 1^{\circ}$. The navigation algorithm runs on a laptop that communicates with the robot in real time. The laptop has an i7-12700H CPU and an NVIDIA GeForce RTX 3060 GPU.

A total of 14 test scenes are constructed in the lab. The scenes and the sampled navigation trajectories are shown in Fig. 7. The order of the scenes is sorted by the local similarity score, arranged from the highest to the lowest. Without loss of generality, the trained agent #10 from the simulation is selected and used in the experiment. In each scene, 30 goal destinations are generated to perform the proposed local map-based DQN navigation algorithm.

Figure 8 shows the scatter plot of the experimental results between the relative scene similarity score and the navigation success rate. The proposed local map-based DQN navigation algorithm successfully steers the robot toward its destinations in the real world. The success rate varies with the local similarity score following the same pattern as observed in the simulation. As the local scene similarity score decreases, the success rate has a smaller mean value and a larger standard deviation, indicating increased uncertainties in the model.

The simulation and experimental results confirm that the local scene similarity metric and the navigation success rate of the proposed local map-based navigation algorithm are strongly correlated. The proposed scene similarity metric has a great potential in providing guidance for the design of training scenes and the transfer learning algorithm so that an improved navigation performance (e.g., increased safety) can be achieved.

To showcase the transferability and robustness of using the local map as the observation, comparison studies are conducted with selected classic and state-of-the-art observations. The global scene similarity $SS_{\rm global}$ is adopted as the evaluation metric.

The selected observations and corresponding network architectures are shown in Fig. 9. The first observation shown in Fig. 9(a) is the sparse LiDAR data proposed in [5]. The network architecture similar to [5] is used to train. The spare LiDAR data is a normalized 10-dimensional vector uniformly sampled from the raw laser measurements from angle $-90^{\circ}$ to $90^{\circ}$. The vector is concatenated with the goal information ${\bm{g}=[d,\alpha]}$ as the input, where $d$ and $\alpha$ are the distance and the orientation angle of the goal with respect to the agent, respectively. The second observation shown in Fig. 9(b) is the LiDAR data after the min pooling operation proposed in [6]. The similar architecture proposed in [6] is used to train. The operation compresses the raw 360 laser measurements into a 36-dimensional vector. The third observation is the LiDAR data after the min pooling and the IPAPRec operation proposed in [10], which is a powerful input preprocessing approach with adaptively parametric reciprocal functions. The initial value of IPAPRec trainable parameter $\beta_{i}$ is set to -1 in this paper. The last observation shown in Fig. 9(c) is the most commonly used dense LiDAR data, which uses the raw laser range measurements as a 360-dimensional vector. The adopted network architecture is similar to [3] which utilizes 1D convolutional layers to extract features of laser measurements.

For each observation, 5 independent agents are trained using DQN and TD3 with the hyper parameters listed in Table I and the reward function in Eq. (4). The trained agents are deployed in 8 test scenes in Fig. 5. Those test scenes are selected for clear presentation of the results considering that their global scene similarity scores $SS_{\rm global}$ with respect to the training scene differ considerably apart from each other. In each test scene, the same 100 destination points are generated for each agent. The average success rate and the average navigation time of each DRL observation with respect to $SS_{\rm global}$ are shown in Fig 10.

As shown in Fig. 10(a) and Fig. 10(b), the success rates of all DRL navigation algorithms using different observations show a decreasing trend as $SS_{\rm global}$ decreases. However, compared to other observations, using the local map achieves significantly better navigation performance with much higher navigation success rates in test scenes with lower $SS_{\rm global}$. Although there is no small obstacle in the training scene (scene a), the local map-based DRL navigation still avoids colliding with small obstacles when deployed in test scenes h, k and l. The results confirm the enhanced robustness and transferability of the local map observation design for DRL navigation. The reason for the higher transferability of the local map is that it converts LiDAR data and the agent’s position to the map, which helps the agent recognize the shapes of the surrounding obstacles as well as the colliding situation by recognizing the overlap between the obstacles and current position on the map. In comparison, Minpool, for example, only takes the minimum LiDAR measurement within a certain angle interval, not able to reflect the shape of obstacles in the angle interval and resulting in collisions when the shapes of the obstacles are dissimilar or more complex than the training scene. In Fig. 10(c) and Fig. 10(d), we observe that the navigation time of the local map in scenes (h) and (l) is slightly longer than other observations. We conjecture the reason is that the local map-based navigation takes safer and more conservative actions to avoid collisions in scenes with lower similarity scores.

While the goal and velocity information are often modeled as a vector input, the local map fused all information onto a map, which helps the CNNs learn all the spatial information together. The proposed local map $\bm{M},\bm{M}_{a}$ is compared to other inputs that use the goal and/or velocity as a vector and the results are shown in Fig. 11. We use $M$ to denote the map input and $V$ to denote the vector input. All the inputs are trained using the TD3 algorithm in Fig. 2 except $M_{\rm o},\bm{V}_{\rm g}$ being trained using DQN because the TD3 agent fails to learn a good policy when using this kind of input. The four selected inputs are detailed as follows.

1) $\bm{M},\bm{M}_{a}$ denotes the use of the proposed local map $\bm{M}$. It converts the velocity information to the action map $\bm{M}_{a}$ as the input to the TD3 critic network. 2) $\bm{M},\bm{V}_{a}$ inputs the action as a vector $\bm{V}_{a}$ to the TD3 critic network and concatenates it with the extracted features of the local map $\bm{M}$ after CNNs. 3) $M_{\rm o,p},\bm{V}_{\rm g},\bm{V}_{a}$ adds the robot position onto the obstacle map [1] denoted by $M_{\rm o,p}$, and inputs goal and velocity information as vectors $\bm{V}_{\rm g}$ and $\bm{V}_{a}$, respectively. 4) $M_{\rm o},\bm{V}_{\rm g}$ only inputs the obstacle map $M_{\rm o}$ and the goal information as a vector $\bm{V}_{\rm g}$.

As shown in Fig. 11, the proposed local map input $\bm{M},\bm{M}_{a}$ achieves the best overall performance with a shorter average navigation time and higher average navigation success rates in almost all test scenes. $M_{\rm o},\bm{V}_{\rm g}$ performs worst since it only inputs the obstacle map without the agent’s position, thus very challenging for the network to learn the colliding situations with obstacles.

Converting the LiDAR data to the local map allows an arbitrary number of LiDAR measurements. The FoV and the angular resolution of the LiDAR sensor are changed at the deployment time to test if the local map-based DRL navigation algorithm still performs robustly well with respect to different sensor setups. The average navigation success rates and average navigation time using different sensor setups tested in scene (a) of Fig. 5 are listed in Table II. The local map-based DRL navigation algorithm maintains a high success rate in almost all presented LiDAR setups. Only when the number of measurements is significantly small (# of Meas.=30), the navigation success rate of the DQN-based algorithm decreases slightly. The results show the robustness and transferability of the local map-based navigation algorithm that allows the interchange of the sensor’s FoV and angular resolution at the deployment time.

A new simulation test scene is constructed to compare the local map-based DRL navigation with the traditional local planners APF[12] and DWA[13]. We use the same agents trained in Section VII-A and the results are shown in Fig.12. APF and DWA fail to reach destinations due to the local optimum, while the local map reaches a success rate of over 70% in the test scene with a low $SS_{\rm global}$ (0.374). When it is trained in a scene similar to the test scene, it is able to achieve better navigation performance. The results confirm the generalization of the local map-based DRL navigation in a new environment.

To demonstrate the advantage of $SS_{\rm global}$ and $SS_{\rm local}$, the proposed metrics are compared to the $l_{1}$ and $l_{2}$ norms used in the discrepancy function of[19]. The $l_{1}$ and $l_{2}$ norms require the same size of the global maps of scenes to be flattened into vectors with the same dimension, while the proposed transferability metric has no such limitation. We utilized the same data in Fig. 6 and the results of the relationship between navigation success rates and different metrics are presented in Fig. 13. There exists no clear correlation between the navigation success rate and the $l_{1}$ and $l_{2}$ norms (Fig. 13(a) and (b)). In comparison, as $SS_{\rm global}$ and $SS_{\rm local}$ decrease, the success rate decreases with a larger deviation (Fig. 13(c) and Fig. 6(d)). The results demonstrate the effectiveness of the proposed transferability metric.

To test the generalization of the local scene similarity, 3 independent agents in simulation using DQN for each observation including Dense, Minpool and IPAPRec with the same setting in Section VII-A are trained, since they use the same amount of LiDAR data as the local map. The success rate with respect to $SS_{\rm local}$ is shown in Fig. 14. With decreasing $SS_{\rm local}$, the success rates of three observations show a decreasing trend with a wider distribution. Although the local scene similarity is designed to quantify the transferability of the local map-based navigation algorithm, the results show a great potential in applying $SS_{\rm local}$ to other observations.