Random Network Distillation Based Deep Reinforcement Learning for AGV Path Planning

The key to solving the AGV path planning problem using RL is how the AGV agent updates its own action policy based on the received rewards to obtain the maximum cumulative reward value. In the real AGV path planning environments, the rewards are sparse, in this case we need to use intrinsic rewards to guide the AGV agent to fully explore the state space and action space in the environment, so we design a new exploration mechanism RND-PPO to motivate the agent to explore the environment.

The structure of our proposed RND-PPO is shown in Fig. 2. The AGV agent training process is divided into two stages. The yellow box is the RND stage and the other part connected to it is PPO training stage. RND defines a new training stage and the RND training alternates with the training of the agent. The model obtained from the RND training is input to PPO and used to generate the corresponding intrinsic rewards. The next stage is the agent training stage, which is a stage of using the trained RND model, combining the intrinsic rewards predicted by the RND model with the RL algorithm PPO. In the end, the agent completes the learning of the optimal policy by using the obtained extrinsic rewards $r_{t}^{e}$, intrinsic rewards $r_{t}^{i}$ and environment state $s_{t}$.

To address the lack of exploration of PPO in sparse reward environments, among the intrinsic reward methods used for exploration, we invoke Random Network Distillation (RND) which is a technique based on prediction error. The model is presented in the yellow part of Fig. 2. In RND, the agent first constructs a randomly-fixed target neural network $f$, where fixed means that it will not be updated throughout the learning process, and constructs a prediction network $\hat{f}$, whose goal is to predict the output of the randomly-set target network $f$. The target network $f$ and the prediction network $\hat{f}$ map the observations $\mathbb{S}$ to the reward $\mathbb{R}^{k}$. The target network defined as: $f:\mathbb{S}\rightarrow\mathbb{R}^{k}$, network parameter is denoted as $\varphi$ and remain fixed after random initialisation. The prediction network defined as: $\hat{f}:\mathbb{S}\rightarrow\mathbb{R}^{k}$, network parameter is denoted as $\hat{\varphi}$ which is trained to minimise the prediction error. The parameter $\hat{\varphi}$ is updated by minimising the expected value of the mean square error $\big{\|}\hat{f}(s_{t};\hat{\varphi})-f(s_{t};\varphi)\big{\|}^{2}$ through gradient descent algorithm. The agent will input the observation $s_{t}$ obtained from the environment into the target network $f$, at which time $f(s_{t})$ serves as the prediction target of the prediction network $\hat{f}$. When the prediction network $\hat{f}(s_{t})$ is input with a novel state, due to the large discrepancy between this distribution and inputs it has ever received, the agent will receive a large intrinsic reward as

In the sparse reward environment, we propose an exploration mechanism that uses RND based PPO to motivate the agent to find more novel state $s$. First, we give the concept of state novelty which can be measured by the prediction error. For AGV agent observing the state $s$ at the current moment, the fewer the number of states similar to state $s$ among all previously visited states, the more novel state $s$ is.

The PPO algorithm is essentially a model-free algorithm, and its core architecture remains an Actor-Critic algorithm. The critic network fits the state value function and action value function through the environmental state information $s$ observed by an agent, and updates critic network parameters by calculating advantage function and using the mean square error as critic loss function. The advantage function is shown as

where $\beta$ is a tunable coefficient. Different estimates can be obtained by adjusting $\beta$. When updating the actor network, PPO uses two networks with the same structure to preserve the old and new policy. The policy ratio is used to measure the ratio of the probability of taking a certain state-action pair $(s,a)$ under the new policy to the probability of taking the same state-action pair under the old policy. The policy ratio is defined as

PPO introduces a new clip mechanism, which can effectively reduce the number of computation steps while limiting the magnitude of policy update, and it is defined as follows

where $\epsilon$ is a hyperparameter, $\hat{A_{t}}$ is an estimate of the advantage function at time step $t$. The purpose of setting $1-\epsilon,1+\epsilon$ is to specify the magnitude of the policy update to prevent the update from being too large and causing the training to be unsmooth.

In our proposed method shown in Algorithm 1, the first three lines initialise various parameters of the RND-PPO. After that, the training process of the RND model starts to indicate lines in Algorithm 1. The parameters of the target network $\varphi$ are fixed, and according to the stochastic gradient descent method, the expected value of the mean square error $\big{\|}\hat{f}(s_{t};\hat{\varphi})-f(s_{t};\varphi)\big{\|}^{2}$ is minimised, and prediction network parameters $\hat{\varphi}$ are optimised. This RND process can be regarded as doing distillation between the target network, which is randomly generated with fixed parameters, and the prediction network, whose parameters are to be updated, so that the prediction network is constantly close to the target network. Then the training process of the agent using the PPO algorithm based on intrinsic and extrinsic rewards starts to indicate lines in Algorithm 1. These rewards are first normalised separately to compute the final set of training trajectories. In the last stages, it combines intrinsic motivations with extrinsic rewards to calculate the advantage function and value function, subsequently refining PPO by updating the policy parameters $\theta$.

The AGV agent body and the target object are spheres with a radiuis of $0.5$ per unit length, the size of the simple scene is $20\times 20$, the size of the complex scene is $40\times 40$. The maximum number of steps for each learning episode of the static and dynamic experiments in the simple scene is 2000, and the maximum number of steps for the complex static and dynamic scenes is 3000 and 4000, respectively. The number of learning episodes in each experiment is $1\cdot 10^{6}$. The reward function of AGV agent is shown in Eq. (1). All the experiments are carried out on an AMD Ryzen 7 5800H 3.20 GHz PC with 16GB memory.

The simple scene is a $20\times 20$ map: $(-5.0,0.5,-8.0)$ is the start location of agent, and $(5.0,0.5,-1.5)$ is the location of target object in static experiment. In dynamic experiment, the target object will be randomly generated in $(5.0,0.5,-1.5)$ and $(-8.0,0.5,-1.0)$.

We test our method in the simple scene and choose three metrics including training episodes, environment cumulative reward and episode length, to evaluate our experimental results. First, we test our proposed method in the simple static environment. In the simple static environment, after $0.18\cdot 10^{6}$ episodes of training, our proposed method RND-PPO is able to obtain the environmental cumulative reward of $4.8$ within $280$ steps of an episode. In contrast, as shown in Fig. 3, the PPO algorithm without RND performs poorly, and the agent is able to obtain the same environmental reward value within $238$ steps over $0.39\cdot 10^{6}$ episodes of training. However, performance of PPO is worse in the simple dynamic environment where there are two randomly generated target objects, and since there is no intrinsic reward for exploration of the environment. It is difficult for PPO to explore the location of the other target object that would earn a reward as shown in Fig. 4. Although PPO relies on search by chance to find the first target object faster than RND-PPO, it needs to spend a large number of training episodes in searching the second target object. PPO can find the target object with an average of $170$ steps after requiring $0.52\cdot 10^{6}$ training episodes and get an environmental cumulative reward of $4.8$. In contrast, our proposed RND-PPO method only requires $0.18\cdot 10^{6}$ training episodes to get same environmental cumulative reward with an average of $187$ steps.

The complex scene is a $40\times 40$ map: $(-12.0,0.5,-16.0)$ is the start location of agent, and $(17.0,0.5,15.0)$ is the location of target object in the static experiment. In the dynamic experiment, the target object will be randomly generated in $(15.0,0.5,2.0)$, $(15.0,0.5,-17.0)$ and $(-17.0,0.5,15.0)$. Fig. 5 and Fig. 6 show the trajectory of the AGV agent after training $0.25$, $0.5$, $0.75$ and $1.0\cdot 10^{6}$ episodes in the complex static and dynamic environment respectively. The AGV agent trained using RND-PPO has found the path to reach the target object after $0.2\cdot 10^{6}$ episodes, while the agent trained only by PPO is still exploring the space around the starting position. The two metrics used for evaluation can be found in Fig. 7, and our proposed method is the better performer on both data. In the static environment, after the same training of $0.2\cdot 10^{6}$ episodes, our proposed method is already able to obtain an environmental cumulative reward of $4.85$ in $492$ steps of an episode, while the PPO can hardly get an environmental reward value of $-1$, which means it still expores the environment. The agent trained by PPO is able to reach the same environmental reward value of $4.85$ only after beeing trained for at least $0.34\cdot 10^{6}$ episodes.

In the dynamic experiment, Fig. 6 shows the experimental results of the AGV agent after being trained by PPO and RND-PPO. It can be seen that our proposed method RND-PPO can find the optimal path quickly and accurately when the target object randomly appears in three positions. However, the agent trained only by using PPO can only find the target object located in the upper left corner due to the fact that there is no intrinsic reward that can motivate the AGV agent to explore the whole environment. The relationship between the three metrics is shown in Fig. 8. The agent trained by RND-PPO found the first target object after $0.07\cdot 10^{6}$ episodes of training, while PPO did not complete this goal until around $0.16\cdot 10^{6}$ episodes. During the $0.08-0.16\cdot 10^{6}$ episodes of training, the curve of RND-PPO fluctuated due to the presence of dynamic objects, and fell into a short struggle in exploring the new environment. But soon with the help of the intrinsic rewards of RND, the agent learnt the paths to reach the three target objects. The agent under RND-PPO training is able to reach more than 4.8 environment cumulative reward after $0.24\cdot 10^{6}$ episodes with $257$ steps per episode. The PPO, on the other hand, still failed to complete the entire path planning task until the end of training. In conclusion, our proposed method explore static and dynamic environment faster in both simple or complex scene than the agent trained with PPO only.