A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals

Previous works on next state prediction primarily focus on producing a deterministic prediction and comparing this prediction to the actual next state to produce a loss function, with an optimizer using the loss function to improve future predictions. This method of prediction does poorly in environments with a lot of noise present or if there are stochastic elements present. By using a GAN to generate the next state, the network is not predicting the next state but instead producing one possible next state. In taking this approach the agent is able to learn which transitions are possible and which are impossible, and in doing so gain robustness to noise and stochasticity.

In order to be able to predict a variety of possible transitions we learn a low-dimensional latent space $\mathbf{z}\in\mathbf{R}^{\mathbf{Z}}$ which encapsulates the stochastic aspects of the transition. We then learn a deterministic mapping $G(\mathbf{s}_{t},a_{t},\mathbf{z}_{t})\to\mathbf{\hat{s}}_{t+1}$. Where $a_{t}$ is the action at time step $t$, $\mathbf{s}_{t}$ is the state at time step $t$, and $\mathbf{\hat{s}}_{t+1}$ is a generated configuration of what the state could be at time step $t+1$. We draw the latent vector $\mathbf{z}_{t}$ from a prior distribution $p(\mathbf{z}_{t})$; in this work we use a standard Gaussian distribution $\mathcal{N}(0,1)$. A discriminator network, $D$, is trained to detect synthetic transitions generated by $G$: $D(\mathbf{s}_{t},a_{t},\mathbf{u}_{t+1})\to d_{t}$, where $\mathbf{u}_{t+1}\in\{\mathbf{s}_{t+1},\mathbf{\hat{s}}_{t+1}\}$, and $d_{t}\in(0,1)$ corresponds to the probability that $\mathbf{u}_{t+1}$ is synthetic ($\mathbf{u}_{t+1}=\mathbf{\hat{s}}_{t+1}$).

We use the generator $G$ and discriminator $D$ as a source for an intrinsic curiosity reward similar to those produced from next state prediction error. In order to produce in informative curiosity reward, we force the generator $G$ to attempt to generate a prediction that matches the true next state $\mathbf{s}_{t+1}$. We directly map $\mathbf{s}_{t+1}$ to the latent space $\mathbf{z}$ using an encoding function, $E(\mathbf{s}_{t+1})\to\mathbf{z}_{t}^{\text{enc}}$. The generator $G$ then uses a sample of the encoded latent space $\mathbf{z}_{t}^{\text{enc}}$, the state $\mathbf{s}_{t}$ and action $a_{t}$ to synthesize the desired output $\mathbf{\hat{s}}_{t+1}^{\text{enc}}$. This process can be likened to the reconstruction of $\mathbf{s}_{t+1}$ using an auto-encoder. We define the reconstruction error of $\mathbf{s}_{t+1}$ to be $||\mathbf{s}_{t+1}-\mathbf{\hat{s}}_{t+1}^{\text{enc}}||$, which is used as a component of the curiosity reward function.

We define the curiosity based reward utilizing the reconstruction error and the discriminator’s output $d_{t}$ predicting a synthetic transition:

where $\alpha\in[0,\infty)$ is a scaling hyper parameter.

We consider the traditional RL framework of a Markov Decision Process with a state space $S$, a set of actions $A$ and a transition function $p(\mathbf{s}_{t+1}|\mathbf{s}_{t},a_{t})$ which specifies the distribution over next states given a current state and action. At each time-step $t$, the agent in state $\mathbf{s}_{t}\in S$ takes an action $a_{t}\in A$ by sampling from a goal-conditioned stochastic policy $\pi(a_{t}|\mathbf{s}_{t},g_{t};\theta_{\pi})$ represented as a neural network with parameters $\theta_{\pi}$ where $g_{t}$ is the goal. We assume that some goal verification function $v$ can be specified such that $v(\mathbf{s}_{t},g_{t})=1$, if the state $\mathbf{s}_{t}$ satisfies the goal $g_{t}$, and 0 otherwise. The undiscounted intrinsic goal reward $r^{g_{t}}$ is defined to be:

where $r>0$ is a hyper-parameter. The navigation network is trained to maximize the discounted expected reward, $R^{Nav}_{t}$:

where $\gamma\in[0,1)$ is the discount factor and $r^{c}$ is the curiosity reward.

The subgoal generator neural network is defined as $SG(\mathbf{s}_{t},g_{0})\to g_{t}$, where $g_{t}$ is the current goal, and $g_{0}$ is the initial goal set by the environment. Possible goals are set relative to the agent’s observation space. It selects an input position and a goal value for that position. The navigator network is then tasked with altering the selected input to be the goal value. A new subgoal can be proposed at the start of an episode, once the previous one is reached or the previous one is abandoned. The agent abandons subgoals if it exceeds a hyper-parameter that sets the step limit to attempt to reach the goal. Finally, the network can also choose to set no subgoal, which is taken when the network selects the current goal as the subgoal.

The subgoal generator network is trained to maximize the discounted expected reward:

where $\gamma\in[0,1)$ is the discount factor, $r^{c}$ is the curiosity reward, $r^{e}$ is the extrinsic reward within the environment, $v(\mathbf{s}_{t},g_{t})$ is the goal verification function, and $\beta\in[0,\infty)$ is a scaling parameter which determines the penalty applied to the extrinsic reward when the goal does not align with the reward.

We use a modified version of Door & Key environments within the minigrid package (Figure 1). The environment randomizes the position of the dividing wall, door, and key, as well as the agent’s starting position and orientation. We alter the environment to also randomize the location of the agent’s target position within the second room. The randomization adds increased difficulty by increasing the number of possible trajectories and adds stochastic transitions into the environment.

The environments are comprised of $N\times N$ tiles, where each tile contains at most one of the following objects: door, key, wall, goal. If it contains none, it is the ‘empty’ object. The agent has limited vision, seeing within a $M\times M$ square with the agent positioned on a center edge of the square. The agent cannot see behind walls or closed doors even if they fall within the vision range. Such objects are assigned as ‘unseen’ rather than their true values. All objects and their attributes are transformed using an embedding layer unique for each network before being fed into the remainder of the network. The navigator is able to perform the following actions: turn left, turn right, move forward, pick up an object, drop an object, or toggle (interact with objects such as opening a door).

The navigator and subgoal generator have a network architecture consisting of a fully connected layer with rectified linear units, two LSTM layers, followed by a final fully connected layer. They are trained using the TorchBeast [20] implementation of IMPALA [19].

The GAN’s encoder simply consists of three fully connected layers interleaved with a hyperbolic tangent activation function. Similarly, both the generator and discriminator consist of four fully connected layers interleaved with a hyperbolic tangent activation function. In addition, the singular output from the final layer of the discriminator is activated by a sigmoid function. We train the GAN using the same methods as BicycleGAN [15], with one exception. As BicylceGAN is designed for continuous data sets, we replace the standard generator loss function with the MaliGAN [21] gradient estimator.

For comparison, we implement 3 alternate methods. We use IMPALA without any intrinsic motivation as a standard deep RL baseline. Additionally, we include two alternative methods utilizing intrinsic motivation. Random Network Distillation Exploration (RND) [22] which uses next state feature prediction error as a curiosity based form of intrinsic motivation, where features are produced from a random neural network. AMIGO is also trained where the teacher and student network architectures match those described in the AMIGO paper [18] and accompanying code https://github.com/facebookresearch/adversarially-motivated-intrinsic-goals. The AMIGO networks receive the full absolute view of the environment, as opposed to the agent centric partial view the other methods receive. Both RND and IMPALA share the same network architecture as our navigator and subgoal generator networks. All three alternate methods are trained using the TorchBeast framework.

We use the mean extrinsic reward at the end of training to evaluate the performance of each agent. Where the extrinsic reward defined in the environment is $r^{e}_{t}=1-(0.9t)/t_{\text{max}}$. We summarize our results in Table I.

IMPALA is unable to reach the goal once the environment space becomes too large given the lack of intrinsic motivation for exploration. However it performs better in the low dimensional spaces due to the lack of reward noise the other methods utilize. RND and AMIGO perform comparatively across all environment dimensions. Our agent, CSG, performs better than both RND and AMIGO in the highest dimensional environment.

The novel subgoal generation has the added benefit of increasing the agent’s explainablity. Figure 2 highlights an example set of goals used to direct the navigator network in order to solve the environment. The subgoals listed are the human interpretation of the subgoal generator output. In figure 2 the agent receives a $5\times 5$ field of view:

where $\mathbf{b}_{ij}$ is an object. The field of view moves with the agent such that the agent is always positioned at $\mathbf{b}_{35}$, with $\mathbf{b}_{34}$ directly in front. At time step $0$, in Figure 2(a), the subgoal generator selects $\mathbf{b}_{35}$ to be changed to the object index value corresponding to the yellow key, which can be interpreted as the agent holding the key, and hence the subgoal is to pick up the key. Similarly, at time step $5$, in Figure 2(b), the subgoal generator selects $\mathbf{b}_{34}$ to be changed to the object index value corresponding to the locked yellow door, which we interpret as the agent setting going to the door as its subgoal. Finally, at time step $10$, in Figure 2(c), $\mathbf{b}_{35}$ is selected to be changed to the object index value corresponding to the green goal zone as its subgoal, which is also the overarching goal set by the environment.

The ability to generate explicit subgoals has two benefits: the first is to developers, as it gives a window into the training process, showing stages along the way to the agent’s final functionality, that are accessible to human language, logic and reasoning. Therefore, more interventions are possible, giving the ability to craft and debug the training process at a finer grain than conventional methods. The second benefit is to end users of the agent. These users will be able to understand in human terms how and why the agent made the decisions it did, leading to greater trust in the agent. Along this track there is a future long-term benefit to end users in that their feedback can be integrated as reward to train the agent, giving them the ability to teach the agent according to their preferences. Users will get a greater measure of control and insight into the agent’s reasoning and decision criteria.