IGN : Implicit Generative Networks

There are a lot of datasets we can use to show our models’ performance and compare them. In our work, we want to compare our model with baselines with several classic datasets. One of them is Atari games datasets (see Fig. 1) [7], which is one of the classic datasets in reinforcement learning in game theory. We use the 57 Atari 2600 games in the ALE, and it can help show our contribution to our algorithm in the risk-sensitive policies, in which the Atari 2600 games have  200k frames collected using the OpenAI baseline and implemented based on Advantage Actor Critic (A2C). Also, they use sticky actions to make the problem become more challenging. There is a $25\%$ probability that the agent’s previous action is executed. It will greatly help us show our algorithm performance on risk-sensitive policy.

Before our work, there was a lot of research already work in reinforcement learning. Implicit Quantile Networks (IQN) [4] is one of the models that shows a good result. It brings a huge improvement on the risk-sensitive policy training. However, the model based on the Quantile distribution make a huge fragile on the performance of the model, and it should cost huge time for training the model. We expected to design a state-of-art algorithm that has more robustness and has more effectiveness in training the policy. One of the possible methods is using transfer learning. Transfer learning [8] is an optimization method that can improve the performance of the model or rapid progress in the second task. In our research, transfer learning can help us have a higher start and higher slope from the second training. Also, it will have great help in the policy evaluation. There are some state-of-art transfer learning methods, such as VGG [9], ResNet [10] Model. From several model pooling, there is one algorithm using Gradient method is GAN model, which can help our model have a more competitive algorithm with origin IQN model

Most existing offline policy evaluation methods under the above setting with mean criteria can be grouped into three categories. The first category is direct methods, which directly estimate the state-action value function [11, 12], also known as the Q-function. The second category of offline policy evaluation methods is motivated by marginal importance sampling [13]. The last category of methods combines direct and marginal importance sampling methods to construct some doubly robust estimator, which is also motivated by the efficient influence function like equation 1. [14]. In which, $Q^{\pi}$ and $\omega^{\pi}$ are two nuisance functions.

In our work, we hope to use the generative method to improve our model. The most popular method on the generative method is Generative Adversarial Networks (GAN) [15]. The GAN model shows a great improvement for our model in the gradient periods and make our model have high efficiency to get the best reward score.

GANs can train two competing models (typically NNs)[15]. The generator takes noise $z\sim P_{z}$ as input and generates samples according to some transformation, $G_{\theta}(z)$. The discriminator takes samples from both the generator output and the training set as input and aims to distinguish between the input sources. Goodfellow[15] measured the discrepancy between the generated and the real distribution using the Kullback-Leibler divergence. However, this approach was improved by Arjovsky[16] by using the Wasserstein-1 distance [17]. WassersteinGANs exploit the Kantorovich-Rubinstein duality[18],

where $1-\text{Lip}$ is the class of Lipschitz functions with Lipschitz constant 1, in order to approximate the distance between the real distribution, $\mathbb{P}_{r}$, and the generated one, $\mathbb{P}_{g}$. The GAN objective is then to train a generator model $G_{\theta}(z)$ with noise distribution $P_{z}$ at its input, and a critic $f\sim 1-\text{Lip}$, achieving

The WGAN-GP[16] (Fig. 2(b)) is a stable training algorithm for equation 4 [17] that employs stochastic gradient descent and a penalty on the norm of the gradient of the critic with respect to its input.

The problem is driven by recent development in distributional reinforcement learning. [4] Consider a single trajectory ${(S_{t},A_{t},R_{t})}_{t}\geq 0$ where $(S_{t},A_{t},R_{t})$ denotes the state-action-reward triplet collected at time t. We use $S\subseteq\mathbb{R}_{p}$ and $\mathcal{A}$ denote the state and action space, respectively. We assume A is discrete and finite, and rewards $R_{t}$ are uniformly bounded by $R_{max}$. Consider the offline setting. Then we observed data consisting of N trajectories, corresponding to N independent and identically distributed copies of ${(S_{t},A_{t},R_{t})}_{t}\geq 0$. For any i = $1,\cdots,N,$ data collected from the $i^{th}$ trajectory can be summarized by ${(S_{i,t},A_{i,t},R_{i,t},S_{i,t+1})}0\leq t<T_{i}$ , where $T_{i}$ denotes the termination time. For simplicity, we assume $T_{i}$ = T for i = $1,\cdots,N.$

A policy defines the agent’s way of choosing the action at each decision time. We focus on the stationary policy. Specifically, at each time point t, a stationary policy $\pi$ maps the current state value into a probability mass function over the action space, i.e., $\pi(a|s)$ denotes the probability of choosing action a given the state value s.

In our work, we will work on the policy evaluation and policy optimization. In policy evaluation, we want to see the performance of the model what would have happened in the absence of a policy. We will use the Pong environment agent as a fixed agent, which we will use as one of the offline policy, and we will use our model to train the online policy and see the performance of the model training. In our work, we will compare the reward scores and efficiency of the timestamps because comparing with the complex model with a simple model in the resource using is not a fit game, we will compare the timestamps( measured with the training cycle time, not the real time). In our future work, if there is one model that requires less real time is one of the potential research directions in our work. For the policy optimization, we will train a model with different environments to evaluate our model performance in timestamps and reward gain with other baselines models.

We introduce a state that uses quantile regression to approximate risk-sensitive strategies, in other words, an optimization method for reinforcement learning via a full quantile function of the behavioural reward distribution.

We always model the agent and the environment as a Markov decision model[19] in classical reinforcement learning designs. Here, we define the rule $\pi(\cdot|x)$ as a mapping from states to distributions constantly changing with actions. For any agent that follows the rule $\pi$, we define the discounted sum of future rewards as a function $Z$. Clearly, $Z^{\pi}(x,i)$ varies with state and action as following

, where $\beta$ is a discount factor and $\beta\in(0,1)$. The action-value function can be defined as $Q^{\pi}(x,i)=\mathbb{E}[Z^{\pi}(x,i)]$, in which $\mathbb{E}$ is the Bellman equation operator[20].

Azer’s research[21] on reinforcement learning proposed a distributed reinforcement learning theory that values the benefit distribution $Z^{\pi}(x,i)$ rather than the action-value function $Q^{\pi}(x,i)$. According to the analysis[22], it can be found that distributed reinforcement learning methods have substantial advantages in terms of sample complexity, final performance, and hyperparameter processing robustness.

According to Bellemare’s study[23], the distributed Bellman operator is a contraction in the p-Wasserstein metric. There are two main theories for minimizing the Wasserstein metric. One is distributed reinforcement learning using quantile regression and shows that the resulting predictive distributed Bellman operator is a contraction of the $\infty$-Wasserstein metric by properly selecting the quantile target[24]. Another study shows that the original class of classification algorithms is a contraction in the Cramér distance, the L2 metric on the cumulative distribution function.[25]

QR-DQN minimizes the Wasserstein distance distribution Bellman objective by estimating the quantile function at precisely chosen points. This estimation uses quantile regression, which has been shown to converge to the true quantile function value when minimized using stochastic approximation. In QR-DQN, the stochastic returns are approximated by a homogeneous mixture of N Diracs,

in which, $\theta_{i}$ is assigned a fixed quantile target $\tau$. We can use Huber quantile regression loss to train this quantile target $\tau$ with threshold $\delta$.

In practice, we use the IQN as the baseline and implement the gradient method with the GAN model and calculate the GAN score as the gradient score and calculate the new gradient with the Huber loss (see equation 9) to get the reward score and use this new reward score into the agent.

Define a random variable $Y=\sum\limits_{t=0}^{+\infty}\gamma^{t}R_{t}$, whose support is
$\left[-\frac{R_{max}}{1-\gamma},\frac{R_{max}}{1-\gamma}\right]$ by assumption. Given a target policy $\pi$, our goal is to estimate the distribution of $Y$ given any state $s\in\mathcal{S}$ and $a\in\mathcal{A}$. We use $\mathcal{U}$ to denote the product space of cumulative rewards, state and actions, i.e., $\mathcal{U}=\left[-\frac{R_{max}}{1-\gamma},\frac{R_{max}}{1-\gamma}\right]\times\mathcal{S}\times\mathcal{A}$. For any $y\in\mathbb{R}$, define either a conditional density or a conditional probability mass function of $Y=y$ given a state-action pair $(s,a)$ as

where $f^{\pi}_{Y}$ indicates actions are selected according to $\pi$. In order to estimate $f^{\pi}_{Y}$ given any $(s,a)$, we have the distributional Bellman equation that under Assumptions (A1) and (A2), for any $t\geq 0$, $y\in\mathbb{R}$, $s\in\mathcal{S}$ and $a\in\mathcal{A}$, the following equation holds.

Motivated by the distributional Bellman operator is $\pi$-contraction under a modified Wasserstein distance, and the recently developed generative adversarial networks (GANs), we propose a Wasserstein GAN-based distributional policy evaluation method to estimate $f^{\pi}_{Y}$ will discuss the benefits of using GANs. For any continuous function $h$ defined over $\mathbb{R}\times\mathcal{S}\times\mathcal{A}$, the following equation holds. {strip}

Motivated by equation 11, we propose to use generative adversarial networks to estimate $f^{\pi}_{Y}$ of $Y$ given state-action pair $(s,a)\in\mathcal{S}\times\mathcal{A}$. Specifically, we aim to learn a generator $\mathbb{G}^{\pi}$ that takes $(s,a)$ a uniform random variable $Z$, i.e.,$Z\sim$ uniform (0, 1), as the input, and the output is a pseduo sample $\tilde{Y}$ as one realization of target conditional distribution $f^{\pi}_{Y}$. This is also inspired by the conditional GANs introduced by Mirza and Osindero[26]. In typical GANs, the generator is trained by minimizing some divergence between the conditional distribution of $Y|s,a$ and $\tilde{Y}|s,a$. Alternatively, a generator $\mathbb{G}^{\pi}(Z,S,A)$ is trained such that the joint distribution of $(Y(S,A),S,A)$ is the same as $(\mathbb{G}^{\pi}(Z,S,A),S,A)$, where $Y(S,A)$ is a random variable that has the same distribution of $Y$ given $s\in\mathcal{S}$ and $a\in\mathcal{A}$. However, different from conventional distribution learning tasks, neither $Y$ nor $Y(S,A)$ is not observed in our problem. Nevertheless, the distributional Bellman equation and the equation11 indicate us to obtain $\mathbb{G}^{\pi}$ by solving the following min-max optimization problem, which has the similar spirit of Wasserstein-GANs[27]. {strip}

where $\{Z_{t}\}_{0\leq t\leq(T-1)}$ are set to independently follow uniform(0,1), and $h$ is a function mapping from $\mathbb{R}^{p+2}$ to $\mathbb{R}$.

In practice, as in conventional GANs, we restrict $h$ to some set of neural networks $\mathcal{R}_{D}$ in addition, which can provide tractable computation. In addition, by powerful deep neural networks, we can handle potential high-dimensional state space $\mathcal{S}$. Similarly, we also use some set of neural networks $\mathcal{R}_{G}$ to approximate the generator $\mathbb{G}^{\pi}$.

Our goal is to find a generator $\mathbb{G}^{\pi}(\bullet,S,A)$ to approximate the conditional distribution of $Y$ given $S$ and $A$. This is, in general, a very challenging problem, especially when $S$ is a high-dimensional state, e.g., the number of states in $\mathcal{S}$ is huge. Thus, suggested by Haas and Richter[28], we impose some structure assumptions on the underlying true distribution. In particular, we assume the distribution of $(Y(S,A),S,A)$ lie in a $d_{\mathbb{G}}$-dimensional subspace with $d_{\mathbb{G}}<p+2$. Based on this consideration, we define the following function class for the generator.

Let $\mathcal{G}(d_{\mathbb{G}},\beta,K)$ be the class of all measurable functions $\mathbb{G}:\mathbb{R}^{p+2}\rightarrow\mathbb{R}$ such that any component only depends on $d_{\mathbb{G}}$ arguments and lies in $C^{\beta}(\mathcal{U},K)$.

Given the offline dataset $\mathcal{D}_{N}=\{(S_{i,t},A_{i,t},R_{i,t},S_{i,t+1})\}_{0\leq t\leq(T-1),1\leq i\leq N}$, we use the empirical average to approximate the objective function and solve the following optimization to estimate the generator $\mathbb{G}^{\pi}$. {strip}

where $\left\{Z_{i,t}\right\}_{1\leq i\leq N;0\leq t\leq(T-1)}$ and $\left\{\tilde{Z}_{i,t}\right\}_{1\leq i\leq N;0\leq t\leq(T-1)}$ are two independent samples from uniform (0, 1). The optimal solution of the optimization above always exists because the inner maximization problem is Lipschitz with respect to $\mathbb{G}$ and $\mathbb{G}$ is Lipschitz with respect to all parameters in $\mathcal{R}(L_{\mathbb{G}},p_{\mathbb{G}},s_{\mathbb{G}})$ with bounded support. Similarly, the inner maximization problem also has an optimal solution. Stochastic gradient descent algorithms as the state-of-the-art can be implemented to solve the above optimization problem. We denote the final minimizer as by $\hat{\mathbb{G}}^{\pi}$, which can generate a sequence of pseudo samples $\{\tilde{Y}^{\pi}_{m}(s,a)\}_{1\leq m\leq M}$ for some integer $M$ for approximating the distribution of $Y$ given state-action pair $(s,a)$ under the target policy $\pi$. This provides a flexible way for estimating interesting quantities related to the target policy $\pi$.

In all our examples, if $\hat{\mathbb{G}}^{\pi}$ is consistent and $M$ (or $K$) diverges to infinity, $\hat{\mathcal{V}}(\pi)$, $\hat{q}^{\pi}_{\tau}(Y)$ and $\widehat{Var}^{\pi}(Y)$ are all consistent.