A Comparison of Reward Functions in Q-Learning Applied to a Cart Position Problem

The RL algorithm used in this paper is Q-Learning. Q-Learning was first coined by Watkins (1989) in his PhD thesis [3]. It finds an optimal policy $Q(s,a)$ where $s$ is the state and $a$ is the action. This optimal policy maximizes the total reward in the system. $Q(s,a)$ can be thought of a table of values for all states and actions.

At first, $Q(s,a)$ is initialized to $0$ for all $s,a$. At every time step $t$, the action $a_{t}$ is determined from the state $s_{t}$ through an $\epsilon$-greedy process. In order to find the best action $a_{t}$ from $Q(s,a)$, we must first use a random process to figure out the result of the action. When the training of the agent starts, we let $\epsilon=1$. At every time step, we take a random sample from the $Uniform(0,1)$ distribution. If the random sample is greater than $\epsilon$, then $a_{t}=\operatorname*{arg\,max}_{a}(Q(s_{t},a))$. Otherwise, $a_{t}$ is a randomly selected action.

Let $Q^{[n]}(s,a)$ be the policy after $n$ updates. When $s_{t+1}$ is calculated using $s_{t}$ and $a_{t}$, then $Q^{[n]}(s,t)$ is updated using the following equation:

Where $r_{t}$ is the reward, $\zeta$ is the learning rate. $\max_{a}Q(s_{t+1},a)$ is a measure of the guess of the future reward, and $\gamma$ is a measure of the weight we give to future rewards. $\zeta$ and $\gamma$ are hyperparameters to this agent. The hyperparameters used in this paper are: $\zeta=0.05,\gamma=0.9$

In $Q(s,a)$, the state $s$ refers to the position $x$ of the cart, and the action $a$ refers to the voltage $V$ of the cart.

The Q-Learning code used in this paper is taken from Lin’s (2018) GitHub repository. This code was originally used to solve the Blackjack problem using Q-Learning. The equation for updating $\epsilon$ after every time step $t$ according to Lin’s code is given by the equation below [4]:

Where $n_{train}=30000$ is the number of time steps used to train the agent.

As shown previously, reward functions are used to update the policy function at each time step with the intention that the RL agent maximizes the total reward. This paper will explore three reward functions. The first reward function takes the negative of the square of the distance between $x$ and $r$ with the intention that the RL agent will be rewarded better if it is closer to $r$.

The second reward function is piece-wise and is always positive. This function is linear instead of quadratic.

The third reward function is discontinuous. It only rewards a $1$ if the cart is a distance of less than $1$ unit away from $r$ and it rewards a $5$ if the cart is a distance of less than $0.1$ units away from $r$.

To reproduce our results, we provided the code used for this problem [7].

The first reward function takes the negative of the square of the distance between $x$ and $r$ with the intention that the RL agent will be rewarded better if it is closer to $r$, as shown in Equation 6. Figure 3 below shows how the average reward varies with each sample. The average reward is calculated by taking the mean of the total reward of each of the trajectories in the sample.

It is clear that, after training for $70$ samples, the training is complete and the average reward is approximately $-468$. Figure 4 shows the trajectory of each cart in sample 1. These carts follow a random motion. All trajectories of carts in later samples will be compared to this plot.

In this graph, most of the carts go out of bounds within the first 10 time steps. During the first sample, the value of $\epsilon$ used in the $\epsilon$-greedy algorithm is almost $1$, which is why the voltage applied at each time step is picked randomly. Thus, the trajectory that each cart takes is random.

At samples $60$ to $70$, the average reward plot in Figure 3 shows an increasing trend. Figure 5 shows the trajectory of each cart at sample $65$.

In this plot, clearly the RL agent has learned that the cart needs to move towards $x=r$ during the start of the round. The trajectories are less likely to move out of bounds. Several trajectories appear to move back and forth $x=r$. It shows a significant improvement compared to Figure 4.

Figure 6 shows the trajectory of each cart in sample $100$.

This graph shows that the cart reaches a steady-state motion after $5$ time steps (or $1.0$ seconds). The position alters between $x=10.0$ and $x=8.0$ and the voltage alters between $V=-1$ and $V=1$. The total reward in this trajectory is $-468$, which is significantly high compared to the average rewards in the first few samples shown in Figure 3.

The steady-state position is within $\pm 2$ of $r$. If $V=0$ at $x=10$, then the cart is expected to drift past $x=r$, which is why $V=-1$ is the best action to take here.

The second function gives a positive reward only if the position of the cart is in the range $[\frac{r}{2},\frac{3r}{2}]=[5,15]$, as shown in Equation 7. Contrary to reward function 1 and 3, the step-size used here is $0.1$s because it leads to better results, and the number of time steps in a round is still $50$. Figure 7 below shows how the average reward varies with each sample.

It is clear that, after training for $40$ samples, each trajectory of the cart has a total reward of $86$. Figure 8 shows the trajectory of each cart in sample 1. These carts follow a random motion. All trajectories of carts in later samples will be compared to this plot. The distance from origin is the value $x(t)$ at time $t$ and the time step is $t/\delta t$.

In this graph, most of the carts go out of bounds within the first 40 time steps. This makes sense since the time step size used here is half the step size used for the first reward function. For this reason, the trajectories shown in this graph are different from the trajectories shown in Figure 4. The trajectory that each cart takes is random because the voltage applied at each time step is picked randomly.

At samples $20$ to $30$, the average reward in Figure 7 shows an increasing trend. Figure 9 shows the trajectory of each cart in sample 25.

In this plot, it is clear the cart has learned to move towards $x=r$ during the start of the round. Several trajectories appear to stay near $x=r$ and the cart is less likely to go out of bounds. It shows a significant improvement compared to Figure 8.

Figure 10 shows the trajectory of each cart in sample 100.

This graph shows that the cart reaches a steady-state motion after $10$ time steps (or $1.0$ seconds). The validation plot is the same as this plot. The position alters between $x=12.0$, $x=13.0$ and $x=17.0$ and the voltage alters between $V=1$, $V=4$ and $V=-5$. The total reward in this trajectory is $86.0$, which is significantly high compared to the average rewards in the first few samples shown in Figure 7.

However, the variation in the position during the steady-state motion is too big. The steady-state position is within $\pm 7$ of $r$. And $r$ is not in the range of the positions in the steady-state motion. This is because the agent has learned that a positive reward comes if the cart is in the range $[5,15]$. Thus, there is less of an incentive to stay closer to $x=r$ since the current trajectory already significantly maximizes the total reward. This shows that the second reward function is not as reliable as the first reward function.

As a measure of ensuring that the variability of the steady-state position is minimized, the third reward function gives a positive reward only if the position of the cart is in the range $[r-1,r+1]=[9,11]$, as shown in Equation 8. Figure 11 shows how the average reward varies with each sample.

It is clear that, after training for $85$ samples, each trajectory of the cart has a total reward of $230$. Figure 12 shows the trajectory of each cart in sample 1. These carts follow a random motion. All trajectories of carts in later samples will be compared to this plot.

In this graph, most of the carts go out of bounds within the first 10 time steps. This graph shows similar trends to Figure 4. The direction each cart goes at the start of the round is random as the voltage applied at each time step is picked randomly.

At samples $80$ to $85$, the average reward in Figure 11 shows an increasing trend. Figure 13 shows the trajectory of each cart in sample $84$.

In this plot, it is clear that the agent has learned to move the cart towards $x=r$ during the start of the round. Trajectories are less likely to go out of bounds. In some trajectories, the cart moves back and forth the interval $[r-1,r+1]$ multiple times in order to increase its total reward.

Figure 14 shows the trajectory of each cart in sample 100.

This graph shows that the cart reaches exactly $x=r$ after $4$ time steps ($0.8$ seconds) and stays there. The validation plot is the same as this plot. The steady-state motion has no variability compared to the first and second reward function. On the other hand, a drawback of using this reward function is that training this agent takes more samples compared to training RL agents that use the first or second reward function, as shown in Figure 11.

We first compare which RL agent reaches steady-state motion the earliest. For the first and second reward function, the Q-Learning agents take equally as long to reach steady-state motion ($1.0$s). The Q-Learning agent using the third reward function reaches steady-state motion the quickest ($0.8$s).

We then compare which RL agent has the lowest variability in its steady-state motion. In the first reward function, the steady-state motion alternates between positions $8.0$ and $10.0$, thus having a width of $2.0$. In the second reward function, the steady-state motion alternates between positions $12.0,13.0$ and $17.0$, thus having a width of $5.0$. In the third reward function, the steady-state motion just takes values of $10.0$, thus having a width of $0.0$. This shows that, while the first reward function leads to a smaller width compared to the second reward function, the third reward function has the smallest width.

Lastly, we compare which RL agent is closest to $x=r$ in its steady-state motion. Clearly, the third reward function performs the best since it’s steady state position is exactly $r$. And the second reward function performs the worst since $r$ is not in its set of steady-state positions.

Thus, the third reward function gives the best results.

In this paper, the cart position problem is thought of as a toy problem where Q-Learning may be useful. Suppose a real cart follows the model shown in Equation 1 and its goal is to get from a position $x=0$ to $x=r$. This means the position is updated continuously as opposed to time steps shown in Euler’s method. In Q-Learning, the policy function $Q(s,a)$ picks actions based on trial-and-error. The set of states $s$ in $Q(s,a)$ are finite and discrete. Thus, Q-Learning is impractical on a real cart where the position can be any real number. In this situation, the theoretical solution is more effective as it adjusts the voltage based on a continuous set of positions.

Having a discrete set of actions can also be undesirable in this problem. In the situation with reward function 1 (Figure 6), it is clear that $V$ has to alter between $-1$ and $1$ in order to maintain a steady-state motion. If $V=0$ at $x=10$, then the cart will drift away from $x=10$ since its velocity is non-zero. Thus, further areas of exploration involve adding values of $V$ with a smaller magnitude (i.e. $0.1,0.2$) into the range of actions the Q-learning agent could take.

Further areas of exploration also involve using Deep Q learning. Artificial neural networks (ANNs) are trained by modifying weights rather than entries in a $Q(s,a)$ table, thus, ANNs may be more reliable in problems involving continuous states or continuous actions.

Consider the trajectory of the Q-Learning agent shown in Figure 14. Between time $t=0$ and $t=\delta t=0.2$, the velocity of the cart increases from $\dot{x}=0ms^{-1}$ to $\dot{x}=40ms^{-1}$. This means the acceleration is approximately $\ddot{x}=200ms^{-2}$. In a real cart, this could be dangerously high, given that cars that have an acceleration of approximately $10ms^{-2}$ to $20ms^{-2}$ are considered one of the fastest accelerating cars [5]. This shows that the Q-Learning agent is impractical on real carts.

Lastly, in real world carts, there is always an error associated with measuring the mass of the cart, the coefficient of friction, or any of the constants described in Equation 1. This means there can be an error associated with measuring $\alpha$ or $\beta$. Suppose we trained a Q-Learning agent with $\alpha=-1$ and we intend to test it on a cart with $\alpha=-1.01$. Figure 15 shows what the trajectory will look like.

If we use a cart with $\alpha=-1$, we get the trajectory shown in Figure 14. If we use a cart with $\alpha=-1.01$, we get the trajectory shown in Figure 15. This shows that a small error in the measurement of $\alpha$ can lead to a significant change in the trajectory of the cart. This occurs because the set of states $s$ in $Q(s,a)$ are finite and discrete. This is why a small change in $\alpha$ means that the cart will reach positions that are either not in the set of states in $Q(s,a)$, or have a different action associated with it. This again shows that Q-Learning is not reliable for real carts.

Writing this in matrix form with $X_{h}(t)=\begin{bmatrix}x_{h}(t)\\ \dot{x}_{h}(t)\end{bmatrix}$ gives the following equation:

The solution to this equation is:

The eigenvalues of $A_{h}$ are:

Since $\beta>0$, this means:

And since $\alpha<0$, this means:

This shows that $A_{h}$ is Hurwitz, and by extension:

Thus, the steady-state value of the homogenous equation is: $x_{h}=0$.

Consider:

Substituting it into equation 2 gives:

Thus, $x_{p}(t)=r$ is a particular solution to the non-homogenous equation. The steady-state value of $x_{p}(t)$ is $r$.

All steady-state values of the homogenous equation are $0$ and the steady-state solution to the particular solution of the non-homogenous equation is $r$. This means the steady-state value of equation 11 is $r$.

This shows that using the input function $V(t)=K_{p}(r-x(t))$ is effective in moving the cart from position $x=0$ to position $x=r$.