Deep Q-Network for Stochastic Process Environments

The Deep Q-Network (DQN) algorithm used for the Flappy Bird environment is based on an implementation of a neural network with three linear layers and two dropout layers which takes in the number of input features, the number of hidden units, and the number of output actions as inputs. The forward function of the network applies ReLU activation to the output of each linear layer and applies dropout with a probability of 0.1 to the output of the first two linear layers. The output of the final linear layer represents the Q-values for each possible action. The DQN algorithm learns the optimal Q-values by minimizing the mean squared error loss between the predicted Q-values and the target Q-values, calculated using the Bellman equation:

where $Q(s,a)$ is the Q-value for a state-action pair, $s$ is the current state, $a$ is the current action, $r$ is the reward for taking action $a$ in state $s$, $\gamma$ is the discount factor, and $\alpha$ is the learning rate. The agent selects actions according to an $\epsilon$-greedy policy, where with probability $\epsilon$ it selects a random action and with probability $1-\epsilon$ it selects the action with the highest Q-value.

We utilized the Kaiming initialization method for the weights of our DQN network. The Kaiming initialization is a method of weight initialization to improve the convergence speed and performance of deep neural networks. This method initializes the weights using a Gaussian distribution with mean 0 and variance $\frac{2}{n}$, where $n$ is the number of input features. The Kaiming initialization is applied to the weights of the linear layers using the PyTorch with a nonlinearity of ReLU. These weight and bias initialization methods help to improve the performance of our DQN agent in learning the optimal Q-values and navigating the Flappy Bird environment.

Our implementation of the DQN algorithm in the Flappy Bird game involved the replay memory method. The ReplayMemory class stored recent transitions consisting of the current state, action taken, next state observed, and corresponding reward, in a deque data structure with a maximum capacity. The push method added a new transition to the memory buffer, and the sample method randomly selected a batch of transitions for training. This method helped decorrelate transitions, improve sample efficiency, and provide a more stable training process.

The optimize model function is realized by computing the expected state-action values using the Bellman equation defining the Huber loss function:

The function calculates the loss between expected and predicted state-action values, backpropagating the loss, clipping gradients, and performing an optimization step. It utilizes the policy and target networks to update the Q-values of the agent’s policy.

With the previously stated methods above, we build the full training pipeline to train the agent for Flappy bird environment in algorithm 1. The technique used in our approach is Q-learning with experience replay, where we store every experience at every frame in a replay memory.

In this part, we aim to develop a reinforcement learning algorithm to optimize stock trading decisions. To achieve this goal, we define a custom TradingEnv class to simulate stock trading in the real world. The initial state of the environment is set to a cash balance of $ 1 million, and the commission fee is set to 0.1%. The TradingEnv class has two methods: reset() and step(). The reset() method initializes the environment variables to their initial values and creates an observation consisting of four elements: the $currentprice$ of the stock, the $currentvalue$ of the account, the $currentposition$ of the account (i.e., how many stocks the account holds), and the $currentcashbalance$ of the account. The step() method takes an action as input and updates the environment variables accordingly. The action can be to buy or sell any action value of the stock. The current market value and balance of the account are updated after each action. The $dailyprofit$ and $totalprofit$ from the initial value of the account are calculated. A $reward$ is determined based on the total profit made by trading. The method also returns the observation, reward, and done flag. The episode is considered done when the time comes to an end, which is the last timestamp of the training data. The stock data consists of four symbolic stocks with the highest trading volume from 2018 to 2020. We will use this environment to evaluate the performance of the reinforcement learning algorithm on simulated stock trading scenarios.

The agent class in this project implements the deep Q-learning algorithm with experience replay to train a stock trading agent. The constructor initializes hyperparameters and memory parameters such as $\gamma$, $\epsilon$, learning rate, input dimensions, batch size, memory size, $\epsilon$ end, and $\epsilon$ decay. The action space is continuous and ranges from $-1$ to $1$. The Q-network is initialized, and the memory buffers are allocated using the input dimensions and memory size. The store_transition() method stores a transition tuple of state, action, reward, next state, and done in memory. The choose_action() method chooses the action with the highest value for greedy exploration or a random action for exploration based on the $\epsilon$ value. The learn() method computes gradients and updates the weights using a randomly generated batch index, and the Bellman equation is used to compute the Q-targets. The $\epsilon$ value is decayed over time to ensure the agent performs less exploration as it gains more experience.

With the previously stated methods above, we build the full training pipeline to train the agent for the stock trading environment in algorithm 2.

We implemented an enhanced DQN for the Flappy Bird game, which incorporated dropout layers and utilized the Kaiming initialization method to improve performance. Compared to the Lunar Landing environment, where the agent receives continuous rewards, Flappy Bird only provides rewards when the agent passes through the pipes correctly. To address this, we trained a more robust network with a larger memory replay size of $MEMORY=50000$, and a batch size of $BATCH\_SIZE=256$, which resulted in stable activation. The enhanced DQN also employed the following hyperparameters: discount factor $\gamma=0.99$, exploration rate starting at $\epsilon=EPS\_START=0.99$ and decaying exponentially to $\epsilon=EPS\_END=0.01$ over $EPS\_DECAY=1000$ steps, target network update rate $\tau=0.005$, learning rate $LR=1e-4$, and hidden layer size $HIDDEN=256$, which resulted in more stable and faster activation. The enhanced DQN triggered a faster training process and led to a significant improvement in test results. Specifically, the Average Reward over 20 Episodes increased from $158.70$ to $801.00$, while the average trigger episode decreased from $2212.73$ to $1623.58$.2.

Dropout layers randomly drop out a certain percentage of neurons during each training iteration, which prevents overfitting and encourages the network to learn more robust features. The Kaiming initialization method takes into account the non-linearity of activation functions, which helps to mitigate the vanishing/exploding gradient problem during training. Overall, the enhanced DQN achieved significant improvements in performance for the Flappy Bird game.

In this part, we developed a reinforcement learning algorithm to optimize stock trading decisions in both single-stock and multiple-stock environments. In the single-stock environment, the results were impressive. After training the policy for 2000 episodes, the average profit for 1000 steps was 121 % of the starting fund. The policy tends to sell when the stock surges, mimicking the behavior of a professional investor. See fig 3. However, this is just a simulation using historical data, and the model may overfit the data. To evaluate the performance of the model, we used the backtest class to analyze the pattern of the position and profit. We found that the model’s risk preference could be tuned to achieve even better results.

The backtest class allows us to analyze the performance of the policy over multiple episodes and identify patterns in the position and profit. By tuning the model’s risk preference, we can further improve the performance of the policy in future simulations.