Adversarial Attacks on Deep Algorithmic Trading Policies

Reinforcement learning is concerned with agents that interact with an environment and exploit their experiences to optimize a decision-making policy. The generic RL problem can be formally modeled as learning to control a Markov Decision Process (MDP), described by the tuple $MDP=(\mathbb{S},\mathbb{A},\mathbb{R},\mathbb{P})$, where $\mathbb{S}$ is the set of reachable states in the process, $\mathbb{A}$ is the set of available actions, $\mathbb{R}$ is the mapping of transitions to the immediate reward, and $\mathbb{P}$ represents the transition probabilities (i.e., state dynamics), which are initially unknown to RL agents. At any given time-step $t$, the MDP is at a state $s_{t}\in\mathbb{S}$. The RL agent’s choice of action at time $t$, $a_{t}\in\mathbb{A}$ causes a transition from $s_{t}$ to a state $s_{t+1}$ according to the transition probability $\mathbb{P}(s_{t+1}|s_{t},a_{t})$. The agent receives a reward $r_{t+1}=\mathbb{R}(s_{t},a_{t},s_{t+1})$ for choosing the action $a_{t}$ at state $s_{t}$. Interactions of the agent with MDP are determined by the policy $\pi$. When such interactions are deterministic, the policy $\pi:\mathbb{S}\rightarrow\mathbb{A}$ is a mapping between the states and their corresponding actions. A stochastic policy $\pi(s)$ represents the probability distribution of implementing any action $a\in\mathbb{A}$ at state $s$. The goal of RL is to learn a policy that maximizes the expected discounted return $E[R_{t}]$, where $R_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}$; with $r_{t}$ denoting the instantaneous reward received at time $t$, and $\gamma$ is a discount factor $\gamma\in[0,1]$. The value of a state $s_{t}$ is defined as the expected discounted return from $s_{t}$ following a policy $\pi$, that is, $V^{\pi}(s_{t})=E[R_{t}|s_{t},\pi]$. The action-value (Q-value) $Q^{\pi}(s_{t},a_{t})=E[R_{t}|s_{t},a_{t},\pi]$ is the value of state $s_{t}$ after applying action $a_{t}$ and following a policy $\pi$ thereafter.

Value iteration refers to a class of algorithms for RL that optimize a value function (e.g., $V(.)$ or $Q(.,.)$) to extract the optimal policy from it. As an instance of value iteration algorithms, Q-Learning aims to maximize for the action-value function $Q$ through the iterative formulation of Eq. (1):

$\displaystyle Q(s,a)=R(s,a)+\gamma max_{a^{\prime}}(Q(s^{\prime},a^{\prime}))$

(1)

Where $s^{\prime}$ is the state that emerges as a result of action $a$, and $a^{\prime}$ is a possible action in state $s^{\prime}$. The optimal $Q$ value given a policy $\pi$ is hence defined as: $Q^{*}(s,a)=max_{\pi}Q^{\pi}(s,a)$, and the optimal policy is given by $\pi^{*}(s)=\arg\max_{a}Q(s,a)$.

The Q-learning method estimates the optimal action policies by using the Bellman formulation to iteratively reduce the TD-Error given by $Q_{i+1}(s,a)-\mathbf{E}[r+\gamma\max_{a}Q_{i}]$ for the iterative update of a value iteration technique. Practical implementation of Q-learning is commonly based on function approximation of the parametrized Q-function $Q(s,a;\theta)\approx Q^{\ast}(s,a)$. A common technique for approximating the parametrized non-linear Q-function is via neural network models whose weights correspond to the parameter vector $\theta$. Such neural networks, commonly referred to as Q-networks, are trained such that at every iteration $i$, the following loss function is minimized:

$\displaystyle L_{i}(\theta_{i})=\mathbf{E}_{s,a\sim\rho(.)}[(y_{i}-Q(s,a,;\theta_{i}))^{2}]$

(2)

where $y_{i}=\mathbf{E}[r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i-1})|s,a]$, and $\rho(s,a)$ is a probability distribution over states $s$ and actions $a$. This optimization problem is typically solved using computationally efficient techniques such as Stochastic Gradient Descent (SGD).

Classical Q-networks introduce a number of major problems in the Q-learning process. First, the sequential processing of consecutive observations breaks the i.i.d. (Independent and Identically Distributed) assumption on the training data, as successive samples are correlated. Furthermore, slight changes to Q-values leads to rapid changes in the policy estimated by Q-network, thus giving rise to policy oscillations. Also, since the values of rewards and Qs are unbounded, the gradients of Q-networks may become sufficiently large to render the backpropagation process unstable.

A Deep Q-Network (DQN) [16] is a training algorithm designed to resolve these problems. To overcome the issue of correlation between consecutive observations, DQN employs a technique called experience replay: instead of training on successive observations, experience replay samples a random batch of previous observations stored in the replay memory to train on. As a result, the correlation between successive training samples is broken and the i.i.d. setting is re-established. In order to avoid oscillations, DQN fixes the parameters of a network $\hat{Q}$, which represents the optimization target $y_{i}$. These parameters are then updated at regular intervals by adopting the current weights of the Q-network. The issue of instability in backpropagation is also solved in DQN by normalizing the reward values to the range $[-1,+1]$, thus preventing Q-values from becoming too large.

Mnih et al. [16] demonstrate the application of this new Q-network technique to end-to-end learning of Q values in playing Atari games based on observations of pixel values in the game environtment. To capture the movements in the game environment, Mnih et al. use stacks of four consecutive image frames as the input to the network. To train the network, a random batch is sampled from the previous observation tuples $<s_{t},a_{t},r_{t},s_{t+1}>$, where $r_{t}$ denotes the reward at time $t$. Each observation is then processed by two layers of convolutional neural networks to learn the features of input images, which are then employed by feed-forward layers to approximate the Q-function. The target network $\hat{Q}$, with parameters $\theta^{-}$, is synchronized with the parameters of the original $Q$ network at fixed periods intervals. i.e., at every $i$th iteration, $\theta^{-}_{t}=\theta_{t}$, and is kept fixed until the next synchronization. The target value for optimization of DQN thus becomes:

$\displaystyle y_{t}\equiv r_{t+1}+\gamma max_{a^{\prime}}\hat{Q}(s_{t+1},a^{\prime};\theta^{-})$

(3)

Accordingly, the update rule for the parameters in the DQN training process can be stated as:

$\displaystyle\theta_{t+1}=\theta_{t}+\alpha(y_{t}-Q(s_{t},a_{t};\theta_{t}))\nabla_{\theta_{t}}Q(s_{t},a_{t};\theta_{t})$

(4)

In recent years, electronic trading platforms have made access to global capital markets easier for the general public, resulting in a lower barrier to entry and an influx of traffic across these platforms. The growing interest in such trading platforms and technologies is however accompanied by the increasing risks of cyber attacks. While the literature on the cybersecurity issues of current trading platforms is scarce, few industry-sponsored studies report concerning issues in deployed trading platforms. One such study on the exposure of security flaws in trading technologies [17] evaluates various popular desktop, mobile and web trading service platforms against a standard list of security checks, and reports that these trading technologies are in general far more susceptible to cyber attacks than previously-reviewed personal banking applications from 2013 and 2015. The security checks consisted of features such as 2-Factor Authentication (2FA), encrypted communications, privacy mode, anti-reverse engineering, and hard-coded secrets. This study reports that 64% of the reviewed trading applications rely on unencrypted communication channels for authentication and trading data. Also, the author finds that many trading applications utilize poor session management and SSL certificate validation, thereby enabling Man-in-The-Middle (MITM) attacks. Furthermore, this report points out the wide-scale susceptibility of such platforms to remote Denial of Service (DoS) attacks, which may render the applications useless. Building on the findings of this study, our paper investigates attacks that leverage the aforementioned vulnerabilities to manipulate deep algorithmic trading policies.

Adversarial attacks may target all components of a DRL agent, including the environment, agent’s observation channel, reward channel, actuators, and training components (e.g., experience storage and selection), as identified by Behzadan [8].

Figure 1 illustrates the components of a DRL trading agent at test-time. In the context of algorithmic trading, the observation of the environment is gathered from various sources such as market indicators, social media indicators, and exchanges– we refer to these sources as input channels. This data is prepossessed and feature engineered to create the agent’s observation of the state. These states are part of the observation returned by the environment to the agent along with the reward. Through the observation channel, an adversary may intercept the observation and exchange it for a perturbed observation, otherwise called a Man-In-The-Middle (MITM) attack. An adversary may also delay the observation channel through a Denial of Service (DoS) attack. The reward channel is often tied to internal securities such as bank accounts or portfolios, and hence are less susceptible to external adversarial manipulation. However, any external component reachable by the agent can be compromised implicitly.

The capabilities of an adversary are defined by two factors, namely the actions available to the adversary, and information available about the target. These define the extend and impact of eligible attacks on a DRL agent. This section presents a classification of attacks and adversaries at the inference phase based on the aforementioned factors.

Inference-time (also referred to as test-time) attacks may be active or passive. Passive attacks aim to gather information about the target agent by observing the behavior of the target in various states. With sufficient observations of state-action pairs, the adversary can reconstruct the targeted policy and thereby compromise the Confidentiality of the targeted, proprietary agents [18]. On the other hand, active attacks are those that perturb one or more components of a DRL agent to manipulate the behavior of its policy. According to the available information, such attacks can be classified as whitebox or blackbox. Whitebox attacks refers to cases where the adversary has sufficient knowledge of the target’s parameter to directly craft an effective perturbation, and blackbox attacks refer to the vice versa scenario. Imitation Learning and Inverse Reinforcement Learning are avenues an adversary may use to either attack their target agent or steal components of the agent like its learned policy. As demonstrated in [18], adversaries can gather additional information through policy imitation, thereby enabling whitebox attacks against blackbox targets.

Furthermore, active manipulations can be classified under targeted and non-targeted attacks. Successful non-targeted or indiscriminate attacks through aim to have the policy select any action other than the optimal action $a$ by providing a perturbed observation instead of the true observation at a timestep $t$, resulting in less reward given by the environment. Targeted attacks aim to craft perturbations such that the policy selects a particular sub-optimal action $a^{\prime}$, as chosen by the adversary. For each attack, a perturbation vector is crafted that is minutely different to the true observation vector that would otherwise be undetectable by human traders, but produces state values that result in the policy choosing a different action than action $a$. This is similar to how adversarial attacks affect supervised machine learning models.

Perturbations in observation affect both test-time and train-time. While the focus of this paper is on test-time attacks, it is noteworthy that during training, perturbed observations result in state values which are bootstrapped throughout its updates, potentially impacting learned policies and their trajectory which is the sequence of actions taken by the agent. Work by Behzadan and Munir [19] show that training-time attacks under certain conditions with high probability of attack per observation between 0.5 and 1.0 resulted in the agent’s inability to recover performance upon test-time evaluation under non-adversarial conditions.

In this scenario, our data is sourced from Russian stock market prices between the period of 2015-2016. The dataset is comprised of samples representing a one-minute temporal resolution, and the dynamic of the price during that minute is captured by four values:

• •

  open price - price at the beginning of the minute

• •

  high price - maximum price during the minute interval

• •

  low price - low is the minimum price during the minute interval

• •

  close price - last price of the minute time interval

Each minute interval is called a bar.Our agent will have three actions: Buy a Share, Wait, or Close the Position/Sell. Hypothetically, to buy a share, the agent would borrow a stock share and be charged a commission. If the agent already owns a share, nothing will happen and the agent can have at most one share. If an agent owns a share and decides to close the position/sell, the agent will pay a commission which is a fixed percentage of the current price. If the agent does not own a share, nothing will happen. The action Wait is that of taking no action at all. Table 3 details the specifications of the Basic Stock Environment. Table 4 contains hyperparameters of the DQN agent trained in this environment.

The TensorTrade environment (TT) has more components than the basic DQN’s environment. TT can implement a portfolio that holds wallets of various coins or currencies. The data used for this setup is included with TT as a demonstration of training. This dataset is dated from the start of 2020, and contains the open, high, low, close and volume prices at hourly intervals. It also includes technical indicators such as the Relative Strength Indicator (RSI) and Moving Average Convergence Divergence (MACD) as well as $log(C_{t})-log(C_{t-1})$ where $C_{t}$ is the closing price at timestep $t$ as the dataset features. Our portfolio starts with the same setup as TT’s demo which includes 10,000 USD and 10 BTC. We use the risk-adjusted reward scheme and manage-risk action scheme provided by TT. The risk-adjusted reward scheme uses the Sharpe Ratio which is defined by the equation below:

$$S_{a}=\frac{E[R_{a}-R_{b}]}{\sigma_{a}}$$

where $R_{a}$ is the asset return, $R_{b}$ is the risk-free return, and $\sigma_{a}$ is the standard deviation of the asset excess return. The implementation offsets the numerator and denominator by a small decimal to avoid zero division. The manage-risk action scheme scales the action space depending on provided arguments such as trade size, stop and take. The default trade size is 10 which implies there will be a list of 10 trade sizes that are uniformly spaced. For instance, trade size of 3 implies $33.3\%$, $66.6\%$, and $99.9\%$ of the balance can be traded. Take is a list of possible take profit percentages from an order, and stop is a list of possible stop loss percentages from an order. The action space is the resulting product of take, stop, trade size, and action type which is buy or sell. There is one additional action, namely wait/hold placed at index 0. In our case, we have an action space size of 181. This information as well as training hyperparameters are summarized in Table 3 and Table 4, respectively. There are other simpler reward (e.g., SimpleProfit) and action (e.g., BSH stands for Buy Sell Hold) schemes available with TT.

We evaluate through non-targeted attacks on a single, most recent window history tuple of their features used for observation. This is an attack on the observation channel. For the basic DQN, this would be a tuple of relative high, relative low and relative close prices in regards to their open price. For TensorTrade’s DQN, this would be $log(C_{t})-log(C_{t-1})$ where $C_{t}$ is the closing price at timestep $t$, MACD, and RSI. Our first attack will be an observation delay of 1 timestep where a tuple of values seen at timestep $t-1$ will be received at timestep $t$. We choose a delay of 1 timestep because it is both practical and representative of minimal interference. Results are presented in Figure 2. As is demonstrate in the results, when an agent picks an action based on delayed observation from timestep $t-1$, action received by the environment which is at a true timestep $t$ returns a reward which can be at most equivalent to the optimal action reward at the timestep $t$. This type of non-targeted attack should be of concern to traders because of its lack of computational expense to implement, and adversarial predisposition since it depends on time-series locality to mask anomalies.

To investigate the effectiveness of adversarial example attacks on DRL policies, we implemented Fast Gradient Sign [5] and Carlini and Wagner ( C&W) [20] adversarial example attacks using $L_{2}$ loss for both DQNs.

For our basic DQN implementation, the values of high, low, close prices in the observation space are relative prices scaled according to their open price. The data is not normalized along its dimension, but the value range is similar by density since majority of the relative prices fall within a bounded region of values and its range is fairly close to $[0,1]$. Instead of selecting the perturbation threshold $\epsilon$ through data prepossessing such that $\epsilon$ can be a valid representation of allowable perturbation along a dimension, we test a range of small epsilon values and chose the best among the selection that produced the most adversarial-like samples. Additionally, we apply post constraints with a strong assumption of a fixed, uniformal step-size across all dimensions to be acceptable. We have fixed the initial $\epsilon$ to $0.0001$ and for any failure to pick another action at a timestep, the attack will be allowed five iterations at increasing $\epsilon$ values where its last iteration tests at an $\epsilon$ of value $0.001$. Our implementation of the C&W attack on the basic DQN has a max iteration of 100 with a learning rate of 0.5 and initial constant of 0.1. We chose a lower number of iterations and higher learning rate because of computational expense, but the results are sufficient in representing adversarial observations that are similar to their true observation state. Results of the non-target FGSM attack and non-target C&W attack on the basic DQN can be seen in Figure 3. Only successful non-target attacks have their perturbed observation saved for future timesteps. Additionally, if there is a successful attack at timestep $t$ that has impacted a future action a_t+k where $k<0\leq N$, we do not attack the timestep and count the timestep as no change needed (NCN). We define a failure for the basic DQN as the failure to change an agent’s action. Our attacks abide by constraints that makes these perturbations adversarial when compared to the true observations: Relative high prices are non-negative and relative low prices are non-positive. Relative closing price must be bounded within the relative high price and relative low price. We also have considered matching the true relative close price’s behavior whether it was a relative high price, relative low price, or strictly between the prices. By applying these post constraints, we shift the perturbation of a dimension within the correct distribution of values for a given dimension. Table 10 contains failure counts and other notable counts for the basic DQN. Representative samples of a perturbed tuple of values from successful attacks are presented in Table 5 and Table 6.

Our TensorTrade DQN’s action space is larger and we have fewer timesteps to attack, which imply a higher failure ratio for attacks with perturbation per observation probabilities less than 1.0. We adjust our definition of failure as the failure to change an agent’s action $a$ to some action $a^{\prime}$ which is not the same action type. For instance, if $a$ is a buy action type, then the attack has failed to change the agent’s action to either a sell action type or wait action type. We have tighter constraints for non-targeted attacks because we are more interested in change of action type in regards to indiscriminate attacks. However, more lenient failure constraints are also valid since they also impact the agent’s performance. Results on failure count and attempts can for TT’s DQN can be found in Table 9. Unlike the basic DQN’s observations, our observations are not bounded to a small range close to $[0,1]$ nor have similar distributions along its dimensions such that uniform steps in all dimensions are adversarial. This stems from the non-normalized state of the data. Additionally, normalizing data that is not innately bounded makes it difficult to guarantee that FGSM’s $\epsilon$ which is used to represent the amount of allowable perturbation in all dimensions be applicable to newer data that may fall out of the existing normalized boundaries. Therefore, we cannot think of FGSM’s $\epsilon$ as $\epsilon\in(0,1]$ where it represents allowable perturbation in this case. We evaluate values of $\epsilon$ by comparing the adversarial observation with its true observation. Since the features do not share similar ranges of values, we individually scale each perturbed feature by $\epsilon\times k_{d}$, where $0\leq k_{d}\leq 1$, where $d$ is the respected feature/dimension such that our perturbations are similar to their true observation tuple. In our implementation, these $k$ scalars are exponential forms $10^{-m}$ where $m\in\mathbb{N}$. Alternative $k$ scalars can be calculated for each dimension by using their distribution. Our non-targeted FGSM attack follows the same setup as before for the basic DQN but we start at an $\epsilon$ of 0.1 and end at an epsilon of 3.0 with $k_{0}=0.01$, $k_{1}=0.01$, and $k_{2}=0.1$. For example, if given a tuple:

$$(x_{0},x_{1},x_{2})$$

where these values are from our feature vector, then the perturbation tuple where $\delta=(p_{0},p_{1},p_{2})$ will be

$$(x_{0}+\epsilon k_{0}p_{0},x_{1}+\epsilon k_{1}p_{1},x_{2}+\epsilon k_{2}p_{2})$$

.

For our C&W attack, we could not implement in the way we did for the basic DQN. C&W optimizes over a vector $w$ that produces an adversarial example $x+\delta$ with bounded values between $[0,1]$ given$x+\delta=\frac{1}{2}\times(tanh(w)+1)$ with $x$ as the true sample which implies that the feature data must be normalized for a direct C&W attack. Our agent is not trained on normalized data, but we can similarly add an amount of adversarial perturbation to the original observation like in FGSM as long as the adversarial observations falls within the distribution. This C&W attack implementation is weaker than the basic DQN’s C&W attack implementation but still impacts the performance of the agent. We have individual scalars $k_{0}=0.01$, $k_{1}=1.0$, and $k_{2}=1.0$. We consider $\epsilon=1.0$ to be the scalar applied to each $k_{d}$ where $d$ is a feature/dimension. We have the learning rate set to 0.5 and max iterations of 100. Table 7 and Table 8 contain successful samples for the non-target FGSM and C&W attack on TT’s DQN. Despite manually testing scalar values, we have crafted adversarial samples that are convincing. Additionally, we have provided the total reward difference and net-worth difference between the control agent and agents under attack in Figure 4 and Figure 5, respectively. Through these results, we establish that the test-time performance of the target policy in regards to its total reward is negatively impacted by our attacks. We have also shown that the agent’s net-worth is also impacted, but not necessarily reflected by total reward.

It is noteworthy that constraints are applied after the construction of an adversarial tuple, which impact the number of failures. For example, for an FGSM attack at a timestep $t$ on the basic DQN, the gradient provides the direction of the minimum and we craft an adversarial tuple starting from the original tuple either moving away from the minimum (for non-targeted) or moving towards the minimum for a targeted class. We shift the values of the adversarial samples along the respected dimension if they violate the constraint which is not in the direction of gradient. This affects targeted attacks, but can also impact non-targeted attacks. Though not implemented, failures preserved in the history window can impact future actions while in the observation leading to a sub-optimal trajectory. Though all observations that receive positive reward from the environment increase the return, some timesteps contribute more to the return than others emphasizing that any successful attack can have varying impacts on the agent’s performance for timing dependent tasks. We see that even less frequent attacks on the simpler DQN impact the agent’s performance at test-time and weaker attacks like FGSM are also effective against TT’s DQN agent performance.

We also investigate targeted attacks, which aim to manipulate a policy into taking an adversarial action $a_{t}^{\prime}$ instead of action $a_{t}$ at a timestep $t$. We have evaluated against targeted FGSM and targeted C&W attacks using $L_{2}$ loss for both DQNs.

For the targeted FGSM attack on the basic DQN, we allowed up to five iterations of increasing $\epsilon$, starting at $0.0001$ where the last iteration tests at an $\epsilon$ value of 0.001. The adversarial action $a_{t}^{\prime}$ is set to be the action with the least Q value at timestep $t$. The same constraints are implemented from the non-target attacks on the simpler DQN. C&W parameters are the same as non-targeted C&W attack on the simpler DQN. Table 15 contains the failure count and attempt count. We define failure as the failure to change the agent’s action $a_{t}$ to the adversarial action $a_{t}^{\prime}$ at timestep $t$. To reflect only the impact of targeted attacks, only perturbations in observation that resulted in adversarial action $a_{t}^{\prime}$ were kept for future timesteps. Results of the targeted FGSM and C&W attack is outlined in Figure 6 for the simpler DQN. All non-targeted attacks were assigned the optimal action $a_{t}$ for each respected timestep $t$. Our adversarial tuples are simple but should emphasize that adversarial attacks crafted under expensive parameters like low learning rate, high number of iterations, and high confidence can produce more human convincing adversarial samples. Successful samples for these attacks on the simpler DQN can be found in Table 11 and Table 12.

For the targeted FGSM attack through TT’s DQN, we consider a similar setup to non-targeted FGSM attack on the simpler DQN but we will be more lenient on the failure criteria because the action space is large. We will consider an attack a failure if the attack fails to change the agent’s action $a_{t}$ to the adversarial action $a_{t}^{\prime}$ at a timestep $t$. We will consider a partial success if our attack results in an action $a^{m}$ where the action type (buy, sell, wait) of $a^{m}$ is the adversarial action type for action $a_{t}^{\prime}$. Successful attacks and partial successful attacks will be preserved in observation, otherwise all failures reassign the agent to take optimal action $a_{t}$ at timestep $t$. Failure count and attempts are in Table 16 and successful samples for targeted FGSM and targeted C&W for TT’s DQN can be found in Table 13 and Table 14 respectively. Like before for non-targeted attacks on TT’s DQN, we have plots of the reward difference and net-worth difference between the control agent and attacked agents which are Figure 7 and 8. We thus establish the impact of targeted attacks on TT’s DQN on test-time performance as well as its significant impact on the agent’s net-worth.