Reinforcement Learning for Systematic FX Trading

Transfer learning refers to the machine learning paradigm in which an algorithm extracts knowledge from one or more application scenarios to help boost the learning performance in a target scenario [8]. Typically, traditional machine learning requires significant amounts of training data. Transfer learning copes better with data sparsity by looking at related learning domains where data is sufficient. Even in a big data scenario such as with streaming high-frequency data, transfer learning benefits by learning the adaptive statistical relationship of the predictors and the response. An increasing number of papers focus on online transfer learning [11, 12, 13]. Following Pan and Yang [14], we define transfer learning as:

In the context of this paper, the source domain $\mathcal{D}_{S}$ represents the feature space, which consists of the daily returns of the 36 currency pairs that are used in our experiment. The source learning task $\mathcal{T}_{S}$ is the unsupervised compression of this feature space into a clustered form that learns its intrinsic nature. The clusters are formed via Gaussian mixture models, and we transfer their output via radial basis function networks to currency pairs that we wish to trade in the target domain $\mathcal{D}_{T}$. The target learning task $\mathcal{T}_{T}$ is to take financial risk positions in these currency pairs for economic utility maximisation via direct recurrent reinforcement learning.

Williams [15] was one of the first to introduce policy gradient methods in a reinforcement learning context. Whereas most reinforcement learning algorithms focus on action-value estimation, learning the value of actions and selecting them based on their estimated values, policy gradient methods learn a parameterised policy that can select actions without using a value function. Williams also introduced his reinforce algorithm

where $\boldsymbol{\theta}_{ij}$ is the model weight going from the $j^{\prime}th$ input to the $i^{\prime}th$ output, and $\boldsymbol{\theta}_{i}$ is the weight vector for the $i^{\prime}th$ hidden processing unit of a network of such units, whose goal it is to adapt in such a way as to maximise the scalar reward $r$. For the moment, we exclude the dependence on the time of the weight update to make the notation clearer. Furthermore, $\eta_{ij}$ is a learning rate, typically applied with gradient ascent, $b_{ij}$ is a reinforcement baseline, conditionally independent of the model outputs $y_{i}$, given the network parameters $\boldsymbol{\theta}$ and inputs $\mathbf{x}_{i}$. $\ln(\partial{\pi_{i}}/\partial{\boldsymbol{\theta}}_{ij})$ is known as the characteristic eligibility of $\boldsymbol{\theta}_{ij}$, where $\pi_{i}(y_{i}=c,\boldsymbol{\theta}_{i},\mathbf{x}_{i})$ , is a probability mass function determining the value of $y_{i}$ as a function of the parameters of the unit and its input. Baseline subtraction $r-b_{ij}$ plays a vital role in reducing the variance of gradient estimators. Sugiyama [16] shows that the optimal baseline is given as

where the policy function $\pi(a_{t}|s_{t},\boldsymbol{\theta})$ denotes the probability of taking action $a_{t}$ at time t given state $s_{t}$, parameterised by $\boldsymbol{\theta}$. The expectation $\mathbb{E}_{p(r|\boldsymbol{\theta})}$, is distributed over the probability of rewards given the model parameterisation.

The main result of Williams’ paper is

This result relates $\nabla\mathbb{E}[r|\boldsymbol{\theta}]$, the gradient in weight space of the performance measure $\mathbb{E}[r|\boldsymbol{\theta}]$, to $\mathbb{E}[\Delta\boldsymbol{\theta}|\boldsymbol{\theta}]$, the average update vector in weight space. Thus for any reinforce algorithm, the average update vector in weight space lies in a direction for which this performance measure is increasing, and the quantity $(r-b_{ij})\ln(\partial{\pi_{i}}/\partial{\boldsymbol{\theta}}_{ij})$ represents an unbiased estimate of $\partial{\mathbb{E}[r|\boldsymbol{\theta}}]/\partial{\boldsymbol{\theta}_{ij}}$.

Sutton and Barto [17] demonstrate an actor-critic version of a policy gradient model, where the actor references the learned policy and the critic refers to the learned value function, usually a state-value function. Denote the scalar performance measure as $J(\boldsymbol{\theta})$; the gradient ascent update takes the form

With the one-step actor-critic policy gradient algorithm, one inserts a differentiable policy parameterisation $\pi(a|s,\boldsymbol{\theta})$, a differentiable state-value function parameterisation $\hat{v}(s,\mathbf{w})$ and then one draws an action

taking action $a_{t}$ and observing a transition to state $s_{t+1}$ with reward $r_{t+1}$. Define

where $0\ll\gamma\leq 1$ is discount factor. The critic’s weight vector is updated as follows

and finally, the actor’s weight vector is updated as

The actor-critic architecture uses temporal-difference learning combined with trial-and-error learning to improve the learned policy sequentially.

This section describes the foreign exchange market and the mechanics of the foreign exchange derivatives, which are central to the experimentation that we conduct in section IV. The global foreign exchange market sees transactions above 6 trillion US dollars traded daily. Figure II-C shows this breakdown by instrument type and is extracted from the Bank of International Settlements Triennial Central Bank Survey, 2019.

[!t]()[width=0.99]bis_daily_fx_turnover.png Average daily global foreign exchange market turnover in millions of US dollars, source: Bank of International Settlements.

FX transactions implicitly involve two currencies: the dominant or base currency is quoted conventionally on the left-hand side and the secondary or counter currency on the right-hand side. If foreign exchange positions are held overnight, the trader will earn the interest rate of the currency bought and pay the interest rate of the currency sold. The interest rates for specific maturities are determined in the inter-bank currency market and are heavily influenced by the base rates typically set by central banks. Foreign exchange trades settle two business days after the trade date by market convention unless otherwise specified.

Clients fund their positions by rolling them forward via tomorrow/next (tomnext) swaps. Tomnext is a short-term foreign exchange transaction where a currency pair is simultaneously bought and sold over two business days: tomorrow (in one business day) and the following day (two business days from today). The tomnext transaction allows traders to maintain their position without being forced to take physical delivery and is the convention applied by prime brokers to their clients on the inter-bank foreign exchange market. In order to determine this funding cost, one needs to compute the forward rates (prices). Forwards are agreements between two counterparties to exchange currencies at a predetermined rate on some future date.

Forward rates are calculated by adding forward points to a spot rate. These points reflect the interest rate differential between the two currencies being traded and the maturity of the trade. Forward points do not represent an expectation of the direction of a currency but rather the interest rate differential. Let $bid_{t}^{spot}$ denote the spot/cash currency pair rate at which price takers can sell at time $t$. Similarly, let $ask_{t}^{spot}$ denote the spot/cash currency pair rate at which price takers can buy at time $t$. The spot mid-rate is

Forward points are computed as follows

where $e_{2}$ is the secondary interest rate, $e_{1}$ is the dominant interest rate, $T$ is the number of days till maturity, and $\phi$ is the tick size or pip value for the associated currency pair. Example forward points for GBPUSD are shown in figure II-C. GBP= is the Refinitiv information code (ric) for cash GBPUSD and GBPTND= is the ric for tomnext GBPUSD forward points. Note that the forward points are quoted as a bid/ask pair, reflecting the appropriate interest differential applied to sellers and buyers and the additional cost (spread) quoted by the foreign exchange forwards market maker to compensate them for their quoting risk. The tomnext outrights are computed as

As an example of rolling a long GBPUSD position forward, the tomnext swap would involve selling GBPUSD at $bid_{t}^{spot}$ and repurchasing it at $ask_{t}^{tn}$. The cost of this roll is thus $notional\times(bid_{t}^{spot}-ask_{t}^{tn})$, where $notional$ denotes the size of the position taken by the trader. If a trader is short GBPUSD, then to roll the position forward, she would buy $ask_{t}^{spot}$ and sell forward $bid_{t}^{tn}$, with the funding cost being $notional\times(bid_{t}^{tn}-ask_{t}^{spot})$. This funding may be a loss but also a profit. In addition, many currency market participants hold foreign exchange deliberately to capture the favourable interest rate differential between two currency pairs. This approach is known as the carry trade and is extremely popular with the retail public in Japan, where the Yen interest rates have been historically low relative to other countries for quite some time.

[!t]()[width=0.9]gbpfwd.png Refinitiv GBPUSD forward rates.

Sharpe [5] discusses asset allocation as a function of expected utility maximisation, where the utility function may be more complex than that associated with mean-variance analysis. Denote the expected utility at time $t$ for a single portfolio constituent as

where the expected return $\mu_{t}=\mathbb{E}[r_{t}]$ and variance of returns $\sigma_{t}=\mathbb{E}[r_{t}^{2}]-\mathbb{E}[r_{t}]^{2}$ may be estimated in an online fashion with exponential decay, where as before $\tau$ is an exponential decay constant

The risk appetite constant $\lambda>0$ can be set as a function of an investor’s desired risk-adjusted return, as demonstrated by Grinold and Kahn [33]. The information ratio is a risk-adjusted differential reward measure, where the difference is taken between the model or strategy being evaluated and a baseline or benchmark strategy with expected returns $b_{t}=\mathbb{E}\big{[}r_{t}^{(b)}\big{]}$:

The similarity to the Sharpe ratio is apparent. Setting $b_{t}=0$ and substituting the non-annualised information ratio into the quadratic utility and differentiating with respect to the risk, we obtain a suitable value for the risk appetite parameter:

The net returns whose expectation and variance we seek to learn are decomposed as

where $\Delta p_{t}$ is the change in reference price, typically a mid-price

$\delta_{t}$ represents the execution cost for a price taker

$\kappa_{t}$ is the profit or loss of rolling the overnight foreign exchange position, the so-called ’carry’ (see section II-C) and $f_{t}$ is the desired position learnt by the recurrent reinforcement learner

The model is maximally short when $f_{t}=-1$ and maximally long when $f_{t}=1$. The recurrent nature of the model occurs in the input feature space where the previous position is fed to the model input

and $\phi_{j}(.)$ denotes a radial basis function hidden processing unit, in a network of $m$ such units, which takes as input a feature vector $\mathbf{u}_{t}$, see section III-B. The goal of our recurrent reinforcement learner is to maximise the utility in equation 3 by targeting a position in equation 9. To do this, one may apply an online stochastic gradient ascent update

Instead of a static learning rate $\eta$, one may consider the Adam optimiser of Kingma and Ba [34], where an adaptive learning rate is applied. This adaptive learning rate is a function of the gradient expectation and variance. The weight update then takes the form

with $\hat{\mathbf{m}}_{t}=\mathbf{m}_{t}/(1-\beta_{1})$ and $\hat{\mathbf{v}}_{t}=\mathbf{v}_{t}/(1-\beta_{2})$ denoting bias-corrected versions of the expected gradient and gradient variance, respectively. $\beta_{1}$ and $\beta_{2}$ are exponential decay constants. In earlier work, Bottou [35] had considered approximating the Hessian of the performance measure with respect to the model weights as a function of gradient only information. In practice, we find that Adam takes many iterations of model fitting to get the weights large enough to take a meaningful position via function 9; this is not necessarily an Adam problem, but a result of the $\tanh$ position function taking a while to saturate. If the weights are too small, then the average position taken by the recurrent reinforcement learner will be small as well. Therefore, we settle on an extended Kalman filter [36, 9] gradient-based weight update, albeit modified for reinforcement learning in this context.

In algorithm 1, $\mathbf{P}_{t}$ is an approximation to $[\nabla^{2}\upsilon_{t}]^{-1}$, the inverse Hessian of the utility function $\upsilon_{t}$ with respect to the model weights $\boldsymbol{\theta}_{t}$.

We decompose the gradient of the utility function with respect to the recurrent reinforcement learner’s parameters as follows:

The constituent derivatives for the left half of equation 11 are:

In Borrageiro et al. [37], the authors show that online transfer learning via radial basis function networks provides a residual benefit in forecasting non-stationary time series. The residual benefit stems from the feature representation transfer of clustering algorithms. These algorithms are adapted sequentially, as are the supervised learners, which map the clustered feature space to the targets. The feature engineering that we use in this paper uses clusters formed of Gaussian mixture models. The network size is determined by the unsupervised learning procedure of finite mixture models described by Figueiredo and Jain [7]. Finally, we briefly describe the key ingredients of this meta-algorithm here.

The radial basis function network is a network of $m>0$ Gaussian basis functions

Here we learn the $j^{\prime}th$ mean $\boldsymbol{\mu}_{j}$ and covariance $\boldsymbol{\Sigma}_{j}$ through a Gaussian mixture model fitting procedure. Denote the probability density function of a $k$ component mixture as

where

and the mixing weights satisfy $0\leq\pi_{j}\leq 1$, $\sum_{j=1}^{k}\pi_{j}=1$. The maximum likelihood estimate

and the Bayesian maximum a posteriori criterion

cannot be found analytically. The standard way of estimating $\boldsymbol{\theta}_{ML}$ or $\boldsymbol{\theta}_{MAP}$ is the expectation-maximisation algorithm [38]. This iterative procedure is based on the interpretation of $\mathbf{u}$ as incomplete data. The missing part for finite mixtures is the set of labels $\mathcal{Z}={\mathbf{z}_{0},...,\mathbf{z}_{n}}$, which accompany the training data $\mathbf{u}_{0},...,\mathbf{u}_{n}$, indicating which component produced each training vector. Following Murphy [39], let us define the complete data log-likelihood to be

which cannot be computed since $\mathbf{z}_{i}$ is unknown. Thus, let us define an auxiliary function

where $t$ is the current time step. The expectation is taken with respect to the old parameters $\boldsymbol{\theta}_{t-1}$ and the observed data $\mathbf{u}$. Denote as $r_{ic}=p(z_{i}=c|\mathbf{u}_{i},\boldsymbol{\theta}_{t-1})$, cluster $c$’s responsibility for datum $i$. The expectation step has the following form

The maximisation step optimises the auxiliary function $\mathcal{Q}$ with respect to $\boldsymbol{\theta}$

The $c^{\prime}th$ mixing weight is estimated as

The parameter set $\boldsymbol{\theta}_{c}=\{\boldsymbol{\mu}_{c},\boldsymbol{\Sigma}_{c}\}$ is then

As discussed by Figueiredo and Jain [7], expectation-maximisation is highly dependent on initialisation. They highlight several strategies to ameliorate this problem, such as multiple random starts, final selection based on the maximum likelihood of the mixture, or k-means based initialisation. However, the distinction between model-class selection and model estimation in mixture models is unclear. For example, a 3 component mixture in which one of the mixing probabilities is zero is indistinguishable for a 2 component mixture. They propose an unsupervised algorithm for learning a finite mixture model from multivariate data. Their approach is based on the philosophy of minimum message length encoding [40], where one aims to build a short-code that facilitates a good data generation model. Their algorithm can select the number of components and, unlike the standard expectation-maximisation algorithm, does not require careful initialisation. The proposed method also avoids another drawback of expectation-maximisation for mixture fitting: the possibility of convergence toward a singular estimate at the boundary of the parameter space. Denote the optimal mixture parameter set

where

This leads to a modified maximisation step

The maximisation step is identical to expectation-maximisation, except that the $c^{\prime}th$ parameter set $\boldsymbol{\theta}_{c}$ is only estimated when $\pi_{c}>0$ and $\boldsymbol{\theta}_{c}$ is discarded from $\boldsymbol{\theta}^{*}$ when $\pi_{c}=0$. A distinctive feature of the modified maximisation step is that it leads to component annihilation; this prevents the algorithm from approaching the boundary of the parameter space. In other words, if one of the mixtures is not supported by the data, it is annihilated.

We finish the section by showing figure III-B, which provides a visual representation of the feature representation transfer from the radial basis function network to the recurrent reinforcement learning agent. The external input to the transfer learner, represented by the left-most black circles, is a vector of daily returns of the 36 currency pairs used in the experiment, detailed in section IV-A. The grey circles represent the radial basis function network hidden processing unit layer. In addition, we have a blue circle that represents the previously estimated position of the recurrent reinforcement learning agent. The outputs of this hidden layer are stored in a feature vector, as shown by equation 10. These outputs are fed into the recurrent reinforcement learning agent, who learns the position function using equation 9. The weights of the position function are fitted via the extended Kalman filter procedure of algorithm 1. The gradient vector, fed into the extended Kalman filter, is computed using equation 11. This output is fed back into the hidden layer in a recurrent manner represented by the dotted blue line.

[!t]()[width=0.99]rbf_net.png Feature representation transfer from a radial basis function network to a recurrent reinforcement learning agent.

In order to assess the comparative strength of the model of section III-A, we employ two baseline models. The first model is a momentum trader, which uses the sign of the next step ahead return forecast as a target position. This model is also a radial basis function network; except here, the feature representation transfer of the Gaussian mixture model cluster is made available to an exponentially weighted recursive least-squares supervised learner. A visual representation of the model is similar to figure III-B, without any recurrent position unit as represented by the blue circle.

The exponentially weighted recursive least-squares fitting procedure is shown compactly in algorithm 2. The precision matrix $\mathbf{P}_{0}$ may be initialised to the identity matrix scaled by the inverse of the Ridge penalty, $\mathbf{I}_{d}\alpha^{-1}$ and the initial weights $\mathbf{w}_{0}$ are typically initialised to the zero vector. The discount factor $\tau$ is typically close to but less than 1. The particular model form is experimented with by Borrageiro et al. [37] in a multi-step horizon forecasting context.

Our second baseline is the carry trader, hoping to earn the positive differential overnight foreign exchange rate. Denoting the long and short carry as

where the superscript spot denotes the cash price, and the superscript tn denotes the tomorrow/next price, the position of the carry trader is

In other words, the carry trader goes long the base currency if the base currency has an overnight interest rate higher than the counter currency. Equally, the carry trader sells the base currency short if the base currency has an overnight interest rate that is lower than the counter currency. Long and short carry may be a cost rather than a profit due to the bid/ask spread that traders make markets in tomnext swaps. Therefore we allow the carry trader to abstain from trading completely in such circumstances.

We obtain our experiment data from Refinitiv. We extract daily sampled data for 36 major cash foreign exchange pairs with available tomnext forward points and outrights. These foreign exchange pairs are listed in table I. Summary statistics of the distribution of the daily returns for these currency pairs are shown in table II. The dataset begins 2010-12-07 and ends on 2021-10-21, a total of 2,840 observations per pair. Daily spot mid-price returns are constructed for each of these currency pairs. These are used as the features for the recurrent reinforcement learning agent and the exponentially weighted recursive least-squares momentum trader. The mid-price is defined in equation 2, and the return for the $k^{\prime}th$ pair is simply

One of the challenges that the models will face in the experiment is that these daily data show the last known top of book spot and outright prices at the end of the trading day, 5 pm EST. The bid/ask spread for these prices are at their widest statistically at this time. Therefore the execution and funding costs will be more expensive; this contrasts with a trader who can execute at a more liquid time, such as 2 pm GMT. If we try to use intraday data, say data sampled minutely, Refinitiv restricts us to 41 trading days, which is not a huge sample size. Figure II illustrates the challenge succinctly. It shows relative intraday bid/ask spreads

for the 36 currency pairs that we experiment with. The data are sampled minutely over two months ending mid-October 2021. The global maximum bid/ask spread occurs precisely when Refinitiv samples the daily data.

[!t]()[width=0.99]intraday_spreads.png Relative intra-day bid/ask spreads for the 36 Refinitiv currency pairs that we experiment with.

We have a little over 11 years of daily data to use in our experiment. From these data, we construct daily returns for each of the 36 currency pairs, reserving the first third as a training set and the final two-thirds as a test set. The structure of the radial basis function networks of sub-section III-B is determined in the training set, with external input being the returns of the various currency pairs. The recurrent reinforcement learning agent is also fitted in the training set to each currency pair, explicitly learning the weights in the position function 9, using the extended Kalman filter learning procedure of algorithm 1. Additionally, the momentum trader of sub-section III-C is fitted in the training set to each currency pair using algorithm 2. Both models continue to learn online during the test set. However, the carry trader baseline does not require any model fitting.

The test set evaluates performance for each currency pair using the net profit and loss equation 8. This reward, net of transaction and funding cost, is in price difference space. We convert to returns space by dividing by the mid-price computed using equation 2. These returns are accumulated to produce the results shown in figure IV and the middle sub-plots of figures V and V. In addition, the daily returns are described statistically in tables III and IV. In table III, the information ratio (ir) is computed using equation 6. We set the baseline return $b_{t}=0$. In summary, we evaluate performance by considering the risk-adjusted daily returns generated by each model, net of transaction and funding costs.

The following hyper-parameters are set in the experiment:

• •

  $\tau=0.99$; this is the exponential decay constant of moving moment equations 4, 5, 8, extended Kalman filter weight algorithm 1 and exponentially weighted recursive least-squares algorithm 2.

• •

  $\alpha=1$; this is the Ridge penalty of extended Kalman filter weight algorithm 1 and exponentially weighted recursive least-squares algorithm 2.

• •

  $\gamma$, the risk appetite parameter of equation 3, is initially set to $1$ and then updated by passing through the training data once and setting it via the procedure of equation 7.