A Comparison of Reinforcement Learning and Deep Trajectory Based Stochastic Control Agents for Stepwise Mean-Variance Hedging

This paper concentrates on hedging strategies that optimize the mean and variance of hedged portfolios (containing hedged instruments and hedging instruments) by optimizing a cumulative stepwise mean-variance objective, but the setting will apply to many trading strategies with different objectives (and we will discuss other settings in future papers).

To define the set up, we will need to model how the prices of the hedging and the hedged instruments evolve, how the trading strategy to do so performs (including transaction costs), and define the stepwise objectives to be optimized over in terms of the prices, costs, etc. involved in the strategy.

In mean-variance hedging, one assumes that both the prices of the hedging and the hedged instruments to define the hedging objectives are known. This might be the case if one tries to learn hedging strategies from settings in which both hedged instruments (say, vanilla options) and hedging instruments (underlier and cash) are liquidly quoted or where one has a book-keeping model for the hedged instruments that will provide the price. The first setting might make sense if one has over the counter (OTC) or exchange-traded options on the books and needs to hedge them with the underlying even though at least some such options are liquidly traded. The second setting might make sense for a trader in an investment bank that hedges an option on the books which will be valued by that book-keeping model¹¹1 We call a model a book-keeping model if it reflects the price or value of an instrument in some way but does not necessarily take all details and features of the instrument and/or the market into account. For instance, Black-Scholes type models and similar ”simpler” models are often used to mark instruments in systems of record (SOR) and they are validated and approved for such use because even though they do not reflect all features and details, they reflect value well enough for purpose. In particular, if the model is contained in a system of record, we call it a SOR model. Sometimes, for purposes of margins or collateral or trading through central counterparties, using such simpler book-keeping models has the advantage that there will be less uncertainty and fewer arguments about the model parameters and settings, compared to more complicated models. in the system of record (SOR) and the trader will be judged by their performance against the prices contained in the SOR.

There might be settings in which neither the hedged instrument nor any closely related instrument are liquidly quoted or in which there are no appropriate and flexible enough book-keeping models available. In these circumstances, one would need stepwise or global objectives that are defined without reference to (book-keeping or other) prices of the to-be-hedged instruments. These could be for instance objectives that consider how well the trading strategy replicates or risk-manages all cashflows of the to-be-hedged instruments, for instance in a squared difference sense, leading to quadratic hedging (we leave this case to a future paper).

We consider a case in which we have a model for how appropriate risk factors and prices of the hedging and the hedged instruments evolve that are needed to compute the objectives. What we mean by that is that we have some implementation that is able to generate as many trajectories of these instrument prices and other risk factors as needed. Based on these trajectories, we will be able to simulate the trading strategy and their performance and compute the appropriate objectives. We will here assume that the trading from our trading strategies does not impact how the prices of the hedging and the hedged instruments evolve, just how the trading strategy and the objectives evolve, which is a common assumptions in current works. We do not specify how this model is obtained, setup, or run; only that we can obtain those trajectories as needed. As such, the model could be a well-specified traditional model or it could be a trained generative model ([BRMH22], [CRW21]). Here, we will work with simulated data from Black-Scholes model and defer the settings involving synthetic data from generative models to future work.

We will test and validate our agents on simulated data from such models and leave the question whether such models trained on simulated or generated data can perform well on observed data for our settings to future work. There are indications in the literature that models trained on appropriate simulated or synthetic data can perform well when applied to observed data from the market (as in [BRMH22]).

We are not using the model(s) or any information about the model(s) in the learning and training beyond using it to generate trajectories and data including stepwise objectives. In this way, we are operating in a model-free, data-driven setting.²²2Particular setups might include state variables and/or compute features that are based on a particular model or setting, such as an option Delta computed under the assumption that the dynamics of the hedging instrument can be approximated by a Black-Scholes type model, but the algorithms to learn trading strategies or value functions do not use or require any model details beyond what is contained in the state and feature space. Given the above discussion, we are actually operating in a rich-data (even infinite-data) regime, corresponding to a well-defined conventional or generative model.

The rest of the paper is structured as follows: First, we will discuss the instrument and trading strategy setup and mean-variance hedging. Then, we will discuss how Reinforcement Learning (RL) conceptualizes the problem and the setting, introduce the RL techniques and algorithms that we will apply, and then introduce the deep trajectory-based empirical stochastic optimal control (DTSOC) approaches and algorithm. We will then specify the experimental setup and the models, report and analyze outcomes, visualize and interpret outcomes, and analyze sensitivity of the results with respect to various choices. Finally, we conclude and discuss some possible future directions.

We discuss here some of the existing work in the literature on the use of RL and neural networks for pricing and hedging options (see [HXY21] for a broader survey of RL applications in finance).

In [Hal20], a European call option in the Black-Scholes model (trading at discrete times, no transaction costs) is hedged and priced using a $Q$-learning method (there, called QLBS method). The objective is minimizing the sum of the initial cash position and the weighted discounted sum of variances of the hedging portfolio at all subsequent time steps. This setting and objective can be identified with a corresponding MDP setup and thus solved with standard RL/DP approaches such as $Q$-learning [Hal20]. The price at maturity has to be equal to the final payoff and prices respective $Q$-functions at earlier times are determined by Dynamic Programming or other standard RL approaches.

The mean-variance hedging problem in a setting which includes transaction costs is considered in [CCHP21]. However, they define the reward objective in global fashion - it is the expected total cost for the entire hedging and its variance. They use versions of $Q$-learning and other methods to address this problem. They do not treat stepwise mean-variance hedging.

The stepwise mean-variance hedging under transaction costs for a single option is considered in [KR19]. The hedging agent is trained with an RL algorithm which seems to be a version of $Q$-learning. Also, stepwise mean-variance hedging under transaction costs with various RL algorithms is considered in [DJK⁺20].

In [MK21], authors consider a case where the stock price³³3Actually, the authors are using the value of the underlying index, and are assuming that the underlying index is tradeable, which is not completely accurate (futures on the index are traded but the spot index, in particular through changes to the index, is not tradeable and not entirely replicable through self-financing strategies). and the option price are taken from higher frequency data acquired from an option exchange and used to train an agent with an RL algorithm. They use a stepwise hedging objective; however, it is defined in terms of mean and standard deviation (and in terms of P&L and absolute value of P&L for the sample version). If the option is quoted and traded on an exchange, it might be less necessary to hedge or replicate said option.

There exists a large literature on applications of neural networks for option pricing. Most of these works consider the neural networks as a numerical tool to accelerate calculation tasks (such as model calibration [Her16], solving PDEs [WHJ21], etc.) more efficiently and in higher dimensions (also see [RW19] for a literature review). Among these, we distinguish the line of works initiated in [H⁺16] where neural networks are used to solve the stochastic optimal control problem numerically. More concretely, the DTSOC method in [H⁺16] is a setup for trajectory based empirical deep stochastic optimal control with both stepwise and final objectives/costs. Finally, the "deep hedging" paper [BGTW19] presents a trajectory based empirical deep stochastic optimal control approach to minimizing some global objectives related to replication and/or risk management of some final payoff, however, it does not consider a stepwise objective, just a global/final objective.

We consider here dynamic trading, hedging, and/or risk management problems. These problems are typically posed on some universe of given instruments (including cash/bank account) and the trader or agent has to select a strategy that determines how much of each instrument to hold at each time to achieve a certain trading, hedging, or risk management objective. As is common in such settings, we assume that there is a given fixed grid of times at which trades can occur and that such trades can incur transaction costs. We also assume that the strategies are self-financing. The objectives can be defined at a certain time horizon or they could be defined for each time period between trading times (stepwise objectives). Here, we will consider a stepwise objective. These stepwise objectives do not necessarily have to directly correspond to trading gain or loss or transaction cost or any financial item, they can also be defined in terms of moments or other properties of the distribution of specific gains, losses, costs, or appropriate combination thereof.

On a basic level, this is a stochastic sequential decision problem. The prices of the given instruments will follow some stochastic process and it is assumed that we are given a model or generator for this process. Based on those prices and any other appropriate risk factors that can be computed based on the history of said prices, we need to decide at each trading time-step what the new holdings of the given instruments will be and thus what the necessary trade sizes are (so holdings and/or trade sizes are the appropriate decision variables), taking into account the transaction costs and other impacts associated with that trade. Given those holdings and cash-at-hand at the beginning of the no-trade interval, the value of all those holdings and cash at the end of the no-trade interval follows from the stochastic evolution of the prices of the given instruments and interest on the cash account. From this, one can define the stepwise objective, which often is based on an appropriately defined change in strategy value over that period. We assume here that we have an implementation or model that describes the evolution of the trading strategy and also the computation of the stepwise objectives.

We consider the case of a hedged portfolio. This means that we are given a fixed holding $H^{O}$ of the to-be-hedged portfolio.⁴⁴4Which could be easily extended to a priori given holding level $H^{O}_{t}$ over time. It is assumed that the price of the to-be-hedged portfolio is given as $O_{t}$, corresponding to a quoted/observed or a computable book-keeping price. We do not rely on how this price is computed, only that we are provided with some implementation of it. It is also assumed that we are given other "hedging" instruments also and we denote their prices by $S_{t}$ (in general a vector) and we denote the holding of them in our strategy at time $t$ with $H^{S}_{t}$ (a vector of corresponding length).

For our examples, we will typically assume a single hedging instrument. Finally, we assume that excess cash from the trading strategy is deposited in a money-market account with a certain given interest accrual and discount factor (and that negative cash balances can be borrowed at the same interest accrual and discount factor). We assume that we initially start with only the fixed holding $H^{O}$ and some initial cash $Y_{0}$ but no holdings in the hedging instruments (i.e., $H^{S}_{0^{-}}=0$). The trading times are given as $t_{i}$ with $t_{0}=0$. If referred to, $t_{-1}$ is a time just before time 0 with no holding in hedging instruments.

Consider time $t_{i+1}$, right before the trade: The value of the hedging part of the strategy then is

This reflects that at the last trade time $t_{i}$, the portfolio was rebalanced to the appropriate holdings $H^{S}_{t_{i}}$, transaction costs were charged $\mathsf{TC}(H^{S}_{t_{i}},H^{S}_{t_{i-1}},S_{t_{i}})$, the thus resulting cash balance $Y_{t_{i}}-H^{S}_{t_{i}}S_{t_{i}}-\mathsf{TC}(H^{S}_{t_{i}},H^{S}_{t_{i-1}},S_{t_{i}})$ attract stepwise interest (interest accrual corresponds to dividing by appropriate stepwise discount factor $DF_{i,i+1}$). At the same time, the holdings in the hedging instruments are now valued based on the new price of the hedging instruments.

It helps to rewrite the equation as follows:

In this completely linear setting without stochastic interest rates (and assuming that the transaction cost is linear in the instrument prices for positive multipliers $c$, $\mathsf{TC}(\cdot,\cdot,cS)=c\mathsf{TC}(\cdot,\cdot,S)$), it is possible to "hide" the discounting in the value definitions in the following sense: with $\tilde{Y}_{t_{i}}=\mathsf{DF}_{0,i}Y_{t_{i}}$ and $\tilde{S}_{t_{i}}=\mathsf{DF}_{0,i}S_{t_{i}}$, we have

In particular, this means for the increment

and we thus do not get any contribution or impact of the initial cash position $Y_{0}$ beyond an additive shift.

If we are looking at the change of value in the total portfolio (both hedged and hedging instruments), we obtain

where we denoted $V_{t}=H^{O}O_{t}+Y_{t}$ and $\tilde{V}_{t}=H^{O}\tilde{O}_{t}+\tilde{Y}_{t}$ . In the following, we will work with values discounted back to initial time and will omit the tilde over the variables.

Extensions to stochastic interest rates and discounting, differential rates, more involved funding policies are possible, require further notation and details, and can be handled similarly, but are not necessary for the settings we discuss here. Differential rates and more involved funding policies could be included in the framework by adding bank accounts for positive and negative balances and other funding instruments to the hedging instruments and adding constraints to the decision variables (only positive holdings of positive bank account, only negative holdings of negative bank account i.e. bank loan, restricting secured loans/repos to the amount held in the corresponding collateral, etc.).

Often, one would like to control both the mean and variance of the stepwise gains or losses by maximizing mean of gains penalized by a multiple of variance (or equivalent minimizing mean of loss combined with variance). When sampling, one can approximate the variance by the second moment (see [Rit17] for a discussion) which means that one can express that objective by the expectation of costs

This can now be used as a cost or reward in RL or DTSOC algorithms. Costs to unwind the position at maturity (or convert it into the position to match the payoff of the hedged instrument) can be included as final cost if needed.

In some settings, one tries to control only the mean and variance globally, i.e. a mean and variance combination of only the final wealth is optimized over:

where $V_{t}$ is the wealth/value process for the overall hedged portfolio as derived earlier in this section (see, for instance [KR19]).

Using the following discrete time decompositions for the mean and variance of the terminal wealth under appropriate assumptions for the variance, for instance a random walk assumption as in [Rit17],

and

[KR19] rewrite the mean variance hedging (MVH) objective in the following way,

and thus obtain a stepwise objective as the one discussed in the last subsection.

The formulations of costs/rewards in the above sections only depend on discounted prices of the hedging instruments, hedged instrument, and transaction costs; but do not need any further information about the models for these instruments and prices. With the costs/rewards given above, the agent only needs to determine how many units to hold in each of the hedging instruments after rebalancing at each rebalancing time $t_{i}$, i.e. $H^{S}_{t_{i}}$. Alternatively, one could specify the units to be sold or bought at any given time together with the initial position; or assume that the amount sold or bought is proportional to the time between trades or some other variable and then specify that proportional trading rate in that setting.

For the hedged instrument, one needs to know the position in the instrument $H^{O}$ (i.e., whether long or short or some particular holdings at particular times) and additional information about the instrument - at least time to maturity and/or time. The minimal state for modeling this decision problem are these discounted prices (hedging and hedged instruments), the amount of the hedging instruments, time, and whatever state the models for these prices and the transaction costs need.

As for the models for the prices of the hedging and hedged instrument, one can model them under pricing or observational measure, with some differences in parameters and calibration or fitting. For the hedging instruments, one can use conventional quantitative finance models, possibly with hidden factors, such as Black-Scholes, Local Volatility Model, Heston model (with stochastic variance), quadratic rough Heston model, etc. One could use generative models such as GANs or appropriately trained neural Stochastic Differential Equations (SDE)s or similar. Finally, one could use historically observed data over one instrument or a cross section of similar instruments together with appropriate assumptions to generate possible future price movements or future prices.⁵⁵5See also [CSS21] for a discussion of possible approaches together with the associated ”model” risks.

For the hedged instrument, one of the challenges is that the prices of the hedged and the prices of the underlying instruments need to be consistent in an appropriate sense because otherwise they might allow arbitrage strategies. One could implement some book-keeping model which would give the price of that hedged instrument in terms of state variables and other parameters. In a trading setup in an investment bank or a hedge fund, holdings will be marked by some model within a system of record, and the book-keeping model would be that model in the system of record. For faster computation, one could try to learn a surrogate that approximately replicates that SOR model but can be run much faster. Alternatively, if the reference prices for the hedged instrument are observable, one may directly model such prices either as dependent on underlying prices or jointly with underlying prices, with appropriate conditions. This, however, will not work for hedged instruments that are illiquidly or rarely traded or for which no consistent pricing is known.

One could try to determine a consistent model according to some conceptualization, such as replication with controlled risk or market making according to certain strategies and models. However, the construction and validation of such consistent models is very hard, and under such circumstances, one most probably should design trading and risk management strategies that describe and manage the risk without requiring book-keeping and/or consistent modeling, such as hedging that tries to control the replication and risk management of the payoffs of the to-be-hedged instrument rather than targets a certain book-keeping value that does not adequately reflect the nature and risk of the instrument.

Particularly simple examples concern the hedging of a short call option $H_{O}=-1$ that we sold to some counterparty or the hedging of a long call option $H_{O}=1$ that we bought, use the Black-Scholes model with constant (or time-dependent parameters), with log-Euler time discretization (which is exact assuming the parameters are properly chosen), with the Black-Scholes Formula for a call as book-keeping model, and size of holding as action and decision variable.

In formulas, this would mean for the price of the hedging instrument

or

such that $\tilde{S}_{t_{i+1}}=f_{1}(\tilde{S}_{t_{i}},\Delta W_{i})$ for an appropriate function $f_{1}$ and a standard normal $\Delta W_{i}$ with covariance $\Delta t_{i}$. For the price of the call option, we use the Black-Scholes formula as book-keeping model.

The minimal state would consist of $s_{i}=(t_{i},\tilde{S}_{t_{i}},\tilde{C}_{t_{i}},H^{S}_{t_{i}})$ with the last element being the action (or being impacted by the action). One can also add other features and information to the state space that the agent could potentially use to make better decisions or that allow the agent to be learned more easily.

If one considers more complicated models based on conventional quantitative finance models, one would simulate the (discounted) underlying instruments with that model and one would replace the Black-Scholes formula with an appropriate pricing formula or pricer under that model (an appropriate future value computation under the book-keeping model). If these models have additional factors, these factors would be added to the state. If some of these factors are latent or hidden factors, one would need to add some mechanism how these latent factors can be estimated or taken into account by some process on observed quantities, add these observed quantities to the state, and learn agents that only depend on observed and observable quantities, not the latent factors that will in general be unknown (and unknowable) to the agent.

In general, trading in hedging instruments could impact the prices in those hedging instruments either temporarily or permanently. We assume here that the hedging instruments are traded liquidly and that the hedged instrument is such that it can be hedged without impacting the prices of the hedging instruments. To a certain extent, short-term price impact can be modeled by and absorbed into the transaction cost terms.

As discussed in abstract and introduction, here we focus on Black-Scholes type models to investigate the agents and algorithms in a setting where the model and the features are simple enough, and will consider other models in future work.

We begin by defining a Markov decision process (MDP). A discounted finite horizon MDP is defined by a tuple $(\mathcal{S},\mathcal{A},P,R,\gamma,T)$, where $\mathcal{S}$ is the set states, $\mathcal{A}$ is the set of all actions, $P:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{P}(\mathcal{S})$ is the transition probability density , $R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathcal{R}$ is the immediate reward (could also be stochastic), $\gamma\in(0,1]$ is the reward discount factor and $T$ is the time horizon ($T<\infty$). By taking any action $a\in\mathcal{A}$ at the state $s\in\mathcal{S}$, $P(.|s,a)$ defines the probability distribution of the next state and $R(.|s,a,s^{\prime})$, the distribution of the immediate reward.

A policy $\pi:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})$ maps a state $s\in\mathcal{S}$ to a probability distribution $\pi(.|s,a)$ over the set of actions $\mathcal{A}$. The interaction of an agent with the environment is formalized as follows, following a policy $\pi$, starting from the state $s_{0}$, at each time step $t$, the agent observes the state $s_{t}$ and takes an action $a_{t}$ according to the policy $a_{t}\sim\pi(s_{t})$. When the agent performs action $a_{t}$, the environment makes a stochastic state transition to a new state $s_{t+1}$ according to the probability distribution $P(s_{t+1}|s_{t},a_{t})$. The agent receives a reward $r(s_{t},u_{t})$ drawn from the distribution $R(r|s,a,s^{\prime})$. This reward is a random variable, because it depends on all of the stochastic transitions up to time $t$. The agent is allowed to make decisions, however, the transition density $P$ and the reward distribution $R$ are dictated by the environment. While the agent will observe transitions and rewards, the underlying model and details for $P$ and $R$ are not known to the agent in general.

Therefore, the evolution of the MDP following the policy $\pi$ (also called the trajectory) is given by,

The value of a state for a given policy $\pi$ is the cumulative reward across the trajectories starting from $s_{0}=s$ and following $\pi$ along the way. The value function corresponding to the policy $\pi$ is defined as,

The action-value function or the Q-function at state $s$ and action $a$ is the value of taking the action $a$ and following the policy $\pi$ onwards,

The value function can be expressed as the expectation of the Q-function across all possible actions,

The Q-function satisfies the Bellman expectation equation,

The optimal policy $\pi^{\ast}$ gives the maximum action-value function $Q^{\ast}_{t}(s,a)$ for any $s$ and $a$,

The optimal Q-function satisfies the Bellman optimality equation (see [SB18]):

The tabular Q-learning algorithm finds the optimal Q-function (and hence the optimal policy $\pi^{\ast}$) satisfying (16). To formalize the notion of optimal policy, for each Q-function, we define the greedy policy $\pi^{Q}$ as the policy that chooses the action maximizing the Q-function,

The optimal policy $\pi^{*}$ is defined as greedy policy for the optimal Q-function $Q^{*}$⁶⁶6If there is more than one maximizer, the greedy policy can be defined as a uniform distribution over such maximizers..

In order to find the optimal $Q^{*}$-function, define the Bellman optimality operator $T$ as follows,

It is observed that equation (16) describes a fixed point $TQ^{*}=Q^{*}$ of the Bellman operator in a suitable function space. In fact, it can be shown that the Bellman operator is a $\gamma$-contraction and hence admits a fixed point⁷⁷7The proof in the tabular case requires $\gamma\in[0,1)$.. The Q-value iteration algorithm approximates this fixed point by initializing a random $Q_{0}$ and constructing a sequence of action-value functions $\{Q_{k}\}$ by defining $Q_{k}=TQ_{k-1}$. It is shown that $\{Q_{k}\}$ converges to the optimal value function at a linear rate (see [Put14], [SB18], also [FWXY20]).

The Q-learning (tabular) algorithm approximates the updates in equation (19) by replacing the expectation with sample observations. At any step $t=1,2,\cdots,T-1$, after taking some action $a_{t}$, the algorithm observes the immediate reward $r_{t}$ and the state transition $s_{t+1}\sim P(.,s_{t},a_{t})$ and updates the $Q$-values for the state-action pair $(s_{t},a_{t})$ by,

The pseudo-code for the tabular $Q$-learning is given below,

and the target in the $\hat{Q}$-update takes the form of

Methods in the previous section fall under the dynamic programming approach to RL ([SB18]). In tabular and deep Q-learning methods, the goal is to approximate the optimal Q-function using the dynamic programming structure of the MDP. The optimal policy is then the greedy policy according to the optimal Q-function.

In the policy gradient approach, the policy is approximated directly. For an MDP defined by $(\mathcal{S},\mathcal{A},R,P,\gamma,T)$, let $\pi_{\theta}:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})$ denote a stochastic policy parameterized by $\theta$ (this also includes the deterministic policies as a special case). This parametrization can be a deep neural network for instance, defining a distribution over the set of actions for each state.

Let $V^{\pi_{\theta}}$ denote the value function corresponding to the long term discounted reward collected by the agent following policy $\pi_{\theta}$ starting from the state $s$. Therefore, the goal is to find the optimal policy through solving the following optimization problem.

The following classical result ([Wil92]) calculates the gradient of the objective above with respect to $\theta$. Consider a trajectory $\tau=\{s_{1},a_{1},\cdots,a_{T-1},s_{T}\}$ resulting from rolling out the policy $\pi_{\theta}$. Define the accumulated rewards along the path as $R(\tau)$, also the probability of the trajectory as

Policy gradient formula gives an unbiased estimate of $\nabla_{\theta}V^{\pi_{\theta}}$,

This leads us to the vanilla version of the REINFORCE Algorithm ([Wil92]) for policy gradients.

While REINFORCE algorithm works in theory, in practice, it suffers from high variance. The following property of the gradients enables us to lower the variance of the estimator without introducing bias.

Using the above property, one can add a "baseline" $b$ in the empirical policy gradient without introducing bias in $\nabla_{\theta}(V^{\pi_{\theta}})$:

More generally, the baseline can be state-dependent,

We end this section by observing that one can further exploit the temporal structure of an MDP to further reduce the variance of the policy gradient estimator (25). The idea is to split the path-wise sum of rewards into parts prior to step $t$ and after step $t$.

Removing terms in the reward summation that do not depend on the current action will help reduce variance. Therefore, denoting $G_{t}=\sum_{k=t}^{T-1}r(s_{t}^{i},a_{t}^{i})$, we have the following form for the policy gradient estimator,