Hedging and Pricing Structured Products Featuring Multiple Underlying Assets

Monte Carlo simulation is the most commonly used technique in derivatives pricing; however, it is often computationally intensive. To accelerate the calculation process, machine learning (ML), deep learning (DL), and neural networks (NN) are extensively employed to approximate the original pricing function, which is also referred to as model-free pricing (Kondratyev, 2018; Mcghee, 2018; Neufeld and Sester, 2022). In finance, regulators require that these approximations be explainable and predictable. Unfortunately, NN methods typically do not meet these criteria. Several alternative methods satisfy the explainability and predictability conditions while also being efficient, such as Fourier and Chebyshev series expansions, Image rendering methods based on regular and stochastic sampling, and Tensor decomposition methods (Antonov and Piterbarg, 2021).

In this paper, we apply the Chebyshev Tensor method as our model-free machine learning pricing approach. This technique uses Chebyshev polynomials and tensor decomposition to approximate pricing functions efficiently and accurately. By leveraging these mathematical tools, we achieve a stable, computationally efficient method that meets regulatory requirements for explainability and predictability. Previous studies have shown the effectiveness of Chebyshev Tensor methods in financial applications such as FRTB compliance and market risk assessment (Zeron et al., 2023; Maran et al., 2021; Ruiz and Zeron, 2021; Zeron and Ruiz, 2021), and our work aims to further validate its utility in derivative pricing.

In recent years, Reinforcement Learning (RL) methods have been extensively applied in finance for a variety of purposes, including optimal trade execution (Zhang et al., 2023), credit pricing (Khraishi and Okhrati, 2022), market making (Ganesh et al., 2019), learning exercise policies for American options (Li et al., 2009), and optimal hedging (Cao et al., 2020; Kolm and Ritter, 2019; Gao et al., 2023; Murray et al., 2022).

Hedging portfolios that include exotic options is a significant research challenge requiring sequential decision-making to periodically rebalance the portfolio. This has drawn interest in RL-based solutions, which have shown superior performance compared to traditional hedging strategies. For instance, (Kolm and Ritter, 2019) utilized Deep Q-learning and Proximal Policy Optimization (PPO) algorithms to develop a policy for option replication considering market frictions, capable of handling various strike prices. (Giurca and Borovkova, 2021) employed RL for delta hedging, taking into account transaction costs, option maturity, and hedging frequency, and applied transfer learning to adapt a policy trained on simulated data to real data scenarios. (Cao et al., 2020) and (Gao et al., 2023) applied the Deep Deterministic Policy Gradient (DDPG) algorithm to hedge under SABR and Heston volatility models, respectively.

Additionally, (Mikkilä and Kanniainen, 2023) trained an RL algorithm on real intraday options data spanning six years for the $S\&P500$ index. (Buehler et al., 2020) addressed hedging over-the-counter derivatives using RL under market frictions like trading costs and liquidity constraints. (Daluiso et al., 2023) explored RL for CVA hedging. The literature extensively covers RL-based sequential decision-making for European (Kolm and Ritter, 2019; Cao et al., 2020; Halperin, 2019), American (Li et al., 2009), and Barrier options (Chen et al., 2023), demonstrating that RL is a compelling alternative to traditional hedging strategies based on performance and $PnL$ distribution. However, there has been limited focus on hedging high-risk options such as autocallable notes. Recently, (Cui et al., 2023) introduced a method for pricing Autocallable notes and employing a Delta neutral strategy for hedging, highlighting the complexity due to multiple barriers and the high cost of Delta hedging near these barriers.

Our approach involves training an RL policy to hedge a 4-year maturity autocallable note portfolio with three underlying assets. We evaluate the RL agent’s performance under varying transaction costs and with different hedging instruments, demonstrating that the RL agent achieves a better $PnL$ distribution compared to traditional hedging strategies.

The RL Pipeline uses a MDP (Markov Decision Process) to frame the problem of our RL agent interacting with the environment to maximize a potential reward. An MDP is defined as a tuple of elements ($S,\mathcal{A},f,R,\gamma$), where $S$ is the state space, $\mathcal{A}$ is the action space, $f(s_{t},s_{t+1})$ is the state transition function, $R(s,a)$ is the reward function and $\gamma$ is the discount factor. We formulate a finite horizon discounted sum reward problem where the horizon length is the maturity of the option ($4$ year in our experiments).

where $V_{i}$ is the value of the option (hedging instrument) at time $i\Delta t$, $H_{i}$ is the position taken in the hedging instrument, $\kappa$ is the transaction cost, $P_{i}^{-}$ is portfolio’s market value before time $i\Delta t$, $P_{i}^{+}$ is portfolio’s market value after time $i\Delta t$, $-\kappa|V_{i}H_{i}|$ is the transaction cost paid at time $i$, and $(P_{i}^{-}-P_{i-1}^{+})$ is the change in portfolio value from time $(i-1)\Delta t$ to $i\Delta t$.

Using the above MDP, an RL environment was created which gives the next state of the environment based on the actions selected by the RL agent. At any given time, the RL agent interacts with the environment simulator by providing an action and the environment returns the next state and the reward.

We use Distributional Reinforcement Learning (DRL) in our proposal. DRL is an advanced paradigm which focuses on modeling the entire distribution of returns, rather than solely estimating the expected value. In DRL, the return $G_{t}$ (total reward received by the agent in an episode) is modelled as a distribution $Z_{t}^{\pi}$ for a fixed policy $\pi$. The return distribution provides more information and is more robust as compared to only the expectation. Classical RL tries to minimize the error between two expectations, expressed as $\operatorname{\mathbb{E}}_{s,a,s^{{}^{\prime}}}[\{r(s,a)+\gamma max_{a^{{}^{\prime}}}Q(s^{{}^{\prime}},a^{{}^{\prime}})-Q(s,a)\}^{2}]$, where $Q(s,a)$ is the output of the policy and $r(s,a)+\gamma max_{a^{{}^{\prime}}}Q(s^{{}^{\prime}},a^{{}^{\prime}})$ is the target function. In contrast, in DRL, the objective is to minimize a distributional error, which is a distance between the two full distributions (the target PMF and the predicted PMF) (Bellemare et al., 2017; Dabney et al., 2017).

The optimal action is selected from the $Q$ value function:

In DRL, the distribution of returns is represented as a PMF (Probability Mass Function) and generally, the probabilities are assigned to discrete values that denote the possible outcome of the RL agent. Let’s say we have a neural network that predicts this PMF by taking a state $s$ and returning a distribution $Z(s,a)$ for each action. Categorical distributions are commonly employed to model these distributions in some DRL algorithms like C-51 (Bellemare et al., 2017) where the action distribution is modelled using a finite number of possible outcomes. In C-51, probabilities are estimated to these fixed locations. We employ Quantile Regression (QR) to learn the distribution of returns and unlike C-51, QR estimates the quantile locations where each quantile corresponds to a fixed uniform probability. That means, QR provides the flexibility to stochastically adjust the quantile locations in place of fixed locations in C-51. QR is a popular approach in DRL which is combined with several distributional RL algorithms such as in QR-DQN (Dabney et al., 2017). QR-DQN uses quantile regression with traditional DQN to learn a distribution of outcomes.

We are hedging a trader’s portfolio which contains risky options - the trader hedges the portfolio by adding other instruments to a hedging portfolio at every hedging instant. In other words, the trader rebalances the portfolio at every $\Delta t$ time interval for hedging. The hedging action involves selecting the percentage of $Gamma$ to hedge at any rebalancing instant. For example, in the experiments for autocallable note hedging, the trader’s portfolio contains one short Autocallable note with $4$ years of maturity and featuring three underlying indices and $2\%$ transaction cost. The trader also keeps a hedging portfolio and at every hedging instant ($\Delta t=1$ month), the trader adds an at-the-money American call and put option to the hedging portfolio for hedging. A hedging strategy decides how much hedging needs to be performed based on the different portfolio Greeks such as $Delta$, $Gamma$, etc. $Delta$ tells how much the option’s price will change for a one-point change in the underlying asset’s price, while $Gamma$ tells how much the $Delta$ will change for a one-point change in the underlying asset’s price. Trader’s use these Greeks to manage risk and make informed decisions. For example, a Delta neutral strategies hedges the entire $Delta$ of the portfolio by creating a position with a $Delta$ value of zero, or very close to zero. We propose a RL based hedging strategy which learns to decide the percentage of the $Gamma$ that needs to be hedged. We compare the hedging performance of our RL agent with traditional hedging strategies, Delta neutral and Delta-Gamma neutral.

Our RL agent uses the D4PG (Distributed Distributional DDPG) algorithm with quantile regression to learn an optimal policy. The RL agent selects an optimal action using the learned policy from interval $[0,1]$. The RL agent is trained for $40,000$ episodes (where one episode is one stock path till note maturity or auto-call). To compare the performance for all methods, we generate an additional $5000$ episodes on which KPI metrics are reported. The prices for the underlying asset are generated using Geometric Brownian motion (GBM) model. We evaluate the hedging performance using $95\%$ percentile of the value-at-risk ($VaR$) and conditional value-at-risk ($CVaR$) distribution.

Figure 2(a) illustrates the price grid for the first three months of an autocallable note with a four-year maturity, using multiple underlying assets. This figure highlights how the prices evolve over time for the given period. In figure 2(c), we present a comparison of the price estimations obtained using machine learning (ML) approximation (Chebyshev Tensor) and the traditional Monte Carlo (MC) simulation method. The maximum absolute error that we encountered in our experiments is $0.43\%$ between both the pricers. This comparison shows that the ML method can approximate the original pricing function (MC) very well. There is no significant bias in their errors. Figure 2(b) depicts the error percentage in pricing between the ML approximation and the Monte Carlo method, specifically for an autocallable note with three underlying assets. This figure quantifies the pricing discrepancies and provides insights into the performance of the ML approximation relative to the Monte Carlo simulation.

Table 3 provides a detailed comparison of computation times for pricing a four-year autocallable note with three underlying assets. This comparison is essential for understanding how traditional methods like Monte Carlo simulations, which can be computationally intensive, stack up against more modern approaches such as machine learning-based models that promise faster computation. By evaluating these trade-offs on accuracy and computational efficiency, practitioners can make informed decisions on selecting the appropriate pricing method based on their needs for accuracy, computational resources, and operational efficiency.

In this experiment, we hedge a trader’s portfolio containing one short autocallable note with a maturity of 4 years and multiple underlying assets. We use American call and put options as hedging instrument which are added at every hedging instant based on the action selected by the RL agent. For this experiment, the state at any time instant $t$ contains the stock price ($x_{t}$, portfolio gamma, and the days to the next call date). The trader rebalances the portfolio every month by adding one American option to the hedging portfolio, both call and put option are added to eliminate the need to rebalance the $Delta$ exposure in the portfolio. We choose $\Delta t=1$ month as the rebalancing interval because of the longer maturity time of 4 years for the autocallable note. The hedging agent strategy decides how much hedging needs to be performed in terms of the Gamma value of the portfolio. We use $VaR99\%$ as the objective function as the reward to the RL agent which then selects an action for the amount of hedging that needs to be performed. The RL agent generated the best payoff as compared to the traditional hedging strategies (Delta-neutral and Delta-Gamma neutral). The figure 3 shows the $PnL$ distribution of the three hedging strategies and RL agent is found to be performing best than the other strategies. Table 2 shows the $VaR-5\%$, $CVaR-5\%$, $VaR-95\%$, $CVaR-95\%$ of the $PnL$ distribution to compare the values at different percentiles of the $PnL$ distribution. Please note that the values in the table are for the actual $PnL$ distribution. The $PnL$ for $VaR-5\%$ of the RL agent is $33.95$, significantly outperforming both the Delta-neutral strategy, which has a $VaR$ of $-0.04$, and the Delta-Gamma-neutral strategy, which has a $VaR$ of $13.05$. Additionally, the $VaR-95\%$ of the RL agent is $-6.35$, which is also better compared to the Delta-neutral and Delta-Gamma-neutral strategies. Overall, the RL agent shifts the $PnL$ distribution positively. From the table and the figure, we can claim that the RL agent provides substantial improvement in the $PnL$ by hedging of the portfolio.