Reinforcement Learning for Jump-Diffusions, with Financial Applications

Recently there is an upsurge of interest in continuous-time reinforcement learning (RL) with continuous state spaces and possibly continuous action spaces. Continuous RL problems are important because: 1) many if not most practical problems are naturally continuous in time (and in space), such as autonomous driving, robot navigation, video game play and high frequency trading; 2) while one can discretize time upfront and turn a continuous-time problem into a discrete-time MDP, it has been known, indeed shown experimentally e.g., Munos (2006), Tallec et al. (2019) and Park et al. (2021), that this approach is very sensitive to time discretization and performs poorly with small time steps; 3) there are more analytical tools available for the continuous setting that enable a rigorous and thorough analysis leading to interpretable (instead of black-box) and general (instead of ad hoc) RL algorithms.

Compared with the vast literature of RL for MDPs, continuous-time RL research is still in its infancy with the latest study focusing on establishing a rigorous mathematical theory and devising resulting RL algorithms. This strand of research starts with Wang et al. (2020) that introduces a mathematical formulation to capture the essence of RL – the exploration–exploitation tradeoff – in the continuous setting, followed by a “trilogy” (Jia and Zhou 2022a, b, Jia and Zhou 2023) that develops intertwining theories on policy evaluation, policy gradient and $q$-learning respectively. The common underpinning of the entire theory is the martingale property of certain stochastic processes, the enforcement of which naturally leads to various temporal difference algorithms to train and learn $q$-functions, value functions and optimal (stochastic) policies. The research is characterized by carrying out all the analysis in the continuous setting, and discretizing time only at the final, implementation stage for approximating the integrated rewards and the temporal difference. The theory has been adapted and extended in different directions; see e.g. Reisinger and Zhang (2021), Guo et al. (2022), Dai et al. (2023), as well as employed for applications; see e.g. Wang and Zhou (2020), Huang et al. (2022), Gao et al. (2022), Wang et al. (2023), and Wu and Li (2024).

The study so far has been predominantly on pure diffusion processes, namely the state processes are governed by controlled stochastic differential equations (SDEs) with a drift part and a diffusion one. While it is reasonable to model the underlying data generating processes as diffusions within a short period of time, sudden and drastic changes can and do happen over time. An example is a stock price process: while it is approximately a diffusion over a sufficiently short period, it may respond dramatically to a surprisingly good or bad earning report. Other examples include neuron dynamics (Giraudo and Sacerdote 1997), stochastic resonance (Gammaitoni et al. 1998) and climate data (Goswami et al. 2018). It is therefore natural and necessary to extend the continuous RL theory and algorithms to the case when jumps are present. This is particularly important for decision makings in financial markets, where it has been well recognized that using jumps to capture large sudden movements provides a more realistic way to model market dynamics; see the discussions in Chapter 1 of Cont and Tankov (2004). The financial modeling literature with jumps dates back to the seminal work of Merton (1976), who extends the classical Black–Scholes model by introducing a compound Poisson process with normally distributed jumps in the log returns. Since then alternative jump size distributions have been proposed in e.g. Kou (2002) and Cai and Kou (2011). Empirical success of jump-diffusion models have been documented for many asset classes; see Bates (1991), Andersen et al. (2002), and Aït-Sahalia et al. (2012) for stocks and stock indices, Bates (1996) for exchange rates, Das (2002) for interest rates, and Li and Linetsky (2014) for commodities, among many others.

This paper makes two major contributions. The first is mathematical in terms of setting up the suitable exploratory formulation and proving the well-posedness of the resulting exploratory SDE, which form the foundation of the RL theory for jump-diffusions. Wang et al. (2020) apply the classical stochastic relaxed control to model the exploration or randomization prevalent in RL, and derive an exploratory state equation that dictates the dynamics of the “average” of infinitely many state processes generated by repeatedly sampling from the same exploratory, stochastic policy. The drift and variance coefficients of the exploratory SDE are the means of those coefficients against the given stochastic policy (which is a probability distribution) respectively. The derivation therein is based on a law of large number argument to the first two moments of the diffusion process. That argument fails for jump-diffusions which are not uniquely determined by the first two moments. We overcome this difficulty by analyzing instead the infinitesimal generator of the sample state process, from which we identify the dynamics of the exploratory state process. Inspired by Kushner (2000) who studies relaxed control for jump-diffusions, we formulate the exploratory SDE by extending the original Poisson random measures for jumps to capture the effect of random exploration. It should be noted that, like almost all the earlier works on relaxed control, Kushner (2000) is motivated by answering the theoretical question of whether an optimal control exists, as randomization convexifies the universe of control strategies. In comparison, our formulation is guided by the practical motivation of exploration for learning. There is also another subtle but important difference. We consider stochastic feedback policies while Kushner (2000) does not. This in turn creates technical issues in studying the well-posedness of the exploratory SDE in our framework.

The second main contribution is several implications regarding the impact of jumps on RL algorithm design. Thanks to the established exploratory formulation, we can define the Hamiltonian that, compared with the pure diffusion counterpart, has to include an additional term corresponding to the jumps. The resulting HJB equation – called the exploratory HJB – is now a partial integro-differential equation (PIDE) instead of a PDE due to that additional term. However, when expressed in terms of the Hamiltonian, the exploratory HJB equation has exactly the same form as that in the diffusion case. This leads to several completely identical statements of important results, including the optimality of the Gibbs exploration, definition of a $q$-function, and martingale characterizations of value functions and $q$-functions. Here by “identical” we mean in terms of the Hamiltonian; in other words, these statements differ between diffusions and jump-diffusions entirely because the Hamiltonian is defined differently (which also causes some differences in the proofs of the results concerned). Most important of all, in the resulting RL algorithms, the Hamiltonian (or equivalently the $q$-function) can be computed using temporal difference of the value function by virtue of the Itô lemma; as a result the algorithms are completely identical no matter whether or not there are jumps. This has a significant practical implication: we can just use the same RL algorithms without the need of checking in advance whether the underlying data come from a pure diffusion or a jump-diffusion. It is significant for the following reason. In practice, data are always observed or sampled at discrete times, no matter how frequent they arrive. Thus we encounter successive discontinuities along the sample trajectory even when the data actually come from a diffusion process. There are some criteria that can be used to check whether the underlying process is a diffusion or a jump-diffusion, e.g. Aït-Sahalia et al. (2012), Wang and Zheng (2022). But these methods typically require data with very high frequency to be effective, which may not always be available. In addition, noises must be properly handled for them to work.

Even though we can apply the same RL algorithms irrespective of the presence of jumps, the parametrization of the policy and value function may still depend on it, if we try to exploit certain special structure of the problem instead of using general neural networks for parameterization. Indeed, we give an example in which the optimal exploratory policy is Gaussian when there are no jumps, whereas an optimal policy either does not exist or becomes non-Gaussian when there are jumps. However, in the mean–variance portfolio selection we present as a concrete application, the optimal Gibbs exploration measure again reduces to Gaussian and the value function is quadratic as in Wang and Zhou (2020), both owing to the inherent linear–quadratic (LQ) structure of the problem. Hence in this particular case jumps do not even affect the parametrization of the policy and value function/$q$-function for learning.

We also consider mean–variance hedging of options as another application. This is a non-LQ problem and hence more difficult to solve than mean–variance portfolio selection. The mean–variance hedging problem has been studied in various early works, such as Schweizer (1996, 2001) and Lim (2005). Here, we introduce the entropy-regularized mean–variance hedging objective for an asset following a jump-diffusion and derive analytical representations for the optimal stochastic policy, which is again Gaussian, as well as the optimal value function. We use these representations to devise an actor–critic algorithm to learn the optimal hedging policy from data.

We compare our work with four recent related papers. (1) Bender and Thuan (2023) consider the continuous-time mean–variance portfolio selection problem with exploration under a jump-diffusion setting. Our paper differs from theirs in several aspects. First, they consider a specific application problem while we study RL for general controlled jump-diffusions. Second, they obtain the SDE for the exploratory state by taking limit of the discrete-time exploratory dynamics, whereas our approach first derives the form of the infinitesimal generator of the sample state process and then infers the exploratory SDE from it. It is unclear how their approach works when dealing with general control problems. Finally, they do not consider how to develop RL algorithms based on their solution of the exploratory mean–variance portfolio selection, which we do in this paper. (2) Guo et al. (2023) consider continuous-time RL for linear–convex models with jumps. The scope and motivation are different from ours: They focus on the Lipschitz stability of feedback controls for this special class of control problems where the diffusion and jump terms are not controlled, and propose a least-square model-based algorithm and obtain sublinear regret guarantees in the episodic setting. By contrast, we consider RL for general jump-diffusions and develop model-free algorithms without considering regret bounds. (3) Denkert et al. (2024) aim to unify certain types of stochastic control problems by considering the so-called randomized control formulation which leads to the same optimal value functions as those of the original problems. They develop a policy gradient representation and actor-critic algorithms for RL. The randomized control formulation is fundamentally different from the framework we are considering: therein the control is applied at a set of random time points generated by a random point process instead of at every time point as in our framework. (4) Bo et al. (2024) develop $q$-learning for jump-diffusions by using Tsallis’ entropy for regularization instead of Shannon’s entropy considered in our paper and in Jia and Zhou (2022b), Jia and Zhou (2023).¹¹1The paper Bo et al. (2024) came to our attention after a previous version of our paper was completed and posted. While this entropy presents an interesting alternative for developing RL algorithms, it may make the exploratory control problem less tractable to solve and lead to policy distributions that are inefficient to sample for exploration in certain applications.

The remainder of the paper is organized as follows. In Section 2, we discuss the problem formulation. In Section 3, we present the theory of $q$-learning for jump-diffusions, followed by the discussion of $q$-learning algorithms in Section 4. In Section 5, we apply the general theory and algorithms to a mean–variance portfolio selection problem, and discuss the impact of jumps. Section 6 presents the application to a mean–variance option hedging problem. Finally, Section 7 concludes. All the proofs are given in an appendix.

For readers’ convenience, we first recall some basic concepts for one-dimensional (1D) Lévy processes, which can be found in standard references such as Sato (1999) and Applebaum (2009). A 1D process $L=\{L_{t}:t\geq 0\}$ is a Lévy process if it is continuous in probability, has stationary and independent increments, and $L_{0}=0$ almost surely. Denote the jump of $L$ at time $t$ by $\Delta L_{t}=L_{t}-L_{t-}$, and let $\boldsymbol{\text{B}}_{0}$ be the collection of Borel sets of $\mathbb{R}$ whose closure does not contain $0$. The Poisson random measure (or jump measure) of $L$ is defined as

which gives the number of jumps up to time $t$ with jump size in a Borel set $B$ away from $0$. The Lévy measure $\nu$ of $L$ is defined by $\nu(B)=\mathbb{E}[N(1,B)]$ for $B\in\boldsymbol{\text{B}}_{0}$, which shows the expected number of jumps in $B$ in unit time, and $\nu(B)$ is finite. For any $B\in\boldsymbol{\text{B}}_{0}$, $\{N(t,B):t\geq 0\}$ is a Poisson process with intensity given by $\nu(B)$. The differential forms of these two measures are written as $N(dt,dz)$ and $\nu(dz)$, respectively. If $\nu$ is absolutely continuous, we write $\nu(dz)=\nu(z)dz$ by using the same letter for the measure and its density function. The Lévy measure $\nu$ must satisfy the integrability condition

However, it is not necessarily a finite measure on $\mathbb{R}/\{0\}$ but always a $\sigma$-finite measure. The Lévy process $L$ is said to have jumps of finite (infinite activity) if $\int_{\mathbb{R}}\nu(dz)<\infty$ ($=\infty$). The number of jumps on any finite time interval is finite in the former case but infinite in the latter. For any Borel set $B$ with its closure including $0$, $\nu(B)$ is finite in the finite activity case but infinite otherwise. Finally, the compensated Poisson random measure is defined as $\widetilde{N}(dt,dz)=N(dt,dz)-\nu(dz)dt$. For any $B\in\boldsymbol{\text{B}}_{0}$, the process $\{\widetilde{N}(t,B):t\geq 0\}$ is a martingale.

Throughout the paper, we use the following notations. By convention, all vectors are column vectors unless otherwise specified. We use $\mathbb{R}^{k}$ and $\mathbb{R}^{k\times\ell}$ to denote the space of $k$-dimensional vectors and $k\times\ell$ matrices, respectively. For matrix $A$, we use $A^{\top}$ for its transpose, $|A|$ for its Euclidean/Frobenius norm, and write $A^{2}\coloneqq AA^{\top}$. Given two matrices $A$ and $B$ of the same size, we denote by $A\circ B$ the inner product between $A$ and $B$, which is given by $\text{tr}(AB^{\top})$. For a positive semidefinite matrix $A$, we write $\sqrt{A}=UD^{1/2}V^{\top}$, where $A=UDV^{\top}$ is its singular value decomposition with $U,V$ as two orthogonal matrices and $D$ as a diagonal matrix, and $D^{1/2}$ is the diagonal matrix whose entries are the square root of those of $D$. We use $f=f(\cdot)$ to denote the function $f$, and $f(x)$ to denote the function value of $f$ at $x$. We use both $f_{x},f_{xx}$ and $\frac{\partial f}{\partial x},\frac{\partial^{2}f}{\partial x^{2}}$ for the firs and second (partial) derivatives of a function $f$ with respect to $x$. We write the minimum of two values $a$ and $b$ as $a\wedge b$. The notation $\mathcal{U}(B)$ denotes the uniform distribution over set $B$ while $\mathcal{N}(\mu,\Sigma)$ refers to the Gaussian distribution with mean vector $\mu$ and covariance matrix $\Sigma$.