Continuous-time optimal investment with portfolio constraints: a reinforcement learning approach

The classical asset allocation problem can be written as

where $U$ is a concave utility function. The optimization problem (2.4) is solely an exploitation problem, which is a typical set up of classical stochastic control optimization problems. In a financial market with complete knowledge of the model parameters, ones are readily to employ the classical model-based stochastic control theory (see e.g. Fleming and Soner (2006); Pham (2009)), duality approach (see e.g. Chen and Vellekoop (2017); Kamma and Pelsser (2022)) or to use the so-called martingale approach (see e.g. Karatzas and Shreve (1998)) to find the solution of Problem (2.4). When implementing the optimal strategy, one needs to estimate the market parameters from historical time series of asset prices. However, in practice, estimating these parameters is notoriously difficult with an acceptable accuracy, especially the mean return $\mu$, also known as the mean-blur problem, see e.g. Luenberger (1998).

In the RL setting where the underlying model is not known, dynamic learning is needed and the agent employs exploration to interact with and learn from the unknown environment through trial and error. The key idea is to model exploration via a distribution over the space of controls $\mathcal{C}\subseteq\mathbb{R}$ from which the trials are sampled. Here we assume that the action space $\mathcal{C}$ is continuous and randomization is restricted to those distributions that have density functions. In particular, at time $t\in[0,T]$, with the corresponding current wealth $X_{t}$, the agent considers a classical control $(\pi_{t})$ sampled from a policy distribution $\lambda_{t}(\pi):=\lambda(\pi|t,X_{t})$. Mathematically, such a policy $\lambda$ can be generated e.g. by introducing an additional random variable $Z$ that is uniformly distributed in $[0,1]$ and is independent of the filtration of the Brownian motion $W$, see e.g. Jia and Zhou (2022a). In this sense, the probability space is now expanded to ($\Omega,\mathcal{F},\{\mathcal{F}_{t}\}_{t\in[0,T]},\mathbb{P}$), where $\mathcal{F}:=\mathcal{F}^{W}\vee\sigma(Z)$ and $\mathbb{P}$ is now an extended probability measure on $\mathcal{F}$ whose projection on $\mathcal{F}^{W}$ is $\mathbb{P}^{W}$. This sampling is executed for $N$ rounds over the same time horizon. Intuitively, the utility of such a (feedback) policy becomes accurate enough when N is large by using the law of large numbers. This procedure, known as policy evaluation, is considered as a fundamental element of most RL algorithms in practice. For our continuous-time setting, we follow the exploratory setting suggested in Wang et al. (2020) and refer to e.g., Wang et al. (2020); Wang and Zhou (2020); Jia and Zhou (2022a, b) for motivations and additional details. In particular, we consider the exploratory version of the wealth dynamics (2.3) given by

where the exploration drift and exploration volatility are defined by

where

From (2.6) we observe that

As a result, the agent may want to maximize over all admissible policies $\lambda$ the exploratory expected investment utility $\mathbb{E}[U(X_{T}^{\lambda})].$ Intuitively, the agent must account for exploration effect in her objective. As suggested in Wang et al. (2020), for a given exploration distribution $\lambda$, the following Shannon’s differential entropy penalty at time $t\in[0,T]$

can be used to measure the exploration impact. Note that when the model is fully known, there would be no requirement for exploration and learning. In such a scenario, control distributions would collapse to Dirac measures, placing us in the domain of classical stochastic control Fleming and Soner (2006); Pham (2009).

Motivated by the above discussion, below we consider the following Shannon’s entropy-regularized exploratory optimization problem

where $\mathcal{H}$ is the set of admissible (feedback) exploration distributions (or more precisely density). Mathematically, (2.8) is a relaxed stochastic control problem which has been widely studied in the control literature, see e.g. Fleming (1976); Fleming and Nisio (1984); Nicole el et al. (1987). Note that relaxed controls in general have a natural interpretation that at each time $t$, the agent chooses not a single action (strategy) but rather a probability measure over the action space $\mathcal{C}$ from which a specific action is randomly sampled. In more general settings of relaxed control e.g. Fleming (1976); Fleming and Nisio (1984); Nicole el et al. (1987), the admissible set may contain open-loop distributions that are measure-valued stochastic processes. This is also in line with the notion of a mixed strategy in game theory, in which players choose probability measures over the action set, rather than single actions (pure strategies).

We note that the minus sign in front of the entropy term accounts for the fact that exploration is inherently costly in term of resources. Also, we can see that the optimization problem (2.8) incorporates both exploration (due to the entropy factor) and exploitation. The rate of exploration is determined by the exogenous temporal parameter $m>0$ in the sense that a larger value of $m$ indicates that more explorations are needed and vice versus. Hence, the agent can personalize her exploration rate by selecting an appropriate exogenous temporal parameter $m$.

Next, we specify the set of admissible controls as follows.

For each $m\in\mathbb{R}$, we denote by $v(t,x;m)$ the optimal value function of Problem (2.8), i.e.

By standard arguments of Bellman’s principle of optimality, the optimal value function $v$ satisfies the following HJB equation

with terminal condition $v(T,x;m)=U(x)$. We observe that the formula in the bracket of (2.9) can be expressed as

The optimal distribution $\lambda^{*}$ can be obtained by Donsker and Varadhan’s variational formula (see e.g. Donsker and Varadhan (2006))

which is a feedback form and can be seen as a Boltzmann distribution (see e.g., Tang et al. (2022) for a similar result). Here the notation $\lambda^{*}(\pi|t,x;m)$ is introduced to indicate that $\lambda^{*}$ depends on the exploration parameter $m$. Note that we can write $\lambda^{*}(\pi|t,x;m)$ as

which is Gaussian with mean $\alpha$ and variance $\beta^{2}$ (assuming that $v_{xx}<0$) defined by

Substituting (3.2) back to the HJB (2.9) we obtain the following non-linear PDE

with terminal condition $\quad v(T,x;m)=U(x).$

Below we show that the optimal solution to (2.12) can be given in explicit form for the case of logarithmic utility. Logarithmic utility is widely recognized as the preference function for rational investors, as highlighted in Rubinstein (1977). The study in Pulley (1983) supports this assertion by showcasing that the logarithmic utility serves as an excellent approximation in the context of the Markowitz mean-variance setting and remarkably, the optimal portfolio under logarithmic utility is almost identical to that derived from the mean-variance strategy. It is crucial to emphasize that while our optimal strategy with logarithmic utility maintains time consistency, the mean-variance strategy is only precommitted, meaning it is only optimal at time 0. This stands out as one of the most critical drawbacks within the mean-variance setting. Logarithmic utility’s versatility extends to diverse applications, including long-term investor preferences (Gerrard et al., 2023), optimal hedging theory and within semimartingale market models (Merton, 1975), categorizing it under the hyperbolic absolute risk aversion (HARA) class. This class has been explored in portfolio optimization under finite-horizon economies (Cuoco, 1997), infinite-horizon consumption-portfolio problems (El Karoui and Jeanblanc-Picqué, 1998), and scenarios involving allowed terminal debt (Chen and Vellekoop, 2017).

We summarize the results for the unconstrained case i.e. $\mathcal{C}=\mathbb{R}$ in the following theorem .

From Theorem 3.1 we can see that the best control distribution to balance exploration and exploitation is Gaussian. This demonstrates the popularity of Gaussian distribution in RL studies. The mean of the optimal distribution control is given by the Merton strategy

whereas the variance of the optimal Gaussian policy is controlled by the degree of exploration $m$. We also observe that at any $t\in[0,T]$, the exploration variance decreases as $\sigma$ increases, which means that exploration is less necessary in more random market environment.

The following shows solvability equivalence between the classical and exploratory EU problem, meaning that the solution to one directly provides the solution to the other, without requiring a separate resolution.

The exploration cost for a general RL problem is defined by the difference between the expected utility following the corresponding optimal control under the classical objective and the exploratory objective, net of the value of the entropy. Remark that the exploration cost is well defined only if both the classical and the exploratory problems are solvable. Let $v(t,x;0)$ be the value function of the classical EU problem. Below, we often write for shorthand $\lambda_{s}(\pi)=\lambda(\pi|s,X^{\lambda}_{s})$ when there is no confusion. The exploration cost at time $t=0$ is defined by

The first term is the classical value function at time $t=0$. The second term is the expected utility (before being regularized) of the exploration problem at time $t=0$.

Observe that the exploration cost converges to zero as $m\to 0$. Since the exploration weight as been taken as an exogenous parameter $m$ which reflects the level of exploration desired by the learning agent, it is intuitive to expect that the smaller $m$ is, the more emphasis is placed on exploitation. Moreover, when $m$ is sufficiently close to zero, the exploratory formulation is getting close to the problem without exploration. The following theorem confirms a desirable result that the entropy-regularized EU problem converges to its classical EU counterpart when the exploration weight $m$ goes to zero.

The following policy improvement theorem is crucial for interpretable RL algorithms as it ensures that the iterated value functions is non-decreasing and converges to the optimal value function. Below, for some given policy $\lambda$ (not necessarily Gaussian), we denote the corresponding value function by

Note that Theorem 3.4 holds true for general utility function $U$. One important implication of Theorem 3.4 is that for any given (not necessarily Gaussian) policy $\lambda$, there are always policies in the Gaussian family that improve the value function of $\lambda$ (i.e. providing higher expected utility values). Therefore, it is intuitive to focus on the Gaussian policies when choosing an initial exploration distribution. Note that the optimal Gaussian policy given by (3.2) also suggests that a candidate of the initial feedback policy may take the form ${\lambda}^{0}(\pi|t,x,m)=\mathcal{N}\big{(}\pi\big{|}a,b^{2}\big{)}$ for some real number $a,b$. As shown below, such a choice leads to a fast convergence of both value functions and policies in a finite number of iterations.

As shown in Appendix A.3, starting with a Gaussian policy the optimal solution of the exploratory EU with logarithmic utility is attained rapidly (after one update).

Following Jia and Zhou (2022a, b) we consider in this section the policy evaluation that takes the martingale property into account. Recall with an abusive notation the value function

where $\lambda_{t}$ is a given learning (feedback) policy. By Feymann-Kac’s theorem, it can be seen that $J(t,x;{\lambda})$ solves (e.g. in viscosity sense) the following average PDE

where $\mathcal{L}$ is the infinitesimal operator associated to $X_{t}^{\lambda}$, i.e.

As in Jia and Zhou (2022a, b), the value function of a policy $\lambda$ can be characterized by the following martingale property.

It follows from Itô’s Lemma and Theorem 3.6 that $\mathbb{E}[\int_{0}^{T}h_{s}dY_{s}]=0$ for all adapted square-integrable processes $h$ satisfying $\mathbb{E}[\int_{0}^{T}h^{2}_{s}d\langle Y\rangle_{s}]<\infty$. Equivalently,

Such a process $h$ is called test function. Let $J^{\theta}(t,X^{\lambda}_{t};{\lambda})$ be a parameterized family that is used to approximate $J$, where $\theta\in\Theta\subset\mathbb{R}^{n},n\geq 1$. Here $J^{\theta}$ satisfies Assumption 3.1 below and our goal is to find the best parameter $\theta^{*}$ such that the martingale property in Theorem 3.6 holds. In this sense, the process

should be a martingale in $[0,T]$ with terminal value

As discussed in (Jia and Zhou, 2022a, page 18), the martingale condition of $Y^{\theta^{*}}$ is further equivalent to a requirement that the process at any given time $t<T$ is the expectation of the terminal value conditional on all the information available at that time. A fundamental property of the conditional expectation then yields that $Y^{\theta_{t}^{*}}$ minimizes the $L^{2}$-error between $Y^{\theta}_{T}$ and any ${\mathcal{F}}_{t}$-measurable random variables. Therefore, our objective is to minimize the martingale cost function defined by $\mathbb{E}[\int_{0}^{T}(Y_{T}^{\theta}-Y_{t}^{\theta})^{2}dt]$. In other words, for the policy evaluation (PE) procedure, we look at the following (offline) minimization problem

and check the martingale orthogonal property

The orthogonality condition (3.23) can be assured by solving the following minimization problem (which is a quadratic form of (3.23)) ²²2Our procedure is inspired from (Jia and Zhou, 2022a, page 22). In particular, we are optimizing with respect to the parameter $\theta$ which is a vector in general. In fact, we have to make sure that the parameterization satisfies the martingale orthogonality condition (3.23)) by choosing a suitable test function $h$. For numerical approximation methods involving a parametric family $\{J^{\theta},\theta\in\mathbb{R}^{n}\}$, in principle, we need at least $n$ equations as our martingale orthogonality conditions in order to fully determine $\theta$. Consequently, $h$ may be vector-valued, making (3.23) a vector-valued equation or, equivalently, a system of equations.

where $\Sigma$ is a positively definite matrix with suitable size (e.g. $\Sigma=I$ or $\Sigma=\mathbb{E}[\int_{0}^{T}h_{t}^{\intercal}h_{t}dt]$) for any test process $h$. A common choice of test process is to take $h_{t}=\frac{\partial}{\partial\theta}J^{\theta}(t,X^{\lambda}_{t};\lambda)$. The following assumption³³3There is a rich theory on solution existence and regularity of general parabolic PDEs, see e.g. Friedman (2008). However, the average PDE (3.18) appears to be a new type of PDEs for which further studies on well-posedness (existence and uniqueness) and regularity of solution are needed. See Tang et al. (2022) for related discussions on a similar exploratory elliptic PDEs. is needed in such an approximation procedure, see e.g. Jia and Zhou (2022b).

Suppose that a PE step can be proceeded to obtain an estimate of the value function for an given admissible policy $\lambda$. If the policy $\lambda^{\phi}$ can be parameterized by $\phi\in\Phi\subset\mathbb{R}^{d},{d\geq 1}$, it is possible to learn the corresponding value function by following a policy gradient (PG) step. Recall first that the value function $J(t,x,\lambda^{\phi})$ satisfies the average PDE

By differentiating (3.25) w.r.t. $\phi$ we obtain the following average PDE for the policy gradient $g(t,x;\phi):=\frac{\partial}{\partial\phi}J(t,x,\lambda^{\phi})$

with terminal condition $g(T,x;\phi)=\frac{\partial}{\partial\phi}J(T,x;\lambda^{\phi})=0$, where

By Feynman-Kac’s formula, $g(t,x;\phi)$ admits the following representation

Since $\psi$ cannot be observed nor computed without knowing the environment, it is important to replace $\psi(s,X^{\lambda^{\phi}}_{s},\pi;\phi)ds$ by a suitable approximation. To this end, using Itô’s formula, the term $\mathcal{L}J(s,X^{\lambda^{\phi}}_{s};\lambda^{\phi})ds$ can be replaced by $dJ(s,X^{\lambda^{\phi}}_{s};\lambda^{\phi})-\frac{\partial}{\partial{x}}J(s,X^{\lambda^{\phi}}_{s};\lambda^{\phi})\hat{B}(\pi,\lambda_{s}^{\phi})dW_{s}$. Taking into account the fact that the term $\frac{\partial}{\partial{x}}J(s,X^{\lambda^{\phi}}_{s};\lambda^{\phi})\hat{B}(\pi,\lambda_{s}^{\phi})dW_{s}$ is a martingale, we obtain the following.