An intelligent algorithmic trading based on a risk-return reinforcement learning algorithm

The critic-only method is also called the value-based method. DQN(Deep Q-Learning Network) is a well-known value-based deep reinforcement learning algorithm, the core of which is to use a neural network to replace the q-label, i. e. the action value function. The input of the neural network is state information, and the output is the value of each action. Therefore, The DQN algorithm can be used to solve the problem of continuous state space and discrete action space but can not solve the problem of continuous action space.

DQN and its improved algorithms have received extensive attention in the research of financial trading. [18] proposed the league championship algorithm (LCA) that can extract and save various stock trading rules according to different stock market conditions. Every agent in LCA is a sports team member and can learn from the weaknesses and strengths of others to improve their performance. They proved that their algorithm outperforms Buy-and-Hold and GNP-RA strategies. [19]trained an LSTM network using double Q-learning [20] and obtained positive rewards in a cryptocurrency market with a decreasing tendency. [21] proposed a Deep Recurrent Q- Network(DRQN), which can process sequences better. They found that the proposed algorithm outperformed the buy and hold strategy in the S&P500 ETF history dataset. [22] also adopted a DRQN-based algorithm, which was applied to the foreign exchange market from 2012 to 2017, and developed a motion enhancement technology to mitigate the lack of exploration in Q-learning. Experimental results showed that the annual return of some currencies was 60%, with an average of about 10%. [23] proposed a novel deep Q-learning portfolio management strategy. The framework consists of two main components: a group of local agents responsible for trading a single asset and a global agent that rewards each local agent based on the same reward function. The results showed that the proposed approach was a promising method for dynamic portfolio optimization. The deep Q-learning network proposed by [24] is based on an improved DNN structure consisting of two branches, one of which learned action values while the other learned the number of shares to take to maximize the objective function. [25] used a LSTM based Q network to implement portfolio optimization. In addition, to consider the risk factors in portfolio management, the volatility of portfolio return was pulsed to the rewards function. [12] discretized the action space by fixing the trading amount, in which trader can buy or sell a specific asset for a fixed amount. Short selling is not allowed, fixed the trading amount for each asset, and a mapping function is designed to transform the infeasible action into feasible action.

In the actor-only method, the action taken by the agent can be learned directly by a neural network. The advantage of this method is that it makes it available for continuous action space. [26] constructed a recursive deep neural network for environmental perception and recursive decision-making. Then, deep learning is used for feature extraction and combined with fuzzy learning to reduce the uncertainty of input data. Another novelty of this paper is a variant of Backpropagation through time(BTT) to handle the vanishing gradient problem. [27] used Policy Gradient(PG) approach for financial trading. Its main contribution is to analyze the advantages of the LSTM network structure over the fully connected network and to compare the impact of some combination of technical indicators on revenue. The drawback of the Actor-only method is that, due to the policy strategy, the number of interactions between agents and the environment is greatly increased compared with a value-based method. As a result, the training is very time-consuming.

The actor-critic method contains two parts: an actor, which selects action based on probability or certainty, and a critic, which evaluates the score of the action taken by the actor. The actor-critic method combines the advantages of the value-based and policy-based methods, which can not only deal with continuous and discrete problems but also carry out the one-step update to improve learning efficiency. [28] exploited Gated Recurrent Unit(GRU) to extract financial features and proposed a critic-only method called Gated Deep Q-learning trading strategy(GDQN) and Actor-Critic method called Gated Deterministic Policy Gradient trading strategy(GDPG). Experimental results show that GDQN and GDPG outperformed the Turtle trading strategy[29] and DRL trading strategy[26], and the performance of GDPG is more stable than the GDQN in the ever-evolving stock market. [30] adopted three versions of the RL algorithm based on Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Policy Gradient (PG) for portfolio management. A so-called Adversarial Training was proposed to reach a robust result and to consider possible risks in optimizing the portfolio more carefully. [16] proposed a novel reinforcement learning algorithm for portfolio optimization using Proximal Policy Optimization (PPO). An essential characteristic of the proposed model is that the number of assets is dynamic. Experiments on a cryptocurrency market showed that the proposed algorithm outperformed three state-of-the-art algorithms presented by [19, 31, 32]. [33] compared the performance of deep double Q-learning and proximal policy optimization (PPO) to several benchmark execution policies and found that PPO realizes the lowest total implementation short-fall across exchanges and currency pairs.

To sum up, these studies used various RL algorithms for financial trading problems in a different settings. The existing literature also has the following problem. First, most literature determines the optimal action(portfolio decision) based on the mean of the portfolio cumulative returns(for example, Q- value) and pays less attention to the risk. Although [23, 28, 16] consider the risk factor in the reward function, these authors can not measure the risk of portfolio cumulative return. Second, short selling is allowed in many strategies[34, 26, 13, 24], but the above strategies consider trading for only one asset, which is often inconsistent with the facts; most of the investors often hold multiple assets to diversify risk. Although some strategies[31, 13, 12] consider trading various assets, short selling was not allowed, which means that investors will not be able to make profits in the bear market. Third, the interaction between agent and environment is very time-consuming in training. Most existing literature does not consider how to improve the training speed. Thus, the algorithm proposed in this paper can effectively overcome the shortcomings mentioned above.

Markov decision process (MDP) is a mathematical model for sequential decision problems, which is used to simulate the random strategies and rewards that agents can realize in the environment where the system state has Markov properties. Portfolio optimization is a typical sequential decision problem. Investors dynamically adjust portfolio weights according to market information. Thus, we can consider the portfolio allocation problems as a Markov Decision Process (MDP). If a state transition conforms to Markov, the next state only depends on its current state and has nothing to do with the state before its current state. A finite MDP (as considered here) is a four tuple, denoted by $(S,A,P_{a}(s,s^{\prime}),R_{a}(s,s^{\prime}))$.

where:

$\bullet$ S is a finite set of states.

$\bullet$ A is a finite set of actions (and As is the finite set of actions available from state s).

$\bullet$ $R_{a}(s,s^{\prime})$ is reward function.

$\bullet$ $P_{a}(s,s^{\prime})=Pr(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)$ is the state transition probability.

The interaction between an agent and financial environment will produce a trajectory $\tau=\{s_{0},a_{0},R_{0},s_{1},a_{1},R_{1},...\}$. $G_{t}$ is the discounted cumulative reward, which the agent can obtain at time t expressed as follows:

$$G_{t}=\sum_{i=t}^{T}\gamma^{i-t}R(s_{i},s_{i+1})$$

(1)

where $\gamma\in[0,1]$ is the discounted rate.

To learn the optimal strategy, we use the value function. There are two types of value functions in reinforcement learning: state value function, denoted by $V(s)$, and action-value function, denoted by $Q(s,a)$. The state value function, shown in Eq.(2), represents the expectation of cumulative rewards from a certain state s.

$$V(s)=E[G_{t}|s_{t}=s]$$

(2)

The action value function, given in Eq.(3), is the expected return obtained after the agent executes an action in the current state s.

$$Q(s,a)=E[G_{t}|s_{t}=s,a_{t}=a]$$

(3)

An important and difficult problem in algorithmic trading is the low observability of the market environment. The information available to investors is extremely limited compared to the complex market environment. Therefore, how to deal with this limited information is very important. In this paper, the state at the period t, denoted by $s_{t}$, consists of three types of variables: historical data for the assets($K_{t}$), portfolio weights($w_{t}$) and trading time step(t). The input of the actor network is historical data, and the input of the critic network is the concatenation of historical data, portfolio weight, and the time index. The historical data of the selected portfolio consists of raw data and technical indicators. The raw data includes the price open, close, high, low, and volume(OCHLV). Technical indicators consist of a list of candlestick patterns, including bearish, bullish, significant, hammer, inverse hammer, bullish engulfing, piercing line, morning star, bullish harami, hanging man, shooting star, bearish engulfing, evening star, three black crows, dark cloud cover, bearish harami, doji, spinning top. See [35] for details. All historical data can be represented by a tensor with dimension $n\times g\times h$, where n is the number of risky assets, g is the number of features per asset, h denotes a window size which is the number of latest feature value the agent can observe. Portfolio weight, also called portfolio vector, is the percentage of a total portfolio represented by a single asset. The rationality of adding a time index to the critic network is that it reflects the time value of money.

$$\begin{split}&s_{t}=(K_{t},w_{t},t)\\ &w_{t}=(w_{1t},w_{2t},...w_{n,t})\\ &K_{t}=(K_{1t},K_{2t},...K_{n,t})\end{split}$$

(4)

A problem in algorithmic trading is the low observability of the market environment. Compared with the complexity of the market environment, the information available to investors is extremely limited. Therefore, how to deal with little details is very important. At each time step t, the agent executes a trading action resulting from the actor network. Specifically, the agent needs to redetermine the optimal portfolio weight according to the updated state at the end of each period. In addition, to realize short selling, the actor network needs to be modified as follows: Firstly, to meet self-financing conditions, we use the softmax function to transform the output of the actor network into a new variable, denoted by $SA=\{SA_{1},SA_{2},...SA_{n}\}$. This transformation function is given in Eq.(5).

$$SA_{i}=\frac{exp^{A_{i}}}{\sum_{i=0}^{n-1}exp^{A_{i}}},\quad\quad i=1,2,...n$$

(5)

where $A=\{A_{1},A_{2},...A_{n}\}$ is the initial output of the actor network.

The default of this transformation is that the portfolio weight of each asset is greater than zero, which makes short selling impossible. Short selling occurs when an investor borrows a security and sells it on the open market, planning to repurchase it later for less money. It allows investors to benefit not only from a bear market but also to use capital proceeds to overweight the portfolio’s long-only component of the portfolio. Different from the existing literature, we realize short selling of assets through the following transformation in Eq.(6), in which $w_{i}(i=1,2,\cdot\cdot\cdot n)$ represents the proportion of asset i after transformation and $\delta$(called Delta) represents adjustment parameter.

$$w_{i}=SA_{i}\times\delta-\frac{\delta-1}{n},\quad\quad i=1,2,...n$$

(6)

The self-financing constraint can be satisfied through the above linear transformation, and this transformation can also realize short selling. In particular, $\delta=1$ indicate no change of portfolio weight, $\delta=0$ indicate that the portfolio weight of each asset is equal to $1/n$. Furthermore, in this paper, our portfolio includes four risky assets and a risk-free asset(n=5); the adjustment parameter is set to 3($\delta=3$), so the portfolio weight of a single asset ranges from -40% to 260%, which means the maximum proportion of short selling in total assets is 160%.

The reward reflects the performance of an agent’s action. Let $asset_{t}$ denote portfolio value at the end of period $t$. The reward, denoted by $reward_{t}$, is defined as portfolio return in period t, computed as the following equation:

$$reward_{t}=\frac{asset_{t}}{asset_{t-1}}-1$$

(7)

If we do not take into account transaction cost, portfolio value at the end of period $t$ equals portfolio value at the end of period $t-1$ plus portfolio return, expressed as follows:

$$asset_{t}=asset_{t-1}+asset_{t-1}\times\sum_{i=1}^{n}w_{i,t-1}\times R_{i,t}$$

(8)

If we do not take into account transaction cost, portfolio value at the end of period $t$ equals portfolio value at the end of period $t-1$ plus portfolio return, expressed as follows

$$reward_{t}=\sum_{i=1}^{n}w_{i,t-1}R_{i,t}$$

(9)

If we consider transaction cost, portfolio value at the end of period $t$ equals $asset_{t-1}$ plus portfolio return minus transaction costs, computed as follows:

$$\begin{split}asset_{t}=&asset_{t-1}+asset_{t-1}\times\sum_{i=1}^{n}w_{i,t-1}\times R_{i,t}-\\ &c1\times\sum_{i=1}^{n}sell_{i,t}\times sellin_{i,t}\times price_{i,t}-\\ &c2\times\sum_{i=1}^{n}buy_{i,t}\times buyin_{i,t}\times price_{i,t}\\ \end{split}$$

(10)

where:

$\bullet$ c1 and c2 is the transaction cost for selling and buying.

$\bullet$ $sell_{i,t}$ and $buy_{i,t}$ are dummy variables, indicating whether or not to buy or sell asset $i$ at the end of period t.

$\bullet$ $buyin_{i,t}$ and $sellin_{i,t}$ denote the trade size of asset i at the end of period t, which should be greater than or equal to zero.

In addition, the share of asset i held by an investor at the end of period t, represented by $asset_{t}\times w_{i,t}$, satisfies the following equation:

$$\begin{split}asset_{t}\times w_{i,t}=&asset_{t-1}\times w_{i,t-1}\times(1+R_{i,t})+\\ &buy_{i,t}\times buyin_{i,t}\times price_{i,t}-\\ &sell_{i,t}\times sellin_{i,t}\times price_{i,t}\end{split}$$

(11)

Obviously, $asset_{t}\times w_{i,t}$ equals to the cumulative value of $asset_{t-1}\times w_{i,t-1}$ in period t plus the shares bought($buy_{i,t}\times buyin_{i,t}\times price_{i,t}$) minus the shares sold($sell_{i,t}\times sellin_{i,t}\times price_{i,t}$). Since it does not make sense to buy and sell an asset simultaneously, which increases transaction costs, one of $sell_{i,t}$ and $buy_{i,t}$ must be zero.

If $asset_{t-1}$ , $w_{i,t-1}$ and $w_{i,t}$ are given, we could adjust trading strategies to maximize the portfolio value $asset_{t}$. This optimization problem can be formulated as a nonlinear programming, the objective function of which is to maximize $asset_{t}$, and the decision variables are $sell_{i,t}$ , $buy_{i,t}$, $buyin_{i,t}$ and $sellin_{i,t}$. The nonlinear programming model can be expressed as follows:

$$\begin{split}&\mathbf{Maximize:}\quad asset_{t}\\ &\mathbf{Subject\quad to}:\\ &\quad asset_{t}\times w_{i,t}=asset_{t-1}\times w_{i,t-1}\times(1+R_{i,t-1})-\\ &\quad\quad\quad\quad\quad\quad sell_{i,t}\times sellin_{i,t}\times price_{i,t}+\\ &\quad\quad\quad\quad\quad\quad buy_{i,t}\times buyin_{i,t}\times price_{i,t}\\ &\quad asset_{t}=asset_{t-1}+asset_{t-1}\times\sum_{i=1}^{n}(w_{i,t-1}\times R_{i,t-1})-\\ &\quad\quad\quad\quad c1\times\sum_{i=1}^{n}sell_{i,t}\times sellin_{i,t}\times price_{i,t}-\\ &\quad\quad\quad\quad c2\times\sum_{i=1}^{n}buy_{i,t}\times buyin_{i,t}\times price_{i,t}\\ &\quad sell_{i,t}\times buy_{i,t}=0\\ &\quad sell_{i,t}+buy_{i,t}=1\\ &\quad sellin_{i,t}\geq 0\\ &\quad buyin_{i,t}\geq 0\end{split}$$

(12)

The algorithm adopted by this paper is based on actor-critic architecture, which includes an actor network and a critic network. In addition, each network has its corresponding target network, so the algorithm includes four networks, namely the actor network, denoted by $\mu_{\varphi}(s)$, and the critical network, denoted by $K_{\omega}(s,a)$, the target actor network, denoted by $\mu^{\prime}_{\varphi^{\prime}}(s)$ and the target critical network, denoted by $K^{\prime}_{\omega^{\prime}}(s,a)$ . In classical actor-critic architectures such as A3C, TD3, and DDPG, the actor network updates $\varphi$ by maximizing the expectation of cumulative rewards, and the critic network updates $\omega$ by minimizing the error between the evaluation value and the target value. The algorithm proposed in this paper is different from the above algorithms. Inspired by distributional reinforcement learning(DRL) initially proposed by[36], we estimate the distribution, rather than the expectation, of cumulative rewards by the critic network.

Distributional reinforcement learning is a new kind of reinforcement learning algorithm, mainly learning the distribution of cumulative rewards. The distributional Bellman operator($\tau$) is shown in Eq.(13).

$$\tau Z(s,a)\overset{D}{=}R(s,a)+\gamma Z(s^{\prime},a^{\prime})$$

(13)

where $Z(s,a)$ represents the cumulative return obtained by taking action $a$ in state $s$, which is a random variable, $R(s,a)$ is the rewards function.

Under Wasserstein metric, [36] proved that the distributional Bellman operator is a $\gamma$-contract operator. The learning task of DRL is to make the distribution $Z(s,a)$ and the target distribution $R(s,a)+\gamma Z(s^{\prime},a^{\prime})$ as similar as possible. Following [15], we utilize quantile regression to estimate network parameters. Quantile regression, projecting the update of distributional Bellman to the quantile distribution, uses a parameterized quantile distribution to approximate the value distribution. Let $[\theta_{1},\theta_{2},\ldots,\theta_{N}]$, which is the output of critic network, denotes the N quantiles of $Z(s,a)$. The target distribution is shown in Eq.(14), which can be regarded as ground truth.

$$\tau\theta^{\prime}_{j}=r+\gamma\theta^{\prime}_{j},\quad\forall j$$

(14)

The loss function of critic network is defined in Eq.(15).

$$L_{\omega}=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{N}[\rho_{\hat{\tau}_{i}}(\tau\theta^{\prime}_{j}-\theta_{i})]$$

(15)

where

$$\rho_{\tau}=|\tau-\delta_{\mu<0}||u|=(\tau-\delta_{\mu<0})u$$

(16)

Because $|u|$ is not differentiable at zero, we take the Huber loss function, given in Eq.(17), instead of $|u|$.

$$\Gamma_{\kappa}=\left\{\begin{aligned} &\frac{1}{2}u^{2}&if\quad|u|\leq\kappa\\ &\kappa(|u|-\frac{1}{2}\kappa)&otherwise\\ \end{aligned}\right.$$

(17)

Thus, we get a new loss function, also called the quantile Huber loss function, expressed as follows:

$$\rho_{\tau}^{\kappa}=|\tau-\delta_{\mu<0}|\Gamma_{\kappa}$$

(18)

Since portfolio weight is a continuous variable, we choose a deterministic policy to generate actions, given the Eq.(19).

$$a=\mu_{\varphi}(s)$$

(19)

As mentioned in the introduction section, the objective function of portfolio optimization is the weighted sum of the mean and VaR of portfolio cumulative return. If we take N uniform quantiles, the mean of portfolio cumulative return, denoted by $MR$, can be approximately equal to the average of quantiles of cumulative return, shown in Eq.(20).

$$MR=\frac{\sum_{i=1}^{N}\theta_{i}}{N}$$

(20)

VaR is used to measure risk in this paper. VaR refers to the maximum portfolio loss for a given confidence level in a specific period. It can be described in Eq.(21), in which $R_{p}$ and $\alpha$ denote portfolio cumulative return and confidence level respectively.

$$Prob(R_{p}\leq-VAR_{\alpha})=1-\alpha$$

(21)

Because ${VaR}_{\alpha}$ is a quantile of portfolio cumulative return, in this paper, it can also be expressed as the following:

$$VaR_{\alpha}=-\theta_{N\times(1-\alpha)}$$

(22)

Given the value of N and $\alpha$, we can easily get $VaR_{\alpha}$. For example, if $N=100,\alpha=95\%$, $VaR_{95\%}=-\theta_{5}$, if $N=200,\alpha=90\%$, $VaR_{90\%}=-\theta_{20}$.

According to classical modern portfolio theory, portfolio selection aims to construct an optimal portfolio model that maximizes expected returns under a given acceptable risk level($\varpi$). This optimal model is shown in the following:

$$\begin{split}Max&\quad MR\\ &s.t.VaR_{\alpha}=\varpi\end{split}$$

(23)

According to the above optimization model, different risk levels correspond to optimal weights. In other words, investors have various portfolio choices, which undoubtedly increases the difficulty of decision-making. To obtain a single optimal solution, The above programming problem can be transformed into a single objective programming problem through the following function:

$$\begin{split}U&=MR-\zeta\times VAR_{\alpha}\\ &=\frac{\sum_{i=1}^{N}\theta_{i}}{N}+\zeta\times\theta_{N*(1-\alpha)}\end{split}$$

(24)

where $\zeta$ (called Zeta) denotes the risk attitude of the investor, the higher $\zeta$, the higher the risk aversion of investors, and the more conservative investment strategies will be adopted.

Exploration is crucial for agents, and deterministic strategies cannot explore, so we need to artificially add noise to the output actions. In DDPG algorithm, [37] uses Ornstein Uhlenbeck process(OU) as action noise. [38] found noise drawn from the OU process offered no performance benefits, so in this paper, following [38], we add Gaussian noise, which is given in Eq.(25), to each action. The standard deviation of Gaussian noise decreases exponentially as the number of parameter updates continues.

$$\begin{split}noise\_output_{i}&=actor\_output_{i}+\epsilon\\ &\epsilon\sim clip(N(0,\sigma),-c,c)\end{split}$$

(25)

We need to model the neural network structure to explore functional patterns and extract informative features. Based on the time series nature of stock data, the gated recurrent unit(GRU) is utilized to construct an informative feature representation. The GRU has two gates, i.e., a reset gate and an update gate. These two gates determine which information can ultimately be used as the output of the gating recurrent unit. They are unique in that they can preserve information in long-term sequences and will not be cleared over time or removed because they are unrelated to prediction. Intuitively, the reset gate determines how to combine the new input information with the previous memory. The update gate defines the amount of prior memory saved to the current time step. The actor and critic network structure is shown in Fig.1 and Fig.2. Since GRU, LSTM, and RNN have similar structural features, we also used LSTM and RNN networks in the sensitivity analysis.

In addition, Several authors have leveraged Convolutional Neural Networks(CNN) to perform financial trading strategy[39, 40], treating the stock trading problem as a computer vision application by using time-series representative images as input. The convolutional neural network has unique advantages in speech recognition and image processing with its particular structure of local weight sharing. Its design is closer to the actual biological neural network. Weight sharing reduces the complexity of the network, especially the feature that the images of multi-dimensional input vectors can be directly input into the network, which avoids the complexity of data reconstruction in the feature extraction and classification process. Because of CNN’s robust feature representation ability, we studied the impact of CNN on the performance of the proposed algorithm in the sensitivity analysis.

In this paper, to make the training result robust, the actor must interact with the environment at different epochs(or iterations) several times, which is very time-consuming. We utilize Ape-X architecture, proposed by [17], to speed up the training procedure. This algorithm decouples acting from learning and decomposes the learning process into three parts. In the first part, there are multiple actors. Each actor interacts with its respective environment based on the shared neural network, accumulating experience and putting it into the shared experience replay memory. We refer to this part, running on CPUs, as acting. In the second part, the (single) learner samples data from the replay memory and then updates the neural network. We refer to this part as the learning part running on a GPU. The third part is mainly responsible for data transmission. We refer to this part, running on CPUs, as communication.

We use a multiprocessing method to implement Ape-X. Specifically, there are 22 parallel processes, of which 20 are responsible for interacting with the environment, one process is accountable for updating network parameters, and one process is responsible for data exchange. The general architecture of the proposed method and the pseudocode for the algorithm are shown in Fig.3 and Table 1.

We experiment with two different portfolios. The first portfolio consists of four famous exchange trading funds (ETFs) in the US market and a risk-free asset. The ETFs portfolio includes SPDR S&P 500 ETF Trust (SPY), Invesco QQQ Trust ETF(QQQ), SPDR Dow Jones Industrial Average ETF(DIA), and iShares Russell 2000 ETF(IWM). The second portfolio consists of stocks of four technology companies and a risk-free asset. The four technology companies are ORCL, AAPL, TSLA, and GOOG. All data used in this paper is available on Yahoo Finance. The trading horizon of 13 years is divided into both training and testing sets as following:

$\bullet$ $\mathbf{Training\enspace set:}$ $01/01/2008\rightarrow 31/12/2018.$

$\bullet$ $\mathbf{testing\enspace set:}$ $01/01/2019\rightarrow 31/12/2022.$

Risk and return are inseparable in the investment decision process. Under normal circumstances, the risk is highly correlated with the return, and a high return means high risk. Therefore, the performance measures must include these two aspects. In this article, we use four types of performance measures to evaluate the proposed trading strategy. The first type of metric measures the profitability of the investment strategy, i.e., total return. The second type of metrics measures investment risk, including variance and VaR. The third metric type considers risk and returns, including sharpe ratio and Sortino Ratio. The last type of metric is average turnover. More detail about performance measures are as following.

$\bullet$ $\mathbf{Total\enspace return(TR).}$ The total return is the rate of return over a period. Total return, computed using Eq.(26), includes capital gains, interest, realized distributions, and dividends.

$$TR=(Q_{[T]}-Q_{[0]})/Q_{[0]}$$

(26)

Where $Q_{[0]}$ is the value of the initial investment;$Q_{[T]}$ is the value of the portfolio at the end of the investment period.

$\bullet$ $\mathbf{Value\enspace at\enspace risk(VaR).}$ As mentioned above, VaR calculates the maximum loss of a portfolio over a given period on specified confidence level.

$\bullet$ $\mathbf{Sharpe\enspace ratio(SR1).}$ The Sharpe ratio, first proposed by citesharpe1998sharpe, is a measure of risk-adjusted return. This ratio, computed using Eq.(27), reflects how much the return on risk per unit exceeds the risk-free return. If the Sharpe ratio is positive, a portfolio’s average net value growth rate exceeds the risk-free interest rate.

$$SR=(E[R_{p}]-R_{f})/\sigma_{p}$$

(27)

where $E[R_{p}]$ is the expectation of portfolio return, $R_{f}$ is risk-free rate, $\sigma_{p}$ is standard deviation of portfolio return.

$\bullet$ $\mathbf{Sortino\enspace ratio(SR2).}$ The Sortino ratio is a risk-adjustment metric used to determine the additional return for each unit of downside risk. It is similar to the Sharpe ratio, but the Sortino ratio uses the lower partial standard deviation rather than the total standard deviation to distinguish between adverse and favorable fluctuations. The Sortino ratio can be expressed as follows:

$$SR=(E[R_{p}]-R_{f})/DR$$

(28)

where $DR$ is the lower partial standard deviation of portfolio return.

$\bullet$ $\mathbf{Standard\enspace deviation(SD).}$ The standard deviation of the portfolio variance can be calculated as the square root of the portfolio variance.

$\bullet$ $\mathbf{Average\enspace turnover(AT).}$ Average turnover represents the average level of change in portfolio weight, which is defined in Eq.(29).

$$AT=1/(2t_{f})\sum_{t=0}^{t_{f}-1}\sum_{i=1}^{l}\mid{w_{t+1,i}-w_{t,i}}\mid$$

(29)

where $t_{f}$ is the investment horizon,$w_{t,l}$ denote weight parameter of asset i in investment period t.

To analyze the effectiveness of the proposed strategy, some benchmark strategies, summarised hereafter, are selected for comparison.

$\bullet$ $\mathbf{Buy\enspace and\enspace hold(B\&H).}$ B&H is used as a benchmark strategy by many researchers compared with their proposed strategies. Suppose that the holding proportion of all five assets is 20% in the B&H strategy and remains unchanged throughout the investment period.

$\bullet$ $\mathbf{Sell\enspace and\enspace hold(S\&H).}$ S&H is also widely used as a benchmark strategy. Assuming that in the S&H strategy, the holding proportion of all four risky assets is -25%, the proportion of risky-free assets is 200%, all of which remain unchanged throughout the investment period.

$\bullet$ $\mathbf{Random\enspace selected(RN).}$ According to the efficient market hypothesis (EMH), all valuable information has been timely, accurate, and fully reflected in the stock price trend, including the current and future value of the enterprise. Without market manipulation, investors cannot obtain excess profits higher than the market average by analyzing past prices. Any trading strategy based on historical data differs from the randomly selected strategy.

$\bullet$ $\mathbf{Mean-variance\enspace model.}$ The mean-variance model, introduced by Markowitz in 1952, aims to find the best portfolio only by the first two moments of cumulative return. Suppose there are n kinds of assets, $R=(r_{1},...r_{n})^{T}$ represents the expected return of a portfolio, $W=(W_{1},...W_{n})^{T}$ is the weight vector, $\Sigma$ is the covariance vector of return, $\zeta$ is risk aversion coefficient, $\mathbf{1}$ represents $n\times 1$ dimensional unit vector, we establish the following optimization model based on utility maximization:

	$\displaystyle max\enspace U=$	$\displaystyle W^{T}R-\zeta W^{T}\Sigma W$			(30)
	$\displaystyle s.t.$	$\displaystyle 1^{T}W=1$			(31)

We obtain time window size and other hyper-parameters, including replay buffer size, batch size, discount factor, etc., through several rounds of tuning. In addition, we need to set some other hyper-parameters before training. Suppose the risk-free rate of return is 2%, and the equivalent weekly return is 0.038%. The transaction cost of buying and selling an asset is set to 0.02%. Risk aversion parameter($\zeta$) and short selling parameter($\delta$)are set to 0.5 and 3; because these two parameters have significant impacts on investment decisions and portfolio returns, we perform sensitivity analysis for them in subsection 6.3. The networks were trained through the ADAM optimization procedure with a learning rate of $10^{-5}$. The activation function is set as the Leaky Relu function. All parameter are summarised in Table 2. Finally, the algorithms proposed in the paper are implemented in python3.7.11 using pytorch1.10.2 and were run on a PC that contains a sixteen-core 2.50GHz CPU with 16GB RAM and NVIDIA Geforce RTX 3060 GPU.

The first detailed analysis concerns the execution of the proposed algorithm on the ETFs portfolio. Fig.4 illustrates the average performance of the proposed algorithms for both the training and testing sets as the parameter update step continues. It can be noticed that the cumulative return of the ETFs portfolio tends to converge in the training and testing set after about 20000 parameter updates. Moreover, we note that the convergence value of the cumulative return in the testing set is greater than in the training set. On the other hand, the risk levels measured by VaR are almost identical in training and testing sets. One possible explanation is that, as pointed out by[41], because the training and testing set does not share identical distributions, it simply indicates an easier-to-trade and more profitable market for the testing set. Although the training and testing sets do not share identical distributions, the training set is representative enough for the R3L algorithm to achieve good results in testing, so we can still use historical data to train the model and then use it for future trading. Finally, from the perspective of the Sharpe ratio, the performance of the R3L algorithm in the testing set is still better than that in the training set.

Fig.5 illustrates the portfolio value trend when applying the proposed algorithm and benchmark strategies in the testing set. We observe that the final portfolio value of the proposed algorithm is 19.51% higher than the S&H strategy, 18.0% higher than the RN strategy, 8.1% higher than the MV strategy, and 7.9% higher than the B&H strategy.

Table 3 further presents output performance measures results when using the R3L algorithm and the benchmark strategies. The R3L algorithm is optimal regarding risk, return, and overall performance. The Sharpe ratio of the proposed algorithm is 19.51% higher than the S&H strategy, 18.0% higher than the RN strategy, 8.1% higher than the MV strategy, and 18.0% higher than the B&H strategy. In addition, as the stock market was primarily bullish throughout the test period, the B&H strategy outperformed other benchmark strategies most of the time, while the performance of the S&H strategy was the worst. The performance of the RN strategy is abysmal. Since we did not consider the company’s financial and internal non-public information, it does not mean that the efficient market hypothesis is not tenable.

The same detailed analysis is performed on the stock portfolio, which shows different characteristics compared to the ETFs portfolio. Fig.6 illustrates the average performance of the proposed algorithm for both the training and testing sets. We observe that the cumulative return of the stock portfolio is almost identical in the training and testing set after 20000 parameter updates. At the same time, the risk level(VaR) is higher in training sets than in testing sets, which is very different from the ETFs portfolio. Finally, from the sharpe ratio perspective, the algorithm’s performance in the training set is better than in the testing set. Fig.5 illustrates the portfolio value trend of different strategies. Similar to the ETFs portfolio, the R3L strategy has the highest cumulative value at the end of the investment period. Table 3 presents output performance measures results when using different trading strategies. It can be noticed that from the perspective of the sharpe ratio, sortino ratio, and VaR, the R3L algorithm is also optimal.

In this paper, delta($\delta$) and zeta($\zeta$) are two critical parameters that affect the overall performance of the proposed algorithm. Delta represents the maximum ratio of short selling allowed; the larger the delta, the more profit opportunities investors have in the bear market. Zeta represents investors’ risk attitude; the larger zeta is, the higher the risk tolerance is, and investors tend to adopt more radical investment strategies. Finally, as mentioned in subsection 4.2, different network structures have different feature extraction abilities. Therefore, it is necessary to analyze the impact of network structures on the algorithm’s effectiveness to select the best one. We conduct the sensitive analysis of delta, theta, and network structure on the R3L strategy. The performance of the proposed model is evaluated concerning different values for delta, theta, and network structures.

$\bullet$ $\mathbf{Sensitive\enspace analysis\enspace of\enspace delta.}$ Fig.7 illustrates the portfolio value trend when applying the R3L algorithm with a different delta. Portfolio value shows an upward trend over time for the ETFs and stock portfolios when delta equals 1 and 3. In this case, investors can obtain positive returns at the end of the investment period. In contrast, in other cases, the portfolio value shows significant volatility over time, and investors receive negative returns at the end of the investment period. Table 4 further shows the overall performance of the proposed strategy concerning different delta values. From the perspective of total return, Sharpe ratio, and Sortino ratio, $\delta=3$ is optimal for the ETFs portfolio, and $\delta=1$ is optimal for the stock portfolio.

It can be seen that the maximum amount of short selling allowed is not the bigger, the better, which is explained by the fact that, although short selling can provide investors with profit opportunities in a bear market, it also brings more risks. Therefore, it is essential to control the scale of short selling appropriately.

$\bullet$ $\mathbf{Sensitive\enspace analysis\enspace of\enspace zeta.}$ Fig.8 illustrates the portfolio value trend when applying the R3L algorithm with different zeta. Table 5 further shows the overall performance of the proposed strategy. It can be noticed that the change in portfolio value over time is similar for different theta. At the end of the investment period, the portfolio value is the greatest for the ETFs portfolio when theta equals 3. From the Sharpe and Sortino ratios perspective, $\zeta=3$ is also optimal. The portfolio value is the greatest for the stock portfolio when theta equals 4. From the standpoint of Sharpe ratio and Sortino ratio, $\zeta=4$ is also optimal. Nevertheless, the VaR is the lowest when theta equals 2 for the ETFs and stock portfolios.

From the above analysis, it can be seen that a higher risk aversion coefficient does not necessarily lead to a smaller return and risk level, which contradicts the classical portfolio theory. One possible explanation is that our algorithm can predict the stock price based on historical data to improve the portfolio’s return and reduce the risk. To sum up, the algorithm proposed in this article can enhance portfolio returns and minimize risk.

$\bullet$ $\mathbf{Sensitive\enspace analysis\enspace of\enspace network\enspace structure.}$ Fig. 9 illustrates the portfolio value trend when applying the R3L algorithm with a different network structure. It can be observed the changing trend of portfolio value is consistent under different network structures, and only the final cumulative value is slightly different. GRU and RNN obtained the maximum cumulative value for the ETFs and stock portfolios. Table 6 further shows the overall performance. We find that GRU performs best in the ETFs portfolio from the perspective of Sharpe ratio and Sortino ratio, and RNN performs best in the stock portfolio from the perspective of Sortino ratio. In contrast, GRU performs best from the perspective of the sharpe ratio.