Cost-Sensitive Portfolio Selection via Deep Reinforcement Learning

Inspired by [63], we use the causal convolution operation, built upon 1D convolutions, to extract the sequential information. Specifically, it can keep the sequence order invariant and guarantee no information leakage from the future to the past by using padding and filter shifting. A simple example is presented in Fig. 4 (a), which depicts a stack of causal convolutions with kernel size $3\small{\times}1$. However, the causal convolution usually requires very large kernel sizes or too many layers to increase the receptive field, leading to a large number of parameters.

To overcome this, inspired by [5], we use the dilated operation to improve the causal convolution, since it can guarantee exponentially large receptive fields [66]. To be specific, the dilation operation is equivalent to introducing a fixed step between every two adjacent filter taps [5]. A simple example is provided in Fig. 4 (b), which depicts a stack of dilated causal convolutions with kernel size $3\small{\times}1$. One can find that the receptive field of the dilated causal convolution is much larger than the causal convolution. Specifically, the gap of receptive fields between the two convolutions increases exponentially with the increase of the network depth.

Note that existing fully convolution networks [5, 63, 42], e.g., dilated causal convolutions, can hardly extract asset correlations, since they process the price of each asset separately by using 1D convolutions. To address this, we devise a correlational convolution operation, which seeks to combine the price information from different assets, by fusing the features of all assets at every time step. Specifically, we apply padding operations to keep the structure of feature maps invariant. With this operation, the correlation information net can construct a multi-block architecture as shown in Fig. 2 (bottom), and asymptotically extract the asset correlation without changing the structure of asset features.

In addition, we denote a degenerate variant of TCCB as TCB which does not use the correlational convolution operation. Concretely, TCB only extracts the price sequential information by using dilated causal convolutions. We empirically show that the correlation information net with TCCB can extract good asset correlations and helps to gain more profits, compared to TCB (See results in Section 6.3). This result further demonstrates the significance of the asset correlation in portfolio selection and confirms the effectiveness of the correlation information net.

Assuming there is no transaction cost, most existing methods use the log-return ($\log r_{t}$) as the reward, since it helps to guarantee a log-optimal strategy.

In practice, we can use the empirical approximation of the expected log-return $\mathbb{E}\{\log r_{t}\}$ as the reward: $R=\frac{1}{T}\sum_{t=1}^{T}\hat{r}_{t}$, where $\hat{r}_{t}\small{:=}\log r_{t}$ is the log-return on the $t$-th period, and $T$ is the total number of sampled portfolio data.

However, this reward ignores the risk cost, thus being less practical. To solve this issue, we define the empirical variance of log-return on sampled portfolio data as the risk penalty, i.e., $\sigma^{2}(\hat{r}_{t}|t\small{=}1,..,T)$, shortly $\sigma^{2}(\hat{r}_{t})$, and then develop a risk-sensitive reward function as:

where $\lambda\geq 0$ is a trade-off hyperparameter.

We next show the near-optimality of the risk-sensitive reward. It represents the relationship between the policy regarding this risk-sensitive reward and the log-optimal strategy in Prop. 2 which cannot constrain the risk cost.

See the supplementary for the proof. From Theorem 1, when $\lambda$ is sufficiently small, the growth rate of the optimal strategy regarding this reward can approach the theoretical best one, i.e., the log-optimal strategy.

Despite having theoretical guarantees, the risk-sensitive reward assumes no transaction cost, thus being insufficient. To solve this issue, we improve it by considering the proportional transaction cost. In this setting, the log-return will be adjusted as $\hat{r}_{t}^{c}\small{:=}\log r^{c}_{t}\small{=}\log r_{t}\small{*}(1\small{-}c_{t})$. Specifically, the expected rebalanced log-return can also guarantee the optimality when facing the transaction cost.

However, optimizing this rebalanced log-return cannot control transaction costs well. To solve this, we further constrain the transaction cost proportion $c_{t}$. Let $\omega_{t}\small{:=}1\small{-}c_{t}$ be the proportion of net wealth, and let $\psi_{p}$ and $\psi_{s}$ be transaction cost rates for purchases and sales. On the $t$-th period, after making the decision $a_{t}$, we need to rebalance from the current portfolio $\hat{a}_{t-1}\small{=}\frac{a_{t-1}\odot x_{t-1}}{a_{t-1}^{\top}x_{t-1}}$ to $a_{t}$, where $\odot$ is the element-wise product. During rebalancing, the sales occur if $\hat{a}_{t-1,i}\small{-}a_{t,i}\omega_{t}\small{>}0$, while the purchases occur if $a_{t,i}\omega_{t}\small{-}\hat{a}_{t-1,i}\small{>}0$. Hence,

where $(x)^{+}\small{=}\max(x,0)$. Following [39], we set $\psi_{p}\small{=}\psi_{s}\small{=}\psi\in[0,1]$, and then obtain:

Getting rid of $\omega_{t}$, we can bound $c_{t}$ as follows.

See the supplementary for the proof. Prop. 4 shows that both upper/lower bounds of $c_{t}$ are related to $\|a_{t}\small{-}\hat{a}_{t-1}\|_{1}$: the smaller the L1 norm, the smaller the upper/lower bounds and thus the $c_{t}$. By constraining this L1 norm, we derive the final cost-sensitive reward based on Theorem 1 and Prop. 4 as:

where $\gamma\geq 0$ is a trade-off hyperparameter.

We next show the near-optimality of the cost-sensitive reward, which reflects the relationship between the strategy regarding this cost-sensitive reward and the theoretical optimal strategy in Prop. 3 which cannot control both costs.

See the supplementary for the proof. Specifically, when $\lambda$ and $\gamma$ are sufficiently small, the wealth growth rate of the strategy regarding the cost-sensitive reward can be close to the theoretical optimum.

We highlight that this reward can be helpful to design more effective portfolio selection methods with the near-optimality guarantee when facing both transaction and risk costs in practice tasks. Specifically, by optimizing this reward with the direct policy gradient method, the proposed PPN can learn at least a sub-optimal policy to effectively maximize accumulated returns while controlling both costs as shown in Prop. 1.

We compare PPN with several state-of-the-art methods, including Uniform Buy-And-Hold (UBAH), best strategy in hindsight (Best), CRP [12], UP [12], EG [26], Anticor [6], ONS [2], CWMR [35], PAMR [36], OLMAR [37], RMR [29], WMAMR [18] and EIIE [31]. In addition, to evaluate the effectiveness of the asset correlation, we also compare PPN with a degenerate variant PPN-I that only exploits independent price information by using TCB.

Following [57, 39], we use three main metrics to evaluate the performance. The first is accumulated portfolio value (APV), which evaluates the profitability when considering the transaction cost.

where $S_{0}=1$ is the initialized wealth. In addition, $a_{t}$, $x_{t}$ and $c_{t}$ indicate the portfolio vector, the price relative vector and the transaction cost proportion on the $t$-th round, respectively.

A major drawback of APV is that it neglects the risk factor. That is, it only relies on the returns without considering the fluctuation of these returns. Thus, the second metric is Sharpe Ratio (SR), which evaluates the average return divided by the fluctuation, i.e., the standard deviation (STD) of returns.

where $r_{t}^{c}$ is the rebalanced log-return on the $t$-th round.

Although SR considers the volatility of portfolio values, it treats upward and downward movements equally. However, downward movements are usually more important, since it measures algorithmic stability in the market downturn. To highlight the downward deviation, we further use Calmar Ratio (CR), which measures the accumulated profit divided by Maximum Drawdown (MDD):

where MDD denotes the biggest loss from a peak to a trough:

Note that the higher APV, SR and CR values, the better profitability of algorithms; while the lower STD and MDD values, the higher stability of returns. We also evaluate average turnover (TO) when examining the influence of transaction costs, since it estimates the average trading volume.

where $\hat{a}_{t-1}$ and $\omega_{t}$ indicate the current portfolio before rebalance and net wealth proportion on the $t$-th round.

The globalization and the rapid growth of crypto-currency markets yield a large number of data in the finance industry. Hence, we evaluate PPN on several real-world crypto-currency datasets. Following the data selection method in [31], all datasets are accessed with Poloniex¹¹1Poloniex’s official API: https://poloniex.com/support/api/.. To be specific, we set the bitcoin as the risk-free cash and select risk assets according to the crypto-currencies with top month trading volumes in Poloniex. We summary statistics of the datasets in Table I. All assets, except the cash asset, contain all 4 prices. The price window of each asset spans 30 trading periods, where each period is with 30-minute length.

Some crypto-currencies might appear very recently, containing some missing values in the early stage of data. To fill them, we use the flat fake price-movements method [31]. Moreover, the decision-making of portfolio selection relies on the relative price change rather than the absolute change [37]. We thus normalize the price series with the price of the last period. That is, the input price on the $t$-th period is normalized by $P_{t}\small{=}\frac{P_{t}}{P_{t,30}}\small{\in}\mathbb{R}^{m\times 30\times 4}$, where $P_{t,30}\small{\in}\mathbb{R}^{m\times 4}$ represents the prices of the last period.

As mentioned above, the overall network architecture of PPN is shown in Fig. 2. Specifically, there are three main components: Correlation Information Net, Sequential Information Net and Decision-making Module. To make it more clearer, we record the detailed architectures in Table II.

To be specific, in Correlation Information Net, we adopt the temporal correlational convolutional block as the basic module. To be specific, it consists of two components, i.e., the dilated causal convolution layer (DCONV) and the correlation convolution layer (CCONV).

Note that the concatenation operation in the decision-making module has two steps. First, we concatenate all extracted features and the portfolio vector from last period. Then, we concatenate the cash bias into all feature maps.

In addition, we implement the proposed portfolio policy network with Tensorflow [1]. Specifically, we use Adam optimizer with batch size 128 on a single NVIDIA TITAN X GPU. We set the learning rate to 0.001, and choose $\gamma$ and $\lambda$ from $10^{[-4:1:-1]}$ using cross-validations. Besides, the training step is $10^{5}$, the cash bias is fixed to 0, and the transaction cost rate is $0.25\%$, which is the maximum rate at Poloniex. In addition, the training time of PPN is about 4, 5.5, 7.5 and 15 GPU hours on the Crypto-A, Crypto-B, Crypto-C and Crypto-D datasets, respectively. All results on crypto-currency datasets are averaged over 5 runs with random initialization seeds.

In previous experiments, we have demonstrated the effectiveness of PPN, where the transaction cost rate is $0.25\%$. However, the effect of the transaction cost rate has not been verified. We thus examine their influences on three dominant methods on Crypto-A.

From Table V, PPN achieves the best APV performance across a wide range of transaction cost rates. This observation further confirms the profitability of PPN.

Compared to EIIE, PPN-based methods obtain relatively low TO, i.e., lower transaction costs. Since EIIE optimizes only the rebalanced log-return, this finding indicates that our proposed reward controls the transaction cost better.

Moreover, when the transaction cost rate is very large, e.g., $c\small{=}5\%$, PPN-based algorithms tend to stop trading and make nearly no gains or losses, while EIIE, however, loses most of the wealth with relatively high TO. This implies our proposed methods are more sensitive to the transaction cost.

We further evaluate the influences of $\gamma$ in the cost-sensitive reward. From Table VI, we find that with the increase of $\gamma$, TO values of PPN decrease. This observation means that when introducing $\|a_{t}\small{-}\hat{a}_{t-1}\|_{1}$ into the reward, PPN can better control the trading volume, and thus better overcome the negative effect of the transaction cost.

This finding is also reflected in Fig. 6. With the increase of $\gamma$, there are more period intervals remaining unchanged. That is, when the transaction cost outweighs the benefit of trading, PPN will stop the meaningless trading.

Note that, PPN achieves the best APV performance when $\gamma\small{=}10^{-3}$ in Table VI. This observation is easy to understand. If $\gamma$ is too small, e.g., $10^{-4}$, PPN tends to trade aggressively, leading to a large number of transaction costs, thus affecting the profitability of PPN. If $\gamma$ is large, e.g., $10^{-2}$ and $10^{-1}$, PPN tends to trade passively, thus limiting the model to seeking better profitability. As a result, when setting a more reasonable value, e.g., $\gamma=10^{-3}$, PPN can learn a better portfolio policy and achieve a better trade-off between the profitability and transaction costs.

We also examine the influences of $\lambda$ in the cost-sensitive reward, and report the results in Table VII. Specifically, with the increase of $\lambda$, the STD values of PPN asymptotically decrease on all datasets. Since $\lambda$ controls the risk penalty $\sigma^{2}(\hat{r}_{t})$, this result is consistent with the expectation and also demonstrates the effectiveness of PPN in controlling the risk cost. Moreover, with the increase of $\lambda$, the MDD results decrease on most datasets. Since MDD depends on the price volatility of the financial market, this result implies that constraining the volatility of returns is helpful to control the downward risk.