DoubleEnsemble: A New Ensemble Method Based on Sample Reweighting and Feature Selection for Financial Data Analysis

We show the learning trajectory based sample reweighting (SR) process in Algorithm 2. In the process, we first calculate the $h$-value for each sample and then divide all the samples into $B$ bins according to the $h$-value. Later, we assign the same weights to the samples in the same bin.

The calculation of the $h$-value is based on the loss curves of the previous sub-model $C$ and the loss values of the current ensemble $L$. For robustness, we first normalize $C$ and $L$ via ranking. The normalization function $\text{norm}:\mathbb{R}^{N\times d}\to[0,1]^{N\times d}$ replaces each element in the matrix with its rank across other elements in the column, i.e., $\text{norm}(X)_{ij}=0.9$ if $X_{ij}$ is larger than $90\%$ of the elements in the $j$-th column of $X$. Then, we can define normalized loss curves $\widetilde{C}=\text{norm}(C)$ and normalized loss values $\widetilde{-L}=\text{norm}(-L)$ (i.e., reversely normalized loss values). To indicate whether the loss of a sample gets improved during the training, we compare its loss at the start and at the end of training. We use $C_{\text{start}},C_{\text{end}}\in\mathbb{R}^{N\times 1}$ to denote the loss for all the samples at the start and at the end of training respectively. Specifically, they are the average of the first and the last 10% rows of $\widetilde{C}$ respectively. For example, if we train $T=100$ iterations for each sub-model, each element in $C_{\text{start}}$ is the average normalized loss of a sample across the first $10$ iterations. Next, we calculate the $h$-values for all the samples as follows:

where ${\bm{h}},\widetilde{-L},C_{\text{start}},C_{\text{end}}\in\mathbb{R}^{N\times 1}$ and the operations are element-wise.

To avoid extreme values for the weights, we further divide the samples into $B$ bins according to the $h$-values and assign the same weights to the samples in the same bin. Suppose the $i$-th sample is divided into the $b_{i}$-th bin. The weight of this sample is assigned as follows:

where $\langle{\bm{h}}\rangle_{b}$ is the average $h$-value for the $b$-th bin. Further, we use a decay factor $\gamma\in[0,1]$ to encourage the weight distribution to be more uniform in the latter sub-models of the ensemble. This technique is a simplified version from the concept of the self-paced factor in [30].

Now, we explain the intuition behind our design with a small example in Figure 1. Consider three types of samples in a classification task: the easy samples that are easily classified correctly, the hard samples that are close to the true decision boundary and may easily get misclassified, and the noisy samples that may mislead the model. We would like our reweighting scheme to boost the weights of hard samples while reducing the weights of the easy and the noisy samples, since easy samples can be fitted anyway and fitting noisy samples may lead to overfitting. The ${\bm{h}}_{1}$ term helps to reduce the weights of easy samples. Specifically, the loss of an easy sample is prone to be small which leads to a large value for ${\bm{h}}_{1}$ and therefore a small weight. However, this term also boosts the noisy samples since it is hard to distinguish the noisy samples and the hard samples solely based on the loss value. Fortunately, we can distinguish them by their loss curves using ${\bm{h}}_{2}$ (cf. Figure 1b). Intuitively, we assign large weights to the samples with a descending normalized loss curve. Since the training process is driven by the majority of the samples, the loss of most of the samples tends to decrease while the loss of noisy samples usually keeps the same or even increases. Therefore, the normalized loss curves of noisy samples will increase which leads to large ${\bm{h}}_{2}$ values and therefore small weights. For easy samples, their normalized loss curves are more likely to remain the same or fluctuate slightly after a quick decay, which results in moderate ${\bm{h}}_{2}$ values and therefore moderate weights. For hard samples, their normalized loss curves slowly decline during the training which indicates their contribution to the decision boundary. This results in small ${\bm{h}}_{2}$ values and therefore large weights. We show the weights of the three types of samples calculated using ${\bm{h}}_{1}$, ${\bm{h}}_{2}$ and ${\bm{h}}$ as the $h$-value respectively in Figure 1c. We observe that, using ${\bm{h}}_{1}$ not only boosts the weights of hard samples but also those of noisy samples, while using ${\bm{h}}_{2}$ suppresses the weights of noisy samples. With ${\bm{h}}_{1}$ and ${\bm{h}}_{2}$ combined (i.e., ${\bm{h}}$), we can effectively boost the hard samples and reduce the weights for the easy samples and the noisy samples.

We use the shuffling based feature selection (FS) process in DoubleEnsemble to select features for training the next sub-model. We show this process in Algorithm 3. Similar to SR, we first calculate a $g$-value for each features and then divide all the features into $D$ bins according to their $g$-values. Later, we randomly select features from different bins with different sampling ratios.

$g$-value for a feature measures the contribution of this feature to the current ensemble (i.e., feature importance). To calculate the $g$-value for a feature, we shuffle the values of this feature and compare the losses before and after the shuffle (cf. Line 5-7 in Algorithm 3). The $g$-value for a feature is large when the elimination of the feature (via shuffling) significantly increases the losses on the samples, which indicates that this feature is important to the current ensemble. For robustness against extreme $g$-values, we then divide all the features into $D$ bins according to the $g$-values and randomly select features from different bins with different sampling ratios (cf. Line 8-12 in Algorithm 3). The sampling ratios are preset and the ratio is large for the bin with large $g$-values. At last, we concatenate and return all the randomly selected features.

The reason for the design is as follows: To estimate the contribution of a feature to the model, we would like to compare with the performance when the feature is absent. One natural but costly way is to eliminate the feature, retrain and then re-evaluate the model. Instead of training a new model, we perturb the dataset to eliminate the contribution of the feature and compare the performance of the model using the perturbed dataset and that using the original dataset. Our scheme computationally is more efficient since there is no need to retrain a model.

Moreover, we argue that shuffling is more appropriate than replacing with zeros (or the mean of the feature). This is because many machine learning models are sensitive to the input data distribution. Shuffling keeps the marginal distribution of that feature, and replacing with zeros completely changes the distribution. For a simple example, consider a feature whose values are either $+1$ or $-1$ and the mean is $0$. The trained model would focus on the regions where the feature value is $+1$ or $-1$ (regions with denser samples are better fitted). Hence, the region around feature value $0$ is not fitted well and the model may behave arbitrarily for samples with feature value replaced by $0$, and cannot correctly reflect the performance when this feature is eliminated.

In addition, the shuffling based feature selection method has the following advantages: First, it considers the contribution of the feature to the model which is trained along with other features, instead of the quality of the feature itself such as the frequently used information coefficient and information ratio [43] in finance. Second, unlike other feature importance metrics that only apply to specific models (such as the information gain in boosting trees and the coefficients in Lasso [44]), the $g$-value is applicable to different base models.

This set of experiments are based on the data from OKEx. OKEx is a cryptocurrency exchange where traders around the world can trade between different cryptocurrencies in 24 hours a day. In this set of experiments, we use the data from four trading pairs: ETC/BTC, ETH/BTC, GAS/BTC and LTC/BTC. For each trading pair, one sample corresponds to one market snapshot, which is captured for approximately every $0.3$ second. The training samples used in the experiments are from $10$ consecutive trading days, with a total number of $3$ million. The testing samples come from the following $5$ trading days, with a total number of $1.5$ million. We use $31$ features, which are calculated based on the microstructure information of the market (snapshots of the limit order book), such as order flow imbalance (OFI) [47] and relative strength index (RSI) [48].

We compare the algorithms under two settings with different noise levels. In the setting denoted by 30% noise, we add 20 additional random features and 30% random samples (i.e., the values of these features/samples are randomly drawn from $U[0,1]$). In the setting denoted by 50% noise, we add 30 random features and 50% random samples. Next, we introduce the algorithms that we compare and the performance metrics that we use.

DoubleEnsemble variants

We use SR to denote the ensemble model that only uses the SR process, i.e., using all the features. We use 1st only and 2nd only to denote the variants that only use the first term (i.e., ${\bm{h}}_{1}$) or the second term (i.e., ${\bm{h}}_{2}$) in Equation (1) for the SR process respectively. We use FS to denote the ensemble model that only uses the FS process, i.e., using equal weights.

Basic methods

SingleModel: We use the training samples with all the available features and equal weights to train a single model. In the experiments, we use two types of base model: the neural network model (denoted as MLP) and the gradient boosting decision tree model (denoted as GBM). For the MLP model, we use a multi-layer perceptron with two hidden layers (each of which has $64$ neurons) followed by a dropout layer [49] and a batch-norm layer [50]. We use Mish [51] as the activation function and train the model for $200$ epochs with early stopping and exponentially decaying learning rate. For the GBM model, we use LightGBM [52] with $200$ trees, each of which has at most 32 leaves. In the later experiments, unless otherwise stated, the hyperparameters for training the sub-models are the same as used here. Notice that this single model is the same as the first sub-model in DoubleEnsemble.

SimpleEnsemble: This baseline model is an ensemble model that contains $K$ identical sub-models. The only difference between the sub-models is that they use different random seeds. We set this baseline to observe the performance difference brought by constructing an ensemble.

RandomEnsemble: This baseline model is different from the previous baseline SimpleEnsemble in that, the sub-models in this baseline not only use different random seeds but also are trained with the samples assigned with random weights. We notice that randomly reweighting the samples may improve the performance due to the fact that it increases the diversity of the sub-models. We set this baseline to isolate the performance different raised by the above reason. Constructing an ensemble by randomly reweighting samples is similar to bagging where the samples are randomly selected to construct different sub-models [18].

Baseline methods

The following baseline methods are designed for noise robustness and we compare our algorithm with them in terms of noise sensitivity. LDMI [38] uses an information-theory based loss function for training a neural network robust to noisy samples. Latent class-conditioned noise model (LCCN) [45] is another model designed for training a robust deep learning model against the noise by modeling the noise transition. CoTeaching [46] simultaneously trains two neural networks and utilize the communication between the two networks to select clean data. MentorNet [29] trains a mentor network to weight the samples based on their training dynamics for noise reduction. LearnReweight [27] sets the weights of the samples as parameters and learns the weights via gradient descent. The above baseline methods construct single models. Additionally, we design Curriculum to construct an ensemble model with $K$ sub-models, each of which uses the $30\%$ to $100\%$ of the easiest samples (the samples with lowest losses), which can be regarded as an ensemble version of curriculum learning [28].

Performance metrics

Precision: While standard classification problems care about the prediction accuracy on all the samples, the classification problems for financial market prediction care more about the accuracy for the retrieved samples. In financial market prediction, a retrieved sample corresponds to a trading signal and therefore relates to the profit of the trading strategy. Hence, we set the threshold such that approximately 1% of the samples are retrieved, and use precision as the performance metric. This corresponds to trading each pair for every $30$ seconds on average.

AUC: We also use the area under the ROC curve (ROC AUC) as the performance metric to summarize the performances of the predictor under different thresholds.

F1: In financial market prediction, we also care about the recall, which indicates the ability of the model to seize the trading opportunity. Therefore, we also use the F1 score as the performance measure which integrates the precision and the recall and it is defined as $F1=2/(\text{precision}^{-1}+\text{recall}^{-1})$.

PCT: Finally, we directly measure the profitability by PCT, which is the average return for each trading day if we follow the following strategy. Each time the sample corresponding to the current trading time point is retrieved by the predictor (which we call a trading signal), we long the base currency in the next trading time point and then close the position after $20$ seconds.

Experiment results

We show the experiment results for the cryptocurrency prediction in Table I. The first number in each entry is the mean of 5 runs with different random seeds, and the second number in the entry is the standard deviation of the 5 runs.

First, we observe that the DoubleEnsemble variants achieve a good performance in the two settings with different noise levels and the DoubleEnsemble algorithm (i.e., SR+FS) achieves the best performance. Besides, although the AUC difference between the DoubleEnsemble variants and other baselines is not significant, the precision and the profitability difference is notable. This indicates that DoubleEnsemble has a higher accuracy on the key samples (i.e., the distinguishable samples with high future returns) and therefore is more suitable for financial applications.

Second, the experiment result also demonstrates the role of the SR process. We can compare the SR models (the models that use SR) with SingleModel, SimpleEnsemble and RandomEnsemble. When using MLP as the base model, the performance improvement brought by the SR process not only comes from constructing an ensemble or the diversity increase resulted from reweighting, but also comes from the reweighting scheme used in the SR process. When using GBM as the base model, the performance improvement is mainly resulted from the reweighting scheme of the SR process. This quantifies the important role that SR plays in identifying and weighting the key samples. We also found that, although some baselines (such as LCCN) are robust to different noise levels, the SR models outperform the previous baseline methods that reweight the samples to denoise. The reason may be that the SR process is designed not only to denoise but also to promote the performance by boosting the key samples.

At last, the experiment result shows the performance improvement brought by the FS process. We can observe the improvement brought by the FS process by comparing FS with RandomEnsemble or by comparing SR+FS with SR.

In this set of experiments, we train predictors for the stock market and trade the stocks based on the prediction. We base our experiments on China’s A share market where over 3,000 stocks are traded. Each sample corresponds to one trading day of one stock.

Experiment settings

We conduct experiments in two different settings. In the first setting (denoted by DAILY), we long the top 20 stocks suggested by the predictor at the market closing of each trading day, and then sell these stocks upon the closing time of the next trading day. The predictions are based on 182 features that are calculated 3 minutes before the market closing of that trading day. In the second setting (denoted by WEEKLY), after the market closing on each trading day, we calculate 254 features based on the historical market information and make the prediction. In the next trading day, we long the top 10 stocks suggested by the prediction at the open price and hold these stocks for five trading days. Thereafter, we sell these stocks after the opening of the fifth trading day. In this setting, we are holding 50 stocks for most of the time. The features in both settings are composed of technical factors and fundamental factors, such as moving average convergence/divergence (MACD) [53] and price-to-book ratio (P/B) [7]. They are designed for the prediction at different frequencies and created by different trading firms. Therefore, they possess quite different underlying properties. Since there are more features in this experiment, we use three hidden layers with more neurons (256, 128 and 64 neurons respectively) in the MLP model and 250 trees in the GBM model.

We run the backtests for the models following a rolling scheme described as follows. We re-train the model every week and use the features of the latest 500 trading days (i.e., approximately the latest two years) each time we train the model. The trading period for two settings is from January 2017 to November 2019. For trading details, we exclude the stocks that reach daily surged limit or listed within 3 months. We long the top $N$ stocks with equal weights. The transaction fee plus slippage is 0.3%. We did not particularly consider the impact of holidays and suspension when making predictions and conducting backtest.

Models

In this set of experiments, we compare the DoubleEnsemble variants with a set of baselines.

In terms of sample reweighting, we compare the SR process with SimpleEnsemble and two other heuristic reweighting schemes designed for financial market prediction. Based on the observation that the patterns in the market varies with time, TimeWeighted gives larger weights to more recent samples to encourage the model to exploit current patterns. Also, since we care about the accuracy on the samples that trigger trading signals, the model should pay more attention to the samples that are possibly retrieved. Accordingly, we design and compare to PCTWeighted where the historical samples with high returns are assigned with larger weights. We use PCT to refer to the percentage of price movement, i.e., return. .

In terms of feature selection, we compare the FS process with the baseline that uses fixed manually selected features (Manual) or uses all features without selection (All). The manually selected features are obtained based on a careful analysis on various aspects of the features, such as the historical performance, the information source and the risk. The two set of features (for DAILY and WEEKLY respectively) are used in the real trading and shown to be stable and effective in the real practice.

Performance metrics

Ann.Ret.: We use the hedged annualized return to measure how much return the investment portfolio constructed by the model earned exceeds the market. We divide our daily funds into two equal parts to buy stocks and hedge the market respectively. To hedge the market, we short the corresponding stock index futures. Moreover, we consider the compound return, i.e., $(1+\texttt{Ann.Ret.})^{n}=\texttt{Total.Ret.}$ where Total.Ret. the return during $n$ years.

Sharpe: The Sharpe ratio is one of the most commonly used metrics for stock investment, it reflects the risk adjusted profitability. Specificaly, $\texttt{Sharpe}=\texttt{Ann.Ret.}/\texttt{Ann.Vol.}$, where Ann.Vol. is annualized volatility.

MDD: Maximum drawdown (MDD) is the maximum relative loss from a peak to a trough for a portfolio. MDD is an indicator of downside risk over a specified time period. MDD is related to investors’ maximum affordability and needs to be kept as low as possible.

IC/IR: The information coefficient (IC) and information ratio (IR) indicate the quality of the prediction. In our experiments, we use $\text{IC}_{daily}=\text{corr}(\text{rank}(Y_{pred})/\text{rank}(Y_{true}))$ and $\text{IC}=\text{mean}(\text{IC}_{daily})$, $\text{IR}=\text{IC}/\text{std}(\text{IC}_{daily})$, where $Y_{pred}$ is the prediction and $Y_{true}$ is the truth, $\text{IC}_{daily}$ is the IC for each time step.

Experiment results

We run backtests for the aforementioned models and hedge the systemic risk of the market by holding a short position of the corresponding exchange traded funds (ETF). We plot the hedged equity curves for these models under different settings in Figure 2. We also list the performance measures of the the backtest results in Table II.

In Figure 2, we show four sets of experiments. The four sets of experiments are conducted under different settings (DAILY or WEEKLY) and using different base models (MLP or GBM). The curves in the figure are the hedged equity curves for different models, and the blue bars in the background indicate the information coefficient (IC) of the SR+FS model on each trading day. The information coefficient on a trading day is the Spearman’s rank correlation coefficient between the continuous signals outputted by the model on that trading day and the actual future returns. While the equity curve reflects the prediction accuracy on the top retrieved samples, the information coefficient reflects the prediction accuracy on all the samples

We can see that the performance of SR+FS (the red lines) is better than that of SR+ALL (the orange lines) where all the features are used in each of the sub-models without selection. This indicates the effectiveness of the FS process. However, the automatic feature selection by the FS process is not as good as the manually selected features, which is quite a strong benchmark. We leave it as a future research direction to discover an automatic end-to-end feature selection method that is comparable or better than the manual selection.

Moreover, we observe that the models with the SR process achieve better performances than the models without the SR process (i.e., SimpleEnsemble). This can be observed by comparing the SR+Manual model (green solid line) with the SimpleEnsemble+Manual model (green dashed line) or by comparing the SR+ALL model (orange solid line) with the SimpleEnsemble+ALL (orange dashed line). This indicates that the SR process can improve the performance by paying more attention to the key samples.

At last, we observe that the performance of PCTWeighted and TimeWeighted is even not as good as that of SimpleEnsemble in most of the settings, except that PCTWeighted+Manual is better than SimpleEnsemble+Manual in the WEEKLY setting when using MLP as the base model. Also, the performance of these two reweighting schemes varies largely across different settings or different base models. The effectiveness of paying attention to the near samples or the samples with high future returns depends on the market environment. For example, if the market environment changes quickly, paying attention to the near samples may avoid the interference of the past samples which represent different market patterns. Paying attention to the samples with high future returns corresponds to the emphasis on the positive samples instead of all the samples. This may improve the precision when the market environment is stable. Compared with these two heuristic reweighting schemes, the SR process weights the samples in a self-paced style and therefore is more robust across different settings.

In Table II, we use the hedged annualized return (Ann.Ret.), the Sharpe ratio (Sharpe), the maximum drawdown (MDD), the mean of the ICs (IC) and the information ratio (IR) as the performance measure for the trading strategies. The information ratio is the mean of the ICs divided by the standard variation of the ICs.

We found that DoubleEnsemble (SR+FS) achieves an annualized return of more than 50% with low risk. The Sharpe ratio is near $5.0$ and the maximum drawdown is less than $6.0\%$. This demonstrate that the strategy induced by DoubleEnsemble has a superior and stable performance.

First, we observe that sample reweigting (SR) and feature selection (FS) do not significantly increase the training time. Compared with the training sub-models, the cost of the SR and FS process is negligible. Indeed, each FS process uses the existing model to predict multiple times. However, it does not involve additional training and the prediction time is generally far less than the training time of a sub-model. Second, in financial applications, the model can be trained offline and is embedded in real-time trading systems where latency may lead to slippage. Therefore, we care more about the execution time instead of the training time. In terms of the execution time of the whole process, we find the main constraint is the calculation of factors instead of the model prediction in practice. Moreover, the sub-models in the ensemble can predict in parallel to avoid the additional time cost induced by using an ensemble model.