Astock: A New Dataset and Automated Stock Trading based on Stock-specific News Analyzing Model

In recent years, the use of text-based information, especially news and social media, has significantly improved the performance of stock prediction tasks and these methods usually rely on text-based features and sentiment analysis to forecast stock movements Xu and Cohen (2018); Hu et al. (2018); Ding et al. (2015). However, These approaches assume that the real-trading distribution was the same as the training distribution, which is not realistic as it is difficult to generalize to future trading. By contrast, our self-supervised SRL approach pays closer attention to the quality and comprehensiveness of the news, which could help with out-of-distribution generalization on the realistic trading.

Semantic role labeling (SRL) aims to disclose the predicate-argument structure of a given sentence, which could provide a clear overlay that uncovers the underlying semantics of text Conia et al. (2021). However, previous stock movement prediction methods Xu and Cohen (2018); Hu et al. (2018); Ding et al. (2015) adopted the word or sentence level representation to predict the stock movement. Due to the lack of abstract information of the news, these approaches can overfit the training data and fail to distinguish the key features of news. To deal with this problem, we used the SRL’s characteristics for extracting a clear overlay that uncovers the underlying semantics of news.

Recently, self-supervised learning has become a very popular technique in the training stage of NLP, which generates labels without any human intervention and learns common language representations. Some researches Im et al. (2021); Zheng et al. (2021) have proven that self-supervised learning strengthens the generalization ability for models as it improves the performance in many tasks.

There are two main components in our dataset: News and stock factors for the China stock market. In terms of news data, there are 40,963 pieces of listed company news, including company announcements and company-related news from July 2018 to November 2021. The news data are split into two parts: the In-distribution split and the out-of-distribution split. The in-distribution split is from July 2018 to December 2020 for training and testing where the training set occupies by 80%, and the validation set and test set occupy 10% respectively. The out-of-distribution split is selected from January 2021 to November 2021, which is used for OOD generalization testing. Every piece of news includes its published time and a corresponding news summary. Factor investing is an investment approach that involves targeting quantifiable firm characteristics or factors that can explain the differences in stock returns. Factor-based strategies may help investors meet particular investment objectives—such as potentially improving returns or reducing risk over the long term. Our Astock dataset covers the 24 stock factors on each stock of the China A-shares including Dividend yield, Total share, Circulated share, Free Float share, Market Capitalization, Price-earning ratio, PE for Trailing Twelve Months, Price/book value ratio, Price-to-sales Ratio, Price to Sales ratio, Circulate Market Capitalization, Open price, High price,Low price, Close price , Previous close price, Price change, Percentage of change, Volume, Amount, Turn over rate, Turn over rate for circulated Market Capitalization, Volume ratio. Furthermore, We compare Astock with several widely used stock prediction datasets in Table 1. The value is reflected in the following aspects: (1) Astock provides financial news for each specific stock over the entire China A-shares market. (2) Astock provides various stock factors for each stock. (3) Astock provides minute-level historical prices for the news.

We divide the automated trading system into two tasks: stock movement classification and simulated trading.

In this work, we propose to leverage the off-the-shelf semantic role labeling, i.e., Propbank Kingsbury and Palmer (2003), to pool the output embeddings of a pre-trained language model to construct an alternative representation. The rationale is that the semantic roles in Propbank, i.e., verb (V), proto-agent (A0), and proto-patient (A1), are general-purposed and are also strongly associated with the event arguments. We show an example for semantic role labeling for financial news in Figure 3.

More specifically, we first use the Language Technology Platform (LTP) Che et al. (2020) to automatically mark the semantic roles from the sentences of an entire piece of news and then select V, A0, and A1 to represent the roles for each sentence. Secondly, we process each sentence with a pretrained language model to obtain a sequence of output embeddings $\{\mathbf{s}_{1},\mathbf{s}_{2},\cdots,\mathbf{s}_{n}\}$. We use $\mathcal{V}$, $\mathcal{A}_{0}$ and $\mathcal{A}_{1}$ to denote the indices of tokens corresponding to the V, A0, A1 components. At last, we perform pooling for embeddings with their indices falling into $\mathcal{V}$, $\mathcal{A}_{0}$ and $\mathcal{A}_{1}$. We call this scheme Semantic Role Labelling Pooling SRLP in short. Taking A0 as an example, the SRLP feature for A0 is

For a sentence with $N$ sets of V, A0 and A1, we concatenate $\mathbf{e}_{A0}$,$\mathbf{e}_{A1}$,$\mathbf{e}_{V}$ of each sentence and the financial factor $\mathbf{F}$ of the stock-of-interest into a data matrix:

where $\mathbf{E}$ is output of the above process. Each column of $\mathbf{E}$, denoted as $\mathbf{e}_{j}$, is the concatenation of $\mathbf{e}^{j}_{V},\mathbf{e}^{j}_{A0},\mathbf{e}^{j}_{A1}$ and $\mathbf{F}$. $\mathbf{E}$ is then processed by a Transformer encoder in the same way as the standard text classification to generate the stock movement prediction.

Besides standard supervised training loss for stock movement classification, in this work, we further propose to use a self-supervised training task as an auxiliary task to train the network. For stock prediction, good generalization is highly desirable since the training data is usually sampled from a period different from the test period.

A significant problem in practice is to ensure that our model generalizes to scenarios different from the training set. We further create a self-supervised learning method on top of the SRLP. Recent studies Mohseni et al. (2020); Hendrycks et al. (2019) have shown that incorporating a self-supervised learning task along with the supervised training task could lead to better generalization. As shown in Figure 2, the self-supervised task is defined as predicting the position of one randomly masked SRL role from all the roles of SRL in a piece of news. Intuitively, the self-supervised learning task should be designed to encourage the favorable properties of features. In this work, we propose to randomly mask one pooled embedding, i.e., $\mathbf{e}_{V}^{j}$, $\mathbf{e}_{A0}^{j}$ or $\mathbf{e}_{A1}^{j}$, from a randomly selected sentence, and then ask the network to identify the masked embedding from a pool of candidate embeddings. Such a cloze-style task encourages the network to perform reasoning over other unmasked cues to work out the missing item. We hypothesize that such a reasoning capability is beneficial for understanding the financial news and thus helps stock movement prediction.

Formally, we randomly select a $\mathbf{e}_{j}$ from $\mathbf{E}$ and then select one element from $\mathbf{e}_{j}$ = { $\mathbf{e}_{V}^{j}$, $\mathbf{e}_{A0}^{j}$, $\mathbf{e}_{A1}^{j}$}, after that we replace the selected element with an all-zero vector, indicating a ”mask” operation. Taking masked V at the t-th sentence as an example, we denote the $\mathbf{E}$ after this mask operation as $\mathbf{E}^{\prime}$.

Then we feed $\mathbf{E}^{\prime}$ into the transformer to obtain a query vector sequence $\mathbf{q}\in\mathbb{R}^{d}$

where $[:,t]$ means extract the t-th column of the vector sequences calculated by the transformer. The unmasked SRLP-V features (or SRLP-A0, SRLP-A1 features, depending on which type of SRLP feature is chosen) is also send to an encoder to calculate candidate key vectors: Formally, $\mathbf{K}$ is defined as:

where $f_{V}$ is an encoder specified for encoding V-type SRLP feature. Then the query vector is compared against each column vector in $\mathbf{K}$ and is expected to have the highest matching score at the $t$-th location. This process could be implemented via matrix multiplication and the softmax operation:

and we hope the highest probability entry in Eq. 4 is at the $t$-th dimension. This requirement could be enforced via the cross-entropy loss. Finally, the training loss for the models is

where $\mathcal{L}_{CLS}$ is the cross-entropy loss for the text classification and $\mathcal{L}_{SSL}$ is the cross-entropy loss on the self-supervised learning prediction $P_{SSL}$. $\alpha$ here is a trade-off parameter.

We evaluated the stock movement prediction and simulated trading performance. For the Stock movement prediction, we applied the Accuracy, F1 Score, Recall and Precision as evaluation metrics. For simulated trading, we applied the Annualized Rate of Return, Maximum Drawdown and Sharpe Ration as evaluation metrics based on our simulated trading strategy.

We re-implement the current state-of-art stock movement prediction models as baselines, including StockNet Xu and Cohen (2018), HAN Stock Hu et al. (2018). StockNet Xu and Cohen (2018) is a stock temporally-dependent movement prediction model which also uses Twitter data and price information to predict two classes for stock movement. Hybrid Attention Networks (HAN) Stock Hu et al. (2018) is a stock trend prediction model based on a sequence of recent related news to predict three classes for stock movement task, which are the same as ours. We also construct baselines by formulating the stock movement prediction problem as text classification and use four strong pre-trained Chinese language models as backbones such as XLNet-base-ChineseCui et al. (2020), Sentiment Knowledge Enhanced pre-trained language model(SKEP)Tian et al. (2020), Bert ChineseDevlin et al. (2019) and RoBERTa WWM ExtCui et al. (2020). For the above four pre-trained language models, we extract sentence embedding from the [CLS] token and attach a three-way classifier to predict the stock movements. In addition, we also compare the CSI300 index, XIN9 index⁴⁴4Equivalent to the Standard and Poor’s 500 (S&P 500) or the Dow Jones Industrial Average (DJIA) in the US stock market against the proposed method when analyzing the profitability of the proposed system. For our methods, we used RoBERTa WWM Ext as our backbone PLM since the remarkable performance.

We first compare different methods on the task of stock movement prediction. We conduct experiments on two different splits of our dataset: In-distribution split and out-of-distribution split. In the in-distribution split, both training and testing data are sampled from the same period while the out-of-distribution split uses data from different periods to construct the training and testing data.

The results are shown in Table 2. From the results, we made the following two observations:

1. If only text information is used, the proposed SRLP approach achieves the state-of-the-art performance. Interestingly, we find that SRLP achieves superior performance when further combines the stock factors. It outperforms RoBERTa WWM Ext (News+Factors) by more than 2%. We postulate that this is because the compact representation in SRLP make incorporation of stock factors easier. Note that the proposed way of incorporating stock factors (see Section 3.1) does not only introduce extra modalities for the stock movement prediction but also could make the text analysis module adaptive to the stock factors. This could be useful to model the scenario like the effect of a similar event could result in a different impact on the stock movement for a different type of company.

2. The proposed self-supervised SRLP can further boost the performance of SRLP. In the best setting of self-supervised SRLP, i.e., with V being masked, self-supervised SRLP achieves more than 1% improvement over SRLP. The improvement is even larger when the stock factors are provided, showing more than 2% improvement over SRLP (News+Factors), achieving 66.89% prediction accuracy. This validates the effectiveness of the proposed self-supervised learning approach. Interestingly, we observe that masking A0 and A1 usually will not bring improvement in contrast to the case of masking V. Note that V encodes the type of an event, and the argument is encoded by A0 or A1. It seems that predicting the type of events is a more effective self-supervised learning task than working on the argument.

In the experiments above, the training data and testing data are sampled from the same period. Thus the distributions of training data and testing data are similar. For real-world applications, the stock movement prediction model is applied to future data unseen at the training time. Hence, it is critical to evaluate the model in such an out-of-distribution setting.

To this end, we construct a new training/testing split by using the data from July 2018 to December 2020 as training data and the data from January 2021 to November 2021 for the testing data. We first conduct an evaluation on the stock movement prediction task, and the results are shown in Table 3. From the results, we can see that the proposed method is still comparably competitive over other baselines.

In this section, we discuss the possible profitability of the proposed strategy in real-world trading. We use our trading strategy to conduct trading simulation (backtesting) on stock data from January 2021 to November 2021 using the stock movement prediction result of our model trained from July 2018 to December 2020 as mentioned in Section 3.1. In Table 4, we show that our self-supervised SRLP model achieves a remarkable annualized rate of return of 13.85% , which surpasses the previous baselines and market index XIN9 and CSI300. The resulting baseline HAN Stock Hu et al. (2018) and StockNet Xu and Cohen (2018) achieve an annualized rate of return of -13.5% and -22.42% respectively, and the market XIN9 index and CSI300 were overall declining in 2021, which obtains -15.38 % and -9.34% respectively. In addition, our self-supervised learning method also obtains the lowest Maximum Drawdown of -3.6% and the highest Sharpe Ratio of 40.93% , which significantly outperforms the previous methods and indicates that our self-supervised could successfully achieve higher expected returns while remaining relatively less risky as shown in Figure 4.