A Novel Distributed Representation of News (DRNews) for Stock Market Predictions ^††thanks: We acknowledge the support by 2016 Jiangsu Science and Technology Programme: Young Scholar Programme (No. BK20160391)

To determine the news vectors, our network embedding approach shares a similar idea with the Node2vec model [15]. The goal is to learn a latent feature vector $F(v)$ for each node $v\in V$, that maximizes the probability of predicting the node $v^{\prime}s$ network neighborhood $N(v)$. Hence, the objective function is written as:

Given the feature vector of the node $v$, it is assumed that the prediction probabilities of its neighborhoods are independent of one another, so

where network neighborhoods $N(v)$ for each node are sampled by the biased random walk and concatenated into sequences. These sequences are processed by extending the skip-gram model [31] from dealing with word sequences to node sequences. Specifically, negative sampling [32] is used to approximate the conditional probability:

where $u$ is sampled from the context of node $v$ in the sequence while $u^{{}^{\prime}}$ ($u^{{}^{\prime}}\neq u$) are sampled $m$ times from the noise distribution $P(v)$, which is typically the Unigram Distribution. Each sampling is treated as a simple binary classification to be solved by the corresponding logistic regression classifier.

Different from uniform random walks of DeepWalk [34], Node2vec introduces two classic search strategies, namely the Breadth-first Search (BFS) and Depth-first Search (DFS). Suppose that the walk just transitioned from $t$ to $v$ and is now walking to its next step node $x$, in our network, there are two possible situations for the next step. One is revisiting $t$, i.e., $d_{tx}=0$, whereas the other is keeping going deeper and further away from the node $t$, i.e., $d_{tx}=2$. Node2vec defines a search bias $\alpha$ to control the search preference:

Next, the transition probability of edge $(v,x)$ is defined as the product of the weight of the edge and the bias, i.e., $\frac{\alpha_{vx}*w_{vx}}{Z}$ where $Z$ is the normalization constant. If $p<q$, the walk is width-first associated with a greater probability to revisit the original node. Therefore, it is easier to sample nodes around the original node, that is, semantically similar news as they share many same news elements. If $p>q$, the walk is depth-first, and tends to visit nodes that have not been passed before. As such it is more likely to explore news with latent relations. During random walks, each node is used as a starting vertex for a fixed-length walk, producing a group of sequences. Based on the skip-gram model, node representations are finally trained on these sequences by optimizing the objective function using the Stochastic Gradient Descent (SGD).

To address the major limitations that Node2vec can neither infer unseen nodes easily nor integrate node attributes into node representations, we propose a Subnode model to embed the DRNews network.

Essentially, the Subnode model is inspired by the Subword model [2], which represents each word using a bag of character $n-$grams. As an example, the word $where$ is represented as $G=<wh,whe,her,ere,re>$. Based on the skip-gram architecture, each character $n-$gram can learn a vector representation and the word vector is the summation of its character $n-$grams.

By the biased random walk, sequences of news and event element nodes (such as $news_{A}\to element_{A}\to news_{B}\to element_{B}\to\cdots$) are obtained. Similar to Subword, the proposed Subnode rewrites each node as a bag of node attributes. With the focus on the news nodes, the node attributes are news features including semantic features such as event elements with high tf-idf scores, text structure features such as words count and paragraphs count and side information such as published date, news type and news emotion. Each news vertex $e\in V_{e}$ is thus represented as:

where words count, paragraph count and sentiment are ordinal data rather than real values. Similarly, the representation of a news node no longer directly corresponds to a distinct vector, but is instead the sum of the vectors of all its features. The objective function is now updated to:

where $F(g)$ denotes the vector representation of a news feature $g$. It is worth-mentioning that nodes with only one edge are removed to reduce the sparsity of the network. The formulation of Eq. (7) is similar to Eq. (4), (5) and the vector representations of news features are trained by optimizing the updated objective function using SGD with negative sampling [32]. The pseudo-code of the Subnode algorithm is given in Algorithm 1.

In this way, the Subnode representation of unseen news nodes can be computed easily by extracting its news features, looking up for the corresponding vectors in the $feature\ to\ vector$ dictionary, and summing the vectors up. The algorithm code is open source and avaiblable at Github¹¹1https://github.com/yema2018/News2vec.

We use financial news (2009/10/19 to 2016/10/31) from Sohu³³3https://www.jianshu.com/p/370d3e67a18f to predict the daily movement of Shanghai Securities Composite (SSE) index.

To obtain the news embeddings, vectors of the news features are first obtained according to the news network from 2009/10/19 to 2015/12/31 whereas news vectors are then computed by summing up its associated feature vectors. In particular, the news network is trained by setting the length of the biased random walk as $100$, the context size as $10$, the return hyper-parameter ($p$ in Eq. (6)) and the input hyper-parameter ($q$ in Eq. (6)) both as $1$. Beside the semantic features, five additional features, that attach the basic information of time (three features: $month$, $day$, $weekday$), news sentiment polarity (one feature: $sentiment$)⁴⁴4In this study, the sentiment polarity, is determined by the number of positive words over the number of negative words. and words count (one feature: $words\ count$) are incorporated for each piece of news considered.

Next, we use the previous 20 days ($T=20$) of news and market returns to predict the next day’s movement of the SSE stock index. The output layer is a ternary classifier to indicate whether the index moves UP (daily return $>$ $0.33\%$), DOWN (daily return $<$ $-0.29\%$) or PRESERVES ($-0.29\%$ $<$ daily return $<$ $0.33\%$). We employ a rolling window with $size=20$ and $strider=1$ to augment the sample.

In this study, news and market data before the year 2016 is used as the training set, whilst the rest is used as the test-validation set.

As the baseline models, we include the following news embedding approaches that are used for stock movement predictions. Note that to ensure the validity of comparison, the performance measures of all models are computed from the same input data of daily news text. The other two event embedding models, namely Event2vec [38] and knowledge-enhanced event embedding [13], are not considered as the former could only be used in an organized event data set and the latter requires additional knowledge graphs.
- Average BOW: News is represented as the average of word vectors in the news titles and bodies [22, 9]. Word embedding is obtained by training the skip-gram Word2vec model [31].
- Doc2vec [23]: News text is represented as dense vectors by the Paragraph Vector model [1].
- Event embedding [12]: Event tuples are extracted from headlines and represented as vector representations by scoring the correct tuples higher than the corrupted ones.
- Deepclue [39]: Different from the two-stage prediction of DRNews that returns the news representation and the prediction separately, Deepclue adopts an end-to-end architecture to predict next day’s stock price using words in headlines from the day before. To use Deepclue as a baseline of creating news embeddings, we modify the model to a two-stage task. Specifically, we first use Deepclue to train a news encoder, instead of predicting stock prices. After the news representations are obtained from the trained encoder, we then feed the news vectors into our attention-based LSTM model to make stock market predictions. In this study, we only include two out of three news encoders proposed by Deepclue, namely the CNN and LSTM, as the third encoder based on the bi-gram produces no trainable parameters.
In addition, we apply a sentence-level encoder BERT [10] to encode news headlines as another baseline model:
- BERT: Average pooling is used to get a fixed representation of a headline based on the Chinese pre-trained BERT model⁵⁵5https://github.com/google-research/bert#pre-trained-models.
The proposed DRNews model is implemented with and without the five additional, i.e. $month$, $week$, $sentiment$, $words\ count$ and $day$, features:
- DRNews-: News vectors are the summation of solely the semantic features (i.e., event elements).
- DRNews+: News vectors are the summation of all news features including the semantic and the five additional features.

In Table I, two performance measures, i.e. the Acc and the MCC, are given to describe the predicting power of DRNews in comparison to other news embedding approaches. Specifically, the first panel lists the Acc and MCC of predictions based on each individual embedding model, the second panel gives the prediction result solely using the historical data of return, whilst the last panel shows the prediction with the joint information of the historical return and the DRNews embedding. Major implications of Table I are threefold.

First, it is shown that stock movement predictions solely based on the embedded information of news (Panel 1) outperform the predictions taking into account the historical data of market return (Panel 2 and 3). This finding suggests that daily news embeddings contain sufficient information for short-term stock movement prediction, at the same time indicates that the SSE stock market complies the weak-form of efficiency as the historical price data makes no contribution to the prediction of its future movements.

Second, the DRNews model outperforms the baseline models with the most prominent Acc and MCC results in a lower dimension of $d=128$ ⁶⁶6In the experiment of stock movement prediction, the three best-performing models are DRNews+ (higest Acc and MCC), BERT (Second highest Acc) and DRNews- (Second highest MCC). The dimension of BERT is $764$.. As for news-driven stock price prediction, it is more reasonable to take into account potential connections between news, such as sharing similar background story or leading to the same event, and reflect the connection by embedding similar values in certain dimensions of the news vector. Through the biased random walk in the news network, DRNews embeddings capture inter-textual knowledge, i.e. latent connections between news, which enriches the news vectors with vital information that contributes to the high predictive power of the model. On the other hand, Doc2vec, Event embedding and BERT produce stock movement predictions of similar qualities in terms of MCC, as the news vectors generated by the three approaches uniformly limit to the textual information within news articles.

Last, the DRNews algorithm is competent to enrich the news vector representation with additional information brought by supplementary features, as the Acc and MCC results of DRNews+ improve by $1.56\%$ and $2.72\%$ in comparison to DRNews- after incorporating the five labeled features ($month$, $week$, $day$ $sentiment$, and $words\ count$). It is worth-mentioning that in this study, the New2vec model is performed in an unsupervised manner as all features are extracted from common news text without any artificial label or trained classifier. Therefore, we believe that there is a large chance to further improve the result once higher quality labels, such as news sentiment from trained classifier or subcategories of financial news, are incorporated. Despite that the performance of Deepclue is hardly satisfactory, we believe that this supervised news embedding model has great potential once a stronger news encoder and supervised tasks are placed to enhance the capacity of feature extraction and the universality of embedding.

In this section, we use an attention-based LSTM network to construct an early warning model for stock market crises. To classify the crisis/non-crisis dates of the SSE index, the switching ARCH (SWARCH) model [18] is hired through the modeling of high/low volatility regimes that reflect respectively the tranquility and turmoil [44]. To determine whether a crisis occurs, a SWARCH(2,1) process is written as:

Eq.(20) describes an AR(1) process with a normal error term $\epsilon_{t}$ of variance $h_{t}$. The regime switching structure of the residual variance $h_{t}$ is given by Eq.(21) where the $\alpha^{\prime}s$ are non-negative, the $\gamma^{\prime}s$ are scaling parameters that capture the change in each regime, $s_{t}$ is the state variable that $s_{t}=1$ indicates the low volatility state of non-crisis, and $s_{t}=2$ indicates the high volatility state of crisis.

The classification of high/low volatility regimes can be implemented on the basis of the so-called filtering probability, which is a byproduct of the maximum likelihood estimation. The filtering probability, written as:

where $\boldsymbol{\theta}$ is the vector of all model parameters to be estimated, could be interpreted as the conditional probability based on the current information of time $t$. The crisis classifier is defined as the following binary function with a probability threshold $0.5$ [17, 16]:

By inputting news embeddings and historical price data, we aim to investigate the performance of DRNews in news-driven crisis predictions in comparison with other embedding models. Similar to the stock trend prediction, we use the news vectors and market return of the first 20 days to predict whether a stock market turbulence will occur in the next day. The difference is that the crash warning system is basically a binary classifier. Besides, a rolling window with $size=20$ and $strider=1$ is employed to augment the sample. Finally, the time interval for training is between 2009/10/19 and 2013/4/25 and the interval for testing is from 2013/4/26 to 2016/10/31.

According to the Acc and MCC results of the six embedding approaches shown in Panel 1, DRNews+ remains as the best-performing embedding model in terms of both Acc and MCC, whilst BERT produces the second highest Acc and DRNews- produces the second highest MCC. As a concluding remark, the DRNews model demonstrates robust and strong performances in the completion of the two financial prediction tasks, i.e. stock movement prediction and stock crises warning, that requires the representation of latent linkages between news events.

On the other hand, Panel 2 and 3 show that the results of crises warning is significant improved after historical return data is employed. Despite that the market-driven model already offers a fairly good prediction with $94.25\%$ Acc and $87.76\%$ MCC, the joint prediction of DRNews embedding and historical return generates the highest Acc and MCC of $94.64\%$ and $88.61\%$, respectively.

Figure 5 and 6 show the predicted SWARCH filter probability of crises based on the historical return with and without DRNews embeddings, respectively. Despite that the market-driven crises warning based on solely the historical return achieves impressive results in terms of Acc and MCC measures, it creates a delay of warning (see Oct 26, 2014 to Jan 26, 2015, red line in Figure 5) in comparison to the actual occurrence of the crisis (blue line in Figure 5). On the other hand, Figure 6 suggests that the joint prediction of DRNews embedding and historical return generates true ’early’ warning signals with predicted probability of crisis gradually raises at the end of 2014 before the actual crash. It appears that DRNews embeddings contain key information that serves as the driving force of crisis alarming during the pre-turmoil episode. In the next section, the key information inherited in the news will be discussed explicitly according to the attention-based LSTM model based on the DRNews embeddings.

Table II confirms the findings in Figure 5 and 6. During the test-set period between April 2013 to October 2016, two stock crisis onsets are detected by the SWARCH model. The joint predictions based on the historical return and DRNews embeddings forewarn both onsets with an average of $2.5$-days forewarned period, whereas predictions with solely the historical return demonstrate a delay in warning the second onset that take place at the end of 2014.