Automatic Domain Adaptation Outperforms Manual Domain Adaptation for Predicting Financial Outcomes

In this paper, we adopt Ordinary Least Squares (OLS), a common research method in empirical finance: a dependent variable of interest (e.g., excess return, volatility) is predicted based on a linear combination of a set of explanatory variables.

The main focus of this paper is to investigate text-based explanatory variables: we would like to know to what extent a text variable such as occurrence of negative words in a 10-K can predict a financial variable like volatility. Identifying the economic drivers of such a financial outcome is of central interest in the field of finance. Some of these determinants may be correlated with sentiment. To understand the role of sentiment in explaining financial variables we therefore need to isolate the complementary information of our text variables. This is achieved by including in our regressions – as control variables – a standard set of financial explanatory variables such as firm size and book-to-market ratio. These control variables are added as additional explanatory variables in the regression specification besides the textual sentiment variables. This experimental setup allows us to assess the added benefit of text-based variables in a realistic empirical finance scenario.

The approach is motivated by previous studies in the finance literature (e.g., Loughran and Mcdonald (2011)), which show that characteristics of financial firms can explain variation in excess returns and volatility. By including these control variables in the regression we are able to determine whether sentiment factors have incremental explanatory power beyond the already established financial factors. Since the inclusion of these control variables is not primarily driven by the assumption that firms with different characteristics use different language, our approach differs from other NLP studies, such as Hovy (2015), who accounts for non-textual characteristics by training group-specific embeddings.

Each text variable we use is based on a dictionary. Its value for a 10-K is the proportion of tokens in the 10-K that are members of the dictionary. For example, if the 10-K is 5000 tokens long and 50 of those tokens are contained in the L&M uncertainty dictionary, then the value of the L&M uncertainty text variable for this 10-K is 0.01.

In the type of analysis of stock market data we conduct, there are two general forms of dependence in the residuals of a regression, which arise from the panel structure of our data set where a single firm is repeatedly observed over time and multiple firms are observed at the same point in time. Firm effect: Time-series dependence assumes that the residuals of a given firm are correlated across years. Time effect: Cross-sectional dependence assumes that the residuals of a given year are correlated across different firms. These properties violate the i.i.d. assumption of residuals in standard OLS. We therefore model data with both firm and time effects and run a two-way robust cluster regression, i.e., an OLS regression with standard errors that are clustered on two dimensions Gelbach et al. (2009), the dimensions of firm and time.⁴⁴4Loughran and Mcdonald (2011) use the method of Fama and MacBeth (1973) instead. This method assumes that the yearly estimates of the coefficient are independent of each other. However, this is not true when there is a firm effect. We apply this regression-based methodology to test the explanatory power of financial dictionaries with regard to two dependent variables: excess return and volatility. This approach allows us to compare the explanatory power of different sentiment dictionaries and in the process test the hypothesis that negative sentiment is associated with subsequently lower stock returns and higher volatility. We now introduce the regression specifications for these tests.

There are two main questions we want to answer:

     Q1. Is a manually domain-adapted or an automatically domain-adapted dictionary a more effective predictor of financial outcomes?

     Q2. L&M adapted H4N for the financial domain and showed that this manually adapted dictionary is more effective than H4N for prediction. Can we further improve L&M’s manual adaptation by automatic domain adaptation?

The general methodology we employ for domain adaptation is based on word embeddings. We train CBOW word2vec (Mikolov et al., 2013) word embeddings on a corpus of 10-Ks for all words of H4N that occur in the corpus – see §4 for details. We consider two adaptations: ADD and RE. ADD is only used to answer question Q2.

ADD. For adapting the L&M dictionary, we train an SVM on an L&M dictionary in which words are labeled +1 if they are marked for the category by L&M and labeled -1 otherwise (where the category is negative, uncertain or litigious). Each word is represented as its embedding. We then run the SVM on all H4N words that are not contained in the L&M dictionary. We also ignore H4N words that we do not have embeddings for because their frequency is below the word2vec frequency threshold. Thus, we obtain an ADD dictionary which is not a superset of the L&M lexicon because it includes only new additional words that are not part of the original dictionary.

SVM scores are converted into probabilities via logistic regression. We define a confidence threshold $\theta$ – we only want to include words in the ADD dictionary that are reliable indicators of the category of interest. A word is added to the dictionary if its converted SVM score is greater than $\theta$.

RE. We train SVMs as for ADD, but this time in a five-fold cross validation setup. Again, SVM scores are converted into probabilities via logistic regression. A word $w$ becomes a member of the adapted dictionary if its converted SVM score of the SVM that was not trained on the fold that contains $w$ is greater than $\theta$.

To answer our first question Q1: “Is automatic or manual adaptation better?”, we apply adaptation method RE to H4N and compare the results to the L&M dictionaries.

To answer our second question Q2: “Can manual adaptation be further improved by automatic adaptation?”, we apply adaptation methods RE and ADD to the three dictionaries compiled by L&M and compare results for original and adapted L&M dictionaries: (i) negative (abbreviated as “neg”), (ii) uncertain (abbreviated as “unc”), (iii) litigious (abbreviated as “lit”). Our goals here are to improve the in-domain L&M dictionaries by relabeling them using adaptation method RE and to find new additional words using adaptation method ADD.

Table 1 gives dictionary sizes.

Table 2 gives regression results for excess return, comparing H4N${}_{\hbox{RE}}$ (our automatic adaptation of the general Harvard dictionary) with the three manually adapted L&M dictionaries. As expected the coefficients are negatively signed – 10-Ks containing a high percentage of pessimistic words are associated with negative excess returns.

L&M designed the dictionary neg${}_{\hbox{lm}}$ specifically for measuring negative information in a 10-K that may have a negative effect on outcomes like excess return. So it is not surprising that neg${}_{\hbox{lm}}$ is the best performing dictionary of the three L&M dictionaries: it has the highest standard coefficient (-0.080) and the highest significance (-2.56). unc${}_{\hbox{lm}}$ performs slightly worse, but is also significant.

However, when comparing the three L&M dictionaries with H4N${}_{\hbox{RE}}$, the automatically adapted Harvard dictionary, we see that H4N${}_{\hbox{RE}}$ performs clearly better: it is highly significant and its standard coefficient is larger by a factor of more than 2 compared to neg${}_{\hbox{lm}}$. This evidence suggests that the automatically created H4N${}_{\hbox{RE}}$ dictionary has a higher explanatory power for excess returns than the manually created L&M dictionaries. This provides an initial answer to question Q1: in this case, automatic adaptation beats manual adaptation.

Table 3 shows manual plus automatic experiments with multiple text variables in one regression, in particular, the combination of H4N${}_{\hbox{RE}}$ with each of the L&M dictionaries. We see that the explanatory power of L&M variables is lost after we additionally include H4N${}_{\hbox{RE}}$ in a regression: all three L&M variables are not significant. In contrast, H4N${}_{\hbox{RE}}$ continues to be significant in all experiments, with large standard coefficients. More manual plus automatic experiments can be found in the appendix. These experiments further confirm that automatic is better than manual adaptation.

Table 4 shows results for automatically adapting the L&M dictionaries.¹²¹²12Experiments with multiple text variables in one regression (manual plus automatic experiments) are presented in the appendix. The subscript “RE+ADD” refers to a dictionary that merges RE and ADD; e.g., neg${}_{\hbox{RE+ADD}}$ is the union of neg${}_{\hbox{RE}}$ and neg${}_{\hbox{ADD}}$.

We see that for each category (neg, lit and unc), the automatically adapted dictionary performs better than the original manually adapted dictionary; e.g., the standard coefficient of neg${}_{\hbox{RE}}$ is -0.111, clearly better than that of neg${}_{\hbox{lm}}$ (-0.080). Results are significant for neg${}_{\hbox{RE}}$ (-2.96) and unc${}_{\hbox{RE}}$ (-2.77). We also evaluate neg${}_{\hbox{spec}}$, the negative word list of Hamilton et al. (2016). neg${}_{\hbox{spec}}$ does not perform well: it is not significant.

These results provide a partial answer to question Q2: for excess return, automatic adaptation of L&M’s manually adapted dictionaries further improves their performance.

Table 5 compares H4N${}_{\hbox{RE}}$ and L&M regression results for volatility. Except for litigious, the coefficients are positive, so the greater the number of pessimistic words, the greater the volatility.

Results for neg${}_{\hbox{lm}}$, unc${}_{\hbox{lm}}$ and H4N${}_{\hbox{RE}}$ are statistically significant. The best L&M dictionary is again neg${}_{\hbox{lm}}$ with standard coefficient 0.0472 and $t=3.30$. However, H4N${}_{\hbox{RE}}$ has the highest explanatory value for volatility. Its standard coefficient (0.173) is more than three times as large as that of neg${}_{\hbox{lm}}$.

The higher effect size demonstrates that H4N${}_{\hbox{RE}}$ better explains volatility than the L&M dictionaries. Again, this indicates – answering question Q1 – that automatic outperforms manual adaptation. Table 6 confirms this. We see that for manual plus automatic experiments each combination of H4N${}_{\hbox{RE}}$ with one of the L&M dictionaries provides significant results for H4N${}_{\hbox{RE}}$. In contrast, L&M dictionaries become negatively signed meaning that more uncertain words decrease volatility, suggesting that they are not indicative of the true relationship between volatility and negative tone in 10-Ks in this regression setup. Our results of additional manual plus automatic experiments support this observation as well. See the appendix for an illustration.

Table 7 gives results for automatically adapting the L&M dictionaries.¹³¹³13Experiments with multiple text variables in one regression (manual plus automatic experiments) are presented in the appendix. For neg, the standard coefficient of neg${}_{\hbox{RE}}$ is 0.0657, better by about 40% than neg${}_{\hbox{lm}}$’s standard coefficient of 0.0472. neg${}_{\hbox{spec}}$ does not provide significant results and has the negative sign, i.e., an increase of negative words decreases volatility. The lit dictionaries are not significant (neither L&M nor adapted dictionaries). For unc, unc${}_{\hbox{RE}}$ performs worse than unc${}_{\hbox{lm}}$, but only slightly by 0.0344 vs. 0.0356 for the standard coefficients. The overall best result is neg${}_{\hbox{RE}}$ (standard coefficient 0.0657). Even though L&M designed the unc${}_{\hbox{lm}}$ dictionary specifically for volatility, our results indicate that neg dictionaries perform better than unc dictionaries, both for L&M dictionaries (neg${}_{\hbox{lm}}$) and their automatic adaptations (e.g., neg${}_{\hbox{RE}}$).

Table 7 also evaluates unc${}_{\hbox{spec}}$, the uncertainty dictionary of Theil et al. (2018). unc${}_{\hbox{spec}}$ does not perform well: it is not significant and the coefficient has the “wrong” sign.¹⁴¹⁴14Theil et al. (2018) define volatility for the time period [6 28] whereas our definition is [6 252], based on (Loughran and Mcdonald, 2011). Larger time windows allow more reliable estimates and account for the fact that information disclosures can influence volatility for long periods (Belo et al., 2016).

The main finding supported by Table 7 is that the best automatic adaptation of an L&M dictionary gives rise to more explanatory power than the best L&M dictionary, i.e., neg${}_{\hbox{RE}}$ performs better than neg${}_{\hbox{lm}}$. This again confirms our answer to Q2: we can further improve manual adaptation by automatic domain adaptation.

Our dictionaries outperform L&M. In this section, we perform a qualitative analysis to determine the reasons for this discrepancy in performance.

Table 8 shows words from automatically adapted dictionaries. Recall that the ADD method adds words that L&M classified as nonrelevant for a category. So words like “missing” (neg), “reevaluate” (unc) and “assignors” (lit) were classified as relevant terms and seem to connote negativity, uncertainty and litigiousness, respectively, in financial contexts.

In L&M’s classification scheme, a word can be part of several different categories. For instance, L&M label “unlawful”, “convicted” and “breach” both as litigious and as negative. When applying our RE method, these words were only classified as negative, not as litigious. Similarly, L&M label “confusion” as negative and uncertain, but automatic RE adaptation labels it only negative. This indicates that there is strong distributional evidence in the corpus for the category negativity, but weaker distributional evidence for litigious and uncertain. For our application, only “negative” litigious/uncertain words are of interest – “acquittal” (positive litigious) and “suspense” (positive uncertain) are examples of positive words that may not help in predicting financial variables. This could explain why the negative category fares better in our adaptation than the other two.

An interesting case study for RE is “abeyance”. L&M classify it as uncertain, automatic adaptation as litigious. Even though “abeyance” has a domain-general uncertain sense (“something that is waiting to be acted upon”), it is mostly used in legal contexts in 10-Ks: “held in abeyance”, “appeal in abeyance”. The nearest neighbors of “abeyance” in embedding space are also litigious words: “stayed”, “hearings”, “mediation”.

H4N${}_{\hbox{RE}}$ contains 74 words that are “common” in H4N. Examples include “compromise”, “serious” and “god”. The nearest neighbors of “compromise” in the 10-K embedding space are the negative terms “misappropriate”, “breaches”, “jeopardize”. In a general-domain embedding space,¹⁵¹⁵15https://code.google.com/archive/p/word2vec/ the nearest neighbors of “compromise” include “negotiated settlement”, “accord” and “modus vivendi”. This example suggests that “compromise” is used in 10-Ks in negative contexts and in the general domain in positive contexts. This also illustrates the importance of domain-specific word embeddings that capture domain-specific information.

Another interesting example is the word “god”; it is frequently used in 10-Ks in the phrase “act of God”. Its nearest neighbors in the 10-K embedding space are “terrorism” and “war”. This example clearly demonstrates that annotators are likely to make mistakes when they annotate words for sentiment without seeing their contexts. Most annotators would annotate “god” as positive, but when presented with the typical context in 10-Ks (“act of God”), they would be able to correctly classify it.

We conclude that manual annotation of words without context based on the prior belief an annotator has about word meanings is error-prone. Our automatic adaptation is performed based on the word’s contexts in the target domain and therefore not susceptible to this type of error.

Table 9 presents a quantitative analysis of the distribution of words over dictionaries. For a row dictionary $d_{r}$ and a column dictionary $d_{c}$, a cell gives $|d_{r}\cap d_{c}|/|d_{r}|$ as a percentage. (Diagonal entries are all equal to 100% and are omitted for space reasons.) For example, 49% of the words in neg${}_{\hbox{lm}}$ are also members of neg${}_{\hbox{RE}}$ (row “neg${}_{\hbox{lm}}$”, column “neg${}_{\hbox{RE}}$”). This analysis allows us to obtain insights into the relationship between different dictionaries and into the relationship between the categories negative, litigious and uncertain.

Looking at rows neg${}_{\hbox{lm}}$, lit${}_{\hbox{lm}}$ and unc${}_{\hbox{lm}}$ first, we see how L&M constructed their dictionaries. neg${}_{\hbox{lm}}$ words come from H4N${}_{\hbox{neg}}$ and H4N${}_{\hbox{cmn}}$ in about equal proportions; i.e., many words that are “common” in ordinary usage were classified as negative by L&M for financial text. Relatively few lit${}_{\hbox{lm}}$ and unc${}_{\hbox{lm}}$ words are taken from H4N${}_{\hbox{neg}}$, most are from H4N${}_{\hbox{cmn}}$. Only 12% of neg${}_{\hbox{lm}}$ words were automatically classified as negative in domain adaptation and assigned to H4N${}_{\hbox{RE}}$. This is a surprisingly low number. Given that H4N${}_{\hbox{RE}}$ performs better than neg${}_{\hbox{lm}}$ in our experiments, this statistic casts serious doubt on the ability of human annotators to correctly classify words for the type of sentiment analysis that is performed in empirical finance if the actual corpus contexts of the words are not considered. We see two types of failures in the human annotation. First, as discussed in §5.1, words like “god” are misclassified because the prevalent context in 10-Ks (“act of God”) is not obvious to the annotator. Second, the utility of a word is not only a function of its sentiment, but also of the strength of this sentiment. Many words in neg${}_{\hbox{lm}}$ that were deemed neutral in automatic adaptation are probably words that may be slightly negative, but that do not contribute to explaining financial variables like excess return. The strength of sentiment of a word is difficult to judge by human annotators. Looking at the row H4N${}_{\hbox{RE}}$, we see that most of its words are taken from neg${}_{\hbox{lm}}$ (79%) and a few from lit${}_{\hbox{lm}}$ and unc${}_{\hbox{lm}}$ (2% each). We can interpret this statistic as indicating that L&M had high recall (they found most of the reliable indicators), but low precision (see the previous paragraph: only 12% of their negative words survive in H4N${}_{\hbox{RE}}$). The distribution of H4N${}_{\hbox{RE}}$ words over H4N${}_{\hbox{neg}}$ and H4N${}_{\hbox{cmn}}$ is 78:22. This confirms the need for domain adaptation: many general-domain common words are negative in the financial domain.

We finally look at how dictionaries for negative, litigious and uncertain overlap, separately for the L&M, ADD and RE dictionaries. lit${}_{\hbox{lm}}$ and unc${}_{\hbox{lm}}$ have considerable overlap with neg${}_{\hbox{lm}}$ (17% and 14%), but they do not overlap with each other. The three ADD dictionaries – neg${}_{\hbox{ADD}}$, lit${}_{\hbox{ADD}}$ and unc${}_{\hbox{ADD}}$ – do not overlap at all. As for RE, 10% of the words of unc${}_{\hbox{RE}}$ are also in neg${}_{\hbox{RE}}$, otherwise there is no overlap between RE dictionaries. Comparing the original L&M dictionaries and the automatically adapted ADD and RE dictionaries, we see that the three categories – negative, litigious and uncertain – are more clearly distinguished after adaptation. L&M dictionaries overlap more, ADD and RE dictionaries overlap less.