Pruned Wasserstein Index Generation Model and wigpy Package

A major component of WIG model is the Wasserstein Dictionary Learning (Schmitz \BOthers., \APACyear2018). Given a set of document $Y=\left[y_{m}\right]\in\mathbb{R}^{N\times M}$, each doc $y_{m}\in\Sigma^{N}$ is associated with a timestamp and $N,M$ are length of dictionary and number of documents in corpus, respectively. Our first step is to cluster documents into topics $T=\left[t_{k}\right]\in\mathbb{R}^{N\times K},$ where $K\ll M,$ and associated weights $\Lambda=\left[\lambda_{m}\right]\in\mathbb{R}^{K\times M}.$ Thus, for a single document $y_{m},$ we could represent it as $y_{m}\approx t_{k}\lambda_{m}.$ Documents and topics lie in $N$-dimensional simplex, and are word distributions. Another important quantity for computing WIG, is the cost matrix $C^{N\times N}$ and $C_{ij}=d^{2}\left(x_{i},x_{j}\right),$ where each $x_{i}\in\mathbb{R}^{1\times D}$ is the $D$-dimensional word embedding vector for the $i$ -th word in the vocabulary. In other words, matrix $C$ measures the “cost” of moving masses of words, and now we can proceed and define the Sinkhorn Distance.

We could then set up the loss function and minimization problem as follows:

By this formula, we wish to minimize the divergence between original document $y_{m}$ and the predicted (reconstructed) $y_{\mathcal{S}_{\mathcal{E}}}(.)$ given by Sinkhorn distance. Moreover, the constaints of this minimization problem considers Softmax operation on each of the columns of the matrices $R$ and $A$, so that $T$ and $\Lambda$ will be (column-wise) discrete densities, as is required by the Sinkhorn distance.

For computation, we first initialize matrices $R$ and $A$ by drawing from Standard Normal distribution and then perform Softmax to obtain $T$ and $\Lambda.$ During training process, we keep track of computational graph and obtain the gradient $\nabla_{T}\mathcal{L}(\cdot;\varepsilon)$ and $\nabla_{\Lambda}\mathcal{L}(\cdot;\varepsilon)$ with respect to $T$ and $\Lambda.$ $R$ and $A$ are then optimized by Adam optimizer (Kingma \BBA Ba, \APACyear2015) after each batch, and the automatic differentiation is done by PyTorch framework (Paszke \BOthers., \APACyear2017).

After conducting WDL on documents for clustering, the next step of WIG would be to generate time-series indices based on the topics. The model first reduce each topic vector $t_{k}$ to a scalar by SVD and then multiply the weight matrix to get document-wise sentiment score for the whole corpus. We then add up the scores for each month and then produce the final monthly index.

Although enjoying many nice theoretical properties (Villani, \APACyear2003), the computation for Optimal Transport has been known for its complexity. This burden has been eased by Cuturi (\APACyear2013) and it has attracted much attention in machine learning community since then.

However, there are still two aspects that hindering our application to textual analysis. First of all, vocabulary will easily go to a very large one, and the computation for Sinkhorn loss will soon become prohibitive. Moreover, after passing a certain point, it not even possible to fit the distance matrix $C$ into the memory, especially when considering the limited VRAM for GPU acceleration.³³3 My configuration is Nvidia 1070Ti (8G). Under single presicion, each digit will occupy 4 byte and in my case, I can only fit, theoretically at most, a square matrix of dimension 44,721. I have a relative small dataset from The New York Times and my vocabulary is of length 9437, but many NLP applications will have much more tokens than I do. In such a case, the WIG model will become infeasible.

I therefore propose the following procedure to reduce the vocabulary dimension and could avoid feeding the full vocabulary matrix into WIG model. It first clusters all word vectors by $k$-means clustering, and then selects a subset of tokens from each of the cluster to form “base tokens.”⁴⁴4 The number of tokens to be considered as “base tokens” is arbitrary, meaning that the compression ratio could potentially be made arbitrarily small. In other words, the researcher could choose such a number that the model can be fitted into the memory of her machine, regardless of the number of tokens she had for the corpus. And that is exactly the way why we need to compress the dictionary by “pruning” some non-important tokens. We could then use Lasso⁵⁵5 A similar approach (Mallapragada \BOthers., \APACyear2010) was proposed using group-LASSO to prune visual vocabulary, but in the area of image processing. to regress word vectors of all other tokens on the vectors of these “base tokens” to ensure sparse weight vector, which will have zero component on non-import features.

Formally speaking, we set up the following minimization problem for the $k$ -means clustering:

where $\mu_{k}$ is the mean of points in cluster $\mathcal{K}_{k}$ and $k\in\{1,\cdots,K\}.$ We can certainly choose some most frequent tokens from each cluster to form a final subset whose length matches our desire. ⁶⁶6 A very simple choice would be Word per Cluster $=\frac{\text{ Maximum Vocabulary Length }}{\text{ Number of Clusters }}$ By doing so, we also represent the whole vocabulary by the most representative tokens. The indices for these “base tokens” are collected in the index set,

Obviously, $\mathfrak{B}^{C}$ is also defined by excluding “base tokens” from the whole vocabulary. $N$ is the size of vocabulary and $x_{b}$ is the $b$-th token in the vocabulary. Denote word vector for “base tokens” as $v_{b}$ and others as $v_{o},$ we have

For each $o,$ we will have a vector $\alpha_{o,b}$ of length $B,$ where $B$ is the dimension of “base vocabulary”.

Previously in the WIG model, we obtain the word distribution for each single document $y_{m}$ by calculating its word frequency, and that will give us a $N$ -dimensional distribution vector. Here, in the pWIG variant, we replace the non-base tokens by weighted base-tokens and could thus represent the word simplex of documents in only $B$ -dimensional spaces.

Now that we have successfully represent our dataset in $s$ smaller vocabulary, we could proceed to define our distance matrix $C_{ij}=d^{2}\left(x_{i},x_{j}\right),$ where $i,j\in\mathfrak{B}.$ Here we have everything we need for the regular WIG model and we fit it using the shrinkage transformed word distributions and distance matrix.

To carry out the computation of WIG and pWIG model, I also provide the wigpy package under MIT license. Notice that the WIG model in wigpy package is a new implementation, though part of the codes are modified from the codes of original WIG paper.

The main model is wrapped in the “WIG” class, where it contains a set of hyper-parameters ⁷⁷7 For example, embedding depth ($emsize$), batch size ($batch\_size$), number of topics ($num\_topics$), sinkhorn regularization weight ($reg$), optimizer learning rate ($lr$) ($L2$ penalty for optimizer ($wdecay$), L1/LASSO weight ($l1\_reg$), maximum number of tokens allowed by pWIG algorithm ($prune\_topk$). to tune the model, and some parameters to control the behavior of preprocessing and Word2Vec training process.

Notice that the previous implementation of WIG model only supports hand-written Adam optimizer, and the optimization for document weights were optimized column-wise. In other words, each document will only be used to update the column of weight in matrix $\Lambda$ for that given document. The new implementation wraps the whole model in PyTorch, providing many optimizers to choose by PyTorch optimizer class. What is more, each document will accumulate gradient and the whole $\Lambda$ matrix will be updated all together.

To test for the pWIG model’s performance, I run the model on the same dataset from the WIG paper. It consists of news headlines collected from The New York Times from 1980 to 2018. As I am implementing a new version of WIG, as provided by the wigpy module, I run the original WIG model and report its result as well.

I run both variants of WIG model separately, by calling wigpy package, to set for hyperparameters by splitting training, evaluation, and testing data as $60\%,10\%,$ and $30\%$ respectively.

For the original WIG, hyper-parameters are chosen as follows: depth of embedding $D=50$ batch size $s=32,$ number of topics $K=4,$ learning rate for Adam $\rho=0.001,$ Sinkhorn regularization weight $\varepsilon=0.1;$ for the pWIG, depth of embedding $D=50,$ batch size $s=$ 64, number of topics $K=4,$ learning rate for Adam $\rho=0.001,$ Sinkhorn regularization weight $\varepsilon=0.08$ I also report Pearson’s and Spearman’s correlation test on four set of automatically generated EPU indices (one LDA-based EPU (Azqueta-Gavaldón, \APACyear2017), one WIG-based EPU (Xie, \APACyear2020), and two flavor of WIG given by wigpy package in this paper), against the original EPU ⁸⁸8 https://www.policyuncertainty.com/. (Baker \BOthers., \APACyear2016).

Apparently, as is shown in Table $1,$ all three WIG methods outperforms LDA-based method by $3\%$ in Pearson’s test and more than $2\%$ in Spearman’s test. This fact has been established by the previous WIG paper. Moreover, if we compare results within three WIG-related methods, this new implementation of original WIG in wigpy package shows better result than the previous implementation. The pruning method does not differ much from the new implemented WIG algorithm, and is even better than the previous implementation of original WIG!

In Table 2, the correlation statistics between EPU generated by WIGs and two other indices: VIX and Michigan Consumer Confidence Sentiment index. As reported (Baker \BOthers., \APACyear2016), EPU has a correlation of 0.58 between VIX and -0.742 between Michigan index. Since our objective is to produce a similar index of EPU, but using an automatic approach, we should expect our WIG-based EPU to have a similar relationship with these other two indices. This is indeed the case here, and we can certainly observe the positive and negative relationship when comparing the VIX and Michigan indices¹¹¹¹11 It may be confusing why the “sentiment index” generated by WIG models has a negative relationship with “Michigan Consumer Sentiment index,” since both names contain “sentiment.” However, there is a clear distinction of the usage of the same word in two different contexts. The famous Michigan index is expressed as the consumer confidence levels, and the higher the index, the more confident the consumers are. The word “sentiment”, as used by WIG, is to capture the subjective information expressed in the texts. In the application of EPU, it is used to capture the intensity of opinions towards the uncertainty of policy, as conveyed by newspaper articles. It is very obvious that what it captures is negative feelings, and the higher the index, the more uncertain that people feel. In other words, although bearing the same word “sentiment” in their names, the underlying element is strikingly different and thus show a negative relationship between each other. Moreover, the WIG model does not limit its use in EPU. As soon as we apply the WIG models to other (textual) datasets, the meaning of “sentiment” will be changed accordingly. In total, the word “sentiment” used in WIG models is more versatile and should be distinguished from the usage as in the Michigan index. .