Are Word Embedding Methods Stable and Should We Care About It?

We performed experiments using four publicly available real-world datasets. Please refer to Table 2 for the statistics about the datasets. These datasets represent diverse styles of natural language usage. Online collaborative writing style is covered in our Wikipedia (Sample) dataset²²2https://dumps.wikimedia.org/enwiki/. It contains a subset of English Wikipedia articles from October 2017 Wikipedia dump. News articles are covered in the NewsCrawl(2007) dataset³³3http://www.statmt.org/wmt16/translation-task.html. We have used the News Crawl articles for the year 2007. The Lyrics dataset represents creative writing style⁴⁴4Datasets from (Barman et al., 2019) and(Fell and Sporleder, 2014). It contains over 400K English songs of around 7.2K artists from the period 1938 to 2016. Formal parliamentary communication is covered in our Europarl (Sample) dataset⁵⁵5http://www.statmt.org/wmt16/translation-task.html. It includes the English version of the proceedings of Bulgarian, Czech, Slovak, Slovene, Romanian and Polish parliaments. Our Lyrics dataset had around fifty million tokens. We sampled other datasets to match the size of our Lyrics dataset. We have deliberately chosen these diverse corpora that vary in corpus parameters like size of vocabulary and topics to assess our models with realistic examples. We have pre-processed these corpora by tokenizing sentences and removing non-alphabetic characters and converting all characters to lower cases. We have also removed the stop words by using the stop words corpora of NLTK ⁶⁶6http://www.nltk.org/. We have also considered a minimum word occurrence of five and removed all words that appear less than five times in a corpus.

Evaluating all the WEMs that are currently available is beyond the scope of this work. We focused on three popular WEMs: Word2Vec, GloVe, and fastText. For Word2Vec, we used the Continuous Bag of Words (CBOW) variant from the Gensim ⁷⁷7https://github.com/RaRe-Technologies/gensim/tree/develop/gensim/ Python library to train our models. For the fastText, we used both the original fastText ⁸⁸8https://github.com/facebookresearch/fastText library and its Gensim version. The original fastText library does not provide the option for setting a random seed value, which is provided by the Gensim version. We trained the GloVe models using the original GloVe code ⁹⁹9https://github.com/stanfordnlp/GloVe with a slight modification to input random seed as a parameter. For all the models trained on all the three WEMs, we experimented with the vector dimensions, iterations, and window size, keeping all other parameters at default settings. For all the experiments on a corpus, we measured stability across five randomized embedding spaces trained with an algorithm and took the average among them. We used the same random seeds for all the WEMs.

Consider a dataset $C$ with vocabulary of size $VocabSize$. A WEM $A$ is used to learn word representations in an embeddings space using the dataset $C$. Let $E_{1}$ and $E_{2}$ be two such embedding spaces. These two embedding spaces correspond to two distinct executions of WEM $A$ over the dataset $C$. These two executions of $A$ can differ from each other in terms of one or multiple parameters. We evaluate the stability of $A$ across $E_{1}$ and $E_{2}$ as follows

Stability of each word $w$ in the vocabulary is computed as follows

$KNN_{E}(w)$ represents the set of top $k$ nearest neighbors of word $w$ in the embedding space $E$. These nearest neighbors are computed using the cosine similarity between the word embeddings. Existing stability evaluation studies use the similar method for stability computation.

The word stability computation formula in the Section 2.3 involves a parameter $k$. It indicates the number of nearest neighbors considered for stability evaluation. If the value of $k$ is set close to one then it is too restrictive and we expect the stability value for WEMs to be low. As we increase the value of $k$, we expect the stability to increase. However, with very high value of $k$, the stability evaluation will be meaningless. For example, in the extreme case of $k=VocabSize-1$, stability of any WEM will be hundred percent.

Please refer to Figure 1. This figure shows the variation in the stability of WEMs with respect to the value of $k$.We can observe that for Word2Vec and GloVe, the stability increases with increase in the value of $k$. However, it plateaus after $k=10$. This observation correlates with existing sability evaluation studies(Wendlandt et al., 2018). In contrast, we observe unexpected behavior for the fastText. With increase in the value of $k$, its stability actually reduces slightly. This indicates that fastText maintains the top nearest neighbors across the embedding spaces. However, fastText fails to maintain the stability for the lower ranked nearest neighbors. Still, fastText is the most stable WEM amongst the three.

Consider a dataset $C$ with the vocabulary of size $VocabSize$. To observe the effect of word frequency on word stability, we divide the vocabulary into five groups of equal size ($VocabSize/5$). Based on the word frequency, these groups are named as : Very Low Frequency ($VL$), Low Frequency ($L$), Medium Frequency ($M$), High Frequency ($H$), and Very High Frequency ($VH$). Each group accounts for twenty percent of the words ranked by frequency. For example, the $VH$ group will have the top twenty percent most frequent words from $C$ and the $VL$ group will have the bottom twenty percent frequent words from $C$.

Please refer to Figures 2, 3 4. These figures show the boxplot for word stability when words are grouped based on frequency as described before. Previous studies have reported that stability of a word group increases with increase in the average frequency of the group (Wendlandt et al., 2018). Which means that as we move from $VL$ group to $VH$ group, the average or median word stability should increase. Our experimental results correlate with the existing studies. However, Wendlandt et al also report low variance in word stability for low as well as high frequency word groups (Wendlandt et al., 2018). They report high variance in word stability only for medium frequency word groups. In contrast, we observe that all word groups have high variance in word stability.

We can also observe from Figure 5 that the stability of a word varies with word embedding methods used. The words in Figure 5(a) were randomly picked from the Wikipedia (sample) corpus and these words have different frequencies ranging from 7 occurrences for ”lieutenant” to 174163 for ”one”. To generalize this observation to the whole vocabulary, we computed the stability of each word using three WEMs. For each word $w$, now we had three stability values. For the word $w$, we chose the WEM with highest stability value amongst the three values. We observed that out of all words, each WEM was chosen for the following percentage of words: fastText (63%), GloVe (24%), and Word2Vec (13%). This implies that fastText is in general a more stable WEM. However, a particular WEM need not be the most stable method for each word in the vocabulary.

One of the design choices with word embeddings is the number of dimensions. Lower number of dimensions reduce the computation but they might fail to represent all semantic relations. On the other hand, higher number of dimensions increase the computation while providing better semantic representation. Here we explore this design choice from the stability point of view. Figure 6 shows variation in stability with respect to the number of dimensions for word embeddings. For each WEM, we are comparing stability between two embedding spaces with the same dimensionality. These two embedding spaces differ from each other only in terms of the initial random seeds. For example, two embedding spaces of 300 dimensioanlity that are trained using fastText, have stability of eighty percent. As expected, the stability improves with increase in the number of dimensions. However, the improvement plateaus after 300 dimensions for all WEMs. Stability of fastText is higher than GloVe and Word2Vec for all dimension sizes. Stability of GloVe is lower for lower dimensions but increases rapidly till dimension 300. After dimension 300, the stability of GloVe becomes comparable to fastText. Stability for Word2Vec is the lowest compared to the other WEMs. We would recommend users to train their models using a vector dimension of around 300 for optimal stability.

Please refer to Figure 7. Here we are computing stability between embedding spaces of different dimensionality. For each WEM, consider two embedding spaces $E_{1}$ and $E_{2}$. Let $Dim(E_{1})$ and $Dim(E_{2})$ represent dimensionality of the respective embedding spaces. We set $Dim(E_{1})$ to 100. We vary $Dim(E_{2})$ from 50 to 800. We can observe that for all WEMs, stability values peaks when $Dim(E_{2})$ is 100. It is expected as at $Dim(E_{2})=100$, we are comparing embedding spaces of the same dimensionality. For $Dim(E_{2})$, as we move away from 100, the stability decreases. This indicates that as we change the dimensionality of embedding spaces, the underlying captured semantics also change. The decrease in stability is rapid for lower number of dimensions. Probable reason for this rapid decrease is that we lose a lot of semantic information as we move to the lower number of dimensions. However, stability reduction is not that rapid as we move towards higher dimensions. We can observe only slight reduction in stability from dimension size 300 to 800. With higher dimensions, we do capture new semantic information. However, with higher dimension size, the rate of growth of semantic information also slows down. This might be a probable reason for slow reduction in stability for higher dimension size.

While training the word embeddings, another design parameter is the number of training epochs. Ideally, we will expect the word embeddings to stabilize with increase in the number of training epochs. Figure 8 shows trends in the stability with variation in the number of training epochs. Given a particular number of training epochs on the X-axis, we are computing stability between two embeddings spaces that are trained for the same number of epochs. However, these embedding spaces differ in the random initial seeds. We can observe that variations in the stability of fasText are minimal. For Word2Vec, the stability improves significantly with increase in the training epochs. GloVe shows a rather interesting trend. The stability sharply increases as we increase the number of training epochs from 1 to 5. After that, stability starts falling significantly with further increase in the number of training epochs. This appears to contradict the intuition that increasing the number of training epochs will increase the stability. Furthermore, fastText performs better than both GloVe and Word2Vec for all the datasets. Stability of Word2Vec is inferior compared to the other two WEMs. However, its stability approaches that of GloVe with increase in the training epochs. Even after 30 training epochs, stability of Word2Vc and GloVe is below 60%. This indicates that choice of initial seed matters for these two WEMs. In contrast, stability of fastText hovers around 90%, indicating that choice of initial seed is insignificant.

We also wanted to quantify the changes in word embeddings across successive epochs. Please refer to Figure 9. For a given WEM, we are comparing two embedding spaces $E_{1}$ amd $E_{2}$. Both these embedding spaces are identical to each other except for the number of training epochs. Let $Train(E_{1})$ and $Train(E_{2})$ be the number of training epochs for respective embedding spaces. X-axis in the Figure 9 represents $train(E_{1})$. We set $train(E_{2})=Train(E_{1})-1$. For example, using fastText as the WEM when we compare $E_{1}$ and $E_{2}$ with $Train(E_{1})=5$ and $Train(E_{2})=4$, the stability is close to 60%. Even in this scenario, fastText is the most stable WEM. It indicates that word embeddings in fastText do not change significantly across successive iterations. So we can terminate the training of fastText after a few training epochs. In contrast, Word2Vec and GloVe have significantly lower stability across successive iterations. The stability value for this experiment does not reach close to 100% for any of the WEMs. This indicates that even with high number of training epochs, top ten nearest neighbors for many words do not stabilize.

The intuition behind any WEM is to generate the representation for a word by learning from its context. The context is limited only to the sentence that the word belongs to. Size of the context window is also a design parameter while training the word embeddings. Figure 10 shows variation in WEM stability with respect to the context window size. For fastText, there is slight reduction in stability. Word2Vec and GloVe show contrasting behavior.After the context window size of ten, these is not significant change in the stability value for any WEM. This is expected as long sentences are rare in real-world datasets that we have considered. Even for this experiment, fastText turns out to be the most stable WEM.

There is a vast variety of clustering algorithms available in the literature. Out of these algorithms we have chosen Shared Nearest Neighbor Density Based Clustering (SNND) algorithm (Ertöz et al., 2003) for the following two reasons. First, SNND is a well known robust clustering algorithm for high-dimensional data. Second, this algorithm clusters data based on the shared top K-nearest neighbors (KNN). This intuition of clustering correlates with our stability measurement.

SNND is a combination of two influential clustering algorithms: Jarvis-Patrick (Jarvis and Patrick, 1973) and DBScan (Ester et al., 1996). In the first step, SNND builds a graph representation of data using the concepts from Jarvis-Patrick algorithm. Then it clusters this graph using concepts from DBScan algorithm. SNND begins by building KNN list for each data point. Similarity between any two data points is the extent of overlap between their KNN lists. Next step is to build the SNN-Graph. In this graph, each data point is a vertex and two vertices have an edge if their similarity is above a predefined threshold $\delta_{sim}$. A vertex is labelled as core if its degree is above another predefined threshold $\delta_{degree}$. A vertex is labelled as noise if it is not core and it is not connected to any other core point. The remaining vertices are labelled as non-core. All core vertices are initially considered as a separate clusters. In the merging phase, two clusters are merged if at least one core point from each cluster is connected with each other in the SNN-graph. This iterative step is repeated till no more clusters can be merged. Noise vertices are removed from clustering. For each non-core vertex, the cluster having the strongest link with it is chosen.

We experimented with all three WEMs on the Wikipedia (Sample) dataset to evaluate the effect of WEM instability on SNND clustering. Using each WEM, we train $P$ embedding spaces: $E_{1}$ to $E_{P}$. These embedding spaces differ only in the random initial seeds. Corresponding to each embedding space, we obtain the clusterings $C_{1}$ to $C_{P}$. We compute the agreement of these $P$ clusterings as the percentage of edges from $C_{1}$ that are preserved across all $P$ clusterings. Please refer to Figure 11. This figure shows the variation in the clustering agreement while varying the value of $P$ from one to ten. As we increase the value of $P$, the clustering agreement decreases. However, this decrease plateaus for higher values of $P$. This indicates that even with the inherent instability of WEMs, resulting clustering always preserves significant majority of the edges. Here again, we observe that fastText results in the highest clustering agreement.

For POS tagging, we combine a number of corpora using the NLTK library. These include treebank (consists of 5% of Penn Treebank), Brown, CONLL (Conference on Computational Natural Language Learning) 2000, 2002 and 2007. The total corpora size consists of 2.5 million tokens. We have split the data into train, validation and test sets in the ratio 70:15:15.

While performing the POS tagging task, our goal is not to replicate the state of the art. We want to analyze the effect of WEM instability on the task. Like the previous stability studies (Wendlandt et al., 2018), we chose a simple a bidirectional LSTM model (Chollet, 2015). It has 50 dimensional hidden vectors and the outputs are passed through a softmax layer. Similar to the clustering task, we trained $P$ embedding spaces $E_{1}$ to $E_{P}$ using each of the WEMs. The model requires vector representation of words as input. While considering $P$ embedding spaces to generate the vector representation of a word, we simply compute the average of embeddings across all $P$ embedding spaces. This averaging strategy across multiple embedding spaces is shown to be effective in the literature (Coates and Bollegala, 2018). Please refer to the Figure 12. This figure shows variation in the POS tagging accuracy score with respect to the number of embedding spaces considered. Initially, the accuracy fluctuates with increase in the number of embedding spaces. However, after crossing seven embedding spaces, the accuracy increases and then it plateaus. Here the instability of WEMs actually benefits the task. If the WEMs were stable and provided similar input across multiple embedding spaces then we will not see any improvement in the accuracy score. However, the increase in the accuracy score indicates that diferent embedding spaces provide complementary information to the model.

Human actions reflect various biases in our thinking. For example, musical instruments might seem inspiring and weapons might seem harmful to some people. Implicit Association Test (IAT) is designed to measure such biases with human subjects (Greenwald et al., 1998). The Word Embedding Association Test (WEAT) is proposed by Caliskan et al.(Caliskan et al., 2017). The WEAT test is an improvisation over the IAT test such that biases can be measured directly from the word embeddings in the absence of humans subjects. This test computes the similarity between words using cosine similarity. Fairness evaluation task involves measuring biases in the the given set of word embeddings using the WEAT test. We selected this test for our experiments because its intuition of cosine based similarity correlates with our stability evaluation method.

The WEAT test takes in sets of target and attribute words. Target words refer to the set of words which denote a particular group based on a specific criterion such as instrument names (guitar, flute)and weapons (gun, sword). Attribute words refer to the set of words which denote the characteristics such as pleasant words (inspiring, creative) and unpleasant words (dangerous, harmful). Given two sets of target words and two sets of attribute words, the WEAT test checks if the null hypothesis holds. That is, if the word embeddings are unbiased then the similarity of the two sets of target words with the attribute words should be almost same. If the result value for the WEAT test is close to zero then it indicates the absence of bias. Please refer to Table 3. It lists eight target and attribute word set tests that we borrowed from Caliskan et al. (Caliskan et al., 2017). The column $IAT$ reports the bias measurement results from Greenwal et al. using the human subjects (Greenwald et al., 1998). The column $CA$ reports the results from Caliskan et al. They trained their word embeddings using a large scale crawl of the Web.

We ran the WEAT test on the three WEMs and used the Wikipedia (Sample) dataset for experimentation. For each WEM, we trained three sets of word embeddings. Using each set of word embeddigs, we ran the WEAT test. In Table 3, three columns for the WEMs report stability results for each method. These three columns contain two values each. These two values correspond to the maximum and minimum value obtained for a test using the given WEM across multiple embedding spaces. For example, consider the WEAT results for test number 1 (flowers vs insects) using the fastText WEM. Corresponding to three embedding spaces, we obtained three different result values. In the table, we list the maximum and minimum of these three values: $1.17$ and $1.15$. Ideally, our results should be close to the columns $CA$ and $IAT$. However, we have trained our word embeddings on much smaller dataset as compared to the dataset used by Caliskan et al. That is the reason why quality of our results does not match well with $CA$ and $IAT$. But our goal is not to demonstrate the quality of output. We aim to measure the stability of the results and compare it with the stability of the WEMs used. Till now, we have observed that fastText is the most stable WEM amongst the three methods. As a consequence, WEAT results for fastText have the highest stability. There is very little difference in the maximum and minimum values in the $FT$ column. WEAT results for Word2Vec are the least stable. This is expected as we have observed that Word2Vec is the least stable method among the three methods considered in this paper.