Frequency-aware Dimension Selection for Static Word Embedding by Mixed Product Distance

Word embedding has been widely used in numerous NLP tasks, such as recommendation systems Zhang et al. (2019), text classification Chung and Glass (2018), information retrieval Palangi et al. (2016) and machine translation Guo et al. (2019). Though pre-trained language models (PLMs) like BERT Devlin et al. (2018) have become a prevailed paradigm in NLP, static word embeddings, such as Word2Vec Mikolov et al. (2013) and GloVe Pennington et al. (2014), are still useful in scenarios where contexts are missing, such as entity retrieval Liu et al. (2021); Han et al. (2020) and word similarity tasks Halawi et al. (2012); Bruni et al. (2014). Under such scenarios, static word embeddings often perform better than PLMs.

Although there are many algorithms for training static word embeddings, the impact of dimension on word embedding has been less investigated. Yin and Shen (2018) have shown that a critical hyper-parameter, the choice of dimension for word vectors, significantly influences the performance of a model that builds on word embeddings. An extremely low-dimensional word embedding is probably not expressive enough to represent the meanings of words, and in contrast, a high-dimensional word embedding is expressive but possibly leads to over-fitting and high computational cost. For most NLP models building on word embeddings, dimension is selected ad hoc or by grid search, either resulting in sub-optimal model performances. Thus, it is promising to establish a dimension selection criterion that can automatically select a proper dimension for static word embedding algorithms with a good efficiency-performance trade-off.

Yin and Shen (2018) first investigated dimension selection for static word embedding. They proposed a metric called Pairwise Inner Product (PIP) loss which attempts to reflect the relationship between the dimension and the quality of word embedding and then developed a PIP-based dimension selection method that selects a dimension with the minimal PIP loss. Later, Wang (2019) proposed a dimension selection method based on Principal Components Analysis (PCA). The PCA-based dimension selection trains a high-dimensional word embedding and then uses the embedding to conduct a grid search to find a proper dimension, which is time-consuming. However, both the PIP-based dimension selection and the PCA-based dimension selection overlook one dominant element that may influence the dimension selection: word frequency.

Thus, in this paper, we first investigate whether word frequency is influential in the dimension selection of static word embedding. Our results reveal that extreme word frequency may cause an improper dimension for the PIP-based or PCA-based dimension selection. That is, without diminishing the impact of word frequency, it is more likely to select an improper dimension and generate a low-quality word embedding. Then, we explore a way to diminish the impact of word frequency. Inspired by the success of post-processing methods on static word embeddings Mu and Viswanath (2018); Liu et al. (2019), which de-emphasize the influence of some words in an embedding, we attempt to transfer them to the dimension selection.

Specifically, we propose an MPD-based dimension selection, which minimizes a metric called Mixed Product Distance (MPD) to find a proper dimension. The MPD can reduce the impact of word frequency by applying a post-processing function to oracle matrices, which results in a more proper dimension. To investigate the effectiveness of our method, we conduct evaluations on context-unavailable and context-available tasks. Comprehensive experiments demonstrate that the MPD-base dimension selection method can consistently perform better than baselines.

Above all, the main contributions of the paper can be outlined as follows:

• •

  We expose a critical issue overlooked by previous dimension selection methods: word frequency. It can influence the dimension selection and result in degraded word embeddings.

• •

  We design a metric called Mixed Product Distance (MPD) for static word embedding dimension selection, which uses post-process functions to overcome the word frequency problem.

• •

  We evaluate the MPD-based dimension selection on various NLP tasks, including context-unavailable and context-available tasks. Comprehensive experiments demonstrate that our method achieves a better efficiency-performance trade-off.

This section presents a basic procedure for the dimension selection of word embedding. A static word embedding algorithm is chosen to obtain an word embedding matrix $X\in\mathbb{R}^{n\times k}$, where $X$ is composed of $n$ vectors, and word $w_{i}$ is represented by vector $v_{i}\in\mathbb{R}^{k}$, $i\in[n]$. Generally, dimension selection aims to find a proper dimension $k$ with a good performance-efficiency trade-off.

There are two existing dimension selection methods: PIP-based and PCA-based. PIP can directly compute the dimension without training any word embedding, while PCA-based needs to train a word embedding before dimension selection, which leads to much more computational cost. So, the PCA-based method owns a worse performance-efficiency trade-off than the PIP-based method. In this paper, we work on dimension selection without training any word embedding.

Following previous works Levy and Goldberg (2014), we leverage oracle matrix for dimension selection, which can be obtained through matrix decomposition without any training process. Formally, for a static word embedding algorithm with two hyper-parameters ($\alpha$ and dimension $k$) and a signal matrix $M$ (i.e., the co-occurrence matrix), an oracle matrix $X$ is derived from matrix $M$ with the form $X=f_{\alpha,k}(M)\leq U_{\cdot,1:k}D_{1:k,1:k}^{\alpha}$, where $M=UDV^{T}$ is its SVD obtained by matrix factorization Levy et al. (2015).

To implement the dimension selection, we divide the training corpus into two identical partitions $C_{1}$ and $C_{2}$, and then obtain the co-occurrence matrices $M_{1}$ and $M_{2}$ on each partition, respectively. Correspondingly, we can have two oracle matrices $X$ and $\hat{X}$. Then, a common way to select a proper dimension is used: define a proper distance $d$, and then find $k$ that can minimize the distance $d(X,\hat{X})$. Notice that a distance ($\approx 0$) that guarantees the unitary-invariance of the two oracle matrices $X$ and $\hat{X}$ may serve as a proper distance for dimension selection Yin and Shen (2018), where unitary-invariance refers that two matrices are essentially identical if one can be transformed from the other by performing a unitary operation (e.g., a simple rotation), and unitary-invariance defines the equivalence class of the oracle matrices.

Although previous works Mu and Viswanath (2018); Gong et al. (2018) have pointed out that word frequency may bring negative consequences to word embeddings, the impact of word frequency on the dimension selection has been overlooked by previous studies on dimension selection Yin and Shen (2018); Wang (2019). Therefore, we explore a question in this section: does the word frequency influence the dimension selection for word embeddings?.

We first prepare an experiment of extreme word frequency on two tasks: context-unavailable (WordSim353 dataset, Finkelstein et al., 2001) and context-available (SST-2 dataset, Socher et al., 2013). In the experiment, the following two kinds of dimensions, $k^{*}$ and $k^{+}$, are used. Generally, word embedding with $k^{+}$ consistently outperforms the one with $k^{*}$, and the performance gap between $k^{+}$ and $k^{*}$ reflects the effectiveness of the dimension selection method.

• •

  Criterion-selected dimension $k^{*}$: Given a dimension selection method, $k^{*}$ with the minimal distance $d$ is selected as the criterion-selected dimension.

• •

  Empirically optimal dimension $k^{+}$ (Optimal): Given a word embedding algorithm, a word embedding is trained for each dimension in a grid search (i.e., the dimension is selected from 50 to 1000 with an increment step of 2). Then, each word embedding is evaluated on the benchmark, and the one achieving the highest performance is selected as the empirically optimal dimension $k^{+}$.

Given the training corpus, Gigaword 5 corpus which contains $N$ sentences, to obtain varying word frequencies, we randomly select a sentence (length is larger than 20, which will not affect the overall fluency of corpus considering the WordVec window size is 5) and duplicate it for $3\%\cdot N$, $5\%\cdot N$, and $10\%\cdot N$ times, and then merge them with the corpus. The results are listed in Table 1. As word frequency becomes extreme, the performance gap between $k^{*}$ and $k^{+}$ enlarges, indicating that extreme word frequency significantly hurts the effectiveness of dimension selection methods. Thus, it is necessary to diminish the influences of word frequency during dimension selection.

This section carries out three sets of experiments to comprehensively analyze our MPD-based dimension selection. The first two sets of experiments evaluate the effectiveness of our MPD-based dimension selection on context-unavailable and context-available tasks, respectively. The third set of experiments compares the efficiency-performance trade-off of different dimension selection methods.

Specifically, two popular static word embedding algorithms (Word2Vec¹¹1We choose the CBOW version. and GloVe) are chosen to train word embeddings and both of them use Gigaword 5 as the training corpus. For the MPD-based dimension selection method, two commonly-used post-processing methods (CN and ABTT) are used, and CN is the default one. Moreover, for each experiment, a criterion-selected dimension $k^{*}$ and an empirically optimal dimension $k^{+}$ are computed, whose definitions can be found in Sec 4.

In the ablation studies, we principally answer two questions: (1) Is the distance combination in MPD effective? (2) Is the post-processing function effective in the case of extreme word frequency?

In this paper, we investigate an overlooked problem, whether word frequency influences the automatic dimension selection for static word embedding algorithms, and discover that extreme word frequency can degrade existing dimension selection methods. According to the empirical finding, we propose MPD, a novel metric that combines primitive relative distance $d_{r}$ and the post relative distance $d_{p}$. Through $d_{p}$, post-processing functions are introduced into the dimension selection. Then, we develop an MPD-based dimension selection method that automatically selects a proper dimension for static embedding algorithms. Extensive experiments on context-unavailable and context-available tasks show that our MPD-based dimension selection achieves much better performance with a good efficiency-performance trade-off.