Unsupervised Lexical Substitution with Decontextualised Embeddings

Given a sentence that contains a target word $x$ and its surrounding context $c$, we first feed the sentence into a pre-trained transformer model Vaswani et al. (2017) such as BERT and generate the contextualised representations of $x$: ${f^{\ell}(x,c)}\in\mathbb{R}^{d}$, where $\ell~{}(\leq L)$ denotes the layer of the model. We propose to predict substitutes of $x$ by retrieving words that have similar representations to ${f^{\ell}(x,c)}$. To this end, we calculate $S(y|x,c)$: the score of $y$ being a substitute of $x$ in context $c$, as follows:

	$\displaystyle S(y|x,c)$	$\displaystyle=\mathrm{cos}(f(y),f(x,c)),$		(1)
	$\displaystyle f(x,c)$	$\displaystyle=\sum_{\ell\in Z}{f^{\ell}(x,c)},$
	$\displaystyle f(y)$	$\displaystyle=\dfrac{1}{N}\sum_{i}^{N}\sum_{\ell\in Z}{f^{\ell}(y,c^{\prime}_{i})},$

where $f(y)\in\mathbb{R}^{d}$ denotes the decontextualised word embedding of $y$; $Z$ is a set of selected layers; and $\mathrm{cos}(a,b)$ denotes the cosine similarity between $a$ and $b$. To obtain $f(y)$, we randomly sample $N$ sentences ($c^{\prime}_{1},c^{\prime}_{2}...,c^{\prime}_{N}$) that contain $y$ from a monolingual corpus, and take the average of the contextualised representations of $y$ given $c^{\prime}_{i}$: $f^{\ell}(y,c^{\prime}_{i})$. We pre-compute $f(y)$ for each word $y$ in our pre-defined vocabulary $\tilde{V}$, which consists of lexical items (i.e. no subwords) and contains different words from the pretrained model’s original vocabulary $V$. If $y$ is segmented into multiple subwords (using the pretrained model’s tokeniser), we average its subword representations — this way we can include low-frequency words in $\tilde{V}$ and generate diverse substitutes. We obtain $f(x,c)$ and $f(y)$ by summing representations across different layers $\ell\in Z$ to capture various lexical information.²²2We also tried taking the weighted sum of the different-layer embeddings, but we did not see noticeable improvement.

Representing $f(y)$ in Eqn. (1) as a simple average of the contextualised representations of $y$ is clearly limited when $y$ has multiple meanings, since the representations will likely vary depending on its usage. For instance, Wiedemann et al. (2019) show that BERT representations of polysemous words such as bank create distinguishable clusters based on their usages. To address this issue, we first group the $N$ sentences into $K$ clusters based on the usages of $y$, and for each cluster $k$, we obtain the decontextualised embedding $f^{k}(y)$ by averaging the contextualised representations, i.e.,

$$\displaystyle f^{k}(y)=\dfrac{1}{|{C}^{k}|}\sum_{c^{\prime}\in{C}^{k}}\sum_{\ell\in Z}{f^{\ell}(y,{c^{\prime}})},$$

where $C^{k}$ denotes the set of the sentences that belong to the cluster $k$. To obtain clusters, we apply $K$-means Lloyd (1982); Arthur and Vassilvitskii (2007) to the L2-normalised representations of $y$ in $N$ sentences.³³3We concatenate $f^{\ell}(y,{c})$ across multiple layers $\ell\in Z$. We expect that if $y$ has multiple senses, $f^{k}{(y)}$ will to some degree capture the different meanings.⁴⁴4Note that the number of clusters $K$ is fixed across all words, forcing the model to “split” and “lump” senses Hanks (2012) to varying degrees. This methodology has been shown to be effective by Chronis and Erk (2020) on context-independent word similarity tasks. With $f^{k}(y)$, we can refine the similarity score $S(y|x,c)$ in Eqn. (1) as follows:

$$\displaystyle{S}(y|x,c)=\underset{k}{\mathrm{max~{}}}{\mathrm{cos}(f^{k}(y),f(x,c))}.$$

In this way, we can compare $x$ with $y$ based on the sense that is most relevant to $x$. Furthermore, we capture global similarity between $x$ and $y$ as:

	$$\displaystyle\begin{aligned} S(y|x,c)=&\underset{k}{\mathrm{~{}max~{}}}\lambda\mathrm{cos}({f^{k}(y),f(x,c)})\\ &+(1-\lambda)\mathrm{cos}(f^{k}(y),f^{j_{c}}(x)),\\ \end{aligned}$$		(2)
	$$\displaystyle\begin{aligned} j_{c}=\underset{j}{\mathrm{~{}argmax~{}}}\mathrm{cos}(f^{j}(x),f(x,c)),\end{aligned}$$		(3)

where the second term in Eqn. (2) corresponds to the global similarity, which compares $x$ and $y$ outside of context $c$.⁵⁵5To obtain $f^{j}(x)$, we compute the decontextualised embedding of $x$ and apply $K$-means, as we do to compute $f^{k}(y)$. When $x$ is not included in our pre-defined vocabulary $\tilde{V}$, we set $\lambda$ to 1 and ignore the second term in Eqn. (2). However, it still considers $c$ in Eqn. (3) to retrieve the cluster that best represents the meaning of $x$ given $c$.

While Eqn. (2) generally generates high-quality substitutes, we found that it sometimes retrieves words that share the same root word as $x$ and yet do not make good substitutes (e.g. pay and payer). This is mainly due to the fact that the vocabulary $\tilde{V}$ contains a large number of derivationally-related words, some of which are out-of-vocabulary (OOV) in the original vocabulary $V$ (e.g. pay ##er). To address this problem, we add a simple heuristic where $y$ is discarded if the normalised edit distance⁶⁶6The distance normalised by the maximum string length. between $x$ and $y$ is less than a threshold (0.5 for our English and Italian experiments).⁷⁷7We tuned this threshold based on English development data (i.e. the development split of SWORDS).

In Eqn. (2), the context $c$ affects the representation of $x$ but not $y$. Ideally, however, we want to consider the context $c$ on both sides to find the words that best fit the context. Therefore, we first generate top-$M$ candidates based on Eqn. (2), and rerank them using the following score:

$$\displaystyle\mathrm{S(y|x,c)}=\frac{1}{|Z|}\sum_{\ell\in Z}{\mathrm{cos}(f^{\ell}(y,c),f^{\ell}(x,c))},$$

(4)

where $f^{\ell}(y,c)$ denotes the contextualised representation of $y$ given $c$, which can be obtained by replacing $x$ in $c$ with $y$ and feeding it into the model. In Eqn. (4), we calculate the similarity at each layer $\ell\in Z$ and take the average, which yields small yet consistent improvements over averaging the embeddings first and then calculating the similarity.⁸⁸8In Eqn. (1), we obtained similar results by averaging the embeddings or cosine similarities across layers. We limit the use of this scoring method to the $M$ candidates only, since it is computationally expensive to calculate $f^{\ell}(y,c)$ for every single word $y$ in $\tilde{V}$. Previously, a similar method was employed by Lee et al. (2021) but they used the last layer only (i.e. $Z=\{L\}$). We show that using multiple layers substantially improves the performance. Following Lee et al. (2021), we set $M$ to 50.

Our approach contrasts with previous approaches Zhou et al. (2019); Lee et al. (2021); Yang et al. (2022) that employ BERT as a generative model and predict lexical substitutes based on the generation probability $P(y|x,c)$:

$$\displaystyle P(y|x,c)=\dfrac{\exp(E_{y}{f^{\hat{L}}(x,c)}+b_{y})}{\sum_{\acute{y}\in V}\exp(E_{\acute{y}}{f^{\hat{L}}(x,c)}+b_{\acute{y}})},$$

(5)

where $E_{y}\in\mathbb{R}^{d}$ denotes the output embedding of $y$, which is usually tied with the input word embedding; ${f^{\hat{L}}(x,c)}$ is the representation at the very last layer of the model;⁹⁹9Note that this does not always correspond to the last layer of transformer: ${f^{L}(x,c)}$. E.g., BERT calculates ${f^{\hat{L}}(x,c)}$ by applying a feed forward network and layer normalisation to ${f^{L}(x,c)}$, whereas for XLNET, ${f^{\hat{L}}(x,c)}$ = ${f^{L}(x,c)}$.¹⁰¹⁰10When $x$ consists of multiple subwords, the representation of the first or longest token is usually used. and $b_{y}$ is a scalar bias. While this approach is straightforward and well motivated, its predictions are highly influenced by morphosyntax, as discussed in Section 1. Moreover, Eqn. (5) shows three additional limitations compared to our approach: (1) the prediction is conditioned on the last layer only, despite previous studies showing that different layers capture different information, with the last layer usually containing less semantic information than the lower or middle layers Bommasani et al. (2020); Tenney et al. (2019); (2) $y$ is represented by the single vector $E_{y}$, which may not work well when $y$ has multiple meanings — we alleviate this by clustering the embeddings (Section 2.2); and (3) the model is not capable of generating OOV words, unless we force the model to decode multiple subwords, e.g. by using multiple mask tokens or duplicating $x$. Our approach, in contrast, can include rare words in the pre-defined vocabulary $\tilde{V}$ and generate diverse substitutes (Section 4.4).

We conduct experiments in two evaluation settings: generation and ranking. In the generation setting, systems produce lexical substitutes given target words and sentences, while in the ranking setting, they are also given substitute candidates and rank them based on their appropriateness.

For the generation task, we base our experiments on SWORDS Lee et al. (2021), the largest English lexical substitution dataset, which extends and improves CoInCo Kremer et al. (2014) by introducing a new annotation scheme: in CoInCo, the annotators were asked to come up with substitutes by themselves, whereas in SWORDS, the annotators were given substitute candidates pre-retrieved from a thesaurus, and only had to made binary judgements (“good” or “bad”).¹¹¹¹11The annotators were asked if they would consider using the substitute candidate to replace the target word as the author of the context. A word is regarded as acceptable if it is judged to be good by more than five out of ten annotators, and conceivable if selected by at least one annotator. In this way, SWORDS provides more comprehensive lists of substitutes, including many low-frequency words that are good substitutes and yet difficult for humans to suggest — these words are of particular interest to us. For the evaluation metrics, the authors use the harmonic mean of the precision and recall given the gold and top-10 system-generated substitutes.¹²¹²12More precisely, their evaluation script lemmatises the top-50 substitutes first and then extracts the top-10 distinct words. As gold substitutes, they use either the acceptable or conceivable words, and calculate the corresponding scores $F_{a}$ and $F_{c}$, respectively. They also propose to measure those scores in both strict and lenient settings, which differ in that in the lenient setting, candidate words that are not scored under SWORDS are filtered out and discarded.

In the ranking task, we evaluate models on the traditional SemEval-2007 Task 10 (“SemEval-07”) data set (trial+test) McCarthy and Navigli (2007), as well as SWORDS. For the evaluation metric, we follow previous work in using Generalized Average Precision (GAP; Kishida (2005)):

$$\displaystyle GAP=\dfrac{\sum_{i=1}^{N}I(\alpha_{i})p_{i}}{\sum_{i=1}^{R}I(\beta_{i})\bar{\beta}_{i}},~{}~{}~{}p_{i}=\dfrac{\sum_{k=1}^{i}\alpha_{k}}{i},$$

(6)

where $\alpha_{i}$ and $\beta_{i}$ denote the gold weight of the $i$-th item in the predicted and gold ranked lists respectively, with $N$ and $R$ indicating their sizes; $I(\alpha_{i})$ is a binary function that returns $1$ if $\alpha_{i}>0$, and $0$ otherwise; and $\bar{\beta}_{i}$ is the average weight of the gold ranked list from the 1st to the $i$-th items. In our task, the weight corresponds to the aptness of the substitute, which is set to zero if it is not in the gold substitutes. Following previous work Melamud et al. (2015); Arefyev et al. (2020), we ignore multiword expressions in SemEval-07.¹³¹³13We run the evaluation code at https://github.com/orenmel/lexsub with the no-mwe option.

As shown in Eqn. (2), our approach requires only the vector representations of words and hence is applicable to any text encoder model. Therefore, we test our method with various pre-trained models, including five masked language models: BERT Devlin et al. (2019), mBERT, SpanBERT Joshi et al. (2020), XLNET Yang et al. (2019), and MPNet Song et al. (2020); one encoder-decoder model: BART Lewis et al. (2020); and two discriminative models: ELECTRA Clark et al. (2020) and DeBERTa-V3 He et al. (2021).¹⁴¹⁴14See Appendix A for the details of all models. We also evaluate two sentence-embedding models: MPNet-based sentence transformer Reimers and Gurevych (2019) and SimCSE Gao et al. (2021), both of which are fine-tuned on semantic downstream tasks such as MNLI and achieve good performance on sentence-level tasks. Finally, we also evaluate the encoder of the fine-tuned mBART on English-to-Many translation Tang et al. (2021). Note that the discriminative models and embedding models (e.g. NMT-encoder, SimCSE) cannot generate words and hence are incompatible with the previous approach described in Section 2.4.

We use the same vocabulary $\tilde{V}$ for all models, which consists of the 30,000 most common words¹⁷¹⁷17We discard tokens that contain numerals, punctuation, or capital letters. As such, $\tilde{V}$ includes more lexical items (with less noise and no subwords) than the original vocabulary $V$. in the OSCAR corpus Ortiz Suárez et al. (2020). We set the number of sentences we sample from OSCAR to calculate the decontextualised embeddings, i.e. $N$, to 300; the clustering size $K$ to 4; and $\lambda$ in Eqn. (2) to 0.7. For the set of transformer layers, $Z$, we employ all layers except for the first and last two, i.e.  {3, 4, …, $L-2$}. We tune all hyper-parameters on the development split of SWORDS.

Table 1 shows the results on SWORDS, along with baseline scores from previous work (with some bug fixes, as noted). The first row, HUMANS, indicates the agreement of two independent sets of annotators on (a subset of) SWORDS, and approximates the upper bound for this task. The second row, CoInCo, shows the accuracy of the gold standard substitutes in CoInCo, which are suggested by human annotators without access to substitution candidates¹⁸¹⁸18Since all of these words are in the substitute candidates of SWORDS, it cannot be evaluated under the strict setting. — this approximates how well humans perform when asked to elicit candidates themselves. The remainder of the rows above OURS denote baseline systems, all of which employ generative approaches. The first baseline uses GPT-3 Brown et al. (2020), and achieves the state-of-the-art in the strict setting. It generates substitutes based on “in-context learning”, where the model first reads several triplets of target sentences, queries, and gold-standard substitutes retrieved from the development set, and then performs on-the-fly inference on the test set. As such, it is not exactly comparable to the other fully unsupervised models. BERT-K generates substitutes based on Eqn. (5) by feeding the target sentence into BERT, and BERT-M works the same except that the target word is replaced by [MASK]. Both models further rerank the candidates based on Eqn. (4), using the last layer only; we show the performance without reranking as “w/o rerank” in Table 1. Yang et al. (2022) and Zhou et al. (2019) also use BERT to generate substitutes, and rerank them using their own method.

The rows below OURS indicate the performance of our approach using various off-the-shelf models. Our method with BERT substantially outperforms all the BERT-based baselines, even without the edit-distance heuristic (Section 2.2) or reranking method (Eqn. (4)). The best performing models are DeBERTa-V3, XLNet, and BART (Enc-Dec), all of which outperform the weakly-supervised GPT-3 model by a large margin in the strict setting; and XLNet even outperforms CoInCo in the lenient setting. The last three rows show the performance of the top-3 models when they are given the candidate words and rank them based on Eqn. (4), which emulates how the SWORDS annotators judged the words. The result shows that all the models still lag behind HUMANS, suggesting there is still substantial room for improvement. Interestingly, BART performs best when we average the scores obtained by its encoder and decoder, suggesting each layer captures complementary information. It is also intriguing to see that the discriminative models (DeBERTa-V3 and ELECTRA) perform much better than BERT, albeit they are not trained to generate words and not compatible with the previous generative approach. The sentence embedding models (SBERT, SimCSE) perform no better than the original models, which contrasts with their strong performance in sentence-level tasks. The multilingual models (mBERT, NMT) perform very poorly, even though the NMT model was fine-tuned on large English-X parallel corpora.

Table 2 shows the results for the ranking task on SemEval-07 and SWORDS. Michalopoulos et al. (2022) harness WordNet Fellbaum (1998) to obtain synsets of the target word and also their glosses, and employ BERT and RoBERTa Liu et al. (2019) to rank candidates. The models proposed by Lacerra et al. (2021a) are different from the others in that they fine-tune BERT on lexical substitution data sets. They propose unsupervised (BERT) and supervised (BERT, sup) models, which are fine-tuned on automatically-generated or manually-annotated data. Table 2 shows that our method with BERT performs comparably with the unsupervised model of Lacerra et al. (2021a) without any fine-tuning, and outperforms Zhou et al. (2019) and Lee et al. (2021) (except for the score of Zhou et al. (2019) on SemEval-07, which couldn’t be reproduced in previous work). Just like the generation task, DeBERTa-V3 achieves the best performance on both data sets and establishes a new state-of-the-art. Other models also follow a similar trend to the generation results, e.g. BART performs best by combining its encoder and decoder, and the multilingual models perform very poorly. We hypothesise that their poor performance is mainly caused by suboptimal segmentation of English words. This hypothesis is also supported by the fact that DeBERTa-V3 has by far the largest vocabulary $V$ of all models.²¹²¹21Note that $V$ differs from $\tilde{V}$, the pre-defined vocabulary we used for all models. Appendix A compares the size of the model’s original vocabulary $V$ across different models.

We further conduct an additional experiment on Italian, based on the data set from the EVALITA 2009 workshop Toral (2009). We report $F$ scores given top-10 predictions as in the English generation task, plus two traditional metrics used in the workshop, namely oot and best, which compare the top-10 and top-1 predictions against the gold substitutes.²²²²22We report precision only, as it is the same as recall under those metrics when predictions are made for every sentence. We lemmatise all the generated words to make them match the gold substitutes, following the SWORDS evaluation script.²³²³23We used the Italian lemmatiser (it_core_news_sm 3.2.0) in spaCy (ver. 3.2.2) Honnibal et al. (2020). We use the same hyper-parameters as for the English experiments, and Table 3 shows the results. Hintz and Biemann (2016) is a strong baseline that retrieves substitute candidates from MultiWordNet Pianta et al. (2002) and ranks them using a supervised ranker model. We also implement BERT-K using an Italian BERT model Schweter (2020), with and without the reranking method. The results show that our approach substantially outperforms the baselines, confirming its effectiveness. However, our reranking method is not as effective as in English, which we attribute to the influence of grammatical gender in Italian (which we return to in Section 4.2). The heuristic improves best-P but harms $F$ and oot-P, meaning it removes good candidates as well as bad ones, possibly because we used the threshold tuned on English.

We perform ablation studies on SWORDS to see the effect of $\lambda$ and the $K$-clustered embeddings, and also the heuristic based on edit distance. Table 4 shows the results. Overall, our method with $\lambda=0.7$ performs better than $\lambda=1$ or $\lambda=0$, confirming the benefit of considering both in-context and out-of-context similarities. One interesting observation is that while BERT and DeBERTa perform better with $\lambda=1$ than with $\lambda=0$, the opposite trend is observed for XLNet and especially BART (and hence the optimal value for $\lambda$ is smaller than 0.7). This suggests that BART representations are highly influenced by context, containing much information that is not relevant to the semantics of the target word; we further confirm this in the next section. When we set the cluster size $K$ to 1, the performance of all the models drops sharply, indicating the effectiveness of the clustered embeddings. When we retrieve the cluster of $y$ at random instead of the closest one to $x$ in Eqn. (2), the performance decreases substantially, suggesting each cluster captures different semantics. The heuristic consistently improves the performance, filtering out derivationally-related yet semantically-dissimilar words to the target word. Lastly, our reranking method substantially improves the performance of all the models, demonstrating that it is important to incorporate the target context $c$ into both the target and candidate word representations.

Compared to previous generative approaches, our method does not depend on the generation probabilities of language models, and hence we expect it to be less sensitive to morphosyntactic agreement effects. To investigate this, we analyse the performance on noun target words which immediately follow one of the following articles: a or an in English, and una, la, le, un, il or i in Italian. The first three Italian articles are used with feminine nouns, and the rest with masculine ones. Our hypothesis is that generative methods will be highly biased by these articles, despite the gold standard being semantically annotated, and thus largely oblivious to local morphosyntactic agreement effects.

Table 5 shows the percentage of top-10 predicted candidates that agree with the article.²⁴²⁴24We retrieve Italian gender information using a dictionary API (https://github.com/sphoneix22/italian_dictionary), and English phonetic information using CMUdict (https://github.com/cmusphinx/cmudict) accessed via NLTK Steven et al. (2009). It demonstrates that the prediction of BERT-K is highly affected by the proceeding article as expected, resulting in substitutes which don’t satisfy this constraint being assigned low probabilities. In contrast, the results of our method are more balanced and close to the gold standard.²⁵²⁵25Note that the big jump in results for an is based on a small number of instances (5 sentences). Conversely, our reranking method actually increases the bias greatly, suggesting that the contextualised embeddings $f^{\ell}(x,c)$ and $f^{\ell}(y,c)$ in Eqn. (4) become similar when $x$ and $y$ collocate similarly with the words in the context $c$ — overall, this leads to better results, but actually hurts in cases of local agreement effects biasing the results. This is one reason why reranking was not as effective in Italian as in English, as agreement effects are stronger in Italian.

Table 6 shows the result when we use different pre-trained models in English. First, it shows that BERT-M is more sensitive to the articles than BERT-K, indicating the strong morphophonetic agreement effect on the masked word prediction. Among the pre-trained language models used by our method, SpanBERT and BART are the most sensitive to the article a and an, respectively. This suggests that the embeddings $f(x,c)$ obtained from these models are highly sensitive to the context $c$, partly explaining why BART performs very poorly with $\lambda$ = 1, as shown in Section 4.1. Lastly, Table 7 shows examples of predicted substitutes when the article an comes before the target word. It shows that BERT-K and OURS with BART tend to retrieve words that start with a vowel sound, as quantitatively described in Table 6.

We analyse the performance on SWORDS using different layers in Figure 1 (w/o rerank).²⁶²⁶26We perform qualitative analysis in Appendix D. First, we clearly see that middle layers perform better than the first or last ones, for all models.²⁷²⁷27We see a similar trend in the ranking task (Appendix B). The performance of BERT peaks at layer 16, in contrast with previous findings that the first quarter of layers perform best on context-independent word similarity tasks Bommasani et al. (2020), likely because lexical substitution critically relies on context.²⁸²⁸28In fact, Tenney et al. (2019) show that high-level semantic information is encoded in higher layers. Our method using multiple layers performs mostly as well as using the best layer without the need to perform model-wise layer selection (see Table 11 in Appendix C). The last layer performs very poorly for all models, highlighting the limitation of the previous approach which uses the last layer only (Eqn. (5)). The downward trend is particularly evident for MPNet, BART (dec), and SpanBERT; for BART and SpanBERT, we attribute this to the fact that their last-layer representations of the word at position $t$ are used to predict the next word $w_{t+1}$, or $u$ neighbouring words {$w_{t-u}$..,$w_{t-1}$} or {$w_{t+1}$..,$w_{t+u}$}.²⁹²⁹29SpanBERT does this for Span Boundary Objective. This training objective may also lead to their sensitivity to articles before the target word, as shown in Section 4.2. Interestingly, the sentence-embedding models (SBERT and SimCSE) are no exception to the downward trend, which is somewhat counter-intuitive given that their last layer representations are fine-tuned (and used during inference) to perform semantic downstream tasks. Importantly, they do not perform better than the original models (MPNet and BERT), although in the ranking task, both models benefit moderately from fine-tuning (see Appendix B).

One of the strengths of our approach is that it can generate low-frequency substitutes that are OOV words in the original vocabulary. To confirm this, we analyse how well our method can generate low-frequency words from different vocabulary sizes $\tilde{V}$. Table 8 shows the results, in which we experiment with our BERT-based model with the vocabulary sizes of 5k, 10k, 20k, and 30k. The columns under “# Matched Words” show the numbers of correctly-predicted words, grouped by frequency range: low, med, and high denote words with frequency $<$50k, 50k–100k, and $>$100k in a large web corpus. The table shows that our method with 30k words generates nearly twice as many low-frequency substitutes as the baseline. Our method with 10k words still outperforms BERT-K in $F_{c}$, demonstrating its effectiveness. The last three rows show the performance of our method using other models, further demonstrating its ability to predict low-frequency words.

Finally, we analyse the effect of the cluster size $K$ for ELECTRA, as shown in Table 9. While a larger cluster size yields better performance, the improvement is marginal. Rather than using a fixed $K$, in future work we are interested in dynamically selecting the number of clusters per word.