MWE as WSD: Solving Multiword Expression Identification with Word Sense Disambiguation

Until the last few years, most approaches to WSD treated senses only as one of many possible labels in a classification task. This formulation limits the information available to the model about each sense to only what is learnable from the training data, and can lead to poor performance on rare or unseen senses. To mitigate these problems, recent approaches have improved performance by incorporating sense glosses (Blevins and Zettlemoyer, 2020; Barba et al., 2021a; Zhang et al., 2022).

Our work is inspired by this methodology and utilizes gloss information to improve MWE identification. In particular, Blevins and Zettlemoyer (2020) demonstrate that a simple Bi-encoder model consisting of two BERT (Devlin et al., 2019) models can achieve competitive WSD performance, with Kohli (2021) improving Bi-encoder training for WSD and Song et al. (2021) achieving further performance gains through improved sense representations. Bi-encoder models are also particularly efficient at inference time because gloss representations can be computed in advance and cached.

The Poly-encoder architecture was proposed by Humeau et al. (2020) as a middle ground between Bi-encoders and Cross-encoders (which jointly encode all possible input pairs), retaining the speed advantage of the Bi-encoder, but allowing information to flow between the two encoder outputs like the Cross-encoder. It can be used in place of a Bi-encoder in tasks such as information retrieval (Li et al., 2022) text reranking (Kim et al., 2022), or in our case MWE identification and WSD.

Precisely defining what constitutes a MWE has proven to be difficult Maziarz et al. (2015), but they can be broadly defined as groupings of words whose meaning is not entirely composed of the meanings of included words (Sag et al., 2002; Baldwin and Kim, 2010). This includes idioms such as a taste of one’s own medicine, verb-particle constructions such as break up or run down, compound nouns such as bus stop, and any other grouping of words with non-compositional semantics. In fact, a significant portion of noun MWEs are named entities (Savary et al., 2019).

The task of MWE identification is locating these MWEs in a given body of text. Common approaches to solving MWE identification include rule-based systems (Foufi et al., 2017; Pasquer et al., 2020), CRF-based systems Liu et al. (2021), and token tagging systems (Rohanian et al., 2019). Rule-based systems remain competitive with neural models in this task, and many systems including ours use MWE lexicons in order to identify MWEs, which Savary et al. (2019) argue are critical to making progress in MWE identification. Kurfalı and Östling (2020) and Kanclerz and Piasecki (2022) are similar to our work in that they frame the task of MWE identification as a classification problem, although neither use gloss information.

Among all the types of MWEs, verbal MWEs are particularly difficult to identify due to their surface variability — constituents can be conjugated or separated so that they become discontinuous (Pasquer et al., 2020). Much work on verbal MWE identification, especially in languages other than English, has been done as part of recent iterations of the PARSEME shared task (Ramisch et al., 2018).

Bi-encoders for WSD, as defined by Blevins and Zettlemoyer (2020), consist of two BERT (Devlin et al., 2019) models: a context encoder $T_{c}$ and gloss encoder $T_{g}$, which embed the context and sense glosses into the same embedding space. Given an input sentence $c=(w_{0},...w_{n})$ containing the target words to disambiguate, we first tokenize it and use the context encoder to produce representations for each token. Because tokenization may break words or MWEs up into multiple subwords, word or MWE representations $r_{w}$ are computed as an average of all included subwords.

	$$\displaystyle T_{c}(c)=t_{0},...t_{n}$$
	$$\displaystyle r_{w}=\frac{1}{|w|}\sum_{t\in w}t$$

Then, for each target word or MWE, the gloss encoder produces a sense representation $r_{s}$ for each possible sense by encoding its gloss and taking the [CLS] token embedding.

$$r_{s}=T_{g}(g_{s})[0]$$

Scores corresponding to possible senses for each target word are computed as the dot product similarity of the word and sense representations, and the model predicts the highest scoring sense.

	$$\displaystyle\phi(w,s_{i})=r_{w}\cdot r_{s_{i}}$$
	$$\displaystyle pred(w)=\operatorname*{arg\,max}_{s_{i}}\phi(w,s_{i}):s_{i}\in S_{w}$$

Like the Bi-encoder, the Poly-encoder has a context encoder $T_{c}$ for target word contexts and a gloss encoder $T_{g}$ for glosses. There is also a new set of parameters that Humeau et al. (2020) refer to as code embeddings, $Q$. These codes are used as queries to extract information from context representations produced by the context encoder. The inputs to the Poly-encoder are the same as to the Bi-encoder, sense representations $r_{s}$ are computed identically, and predictions are still the highest scoring sense. However, senses are scored differently.

We take the last hidden state of the context encoder as the context representation $r_{c}=T_{c}(c)$, which we use along with the code embeddings $Q=(q_{1},...,q_{m})$ in the first dot-product attention step (code context attention) of the Poly-encoder. We use a different set of embeddings for single words and MWEs. The number of embeddings, $m$, is a hyperparameter and their dimensionality is the same as the encoders’ hidden sizes. The context representation $r_{c}$ is used as both keys and values in this dot-product attention module, yielding a code attended context $Y_{ctxt}$. The representation a code $q_{i}$ extracts is as follows:

	$$\displaystyle(w_{0}^{q_{i}},...,w_{n}^{q_{i}})=softmax(q_{i}\cdot r_{c_{1}},...q_{i}\cdot r_{c_{n}})$$
	$$\displaystyle y_{ctxt}^{i}=\sum_{j=1}^{n}w_{j}^{q_{i}}r_{c_{j}}$$

Sense representations $r_{s}$ are then used as queries and the code-attended context representations $Y_{ctxt}$ are used as keys and values in a final dot-product attention module, yielding a gloss attended code-context. For a given sense $s$ of a word or MWE:

	$$\displaystyle(w_{1},...,w_{m})=softmax(r_{s}\cdot y_{ctxt}^{1},...,r_{s}\cdot y_{ctxt}^{m})$$
	$$\displaystyle y_{final}^{s}=\sum_{i=1}^{m}w_{i}y_{ctxt}^{i}$$

We then take the dot product of the gloss attended code-context $y_{final}$ and each gloss embedding $r_{s_{0}},...r_{s_{k}}$, yielding a score for each gloss: $\phi(w,s_{i})=y_{final}\cdot r_{s_{i}}$.

Since the Poly-encoder was originally designed to compute sentence representations, it contains no mechanism for explicitly focusing on a specific set of target words/subwords. To address this problem, we propose a variation of the Poly-encoder which we call “distinct codes attention” (DCA). We change the code context attention step of the Poly-encoder so that it can attend differently to target words and the surrounding context, using two sets of code embeddings: one set for target words, $Q_{t}$ and one set for non-target words $Q_{nt}$. Since we also maintain different code embeddings for single words and MWEs, this gives us a total of four sets of code embeddings.

In the first attention module, code-context attention, we construct two key matrices, one to be used with the target code queries $Q_{t}$ and one to be used with the nontarget code queries $Q_{nt}$. First we create two masks which pick out target or nontarget subwords: the target mask $M_{t}$, which is $1$ at the indices of target subwords and $0$ otherwise, and the nontarget mask $M_{nt}$ which is the opposite. We then multiply each mask by the encoded context $r_{c}$ to get target and nontarget key matrices $K_{t}=M_{t}r_{c}$ and $K_{nt}=M_{nt}r_{c}$. Next we compute target and nontarget query results ($QK^{T}$) and add them.

$$QK^{T}=Q_{t}K_{t}^{T}+Q_{nt}K_{nt}^{T}$$

Finally, we softmax and multiply $QK^{T}$ by the encoded context $r_{c}$ to yield the code attended context, $Y_{ctxt}=softmax(QK^{T})(r_{c})$. The gloss attended code-context and final scores are then computed identically to the standard Poly-encoder.

We use a rule-based pipeline inspired by Kulkarni and Finlayson (2011) for MWE identification. First, we compute initial candidates as all combinations of words in a sentence whose lemmas correspond to a MWE in our lexicon. That is, any group of words that when lemmatized corresponds to a known MWE, regardless of order or location in the sentence, is a candidate. This ensures we rarely miss known MWEs, but also produces many false positives, such as: in that in “That was back in 1954, 55 years ago.”

Next, we filter the candidates by removing MWEs which are out of order or too gappy ($>$3 words in between constituents), and optionally by discarding MWE candidates judged to be incorrect by our DCA Poly-encoder (or other) model. We refer to the combination of rule-based extraction and filters with no model as the rule-based pipeline. Since the model is applied as a final filter after extraction and the other filters, it can only improve precision. While the heuristic filters involving order and gappyness exclude some valid MWEs as well, they empirically improved performance on development data, and the majority of exclusions made by these filters are correctly removing false positives from candidate generation as can be seen in Table 1. Note that many candidates excluded by one filter would also be excluded by another filter later in the pipeline.

We use WordNet Miller (1995) as our MWE lexicon for all experiments, treating every entry including the character “_” as a MWE. All sense glosses are taken from WordNet 3.0.

We train our models on SemCor (Miller et al., 1993), a WSD dataset containing a total of 226,036 examples annotated with senses from WordNet. In order to make the data usable for MWE identification in addition to WSD, we preprocess it in the following ways. First, we explicitly mark any words whose lemma includes the character “_” as MWEs such that during training the possible labels for these MWEs also include the “not a MWE” sense. Since some discontiguous MWEs in SemCor are labeled only on a subset of the included words, we add stranded constituents to their parent MWE by attaching nearby words whose lemmas match constituents missing from the labeled MWE³³3For example, in “Are they encouraged to take full legal advantage of these benefits?” (ID d000.s015), the verb take is correctly labeled as the MWE take_advantage, but advantage is not labeled as being part of any MWE, so we attach it.. Finally, because SemCor contains no labeled negative examples of MWEs — instances where the constituent words of a MWE are all present but their meaning in context does not match any of the MWE senses — we add these ourselves. We generate synthetic negative examples using the rule-based pipeline with its filters inverted to mark combinations of words whose lemmas correspond to a known MWE but are out of order or very gappy as negative examples whose gold label is the “not a MWE” sense. We randomly add negative examples in this fashion until they account for just over 50% of the MWE examples in the training data.

To mitigate the risk of the model learning only the heuristics used to generate these synthetic negatives, we also manually annotate a small number of examples. We do this by running the rule-based pipeline (Section 3.4) on the SemCor data and annotating output MWEs with their appropriate sense from WordNet or the “not a MWE”, sense based on context. Because we exclude words already marked as MWEs and many MWEs in SemCor have already been annotated, $>$50% of the newly annotated examples are negative.

Like Blevins and Zettlemoyer (2020), we train with cross-entropy loss. The difference is that for MWEs, there is one additional possible label representing the “not a MWE” case. Given a word or MWE $w$, its gold sense $g_{s}$, and $|S_{w}|=j$ possible senses in the lexicon, this formalizes to:

	$$\displaystyle\mathcal{L}(w,g_{s})=-\phi(w,g_{s})+\mathrm{log}\sum_{x\in X}\mathrm{exp(}\phi(w,x)\mathrm{)}$$
	$$\displaystyle X=\begin{cases}\{s_{0},...s_{j},n\}&\text{if MWE}\\ \{s_{0},...s_{j}\}&\text{otherwise}\end{cases}$$

We train for 15 epochs on SemCor and three epochs for fine-tuning, computing F1 on the WSD and MWE identification dev sets once per epoch and use the best performing model as our final model. Batch size and other hyperparameters such as learning rate were determined by hyperparameter search. Further implementation and training details can be found in Appendix A.

We evaluate our system on the English section of the PARSEME 1.1 Shared Task (Ramisch et al., 2018) and the DiMSUM dataset (Schneider et al., 2016). We do not evaluate on STREUSLE (Schneider et al., 2018) as it requires predicting lexical categories and supersenses⁴⁴4The STREUSLE evaluation script rejects input without appropriate lexical categories/supersenses, while our system predicts only the presence or absence of MWEs. To measure WSD performance, we use the evaluation framework established by Raganato et al. (2017) and evaluate on the English all-words task.

Following standard practice, we use the SemEval-2007 dataset (Pradhan et al., 2007) as our dev set, holding out the remaining Senseval-02, Senseval-03, SemEval-2013, and SemEval-2015, as test sets (Palmer et al., 2001; Snyder and Palmer, 2004; Navigli et al., 2013; Moro and Navigli, 2015).

We compare performance on the English WSD all-words task to Blevins and Zettlemoyer (2020), a similar Bi-encoder system trained only for WSD. Recent work in WSD has achieved higher scores (Barba et al., 2021b), but our goal is to understand how the addition of the MWE identification task affects WSD performance.

Comparing models, the DCA Poly-encoder outperforms the standard Poly-encoder on WSD, but its performance does not significantly differ from the Bi-encoder. We leave Poly-encoder architectures better suited for WSD to future work.

We also see that while removing rule-based filters from the DCA pipeline lowers PARSEME F1, it slightly raises DiMSUM F1, suggesting that the necessity of these filters depends on the data. However, removing the rule-based filters only works because the DCA Poly-encoder can accurately exclude false positives: removing the same filters from the purely rule-based pipeline results in a very low F1. Finally, the DCA Poly-encoder substantially outperforms the standard Poly-encoder (PolyEnc) on both datasets and surpasses the Bi-encoder on PARSEME, demonstrating that our DCA Poly-encoder model can improve MWE identification performance.

We perform an error analysis on the output of our SemCor trained and fine-tuned models on both test sets, taking 50 false positives and 50 false negatives from each combination of model and dataset (for a total of 400 examples). Select examples can be seen in Table 6, and detailed statistics about the outcome of our analysis can be found in Appendix B.

We find that for $>$80%⁶⁶6Computed excluding false-positives from the DiMSUM noun phrase detector, which does not use the lexicon of false positives a sense from our lexicon was appropriate for the given context, but the target words were not marked as a MWE in the data. Many of these MWEs were present in our lexicon but nowhere in the test set, suggesting discrepancies between the scope of what WordNet and these datasets respectively define as MWEs. However, there were also a number of false positives that are marked as MWEs in other places in the dataset. This could happen if these combinations of words were only marked as MWEs when they had specific meanings or particularly non-compositional semantics, but this was not the case for the examples we examined. These results speak to the difficult and potentially subjective nature of annotating MWEs, and we hope to see work exploring this area in the future.

For false negatives, $>$85% were cases where the target MWE was missing from the lexicon, confirming that the bottleneck for recall is our system’s lexicon. For the majority of the remaining false negatives, an appropriate sense for the given context was present in our lexicon, meaning that these were failures of our MWE identification system and not the lexicon. However, the fact that errors in matching meaning to context account for $<$20% of false positives and $<$15% of false negatives shows that our model has successfully learned how to judge whether a group of words constitutes a MWE with a given meaning. See Table 6 true negatives for examples of MWEs excluded based on meaning.