Vec2Gloss: definition modeling leveraging contextualized vectors with Wordnet gloss

Dictionary definitions, or word glosses, are “language about language”, or “metalanguage” (Sinclair, 1991; Johnson and Johnson, 1998; Hanks, 2013). One of the popular metalanguage theories is the Natural Semantic Metalanguage (NSM) (Wierzbicka, 1972), which proposes that universal semantic primitives can account for meanings of words. For example, Durst (2004) identifies the features of these primitives as: “indefinability”, “indespensability”, “universality”, and “combinability”. On the other hand, Barque and Polguère (2004) have identified that sense descriptions can be categorized into “word paraphrases” and “word interpretations” according to their formal nature. (cf. Pottier, 1974 and Pustejovsky, 1998)

While previous studies of metalanguage more frequently adopt a logical or formal semantic approach, the Corpus Pattern Analysis (CPA) proposed by Hanks (2004) provides us with a new direction into analyzing word glosses from perspectives of syntagmatic patterns. According to Firth (1957), the meanings of a word are contributed to the context formed by surrounding terms. In the same vein, Hanks (2004) analyzes concordance lines from corpus to generalize typical patterns of certain words. These groups of words constitute lexical set which is united by a semantic type. For example, guns, rifles, pistols are a lexical set related to the verb fire, and they are under the semantic type of firearms (Hanks, 2013).

While not closely following the methodology in CPA, the gloss language in Chinese Wordnet tries to incorporate the lexical sets and semantic types into its gloss. For example, one of the gloss patterns³³3See the manual of CWN (in Chinese), https://lope.linguistics.ntu.edu.tw/cwn/documentation for adverbial senses is in the form of ‘表…的程度’, such as ‘表(超過平常)的程度’ (a sense of 很 hěn ‘very’), which literally translates to ‘describing (exceeding normal) extent.’ The same glossing guidelines are created across lexical categories. Therefore, the CWN glosses provide us with a fertile ground to systematically model its gloss language. However, as the gloss patterns are too complex for logical or formal analyses, definition modeling with deep learning is beneficial for exploring the hidden information under these gloss patterns.

Definition modeling generates a definition for a target word (Gardner et al., 2022; Noraset et al., 2017). Noraset et al. (2017) leverage hypernym embeddings to generate dictionary definitions. Since it is inevitably difficult to capture the sense of a polysemy given a single word input, Gadetsky et al. (2018) incorporate the context words’ embeddings and an attention-based skip-gram model to build definition modeling on polysemous words. Recent definition modeling further incorporates other architectures to capture the semantic vectors and definition generation better. To obtain the semantic representations of the target word, models in past studies use recurrent neural networks, variational generative models, and other pretrained language models (Ishiwatari et al., 2019; Reid et al., 2020; Zhang et al., 2020). Notably, some studies leveraged lexical resources such as HowNet and WordNet to construct latent vectors or use them as guiding signals (Dong and Dong, 2006; Luo et al., 2018a, b; Blevins and Zettlemoyer, 2020; Li et al., 2020; Scarlini et al., 2020a; Yang et al., 2020).

Recently, contextualized embeddings have been shown to capture essential aspects of lexical semantics (Peters et al., 2018; Loureiro and Jorge, 2019) in the word sense disambiguation literature. Utilizing the sense vectors built from the contextual embeddings, Scarlini et al. (2020b) found that a simple 1-nearest-neighbor algorithm achieves comparable performance with other supervised model architectures in the word sense disambiguation task. The result demonstrates that the encoded vectors could carry significant semantic information, which should be not only applicable when disambiguating the polysemous words, but beneficial to definition modeling.

In the following, we propose a Vec2Gloss model to formulate the definition modeling as a sequence-to-sequence problem with an encoder-decoder architecture (Mickus et al., 2019). However, an important distinction is that the goal of Vec2Gloss is to decode the definition out of the encoded vectors while simultaneously tuning the encoder for an optimized semantic vector. Therefore, we leverage the pretrained mT5 (Xue et al., 2021) text-to-text model architecture but impose a tight bottleneck between the encoder and decoder. Furthermore, as the decoder no longer has access to the full sentential context, the generated gloss could not depend on the collocations directly. Specifically, the decoder could only produce the gloss with the encoded semantic vectors and the learned gloss patterns.

We first train the model with a denoising objective to better capture the patterns underlying the gloss language. Following the procedures of previous studies (Lewis et al., 2020), we prepare pairs of examples consisting of corrupted spans as inputs and the dropped-out spans as outputs. Such denoising objective is shown to perform well on downstream tasks and be computationally efficient as having shorter decoding sequences (Raffel et al., 2020). A pair of such examples is shown as follows, and the literal translations are in italics:

The ⟨X⟩ and ⟨Y⟩ stand for the special sentinel tokens; they are unique within an example. The spans are character-based and may not follow the word boundaries. The corrupted locations are randomly selected, and the lengths (number of characters) are randomly drawn from a Poisson distribution with $\lambda=2$, with the value clipped between 1 and 4 (inclusive). If the input sequence is longer than 20 characters, another corrupted span is created with the same parameter. The data are extracted from the word glosses of Chinese Wordnet. There are 26,118 pairs created for the denoising objective.

We use the pretrained T5 encoder-decoder architecture (mt5-base) to train the denoising objective (Xue et al., 2021). In the denoising stage, no bottleneck is applied. The model parameters are updated with AdamW optimizer. The learning rate is $10^{-4}$, $\beta_{1}$ and $\beta_{2}$ are 0.9 and 0.999, respectively, and weight decay is set to 0.01. A linear schedule is applied to the learning rate, and the batch size is set to 8. The model was trained for 3 epochs and took 30 minutes in an A5000 GPU. The trained model parameters are the starting point of the next fine-tuning stage.

The fine-tuning stage aims to learn the relationships between the target words embedded in the sentences and their glosses in CWN. In addition to the standard T5 encoder-decoder transformer-based architecture, a tight bottleneck is introduced between the encoder and decoder. Namely, only the target word’s encoder states, which may consist of more than one token, are selected and averaged as the semantic vector, from which the decoder learns to produce a complete gloss sentence.

The training data are extracted from the CWN’s sense inventories. A training instance is created for each example sentence in a CWN sense. The instance is composed of a pair of input and target sequences. The input is an example sentence in which the target words are marked with a pair of angular brackets. The target sequences are the glosses preceded by the part-of-speeches of the senses and followed by a Chinese full-width period. There are 76,969 instances in the training dataset and 8,553 pairs in the evaluation dataset. A sample instance is shown as follows:

The model architecture closely follows the standard T5; thus, the trained weights in the denoising stage are directly applicable to this model. Notably, the target words’ angular brackets are preprocessed and removed to produce the target mask. The mask is used to select the relevant encoder states and generate the semantic vector for the decoder. Therefore, the cross attention of the decoder will always receive a single vector as input. In training time, the model is trained as a text-to-text task. However, in inference time, the encoder and decoder could work independently. That is, the encoder could be used to obtain a semantic vector from a given sentence. The semantic vector can then be flexibly transformed before sending it into the decoder for gloss generation.

The training procedure is the same as the previous stage, the only difference being that the epoch number is 10 for this stage. The training time is 100 minutes in an A5000 GPU.

The automatic evaluation of the definition generation is shown in Table  1, which presents the BLEU and METEOR scores for each lexical category. The overall score is .41 for BLEU and .62 for METEOR. It is noteworthy that the lowest category is the noun (N), while the highest one is the proper name (Nb). The higher score of the proper name may be attributed to words used as a family or foreign names in CWN. Their definitions are short and are thus more likely to be captured by the model. These names account for 188 items in the proper names category. On the other hand, it is less clear how to interpret the automatic metrics in other categories. The scores only indicate the textual difference between the generated and the reference gloss. At a given score level, the generated gloss might be unintelligible to the human reader or just a paraphrase of the reference gloss. Therefore, we study the generated glosses further with human evaluations, including a rating experiment, a gloss dependency analysis, and a token dependency analysis.

In the experiment, we resort to human raters to evaluate the capability of the generated definitions, i.e., their semantic interpretability and syntactic well-formedness. The task is designed as a multiple-choice task with only one correct answer. A total of 140 entries, each consisting of a definition in Chinese and a list of four-word options, are provided. Materials used for the experiment are derived from two sources: Academia Sinica Balanced Corpus of Modern Chinese (henceforth ASBC, Huang and Chen, 1998) and CWN.

Among the 140 test items, 40 are new words with their definitions derived from our Vec2Gloss model (namely V2G:ex vivo). We extract words only composed of Chinese characters, remove proper nouns, and filter out words occurring less than 10 times in the corpus. The target words, i.e., the correct answers, are randomly and equally selected from four different lexical categories: nouns, verbs, adverbs, and others. The incorrect options of each question are of the same word class, randomly selected from the same collection of words derived from ASBC. The remaining 100 words are all taken from the evaluation dataset, with proper names being excluded. Among the 100 words, 20 use definitions from CWN, 80 from model generation (namely V2G:in vivo). The word class composition is the same for the ones from CWN, and the target words are randomly selected from the dataset and equally divided according to the word class.

Five native Chinese speakers majoring in linguistics were recruited as raters. Their first task was to determine the most suitable term from a set of four options based on its given definition. Next, the raters were asked to evaluate the semantic interpretability of a definition using a five-point acceptability judgment scale. That is, they were required to rate on a scale of one to five, to what extent the definition can well explain the word that had been chosen as the correct answer from the previous task. Finally, the raters were asked to evaluate the syntactic well-formedness of a definition, i.e., how much the rater accepted the definition as being well-formed based on his or her internal grammar, on a five-point acceptability judgment scale. Table 2 shows the evaluation results with these two metrics, where we can see the proposed Vec2Gloss model achieves decent performance in comparison with the original glosses in CWN.

More detailed results for evaluations of vector-generated glosses are shown in Table 3. While the mean values for syntactic well-formedness for both V2G:in vivo and V2G:ex vivo are considerably high across all the 4 lexical categories, the semantic interpretability for V2G:ex vivo scores less than for V2G:in vivo. In spite of the inferior interpretability, the multiple-choice task for V2G:ex vivo still achieves over 80% correct rates in every category, similar to the results of V2G:in vivo. Moreover, the semantic scores of nouns are lower than other categories for both sources. To further investigate possible reasons that account for the results, we conduct a gloss dependency analysis.

Two indices are computed for each token to represent their reliance on the preceding contexts and the semantic vector, respectively. First, the token likelihood under the full context and the original semantic vector ($p_{\textrm{full}}$) is compared to the one with all of its preceding contexts masked when decoding. If a token is mostly determined by the context alone, the context masking would significantly impact the token likelihood ($p_{\textrm{mask}}$). Hence, the negative likelihood ratio ($\delta_{\textrm{sem}}$) will be larger. Similarly, if a token is primarily driven by the semantic vector, replacing it while leaving the preceding context intact will lower the likelihood ($p_{\textrm{rep}}$) and make the ratio ($\delta_{\textrm{ctx.}}$) larger. Specifically, the semantic vector ($v_{sem.}$ coming from the encoder) is replaced with another word’s semantic vector from the same lexical category. The indices are all calculated using the shifted reference glosses of each sense as the decoder inputs.

Each token’s indices, $\delta_{\textrm{sem}}$ and $\delta_{\textrm{ctx.}}$, are averaged to produce the gloss-level indices. The results are shown in Figure 2. We first observe that while the contextual dependency is comparable across four different lexical categories, the semantic vector dependency indices show more differences. Specifically, the nouns’ glosses have higher semantic dependency scores, followed by verbs, adverbs, and others. These results echo the human ratings, in which the syntactic ratings are similar in all categories, while nouns are significantly worse in semantic rating scores. The difference in semantic dependency may indicate that nouns are more likely to be used as nominal predicates, and they categorize referents in a class with a holistic set of properties. In contrast, the adverbs, which have the lowest semantic dependency score here, only describe things by adding a single property to the characterization of the referent (Baker and Croft, 2017; Bolinger, 1980). However, if this is the case, the adverbs present interesting cases to study further. Although they are relatively unaffected by semantic vector manipulation, they still carry semantic meaning such as manner, mean, or instrument into their scopes (Lyons, 1977; Lakoff, 1968). Therefore, we should observe some tokens are more pertinent to the semantic vector than others in a more fine-grained analysis.

Following that, we manually identify chunks (semantic constituency) in the gloss and annotate their semantic types. The chunk as defined here is a significant element which functions as a semantic type-carrying unit (cf. Gerdes and Kahane, 2013). We selected 244 adverbs from CWN, whose gloss contains the word “事件” shjiàn ‘event’, as they describe an explicit event structure in the glosses. Each gloss was first segmented into length-variant chunks and manually tagged with its corresponding semantic type. The gloss of the first words is not annotated with semantic types, as they regularly follow a gloss pattern based on their lexical category. For example, the glosses of adverbs start with the word 表 biǎo ‘indicate,’ such as the following example (the gloss of 接連 jiēlián ‘in a row’).

There are 905 chunks and 19 unique semantic types annotated in this dataset. Six semantic types (event, action, modifier, pre/post-position, negation, others) occurring at least 25 times (10% of the glosses count) are selected for further analysis, which accounts for 59% of annotated chunks. Figures in the Appendix show their distribution in different positions in the gloss.

The token level indices are computed as in Sec. 4.2. Notably, since the annotated glosses may have multiple example sentences in CWN, we extract and average the semantic vectors from each sentence to represent the target words. The context and semantic vector dependency scores are computed for each token and averaged by their semantic types. The results are shown in Figure 3.

In alignment with the observation in gloss-level analysis, there are distinctive dependency patterns across different semantic types. In particular, the Action categories are higher in contextual dependency but relatively low in semantic ones. While Action words are usually the main verbs in the glosses, the type distribution is highly skewed. The first three common action words, 發生 fāshēng ‘occur’, 做 zuò ‘make’ , 進行 jìnxíng ‘undergo’ already account for 50% of all action words. Thus, the contextual dependency scores may reflect the constrained word usage given the adverb gloss. In contrast, the Preposition and Negation types are relatively high in semantic vector dependency. It might result from the fact that prepositions are used to introduce the relating complements, and the decoder needs guidance from the semantic vectors to select the exact relations for the gloss. Similarly, the negation words are hard to capture by syntagmatic relations from the context (Aina et al., 2019; Ettinger, 2020); thus, the decoder has to rely on additional cues from the semantic vectors. Consistently, words are highly predictable given the adverb glosses, i.e., the Event type, are low on both dependency scores. However, it is interesting for future work to analyze all the patterns or semantic types of the glosses and investigate the exact roles of contextual and semantic dependencies in the gloss patterns.