Reconciliation of Pre-trained Models and Prototypical Neural Networks
in Few-shot Named Entity Recognition

Named entity recognition (NER) is a classical task in natural language processing (NLP) which aims to automatically identify entities in the plain text by classifying each word to a set of pre-defined entities, e.g. “person/location”, or to the “others” (no-entity) Yadav and Bethard (2019). As a crucial sub-component of many language understanding tasks, NER has been widely adopted to different applications, e.g. news Sang and De Meulder (2003) and the medical Stubbs and Uzuner (2015).

Neural networks (NNs) have achieved great success in NER Lample et al. (2016). However, NNs face the adaptation challenge Wilson and Cook (2020) as words in different entities can change to a great extent Yang and Katiyar (2020), e.g. "Mr. Bush" in the "person" v.s. "budgets" in the "money", and obtaining sufficient annotations of new entities can be expensive Ding et al. (2021). Few-shot NER, a cost-efficient solution, aims at training a model to be aware of unseen entities given few labeled examples Huang et al. (2021). Few-shot NER has received a rising interest in the NLP community, where new datasets Ding et al. (2021) and methods Das et al. (2022); Yang and Katiyar (2020); Tong et al. (2021) have been constantly proposed.

Low-dim manifold encodes more adaptive information Wang et al. (2018). Prototypical neural networks (PNNs) Snell et al. (2017) learn an embedding space where the same-entity datapoints are clustered around a center, called the prototype, and distances between the query data to all prototypes represent its entity probabilities. In addition to using an embedding network, PNNs calculate the prototypes and distances via a non-parameteric algorithm, gaining popularity for the flexibility and low computing cost Wang et al. (2020). A supplementary enhancement will be using embeddings from large-scale pre-trained models (PTMs), like BERT Devlin et al. (2019), to provide extra knowledge that helps PNNs’ learning of entities. As such, incorporating PTMs with PNNs has become a de-facto paradigm for few-shot NER that achieves competitive results to state-of-the-arts Ding et al. (2021); Huang et al. (2021); Bao et al. (2020). Related works consider NER-specific properties Tong et al. (2021) or new learning algorithms Das et al. (2022); Yang and Katiyar (2020) to enhance the model, but they tend not to examine the coordinating effects between PTMs and PNNs in terms of the information contained in embeddings.

It should be reminded that PNNs calculate distances between word embeddings and prototypes to represent entity probabilities. However, PTMs embeddings may not effectively provide entity information as they prominently contain information on word frequencies Mu and Viswanath (2018); Li et al. (2020b), and we find frequencies are shallow statistics that can cause a loss of in-depth and useful entity-denoting information. By probing into PNNs, we find that words tend to be classified to the entity centred with words of higher frequencies. Therefore, the distance measure is biased towards focusing on frequencies. Such a bias can cause the over-fitting of the PNNs and the unreliability on classifying new entities. As a consequence, when frequencies are changed on a new corpus, the distances can no longer effectively represent the entity probabilities.

Form a mathematical view, the biased distance is mainly caused by the varying prototype $\ell$2-norms. However, we argue that those $\ell 2$-norms contribute little to but actually undermine the correct classification. We propose to normalize all prototypes to unit vectors as a simple yet effective remedy to reconcile PNNs and PTMs for few-shot NER. Our experiments on nine few-shot NER datasets Huang et al. (2021); Ding et al. (2021) demonstrate the effectiveness of our one-line-code remedy. The normalized PNNs achieve competitive results compared to the state-of-the-art methods while retaining all the PNNs’ advantages, such as easy implementation and low computation cost. We also demonstrate normalization can make PNNs learn more effectively about correctly classifying entities, and conduct ablation studies on different normalization strategies.

Our study on reconciling PTMs and PNNs, and the promising performance of the simple normalization method may inspire new research motivations to the few-shot NER, as well as other fields that involve the use of PTMs /or PNNs.

In this section, we describe PNNs’ feed-forward propagation from the mathematical viewpoint focusing on the PNNs’ distance function. In $K$-way $N$-shot, let $\mathbb{S}_{k}$ denote the small support set containing $N$ labeled examples with the class $k$. PNNs calculate the prototype of each class through mean-aggregating the embedded support examples:

where $f_{\phi}$ is the embedding network. The class probabilities of a query data $\mathbf{x}$ are given by a distance function $d$ following a softmax operation:

Proofs are provided in the Appendix B. While $d$ is proposed to be any Bregman Divergence Banerjee et al. (2005); Snell et al. (2017), we prove the optimal distance function should be the squared Euclidean distance: $\|{\mathbf{z}}-{\mathbf{z}}^{\prime}\|^{2}$. ²²2According to corollary 1.1, PNNs require the embeddings to follow Gaussian distribution. Similarly, works Yang et al. (2020); Hu et al. (2022) empirically follow the assumption and propose corresponding embedding post-processions to achieve performance gains.

PNNs consider that distances between embeddings and prototypes represent the entity probabilities. Therefore, we count on the distance to capture the sharing entity information between the word and the prototypes. Factorizing the distance $(-\|f_{\phi}({\mathbf{x}})-{\mathbf{c}}_{k}\|^{2})$ to $(-\|f_{\phi}({\mathbf{x}})\|^{2}+2f_{\phi}({\mathbf{x}})^{T}{\mathbf{c}}_{k}-\|{\mathbf{c}}_{k}\|^{2})$, the entity probabilities are not only proportional to the dot production, but are also reversely proportional to the two $\ell 2$-norms. While $\|f_{\phi}({\mathbf{x}})\|^{2}$ represents query data information, different $\|{\mathbf{c}}_{k}\|^{2}$ implies part of the probabilities are priorly determined, and the word is more likely to be classified to the entity that has the smaller prototype $\ell 2$-norm. Unfortunately, because of the representation degeneration of PTMs, these priorly determined probabilities tend to introduce non-entity information, and bias the PNNs’ distance towards frequencies.

In this section, we introduce the concept of representation degeneration in PTMs and explain its associated effects to PNNs. Small-scale PTMs, like GloVe Pennington et al. (2014) and Word2Vec Mikolov et al. (2013), are argued by researchers as low-capacity models for representing the richness in semantics of natural languages Yang et al. (2018); Zhao et al. (2018). Both theoretical Gao et al. (2018) and empirical Mu and Viswanath (2018) results in literature have proven: the learned word embeddings contain substantial non-semantic statistics information, i.e. the frequencies of the words, causing a lower performance on various downstream tasks, like the task of word classification Mu and Viswanath (2018).

Recent Transformer Vaswani et al. (2017)-based large-scale PTMs Devlin et al. (2019); Liu et al. (2019) are groundbreaking in modeling natural language. However, we are concerned that the learned embeddings might also contain the information regarding the non-semantic word frequencies. In line with Mu and Viswanath (2018), we use the online statistics data³³3Data are taken from the Corpus of Contemporary American English (COCA) that provides 60000 English words with frequencies (COCA_60000)., get the word embeddings from BERT and RoBERTa, and do principal component analysis (PCA) to extract the first two coefficients, and plot them on point diagrams. Figure 2 displays the result. The results show both models’ embeddings have correlation to frequencies.

In addition, Li et al. (2020b) finds in the embedding space, word embedding $\ell 2$-norms are inversely proportional to their frequencies. As in PNNs, the prototypes are the mean-aggregation of the words in the support set. Therefore, the prototype $\ell 2$-norms are also correlated to word frequencies as well as the priorly determined probabilities we find in section 3. However, we hypothesize that word frequencies are shallow statistics that are irrelevant to word entities, and the priorly determined probabilities represent little entity information.

We empirically demonstrate the irrelevance between entities and frequencies in this section. We will demonstrate the irrelevance between prototype $\ell 2$-norms and entities in the next section. In a few-shot NER dataset Ding et al. (2021), we count the mean word frequencies of different entities and the frequencies of each word in two random sampled entities.⁴⁴4The words frequencies are counted on the first 2.5 million sentences in BookCorpus Zhu et al. (2015) processed by HuggingFace Wolf et al. (2020). Figure 3 and Figure 4 display the results. Frequencies can be similar among different entities yet distinct in the same entity. Same as the analysis in the next section, we suppose that this irrelevance introduces non-entity information into PNNs probabilities, and biases the PNNs distance towards focusing on frequencies.

In section 3, we have shown that PNNs have a priori on the distances between word embeddings and different entities: embeddings are more likely to be close to the entity that has a smaller prototype $\ell 2$-norm, and the word is more likely to be classified to that entity. However, in section 4, we argue this priori will introduce non-entity information that confuses the calculation of probabilities in PNNs. We have shown frequencies and entities are irrelevant. In the following two figures, we further show the prototype $\ell 2$-norms vary in a manner that is also irrelevant to entities.

Figure 5 displays the average prototype l2-norms of all classes. The $\ell 2$-norms vary greatly among different classes (min=7.25, max=17.13, coefficient of variation=0.202). In Figure 6, the blue column represents the largest class-prototype $\ell 2$-norm, the orange one the smallest and the green one the average. Even within the same class, the prototypes $\ell 2$-norms demonstrate large variance due to the contrasting difference among episodes.

Distances between prototypes and word embeddings should represent entity probabilities. Unfortunately, with respect to the above problem, the distances in the original PNNs are biased towards frequencies instead of being entity-oriented. As a result, PNNs tend to overfit the training data and be trained with unreliable loss minimization.

In this section, we aim to provide a solution to the above-mentioned problems through a normalizing method. Varying $\ell 2$-norms mainly causes frequency-biased distances and the above two problems. As a result, we consider normalizing the prototypes to $\ell 2$-norm-invariant vectors. Earlier works in Computer Vision find normalizing both prototypes and the query data embeddings can achieve better and more stable results Gidaris and Komodakis (2018). However, we do not normalize the query data embeddings, because word embeddings represent more detail and other useful information that may be eliminated by the normalization.

Representing high-level entity information, prototypes should not be priorly distinguished from each other. Furthermore, observing the following evidence, we argue that prototype $\ell 2$-norms have limited contribution to the correct classification.

In both the BERT’s pre-training (1) and the original PNNs (2), we find the $\ell 2$-norms of class features play limited roles to the correct classification.

• (1)

  In the BERT’s pre-training that predicts a word by its context, the $\ell 2$-norms of the words features, i.e. rows of the classifier, show subtle variance. Figure 7 presents the $\ell 2$-norms of the classifier rows: min=0.766, max=2.045, coefficient of variation=0.138.

• (2)

  Without any intervention to the original PNNs, after the training, the prototype $\ell 2$-norms vary much less compared to the original, i.e. after the training: (min=14.00, max=15.25, coefficient of variation=0.014) compared to the original :(min=7.25, max=17.13, coefficient of variation=0.202), and Figure 8 compared to Figure 5.

Based on the above analysis, we propose to normalize the prototypes to unit vectors before calculating the class probabilities.

To demonstrate the effectiveness of our normalized PNNs, we conduct experiments on nine few-shot named entity recognition datasets proposed by Huang et al. (2021) and Ding et al. (2021).

We briefly introduce the main characteristic of Few-NERD: it defines entity types from two perspectives called the fine-grained (INTER) and coarse-grained (INTRA). Under the fine-grained definition, different entities can share more abstract similarities. For example, entities "Island" and "Mountain" are both "Location", and entities "Director" and "Athlete" are both "Person". Under the coarse-grained definition, entities have more differences, such as "Location" v.s. "Person" and "Event" v.s. "Organization". If the training classes contain "Island", the model can easily identify the entity "Mountain" at the testing because they share the same "Location" information. Therefore, training on the fine-grained set is less challenging for NER on new testing entities.

Table 1 reports our normalized PNNs on Few-NERD as well as the results of state-of-the-art models, and the original PNNs for comparisons. Compared with the original PNNs, the normalization achieves at least 8.14% performance gain (largest: 16.07% and average: 12.82%). The sophisticated contrastive learning-based CONTaiNER outperforms our method in certain settings. On average, our model is slightly superior (49.84% (Proto_ours) v.s. 49.80%). Besides, CONTaiNER needs to be fine-tuned on the testing data in order to alleviate the differences between training and testing entities, which can account for its superior performance on the coarse-grained (INTRA) set. It should be noted that our normalization method shows competitive performances yet maintains the PNNs’ advantages, i.e. the low computation cost and easy implementation. In addition, our model achieves the highest average F1 scores (57.52% (Proto_ours)) on the fine-grained (INTER) set, demonstrating its superiority in a more practical setting Ding et al. (2021).

Because the processed datasets "I2B2" and "Onto" are not open-sourced by Huang et al. (2021), we conduct the experiments on the other eight datasets. For more details of the datasets, please refer to Huang et al. (2021).

Table 2 reports the results on the eight datasets. Results vary among different datasets, but the normalized PNNs consistently outperform the original PNNs (min:0.48%, max:20.07%, average:10.20%). In certain datasets, normalized PNNs achieves extremely close even higher results than the original PNNs plus a pre-training method that is expensive in time cost (Proto+NSP). Furthermore, higher performance gains are obtained when incorporating the normalized PNNs with the NSP technique (Proto+NSP +6.48% v.s. Proto_ours+NSP +7.88%). Our results show that the classical PNNs combined with the simple normalization and NSP can achieve the best results on the eight few-shot NER datasets (the open sources do not provide the ST checkpoints for PNN). This finding is innovative compared to the results in Huang et al. (2021).

We examine the synergistic effects of the large-scale PTMs and the classical PNNs in the few-shot NER. Our theoretical analysis of PNNs shows that PNNs’ distances that represent the query data’s entity probabilities are partly priorily determined in terms of the prototype $\ell 2$-norms. However, on the embeddings of the PTMs, we empirically verify that embedding $\ell 2$-norms contain little entity information, being a type of PTMs’ representation degeneration. Furthermore, we show that such representation degeneration makes PNNs’ distance biased towards frequencies instead of entity-denoting. This distance bias prevents PNNs from learning useful entity information and causes PNNs to overfit the training corpus and become unreliable on new entities. Therefore, We propose a one-line-code normalization remedy to reconcile PTMs and PNNs for few-shot NER. The experimental results based on nine datasets suggest that the normalized PNNs proposed in this work achieve significant performance improvements over the original PNNs and get competitive results compared with the latest sophisticated methods while maintaining PNNs’ all advantages, such as easy implementation and low computation cost. Considering the promising results and the innovation in normalizing the existing models, our results and analysis may be an interest of reference study for researchers and practitioners working with few-shot NER or other relevant tasks that involve the use of PTMs or PNNs.

There are certain limitations in this paper. While our theoretical analysis about PNNs and the concept of PTMs’ representation degeneration are not limited to the few-shot named-entity recognition, our focused problem, e.g. PNNs’ distance is biased towards frequencies, is based on the fact that the greatly varied word frequencies represent limited entity information. It is possible that in other tasks, the corpus frequencies can represent semantic features, or the frequencies change much less. Our normalization remedy, therefore, cannot be directly applied to those tasks. Also, representation degeneration is a crucial intrinsic problem of large-scale PTMs. Our focused aspects, e.g. frequencies and entity information, is one type of practical issue. We argue that such intrinsic problems can result in different practical issues affecting other NLP tasks beyond this current work’s scope.

This work is supported by National Key R&D Program of China (2020AAA0105200).