Unifying Token and Span Level Supervisions for Few-Shot Sequence Labeling

Few-shot learning aims to recognize novel classes with only few samples in each class and is proposed in the computer vision community (Fei-Fei et al., 2006). Motivated by a simple machine learning principle (i.e., train and test must match), matching Networks (Vinyals et al., 2016) first models the few-shot learning as an episode N-way K-shot task, where each episode has N classes and each class has K annotated samples (K is very small, e.g., 5). Afterwards, many methods have been proposed, such as metric learning methods, optimization methods, and generation methods. Metric learning methods aim to learn a suitable embedding space for distance metrics, including cosine similarity (Vinyals et al., 2016), euclidean distance to prototype (i.e., mean class representation) (Snell et al., 2017), learnable relation modules (Sung et al., 2018), nearest neighbor (Yang and Katiyar, 2020), and reachable distance (Zhang et al., 2022a). For example, Snell et al. (2017) first construct prototypes by averaging the representations of each class in the support set and then optimizes the network by using softmax function over Euclidean distances between all prototypes and query representation. Some other methods have focused on improving the transferability of features, including using cross-attention module to exploit the semantic relevance between support and query features (Hou et al., 2019), adaptive margin loss (Li et al., 2020b), and so on. Recently, some methods (Chen et al., 2019; Dhillon et al., 2020; Chen et al., 2021) propose a transfer-learning framework that pre-trains on the training set and fine-tunes on the support set of test episode. Model-agnostic meta-learning (MAML) (Finn et al., 2017) is a typical approach of optimization methods and aims to learn a good model initialization and then quickly adapt on test episode. Finn et al. (2018) further extend MAML, which adapts to new tasks via gradient descent, to incorporate a parameter distribution that is trained via a variational lower bound. Generation methods aim to generate samples (Wang et al., 2018; Zhang et al., 2018; Zhou et al., 2022) or features (Li et al., 2020d) to augment the training set. Specifically, Zhang et al. (2018) propose to augment vanilla few-shot classification models with GAN for better generalization. FlipDA (Zhou et al., 2022) jointly uses a generative model and a classifier to generate label-flipped data, which has a more significant performance improvement compared to label-preserved data. AFHN (Li et al., 2020d) generates diverse and discriminative features conditioned on the few labeled samples based on cWGAN.

Recently, few-shot learning has received increasing attention in the IR and NLP communities, including recommendation (Li et al., 2020c), document filtering (Liu et al., 2020), dense retrieval (Yu et al., 2021b), node classification (Liu et al., 2022), and so on.

Sequence labeling task is a fundamental task in the field of NLP such as named entity recognition (Lample et al., 2016), part-of-speech tagging (Li et al., 2015), slot tagging (Qin et al., 2019), and so on. Recently, the neural sequence labeling models become the mainstream method for sequence labeling.

Typical neural sequence labeling models (Huang et al., 2015; Chiu and Nichols, 2016; Lample et al., 2016; Ma and Hovy, 2016) generally use CNN or LSTM to extract features and utilize softmax or Conditional Random Field (CRF) for classification. For example, Chiu and Nichols (2016) use CNN to extract character-level features, LSTM to extract word-level features, and softmax to classify words. However, Cui and Zhang (2019) find that BiLSTM-CRF does not always lead to better results compared with BiLSTM-Softmax, and propose a hierarchically-refined label attention network, which explicitly leverages label embeddings and captures potential long-term label dependency by giving each word incrementally refined label distributions with hierarchical attention. Afterwards, some work (Yu et al., 2020; Fu et al., 2021; Ouchi et al., 2020; Jiang et al., 2020; Zhu et al., 2022) adopts span-level prediction which first detects spans by enumeration or boundary identification, and then classifies spans. For example, Yu et al. (2020) use a biaffine classifier to assign scores for all spans in a sentence. Fu et al. (2021) first compare and analyze the advantages and disadvantages of the traditional token-level models and span-level models in detail, and then reveal that span prediction can serve as a system combiner to re-recognize named entities from different systems’ outputs.

Besides, four novel paradigms for sequence labeling have recently been proposed, reformulating it as sequence-to-sequence learning, generation, set prediction, and machine reading comprehension tasks, respectively. First, considering that the sequence-to-sequence learning has a good ability to learn complex global label dependency, some work applies it to solve the sequence labeling tasks such as chunking (Zhai et al., 2017), aspect term extraction (Ma et al., 2019), emotion-cause pair extraction (Cheng et al., 2021), and dialogue act prediction problem (Colombo et al., 2020). For example, in order to make sequence-to-sequence learning suitable for sequence labeling, Ma et al. (2019) design gated unit networks to incorporate corresponding word representation into the decoder and position-aware attention to pay more attention to the adjacent words of a target word. Secondly, some work use generation to solve the sequence labeling. Specifically, Athiwaratkun et al. (2020) propose to generate augmented output that repeats the original input sequence with additional markers that indicate the token-spans and their associated labels. Yan et al. (2021) propose a unified generation framework with the pointer mechanism to tackle flat, nested, and discontinuous NER subtasks. Zhang et al. (2022b) analyze the incorrect biases in the generation process from a causality perspective and design intra- and inter-entity deconfounding data augmentation methods to eliminate confounders. Thirdly, Tan et al. (2021) first propose a sequence-to-set network for NER, which provides a fixed set of entity queries to replace the explicit candidate spans and uses self-attention to capture the dependencies between entities. Finally, Li et al. (2020a) first propose a machine reading comprehension framework to solve the NER task and use type-specific queries using semantic prior information for entity categories. However, type-specific queries can only extract one entity type per inference and rely on external knowledge to construct queries. Shen et al. (2022) further set up global and learnable instance queries to extract entities from a sentence in a parallel manner.

Few-shot sequence labeling aims to identify novel classes based on only a few labeled samples and has gained widespread attention due to its application value. We roughly divide the existing few-shot sequence labeling methods into three groups.

The first group of methods is token-level metric learning method. Fritzler et al. (2019) first use prototypical networks to solve this task. Afterwards, L-TapNet (Hou et al., 2020) and StructShot (Yang and Katiyar, 2020) both explore CRF to model label dependence in the few-shot setting. L-TapNet (Hou et al., 2020) extends task-adaptive projection networks (TapNet) (Yoon et al., 2019), which improves label representation with label name. StructShot (Yang and Katiyar, 2020) trains model through traditional cross-entropy loss and uses nearest neighbor for classification. Huang et al. (2021) explore noisy supervised pre-training and self-training in the few-shot sequence labeling setting. Motivated by non-entity class O has rich semantics, MUCO (Tong et al., 2021) introduces a binary classifier to determine whether it is an entity or not. Ma et al. (2022a) propose to leverage the semantics of label names and use two separate BERT to encode label and token for token-level similarity metric. Instead of optimizing class-specific attributes, CONTAINER (Das et al., 2022) models Gaussian embedding for each token and optimizes inter token distribution distance to improve generalizability.

The second group of methods is span-level method. SpanNER (Wang et al., 2021) is a two-step method, which first detects the start and end tokens to extract span and then introduces class descriptions to classify spans. Ma et al. (2022b) further propose a model-agnostic meta-learning (MAML) (Finn et al., 2017) enhanced two-step method, which uses BIESO scheme to detect span and prototypical networks to classify span. Retriever (Yu et al., 2021a) proposes a span-level retrieval method that learns similar contextualized representations for spans with the same label via a novel batch-softmax objective. ESD (Wang et al., 2022) proposes inter-span and cross-span attention to enhance the span representations, and uses instance span attention to construct prototype.

The third group of methods mitigates data scarcity with other paradigms, including machine reading comprehension (Ma et al., 2021), generation (Athiwaratkun et al., 2020; Chen et al., 2022), pairwise cloze (Henderson and Vulic, 2021), and prompt tuning (Cui et al., 2021; Lee et al., 2022; Ma et al., 2022d; Hou et al., 2022). For machine reading comprehension method, Ma et al. (2021) enumerate all the slots to extract the answer from the sentence. The MRC-based method can naturally encode the label semantics in the form of questions. For generation methods, Athiwaratkun et al. (2020) propose to generate augmented natural language output, including the original input sequence, delimiter, and label. Self-describing Networks (Chen et al., 2022) is a sequence-to-sequence generation network which can universally describe mentions using concepts, automatically map novel entity types to concepts, and adaptively recognize entities on-demand. For prompt tuning methods, Template (Cui et al., 2021) treats NER as a language model ranking problem in a sequence-to-sequence framework, where the output sequence is filled by templates. Hou et al. (2022) propose an inverse prompt, predicting slot values given slot types, and further introduce the iterative prediction strategy to consider the relations between different slot types. Ma et al. (2022d) reformulate the NER task as an entity-oriented LM task, inducing the LM to predict label words at entity positions during fine-tuning, and propose a template-free prompt tuning method EntLM. Lee et al. (2022) perform a systematic study of entity-oriented prompt (i.e., only entity tokens) and instance-oriented prompt (i.e., retrieved sentences).

We utilize BERT (Devlin et al., 2019) to encode sentences in the support and query set. Given a sentence $\boldsymbol{x}=\{x_{1},x_{2},...,x_{n}\}$, we insert a [CLS] and [SEP] token for each sentence to get the input of BERT and feed it into BERT to get its contextual representations $\mathbf{U}=\{\mathbf{u_{0}},\mathbf{u_{1}},\mathbf{u_{2}},...,\mathbf{u_{n}},\mathbf{u_{n+1}}\}\in\mathbb{R}^{(n+2)\times d_{1}}$ as follows:

We further use a linear layer to obtain the token representation $\mathbf{H}=\{\mathbf{h_{1}},\mathbf{h_{2}},...,\mathbf{h_{n}}\}\in\mathbb{R}^{n\times d}$ as the input to the token-level network as follows:

where $\mathbf{W_{t}}\in\mathbb{R}^{d\times d_{1}}$ and $\mathbf{b_{t}}\in\mathbb{R}^{d}$ are trainable parameters. Since [CLS] and [SEP] tokens do not need to be classified, we do not feed them into the linear layer.

For the span-level network, we use boundary tokens (i.e., left and right boundary tokens) and a linear layer to obtain the span representation as the input to the span-level network. Specifically, given a span $\mathbf{s}_{k}=\{x_{i},...,x_{j}\}$, we represent the span as:

where $\mathbf{W_{s}}\in\mathbb{R}^{d\times 2d_{1}}$ and $\mathbf{b_{s}}\in\mathbb{R}^{d}$ are trainable parameters, and $\oplus$ is the concatenation operation. It is worth noting that we enumerate all spans in the sentence.

For the token-level network, we adopt an adaptive prototypical network to classify tokens. General prototypical networks (Snell et al., 2017) simply average all the tokens of the same class to get prototype. However, the average operation does not reflect the importance of each token (Gao et al., 2019; Sun et al., 2019). Therefore, we propose to construct adaptive prototypes for each query token by considering the importance of each token. Specifically,

where $\mathbf{h_{i}^{q}}\in\mathbb{R}^{d}$ is representation of query token $x_{i}$, $\mathbf{H^{s}_{j}}\in\mathbb{R}^{C_{j}\times d}$ is representation of all tokens of class $j$ in the support set, $C_{j}$ is the number of $j$-class tokens in support set, $\mathbf{\alpha_{i}^{j}}\in\mathbb{R}^{C_{j}}$ is attention weight, and $\mathbf{c_{i}^{j}}\in\mathbb{R}^{d}$ is adaptive prototype of class $j$ for query token $x_{i}$. For clarity, we denote this attention aggregation as $\phi$, i.e., $\mathbf{c_{i}^{j}}=\phi(\mathbf{h_{i}^{q}},\mathbf{H^{s}_{j}})$.

Then we utilize cross-entropy loss to optimize the network and the predicted probability distribution can be produced by a softmax function over Euclidean distances between all prototypes and query token $x_{i}$. Specifically,

where $p_{i}^{t}(\hat{y}_{i}=j|x_{i})$ is predicted probability of $j$-class for token $x_{i}$, $d$ denotes Euclidean distance, and $y_{i}^{*}$ denotes the ground-truth label for token $x_{i}$. It is worth noting that we use the IO (Inside and Outside) scheme for token-level network same as Ding et al. (2021), not BIESO (Begin, Inside, End, Singleton, and Outside) or BIO (Begin, Inside, and Outside) scheme. For example, the label space of example in Table 1 is {building-other, building-library, Outside}, not {begin-building-other, inside-building-other, end-building-other, singleton-building-other, begin-building-library, inside-building-library, end-building-library, singleton-building-library, outside} or {begin-building-other, inside-building-other, begin-building-library, inside-building-library, outside}.

For the span-level network, we classify spans by adopting an adaptive prototypical network similar to the token-level adaptive prototypical network, the difference mainly lies in that we employ an extra cross-attention module and make more fine-grained division for class O.

Intuitively, the token-level network and span-level network should make consistent predictions in a well-trained model. Therefore, we further propose a consistent loss to make the predicted probability distributions of two networks consistent. It contains two steps, where the first step constructs two token-level probability distributions for each token from two networks and the second step measures the difference between two probability distributions.

For each token, we can obtain the token-level probability distribution directly from the token-level network. Then, among all spans containing this token, we select the span with the maximum predicted probability and take its probability distribution as the token-level probability distribution from span-level network. Finally, we use bidirectional Kullback-Leibler divergence with temperature as consistent loss to allow the two networks to learn from each other and to measure the difference between the two distributions. Specifically,

where $l_{t}(x_{i})$ and $l_{s}(x_{i})$ are the predicted logits of token-level and span-level networks for token $x_{i}$, $\sigma$ denotes softmax function, and $T$ is the temperature to adjust the sharpness of the probability distribution.

Finally, we combine the above three loss functions to form the final loss function to jointly train our model as follows:

where $\lambda$, $\beta$, $\gamma$ are hyper-parameters.

In summary, there are three steps to train our model, i.e., calculate token-level loss, calculate span-level loss, and calculate consistent loss. The whole training flow is illustrated in Figure 1 and Algorithm 1.

Intuitively, we consider extracted entities and slots unreliable if the two networks predict different labels. So we propose a consistent greedy algorithm to combine the outputs of the two networks. As shown in Figure 2 and Algorithm 2, we first adjust the predicted probabilities of spans and then use a greedy algorithm to gradually select non-overlapping spans with the maximum probability based on the adjusted probability.

First, we consider a span to be reliable if all token-level predictions from token-level network in the span agree with the span-level prediction from span-level network. For example, as shown in Figure 2, the prediction of span “powerplant” is consistent, and the prediction of “nimbus powerplant” and “the nimbus powerplant” is inconsistent. We punish the probabilities of inconsistent spans by the following operation:

where $\bar{y}_{i}$ and $\hat{y}_{i}$ is the predicted probability of span $q_{i}$ after and before consistent adjustment, $\delta$ is a hyper-parameter to control the degree of penalty, and $\textrm{count}_{i}$ is the number of inconsistent tokens. Then, we use a greedy algorithm to gradually select non-overlapping spans with the maximum probability based on the adjusted probability. Specifically, at each step of the greedy algorithm, we select the span with the maximum probability and discard the overlapping span. For example, we select span “powerplant” and discard “nimbus powerplant” and “the nimbus powerplant”. The whole inference flow is illustrated in Algorithm 2.

We conduct experiments on three benchmark datasets: FewNERD (Ding et al., 2021), SNIPS (Coucke et al., 2018), and Cross. FewNERD and Cross are NER datasets, and SNIPS is a slot tagging dataset. The statistics of the three original datasets are shown in Table 3. The detailed description of the three datasets is as follows:

FewNERD dataset with a hierarchy of 8 coarse-grained entity types and 66 fine-grained entity types, and has two few-shot settings: Intra and Inter. The two settings divide the dataset using coarse-grained and fine-grained entity types, respectively. For Intra setting, all entities in the training set, development set, and test set belong to different coarse-grained types. For Inter setting, all entities in the training set, development set, and test set belong to different fine-grained types. It is worth noting that in order to better meet the dense entities, i.e., a sentence may have multiple entity types, Ding et al. (2021) adopt the $N$-way $K$~$2K$ shot sampling method to construct FewNERD dataset (each class in the support set has $K$~$2K$ entities). Both Intra and Inter setting have 4 settings: 5-way 1~2 shot, 5-way 5~10 shot, 10-way 1~2 shot, and 10-way 5~10 shot. We use the public sampled dataset¹¹1It is worth noting that we use the latest version of FewNERD dataset from https://github.com/thunlp/Few-NERD, which corresponds to the results reported in https://arxiv.org/pdf/2105.07464v6.pdf. released by Ding et al. (2021) which contains 20,000 episodes for training, 1,000 episodes for validation, and 5,000 episodes for testing.

SNIPS is a slot tagging dataset and contains 7 domains. The domains are Weather (We), Music (Mu), Play List (Pl), Book (Bo), Search Screen (Se), Restaurant (Re), and Creative Work (Cr). Hou et al. (2020) randomly sample a domain for validation, then randomly sample a different domain for testing, and the other domains are used for training. In each episode, all classes have $K$ shot samples in the support set and the number of classes ($N$) is not fixed. Each domain of SNIPS dataset has two settings: 1-shot and 5-shot. We use the public sampled dataset²²2https://github.com/AtmaHou/FewShotTagging provided by Hou et al. (2020) for a fair comparison.

Cross is a NER dataset, consisting of four datasets CoNLL-2003 (Sang and Meulder, 2003), GUM (Zeldes, 2017), WNUT-2018 (Derczynski et al., 2017), and Ontonotes (Pradhan et al., 2013). The four datasets are from News, Wiki, Social, and Mixed domains respectively. Same as the SNIPS dataset, Hou et al. (2020) use two datasets for training, one dataset for validation, and one dataset for testing. Each domain of Cross dataset also has two settings: 1-shot and 5-shot and we also use the public sampled dataset³³3https://github.com/AtmaHou/FewShotTagging provided by Hou et al. (2020) for a fair comparison.

For the FewNERD dataset, we report the micro-F1 over all test episodes following Ding et al. (2021). Specifically,

where $\textrm{P}_{all}$ denotes the precision calculated on all test episodes and $\textrm{R}_{all}$ denotes the recall calculated on all test episodes. For the SNIPS and Cross dataset, we first calculate micro-F1 for each test episode and then report the average F1 for all test episodes as the final result following Hou et al. (2022). Specifically,

where n is the number of episodes and $\textrm{F1}_{i}$ denotes the F1 of i-th episode.

We use BERT-base-uncased (Devlin et al., 2019) as our encoder and use AdamW (Loshchilov and Hutter, 2019) optimizer. For the FewNERD dataset, the learning rate is 2e-5 for BERT and 5e-4 for other parameters, and the batch size is set to 2. For the SNIPS dataset, we use grid search to find the best hyper-parameters. The learning rate of BERT is selected from {5e-6, 1e-5, 2e-5, 3e-5, 5e-5} and other parameters are selected from {5e-5, 1e-4, 3e-4, 5e-4}, and the batch size is selected from {2, 4}. We schedule the learning rate that the first 1000 training steps is a linear warmup phrase and then the rest is a linear decay phrase. $\lambda$ and $\gamma$ are selected from {0.02, 0.05, 0.1, 0.2, 0.3} through grid search, $\beta$ is set to 1, and $\delta$ is selected from {0.01, 0.02, 0.05, 0.1}. The maximum span length is set to 8 for inference.

To evaluate the effectiveness of our method, we compare our method with the following baselines:

• •

  Proto (Fritzler et al., 2019; Ding et al., 2021) represents each class by the mean of its token representation and uses Euclidean distance to predict query labels.

• •

  NNShot (Yang and Katiyar, 2020; Ding et al., 2021) is similar to Proto, but uses nearest neighbor classification to predict query labels.

• •

  StructShot (Yang and Katiyar, 2020; Ding et al., 2021) is similar to NNShot and further uses a Viterbi decoder during the inference phase to model label dependency.

• •

  TransferBERT (Hou et al., 2020) is a domain transfer model which pretrains model on source domains and fine-tunes on the target domain.

• •

  Matching (Vinyals et al., 2016; Hou et al., 2020) is similar to Proto and uses the matching networks (Vinyals et al., 2016) to classification instead of prototypical networks.

• •

  L-TapNet+CDT (Hou et al., 2020) uses label name to extend task-adaptive projection networks (Yoon et al., 2019) and further utilizes collapsed dependency transfer to capture the dependencies between labels.

• •

  Retriever (Yu et al., 2021a) is a span-level retrieval method that learns similar contextualized representations for spans with the same label via a novel batch-softmax objective.

• •

  ConVEx (Henderson and Vulic, 2021) first pre-trains through a novel pairwise cloze task using Reddit data and then directly fine-tunes for slot tagging task.

• •

  MRC (Ma et al., 2021) enumerates all the slots to extract the answer from the sentence in machine reading comprehension framework.

• •

  Inverse (Hou et al., 2022) reversely predicts slot values given slot types prompt and further proposes iterative prediction strategy to consider the relations between different slot types.

• •

  De-MAML (Ma et al., 2022b) proposes a two-step MAML-enhanced model to first detect span and then classify span.

• •

  ESD (Wang et al., 2022) proposes inter-span and cross-span attention to enhance the span representations, and uses instance span attention to construct prototype.

We conduct an empirical study to investigate whether our proposed model can achieve better performance for few-shot sequence labeling to answer RQ1. Table 5 and Table 6 report the performance of our proposed model and baselines on few-shot NER dataset (i.e., FewNERD and Cross datasets) and Table 5 reports the performance on few-shot slot tagging dataset (i.e., SNIPS dataset).

First, we observe the performance on the FewNERD dataset. We can see that our model consistently outperforms all baselines under all settings. Compared with the best baseline ESD, our model achieves 3.68 and 2.58 average F1 improvement under Intra and Inter settings. Compared with the best token-level baseline StructShot, our model achieves 14.85 and 13.18 average F1 improvement under two settings. This shows that combining token-level and span-level can achieve better performance for few-shot sequence labeling. It is worth noting that the improvement under Intra setting is higher than under Inter setting. This suggests that when the domain gap between source domain and target domain is relatively large, combining token and span levels can lead to more significant improvements than the single-granularity baselines.

Secondly, we observe the performance on the SNIPS dataset. Compared with the best baseline ESD, our model achieves 1.12 and 1.38 average F1 improvement under 1-shot and 5-shot settings on the SNIPS dataset. Compared to all baseline methods, our method achieves the best or second best performance in most domains. The results also show the effectiveness of our model. It is worth noting that the standard deviations of our method are the best or second best in most settings. This shows that our model is robust.

Thirdly, we observe the performance on the Cross dataset. Compared with the best baseline L-TapNet+CDT, our model achieves 2.5 and 8.37 average F1 improvement under 1-shot and 5-shot settings. Compared to all baseline methods, our method achieves the best or second best performance in all domains. This shows the effectiveness of our model.

To validate the contribution of each component, we report the performance of removing each of them from our model individually in Table 7 to answer RQ2.

First, we can see that all components improve performance. This shows that all components in our model are effective. Secondly, we remove the token-level loss and span-level loss. When we ablate span loss, the network uses the output of the token-level network as the result. We can see that the performance drops to 26.42 and 36.25 under Intra and Inter settings. This may be because token-level metric learning is difficult to extract entities completely and our token-level model is parameter-efficient. When we remove the token-level loss, the performance drops to 34.53 and 47.80 under two settings. This shows that using token-level labels to jointly train the model can effectively improve performance. Thirdly, we remove the consistent loss and the performance drops to 35.27 and 48.10 under two settings. This shows that consistent loss can effectively regularize the model and improve performance. Fourth, we remove the inference algorithm, i.e., all spans that are predicted as entities or slots from span-level network are extracted. The performance drops to 31.13 and 45.18 under two settings. This illustrates that our inference algorithm can effectively combine the results of two networks and avoid extracting overlapping spans. Finally, we ablate all components (i.e., fine-grained division of O, cross-attention module, and adaptive prototype) in the span network. We can see that all components in the span network improve performance. The cross-attention module brings the largest performance improvement. This indicates that the interaction of the support set and query set is the key to performance improvement. We remove the adaptive prototype and the performance drops to 32.52 and 46.27 under two settings. This shows adaptively constructing prototypes for each query span is more effective. We remove the division of class O and the performance drops to 35.63 and 48.49 under two settings. This shows that the fine-grained division of O can effectively construct prototype of class O.

We explore the effects of different inference algorithms in Table 8 to answer RQ3, including our proposed consistent greedy inference algorithm, using intersection or union operation to combine two outputs from two networks (i.e., a span and its type are extracted by both networks or by one), and using the outputs of token network or span network (i.e., we also use greedy algorithm to avoid overlapping spans).

First, we can see that our proposed inference algorithm achieves the best performance. This shows that using token-level predictions to adjust the probability of span network is effective. Secondly, we can see that the performance of span network is better than token network. This may be because token-level metric learning is difficult to extract entities completely and our token-level model is parameter-efficient. Thirdly, we report the performance of using intersection or union operation to combine the outputs of two networks. We can see that their performance does not exceed span network. This is because the poor performance of token network and simple combination does not improve performance. Finally, it is worth noting the performance of span network exceeds w/o span-level loss. This shows that using token-level loss to jointly train can improve performance of span-level network.

We explore the performance of using unidirectional Kullback-Leibler divergence with temperature as consistent loss in Table 9 to answer RQ4, i.e., only using Span network to supervise Token network (ST) or using Token network to supervise Span network (TS).

First, we can see that the bidirectional consistent loss performs better than ST and TS. This shows that the bidirectional consistent loss is more effective to align outputs from two networks compared to the unidirectional consistent loss. Secondly, ST and TS both exceed w/o consistent loss. This shows that both unidirectional consistent losses are effective to align outputs and improve performance. Thirdly, the performance of TS exceeds ST. This suggests that it is more effective to use supervision information from token network to train the span network. This may be because it is easier for span network to learn a consistent output.

We explore the performance of using different losses as consistent loss in Table 9 to answer RQ4, i.e., mean squared error (MSE), Kullback-Leibler divergence (KL), negative cosine similarity (COS), and Jensen-Shannon divergence (JS).

First, we can see that the KL loss with temperature performs best and the KL loss performs second best. This shows that using temperature can obtain better supervision information compared to KL loss. It is worth noting that KL divergence and JS divergence exceed w/o consistent loss. This shows that KL divergence and JS divergence can effectively measure the difference between two distributions and improves performance for our model. Secondly, the performance of w/o consistent loss exceeds MSE and COS. This illustrates that MSE and COS play a side effect and are not a good choice for measuring the difference between the two probability distributions for our model.

We explore the performance of different hyper-parameters (i.e., the weight of token-level loss $\lambda$ in Eq. (18), the weight of consistent loss $\gamma$ in Eq. (18), and the degree of penalty in inference algorithms $\delta$ in Eq. (19)) while fixing the other hyper-parameters in Fig. 3 on FewNERD dataset under 5 way 1~2 shot setting to answer RQ5.

We first observe the effects of $\lambda$ which is selected from {0.02, 0.05, 0.1, 0.2, 0.3}. From Figure 3(a) and Figure 3(b), we find that the performance is relatively stable under Inter and Intra settings, i.e., the performance fluctuates by 0.6% and 1.5%, respectively. As $\lambda$ changes, the performance first has an upward trend and then has a downward trend under two settings. This shows that setting a suitable $\lambda$ can help improve performance. The optimal value of $\lambda$ depends on the specific setting. The model achieves the best performance when $\lambda$ is 0.1 under Inter setting and $\lambda$ is 0.2 under Intra setting.

Then, we observe the effects of $\gamma$ and $\delta$, choosing from {0.02, 0.05, 0.1, 0.2, 0.3} and {0.01, 0.02, 0.05, 0.1}. The overall trend is the same as $\lambda$. We can see that the performance is stable under two settings, i.e., the performance fluctuates by 0.5% and 0.4% with the change of $\gamma$ and the performance fluctuates by 0.2% and 0.3% with the change of $\delta$. This shows that our model is insensitive to $\gamma$ and $\delta$. As $\gamma$ or $\delta$ changes, the overall performance also first has an upward trend and then has a downward trend under two settings. The model achieves the best performance when $\gamma$ is 0.05 and $\delta$ is 0.02 under two settings.

We compare our division with Wang et al. (2022) under two different models (i.e., CDAP and ESD) in Table 10 to show the effectiveness of our division. Wang et al. (2022) also divide class O into three sub-classes: sub-class $O_{1}$ denotes the span that does not overlap with any entities or slots in the sentence, sub-class $O_{2}$ denotes the span that is the sub-span of an entity or slot, and $O_{3}$ denotes the others.

We can see that our O division achieves better performance under two different models. This illustrates the effectiveness of dividing sub-classes according to the left and right boundary tokens. It is worth noting that our division is more effective on the ESD model. This may be because the capacity of distinguishing sub-classes for the ESD model is limited compared to our model, and our division can help ESD model to distinguish sub-classes more effectively.

Since our model considers both token-level and span-level, we compare our method with token-level methods (i.e., Proto, NNShot, and StructShot) and span-level methods (i.e., ESD and De-MAML) in terms of the number of parameters and inference time in Table 11 to show the efficiency.

First, we observe the number of parameters and we can see that Proto, NNShot, and StructShot have the least number of parameters. This is because these methods only use BERT to encode the tokens. Our model is similar to ESD and significantly smaller than De-MAML. This shows that our method is parameter-efficient and does not introduce too many parameters (i.e., 2M) compared to Proto. In addition, De-MAML uses two BERT to encode two tasks separately resulting in a huge number of parameters.

Secondly, we compare the inference time of our model and baselines with the same hardware and batch size (i.e., 1). We can see that the token-level methods are faster compared to the span-level methods. This is because the span-level methods need to enumerate all spans within a sentence whose length is less than or equal to L. Therefore, the number of spans is approximately L times that of tokens, which may bring extra computation overhead. Proto is faster compared to NNShot, and NNShot is faster compared to StructShot. This is because Proto only needs to measure the distance to the prototype, while NNShot needs to measure the distance to all the tokens in the support set. StructShot further uses a Viterbi decoder to model label dependency based on NNShot, so more inference time is required. Then, we compare our model with span-level methods and we can see that our model is 13.40 ms slower than ESD and 32.49 ms faster than De-MAML under 1-shot setting. Under 5-shot setting, our model is 26.78 ms slower than ESD and 23.44 ms slower than De-MAML. This illustrates that the time cost of our model is acceptable and our model is efficient. In addition, De-MAML needs to forward propagate 2 times through BERT, resulting in a long inference time under 1-shot setting. Our model and ESD both use the cross-attention module to model the interaction of the support set and query set, resulting in a significant increase in inference time under 5-shot setting.

We further analyze the predicted error of our model in detail and divide the error into 2 categories (i.e., FP-Span and FP-Type) according to whether the span is correct. FP-Span denotes extracted entities with the wrong span boundary and FP-Type denotes extracted entities with the right boundary but the wrong entity type.

First, as shown in Table 12, we can see that FP-Span is main prediction error under two settings. FP-span accounts for 74.52% under Inter setting and 65.46% under Intra setting. This indicates that span detection is more difficult than span classification, which is the main bottleneck at present and needs further attention in the future. Secondly, compared to Inter setting, the number of both errors rises under the more difficult Intra setting. Compared to Inter setting, FP-Span increased by 52.79% and FP-Type increased by 126.96% under Intra setting. This suggests that as domain difference between the source domain and the target domain increases, entity classification is more difficult to transfer than span detection.

Apart from the quantitative analysis, we also conduct case study to intuitively show the effectiveness and analyze our model. We list the predictions of two groups on the test set of our model and three strong baselines (i.e., StructShot, De-MAML, and ESD) on FewNERD under Inter 5-way 1~2 shot setting in Table 13.

First, we observe the predictions of the first group (Query 1, 2, and 3). We can see that the extraction capacity of StructShot is limited. For example, it only correctly extracts entity “B.A.” in Query (1) and also extracts impossible entities (i.e., “the” and “of” in Query (3)). This illustrates that StructShot relies heavily on the labeled samples in the support set. Then, we observe the predictions of span-level methods. For query (1), De-MAML and ESD predict None, and our model extracts “B.A.”. This may be because we use token-level loss to jointly train the model. For query (2), we can see that our model correctly extracts and predicts “Monaco” and “Maxim’s”. De-MAML fails to classify “Maxim’s” and ESD fails to extract “Maxim’s”. This shows that even if a sentence has multiple entities, our model can extract them. For query (3), we can see that our model correctly extracts and predicts “faculty of medicine in Paris”. De-MAML and ESD extract “faculty of medicine”, and ESD predicts wrong type. This shows that even if an entity has multiple tokens, our model can locate and extract it.

Secondly, we display some failure examples in the second group (Query 4, 5, and 6). For query (4), method StructShot extracts “Right” and fails to extract simple entity “Republican”. This also shows that the extraction capacity of StructShot is limited and relies heavily on the labeled samples in the support set. Methods De-MAML, ESD, and our model do not extract entity “neo-Confederate Right” at all. This may be because it is more difficult to extract this entity. For query (5), we can see that StructShot does not extract the entity completely, only “bachelor” is extracted. De-MAML and our model also do not extract the entity completely, only “bachelor of law” is extracted. This indicates that there are challenges in locating the boundaries of entities. ESD does not extract entities, perhaps because it is trained with only span labels and the supervision is sparse. For query (6), we can see that all the methods predict None except De-MAML and De-MAML predicts wrong entity “New American”. We observe the corresponding support set and find that it may be due to the weak representation of the prototype constructed from the two samples (i.e., Bobino and Gaumont-Palace) in the support set of the class.