Incremental Sequence Labeling: A Tale of Two Shifts

The conventional sequence labeling task typically involves categorizing data into a predetermined set of fixed categories Lample et al. (2016). However, this approach may need to be revised in natural language processing scenarios, such as the named entity recognition task, where new types of entities continuously emerge. Adapting a fixed set of categories becomes challenging when faced with the dynamic nature of new entity classification requirements. Consequently, continuous model updates are essential to accommodating evolving entity types. Previous studies have advocated for adopting continual learning Parisi et al. (2019); Monaikul et al. (2021), also known as lifelong learning or incremental learning. Continual learning is a paradigm designed to train models capable of adapting to the continual addition of new categories in real-world scenarios while ensuring that knowledge of old categories is retained. For instance, voice assistants like Siri frequently encounter new event types, such as pandemics, need to understand and provide health-protecting information based on the user’s latest intent. Monaikul et al. (2021).

Due to constraints imposed by storage limitations and privacy concerns, there exists a shortage of training data about the old categories He and Zhu (2022). Additionally, the manual relabeling of all categories within the new training dataset would incur substantial costs and time investment De Lange et al. (2021); Bang et al. (2021). Consequently, the model undergoes continuous updates using a freshly acquired dataset comprising the new categories. As depicted in Figure 1, the model undergoes training based on the current ground-truth label and undergoes testing using the full ground-truth label.

The incremental sequence labeling task faces a significant challenge known as the catastrophic forgetting problem, as extensively discussed in previous studies McCloskey and Cohen (1989); Robins (1995); Goodfellow et al. (2013); Kirkpatrick et al. (2017); Zheng et al. (2024). This issue manifests as semantic shifts, leading to a decrease in the discriminative power of entity classes Zhang et al. (2023b); Ma et al. (2023). In this paper, we decompose the problem into two primary semantic shifts in the incremental sequence task: E2O and O2E. The first semantic shift, E2O, arises from the presence of non-entities, potential old entities (mislabelled as non-entities), and new entities in the new dataset. Progress has been made in addressing E2O through methods falling into three categories: (1) Methods based on knowledge distillation: For instance, RDP proposes a knowledge distillation loss incorporating inter-task relations Zhang et al. (2023b). At the same time, CFPD introduces a pooled feature distillation loss to alleviate catastrophic forgetting Zhang et al. (2023a). (2) Methods based on pseudo-labels: OCILNER utilizes class prototypes to label new data Ma et al. (2023), and CPFD employs old models to label predictions of new data. (3) Methods based on freezing models: Examples include ICE Liu and Huang (2023), which freezes the backbone model and old classifiers to maintain the stability of the old classes at the expense of learning new classes.

Existing methods primarily focus on addressing the E2O shift, neglecting the bias towards the emergence of new classes and the consequential second semantic shift, O2E. To address both semantic shifts, we propose a novel framework called Incremental Sequential Labeling without Semantic Shifts (IS3). IS3 consists of two key components: First, we apply the knowledge distillation method to tackle the E2O shift. Second, we address the O2E shift on two fronts. At the loss function level, we introduce a debiased cross-entropy loss function to mitigate the model’s impact on old class distributions, reducing its inclination towards new entities. At the optimization level, we introduce a prototype-based approach to balance the imbalanced contributions of old and new entities during batch updates, which aims to increase the involvement of old entities in the optimization process. Importantly, IS3 adopts a storage-efficient approach, maintaining only one prototype per class with minimal storage costs. Class feature centers serve as prototypes, ensuring no direct correspondence to actual sample information and mitigating privacy leakage concerns.

The contribution of our work can be summarized as follows:

• •

  We propose a novel perspective on the semantic shift problem in incremental sequence labeling task by categorizing the catastrophic forgetting problem into E2O and O2E.

• •

  We propose a novel framework, Incremental Sequential Labeling without Semantic Shifts (IS3), to solve the two semantic shifts simultaneously.

• •

  We conduct experiments under nine CIL settings on three datasets, and our method outperforms the previous state-of-the-art methods.

Incremental Learning The model continually acquires new tasks intending to achieve optimal performance on tasks previously learned Gepperth and Hammer (2016); Wu et al. (2019); van de Ven et al. (2022); Zheng et al. (2023b). There are three main categories of current incremental learning methods: regularization-based, rehearsal-based, and architecture-based. Regularization-based methods place constraints on model weights Kirkpatrick et al. (2017); Zenke et al. (2017), representations of intermediate layer features Hou et al. (2019); Douillard et al. (2020), and output probabilities Li and Hoiem (2017); Zheng et al. (2023a). Rehearsal-based methods overcome forgetting by saving some of the data containing the old classes for learning with the new classes Lopez-Paz and Ranzato (2017); Shin et al. (2017). Alternatively, architecture-based approaches involve dynamically expanding the network structure to allow for more data as new classes are added Hou et al. (2018); Yan et al. (2021).
Incremental Sequence Labeling The traditional sequence labeling task is the task of labeling each token of a one-dimensional linear input sequence, which requires each token to be categorized according to its contextual contentRei et al. (2016); Akbik et al. (2018). However, previous methods can only recognize classes in a fixed set. Therefore, continuous learning paradigms are introduced in sequence labeling tasks, including incremental named entities Monaikul et al. (2021); Zheng et al. (2022); Zhang et al. (2023a), incremental event detection Cao et al. (2020); Yu et al. (2021), and so on.

Methods for incremental sequence labeling tasks can be categorized into distillation-based, rehearsal-based, and other approaches. Distillation-based methods encompass ExtendNER Monaikul et al. (2021), which is the pioneer in applying knowledge distillation to incremental sequence labeling task, RDP Zhang et al. (2023b) with a relational distillation approach, and CPFD Zhang et al. (2023a) utilizing pooled features distillation loss. CFNER Zheng et al. (2022) introduces a causal framework for extracting new causal effects in entities and non-entities. Rehearsal-based approaches include KCN Cao et al. (2020) and KD+R+K Yu et al. (2021), both employing rehearsal samples to address the class imbalance and catastrophic forgetting in incremental event detection. L&R Xia et al. (2022) proposes a learn-and-review framework by training a new backbone model and a generative model simultaneously, generating synthetic samples of the old class to be trained with new samples. OCILNER Ma et al. (2023) uses rehearsal samples to compute class feature centers as class prototypes, generates an entity-oriented feature space through comparative learning, and annotates new data with pseudo-labels using class prototypes. Other methods encompass span-based and freezing model-based approaches, among others.

The mentioned methodologies primarily focus on preserving the existing knowledge of the model and do not explicitly consider the implications of transitioning from non-entity to entity semantics. In contrast, our proposed method, IS3, provides a fresh perspective on model forgetting by addressing the model’s inclination towards new classes during task adaptation. IS3 not only addresses issues related to model mislabeling, indirectly mitigating the problem of semantic migration from entity to non-entity, but also handles the challenge of semantic migration from non-entity to entity. By recognizing and addressing the model’s bias towards new classes during adaptation, our approach offers a comprehensive solution to the dynamic challenges associated with transitioning between different semantic categories.

Formally, the objective of incremental sequence labeling is to acquire knowledge through a series of tasks $\mathcal{T}=\{\mathcal{T}_{1},\mathcal{T}_{2},\ldots,\mathcal{T}_{N}\}$. Each task contains its dataset $\mathcal{D}_{t}=\{(x^{i},y^{i})|y^{i}\in\mathcal{Y}_{t}\}$ where $(x^{i},y^{i})$ is a pair formed by the input token sentence and the label corresponding to each token in the sentence and $\mathcal{Y}_{t}$ stands for the current label set. Notably, $y^{i}$ only labels the token corresponding to the current task $t$, and the other tokens are labeled as O class (potential old entities $\mathcal{Y}_{1:t}$, and unseen entities $\mathcal{Y}_{t+1:N}$). At task $t$ ($t>1$), the new model $\mathcal{M}_{t}$ learns only from the new dataset and is expected to perform well on the learned classes $\bigcup_{i=1}^{t}\mathcal{Y}_{i}$.

In this section, we systematically address the catastrophic forgetting problem by decomposing it into two distinct semantic shift challenges (Section 4.1). Subsequently, we present a comprehensive framework designed to address these semantic shifts individually, focusing on E2O in Section 4.2 and O2E in Section 4.3. The overarching goal is to effectively mitigate the catastrophic forgetting problem, as illustrated in Figure 4.

In this paper, we introduce a novel perspective on the catastrophic forgetting problem in incremental sequence annotation, identifying and addressing both E2O and O2E semantic shifts. Bridging gaps in previous research, we propose the IS3 framework to tackle both issues. Comprehensive experiments on three datasets demonstrate that our IS3 method significantly outperforms previous state-of-the-art approaches. This work provides a fresh outlook on the incremental sequence labeling task and offers effective solutions to mitigate the catastrophic forgetting problem.

While the proposed method effectively mitigates catastrophic forgetting to some extent, its reliance on the predictions of old models for preserving existing knowledge can result in accumulated prediction errors, which may lead to poor model performance in more incremental steps. Moreover, the current method does not thoroughly explore the relationship between the penalty effect and the dataset, leaving potential avenues for future research.

The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant No. 62272173), the Natural Science Foundation of Guangdong Province (Grant Nos. 2024A1515010089, 2022A1515010179), the Science and Technology Planning Project of Guangdong Province (Grant No. 2023A0505050106), and the National Key R&D Program of China (Grant No. 2023YFA1011601).