Incremental Few-shot Text Classification with Multi-round New Classes:
Formulation, Dataset and System

As far as we know, there is no prior work in the NLP domain that studies incremental few-shot text classification. In this section, we mainly introduce some work in the computer vision domain. As we mentioned before, they only assume that a single round of new classes $C_{n}$ is appended to the base classes $C_{b}$. Generally, they will learn class representations for classification. Different approaches differ in the way of representing base classes and new classes. Hereafter, we use $W_{b}$ and $W_{n}$ as the representations for $C_{b}$ and $C_{n}$, respectively.

DBLPSnellSZ17 proposes the Prototypical Network, in which both $W_{b}$ and $W_{n}$ are stored as the average embedding of the few-shot support images for a certain class. Although Prototypical Network was not designed for incremental few-shot learning, it can be easily adapted to the incremental setting by providing the representations for all the classes. It trains a nearest neighbor algorithm on the base classes and tests directly on the union of base and new classes. DBLPQiBL18 proposes an “imprinting” mechanism: the base representations $W_{b}$ are learned regularly through supervised pre-training (e.g., the weight matrix in a softmax classifier), and $W_{n}$ are computed using the averaged representations like Prototypical Network.

In DBLPGidarisK18, the base representations $W_{b}$ are learned through supervised pre-training. The representation of the $i^{th}$ novel class ($W_{n,i}$) comes from two origins: (i) the prototypical averaging, $w_{avg}$; (ii) attention-weighted sum over base representations: $w_{att}$. Namely, $W_{n,i}=\phi_{avg}\odot w_{avg}+\phi_{att}\odot w_{att}$, where $\phi_{avg}$ and $\phi_{att}$ are learnable weight vectors. In the few-shot training stage, the original base classes $C_{b}$ are split into “new base classes” and “fake novel classes” for each episode. The loss is computed when the system predicts “new base classes” as well as “fake novel classes”. In testing, $W_{n}$, the representations of novel classes, are constructed based on the $k$ examples and $W_{b}$.

In DBLPRenLFZ19, both $W_{b}$ and $W_{n}$ are learned through supervised training: $W_{b}$ are classifier parameters pre-trained on base classes, $W_{n}$ are classifier parameters learned in new classes. During the training, the support set and the query set are constructed differently for new classes. The support set consists of examples only from new classes; the query set contains examples from both new classes and base classes (because the training goal is to maximize the performance on all classes). The training in this literature has two phases. The first phase is few-shot episode training which learns $W_{n}$ and optimizes the performance on the support set; the second phase (called meta-learning training) optimizes the performance on the query set. The latter has a regularization term on the $W_{n}$, enforcing $W_{n}$ to be as close as possible to the attention-weighted sum of $W_{b}$.

To summarize, compared with DBLPSnellSZ17 and DBLPQiBL18, both DBLPGidarisK18 and DBLPRenLFZ19 build connections between the representations of base classes and the new classes. However, these methods cannot be directly applied to our problem for the following reasons. (i) Despite the claims in some literature that they are dealing with incremental or dynamic few-shot problems, they only considered a single round of new classes DBLPQiBL18; DBLPGidarisK18; DBLPRenLFZ19. It is unclear if the system can keep the performance when multi-round new classes are considered. (ii) During the training for the new classes, they often rely on extra labeled data other than the $k$ examples, such as the query set in DBLPRenLFZ19. (iii) Different from their setting, we have an extra label “out-of-distribution” in incremental few-shot text classification. It’s not guaranteed that the input, such as the customer’s utterance, always falls into the range of seen labels.

zhangdiscriminative is a state-of-the-art paper for few-shot text classification. They propose a discriminative nearest neighbor classification (DNNC) model by comparing whether two examples are in the same class or not. A matching model S($x_{i}$, $x_{j}$) is trained as a binary classifier, such that S($x_{i}$, $x_{j}$) is closed to 1.0 if $x_{i}$ and $x_{j}$ belong to the same class, otherwise closed to 0.0. Thus, their model can be pre-trained with large-scale textual entailment dataset. Given a test query $x$, they compare the test query with all the previous examples. The final prediction is made by searching the nearest neighbor which has the highest matching score S($x$, $x_{i}$) with the query example. As we mentioned before, their computation cost is high and the performance is highly related to the quality of chosen examples.

Moreover, comparing whether two examples are in the same class is different from doing textual entailment. In textual entailment, a human reading a premise to infer that the hypothesis is true or not. The fact that two examples are in the same class does not mean they can entail each other. Thus, they cannot fully utilize the pre-trained entailment model. Instead, our proposed model, Entailment, entail the label with a given utterance, which is much more efficient and maximize the utilization of the pre-trained entailment model.

DBLPYinHR19 is another work that utilizes textual entailment for zero-shot text classification. They convert the zero-shot text classification as a problem of filling a label candidate for a hypothesis. For example, they combine “emotion” labels with the question “this text expresses ?”, and ask the model if this hypothesis is true, given the text. This work more focus on zero-shot learning and they need to propose different questions for different labels.

For the setting with base classes, the system is provided with a set of base classes $C_{b}=\{C_{b,1},C_{b,2},\cdots,C_{b,g}\}$; each base class $C_{b,i}$ ($i=1,\cdots,g$) has $l$ labeled examples. $l$ is usually a large number, like several hundred or several thousand. For the setting without base classes, $C_{b}$ is ignored. Both settings have totally $m$ rounds of new classes coming sequentially: {$C_{n}^{1},\cdots,C_{n}^{m}$}. Each round $C_{n}^{i}$ has $h$ new classes, namely $C_{n}^{i}=\{C^{i}_{n,1},\cdots,C^{i}_{n,h}\}$. Each new class only has $k$ examples ($k\in[1,5]$). The value of $k$ is not fixed and varies for different new classes in the same round, i.e., $k_{C^{i}_{n,i}}\neq k_{C^{i}_{n,j}}$.

We create the multi-round setting since it can evaluate the system more precisely while learning a long line of new classes. In each round, we set $k\in[1,5]$ and allow the flexibility that $k_{C^{i}_{n,i}}\neq k_{C^{i}_{n,j}}$. This setting is more in line with reality, in which we can only collect a handful of examples for the upcoming classes and the number of examples cannot be guaranteed.

To simulate real scenarios in the experiments, we assume that there are only $k$ examples available for the new classes during the incremental training. Thus, our formulation does not provide any development set to help select the best model. It is recommended to select hyper-parameters based on experience or related tasks.

To evaluate the system, the test data consists of examples across all the classes. For the setting with base classes, the potential label space is $C_{b}\cup C_{n,1}^{1}\cdots C_{n,h}^{1}\cdots C_{n,1}^{m}\cdots\cup C_{n,h}^{m}\cup C_{o}$. For the setting without base classes, $C_{b}$ is excluded and we only search among the classes in $C_{n,1}^{1}\cdots C_{n,h}^{1}\cdots C_{n,1}^{m}\cdots\cup C_{n,h}^{m}\cup C_{o}$. $C_{o}$ is an extra out-of-distribution (OOD) class in which examples falling outside of all the seen classes. It give us a chance to check the system’s ability to detect instances that reject all the known classes. This is crucial for an open-set problem like incremental learning, since there are always examples from upcoming classes that do not belong to any existing class.

(i) For the training of $i^{th}$ round $C_{n}^{i}$, the system can only access the newly added few-shot examples in this round and preceding class names in $C_{b}\cup C_{n}^{1}...\cup C_{n}^{i-1}$. The system is not allowed to re-train on the (full or partial) examples of preceding classes. (ii) For the evaluation, we care about the performance in different types of classes, including base classes, different rounds of new classes and OOD classes in $C_{o}$. We expect a system that can continuously recognize new classes well with few-shot examples. In the meantime, the performance drop for preceding classes is also considered. A system showing severer catastrophic forgetting is less preferred.

To transfer text classification problem into textual entailment, we construct positive and negative entailment pairs for the training. Positive entailment pairs ($x_{i}$, $y_{i}$) are constructed with utterance $x_{i}$ and its gold label name $y_{i}$, where $y_{i}\in C_{b}$ for base classes and $y_{i}\in C_{n}^{i}$ for new classes. Negative entailment pairs consist of ($x_{i}$, $y_{j}$), where $y_{j}$ is an incorrect label in current round. For base classes, $y_{j}\in C_{b}$ but $y_{j}\neq y_{i}$; for new classes, $y_{j}\in C_{n}^{i}$ but $y_{j}\neq y_{i}$.

Compared to zhangdiscriminative, their entailment pairs are constructed with two utterances in the same round. ($x_{i},x_{j}$) is a positive pair if they belong to a same class, otherwise it is a negative pair. In order to explore the potential of different combinations, we also propose a hybrid entailment model that use both (utterance, label) paris ($x_{i},y_{i}$) and (utterance, utterance) pairs ($x_{i},x_{j}$). In this hybrid model, we train the model with entailment pairs from both our proposed model and zhangdiscriminative.

In the setting where we have $g$ base classes and $l$ examples for each base class, $g*l$ positive entailment pairs and $(g-1)*g*l$ negative pairs are generated for them. For round $C_{n}^{i}$ which contains $h$ new classes and $k$ examples for each class, there are $h*k$ positive entailment pairs and $(h-1)*h*k$ negative entailment pairs. For simplicity, we use the same $k$ value for all new classes here; in real datasets, different new classes may have different numbers of few-shot examples. In that case, the number of generated pairs will change accordingly.

For each entailment pair ($x,y$) no matter it is positive or negative, we concatenate its utterance $x$ with the label $y$ and fed into the RoBERTa liu2019roberta encoder. Given an utterance $x=(X_{1},X_{2},...,X_{T_{2}})$ with $T_{1}$ words and a label $y=(Y_{1},Y_{2},...,Y_{T_{2}})$ with $T_{2}$ words, we add a special start-of-sequence ([CLS]) token at the beginning of the input and a special end-of-sequence ([SEP]) token at the end of each sentence. The whole input is ([CLS], $X_{1}$, $X_{2}$, …, $X_{T_{1}}$, [SEP], $Y_{1}$, $Y_{2}$, …, $Y_{T_{2}}$, [SEP]). We use the [CLS] embedding output from the RoBERTa encoder with a fully connected layer for binary textual entailment: \linenomathAMS

	$\displaystyle h=$	$\displaystyle\text{RoBERTa}(x,y),$		(1)
	$\displaystyle p=$	$\displaystyle\text{softmax}(Wh+b),$		(2)

where $h\in\mathbb{R}^{d}$ is the embedding for the [CLS] token, $W\in\mathbb{R}^{2\times d}$ and $b\in\mathbb{R}^{2}$ are parameters.

Our model is a binary classification model that can utilize the indirect supervision from textual entailment. Firstly, we pre-train the model with a large-scale entailment dataset williams2018broad. For the setting with base classes, we fine-tune the model on entailment pairs from base classes to obtain a base model. Then it is fine-tuned on the new classes $C_{n}^{i}$ in each round. In the setting without base classes, the model is directly fine-tuned with the entailment pairs from new classes in each round $C_{n}^{i}$.

After the training, we use the model to infer the class for a test input. For each input, we generate entailment pairs by accompanying the input with all classes except $C_{o}$. Each pair will get a score $\lambda\in[0,1]$ indicating whether this input belongs to the particular class or not. $\lambda>0.5$ indicates “YES”, “No” otherwise. If there is at least one class labeled with “YES”, the class with the maximal $\lambda$ score is returned; otherwise, the system returns $C_{o}$. We choose the threshold as 0.5 because entailment recognition is a binary classification problem.

Next, we compare our model with some related systems that can be potentially applied to the incremental few-shot text classification.

DNNC zhangdiscriminative also converts text classification problem into textual entailment. They discriminate whether two text inputs ($x_{i},x_{j}$) are in the same class. As a result, for round $C_{n}^{i}$ with originally $h*k$ examples, this baseline will generate $h*k*(k-1)$ positive pairs and $h*(h-1)*k^{2}$ negative pairs. In testing, a query needs to compare with all the examples of all classes. The computation cost of this baseline is much higher than our proposed model.

Prototypical Network DBLPSnellSZ17 tries to solve few-shot target tasks given a collection of training tasks. The distributions in the training tasks and the target tasks are required to be similar. Prototypical network uses episode training to learn a nearest neighbor algorithm and hope it can generalize well to the target tasks.

The few-shot learning problem solved in Prototypical network is slightly different with our incremental few-shot setting. In Prototypical Network, the label space for target tasks only contains the new classes. However, in the incremental few-shot setting, the target label space is continuously increasing by adding new classes. Due to this essential distinction, applying Prototypical Network to incremental few-shot are very likely to have performance drop on base classes when fine-tuning on new classes.

In Related Work, we introduced some typical approaches in computer vision that deal with incremental few-shot problem. Those methods consistently try to learn representations for classes and examples separately (i.e,, the $W_{b}$ and $W_{n}$ in Section 2). In our model, there are no individual representation vectors for classes or examples. Instead, the model learns an overall representation vector for the whole (input, class) pair. Our solution enables the learning of the input and the class to interact with each other, which has widely demonstrated its superiority in modeling the relations of two elements DBLP12808; zhangdiscriminative.

In addition, the approaches in computer vision mostly rely on large-scale labeled data for base classes to train a robust system. We would argue that the base classes with rich annotations may not be available in real-world applications. Our system which can be pre-trained with entailment dataset, instead, does not rely on base classes. This makes our system more applicable to various scenarios.