Evaluating and Improving Continual Learning in Spoken
Language Understanding

The main issue that continual learning aims to address was formulated as the stability-plasticity dilemma Mermillod et al. (2013). An ideal system is expected to possess both plasticity, enabling the rapid acquisition of new knowledge, and stability, preventing the forgetting effect of previously learned knowledge.

As the two sides of the same coin, stability and plasticity are often regarded as a trade-off between learning and remembering. Assume that a model is continuously encountering data from different tasks $\{{\mathcal{D}}_{1},{\mathcal{D}}_{2},\cdots,{\mathcal{D}}_{T}\}$ in chronological order, where $T$ indicates the total number of tasks. Among different ${\mathcal{D}}_{t}$s, the distribution of data and labels may shift across tasks. If the model has a high memory stability, it will remain stable and not easily erase the knowledge learned in the past tasks, thus may hinder the capability of the model to swiftly accommodate new data distributions. Conversely, excessive plasticity may impede the stability of the system by prioritizing adapting to distribution changes. Therefore, different metrics are often applied to assess the stability and plasticity of continual learning algorithms.

We formulate the metric evaluation in continual learning with the train-evaluation performance matrix $\mathbf{A}$, which is a $T\times T$ matrix where $T$ is the total number of tasks. Each entry $\mathbf{A}_{i,j}$ represents the evaluated performance on task $j$ after the model has learned task $i$. Figure 2 provides a visual representation of the matrix. The entries can be classified into two categories, where the green circles represent the seen tasks and the grey circles represent the unseen tasks.

In this work, we establish the taxonomy of evaluation metrics based on the property that a metric is trying to assess in the model, namely stability, plasticity, and generalizability. We propose a novel metric and compare it with existing metrics in terms of formulation and experimental results. By considering different perspectives, it provides valuable insights into the evaluation of continual learning systems.

Based on the train-evaluation performance matrix, we provide a formulation of the existing continual learning metrics. Along the diagonal of the train-evaluation matrix as in Figure 2, the circles indicate the immediate evaluation of task $i$ after learning the same task. Therefore, it reflects the plasticity of the model by showing how well and fast the model is adapting to new knowledge. Current task accuracy (cur-ACC) is one common metric to assess the plasticity by calculating the average accuracies of tasks from $1$ to current task $t$ along the diagonal of the evaluation matrix:

The lower triangular matrix of $\mathbf{A}$ reflects the stability. At task $t$, it covers the evaluation results of all previous tasks $1$ to $t-1$. One of the most straightforward stability-based metrics is last accuracy (last-ACC), which assesses the stability by calculating the average accuracy of all tasks after the last task $T$:

However, last-ACC provides only one-time evaluation at the end and does not cover all entries in the lower triangular matrix. Therefore, it provides little information on the stability of the entire learning process. To better quantify the overall stability, backward transfer (BWT) measures the performance degradation of an arbitrary task $t$ on its previous tasks from $1$ to $t-1$ Lopez-Paz and Ranzato (2017). It calculates the average differences between the evaluation on a previous task $j$ after learning current task $i$ ($\mathbf{A}_{i,j}$), and the immediate evaluation of the previous task $j$ ($\mathbf{A}_{j,j}$), i.e.,

Typically, the BWT score is a negative number, indicating that losing past information is almost inevitable during the learning process of new tasks.

Forward transfer (FWT) is a metric used to assess generalizability. It is mostly used as an opposite metric compared to BWT. We note that we refer to the implementation of the Continuum library ¹¹1https://github.com/Continvvm/continuum, which evaluates the influence that learning task $t-1$ has on the performance of the future incoming task $t$. Since the model has not seen task $t$ yet, FWT quantifies the generalizability by calculating the gain of the current model performance on the incoming future task ($\mathbf{A}_{i-1,i}$), compared with zero-shot evaluation results on the prospective task ($\bar{\mathbf{A}}_{i}$), i.e.,

In this work, beyond limited and entangled evaluations of stability, plasticity, or generalizability, we propose an evaluation metric named Dual-transfer Matching Index (DMI), to provide a unified evaluation for continual learning. Unlike existing metrics discussed in Section 2.2 that only show how the knowledge transfers in a unidirectional manner, either backward or forward, our metric evaluate the continual learning behavior of the model in terms of all three properties. Meanwhile, instead of entangling plasticity versus stability and generalizability, our DMI provides a disentangled quantification for each of the three properties.

Assume that our training process elapses along the x-axis of the train-evaluation performance matrix $\mathbf{A}$. The model is continuously learning on a sequence of tasks $1,2,\dots,T$. At the end of training task $i$, instead of evaluating the accuracy of one single previous or future task, we evaluate the current model on all seen and unseen classes. However, since the model has not yet gathered information for tasks $t\succ i$, we can only evaluate these tasks in a class-agnostic manner.

Let us denote the total number of data samples as $N$ and the number of classes as $K$. Specifically, we first extract the predicted intent class embeddings $\{{\mathbf{x}}^{(i)}_{n}\}_{n=1}^{N}$ at each task $i$. Then we perform k-means clustering $\mathcal{K}(\cdot)$ MacQueen (1967); Lloyd (1982) on ${\mathbf{x}}^{(i)}$, with the total number of clusters as $K$. By denoting $\boldsymbol{\mu}_{k}^{(i)}$ as the centroid of cluster $k$ at task $i$, we can obtain the class-agnostic assignment as

At each task $i$, ${\mathbf{k}}^{(i)}=(k_{1}^{(i)},k_{2}^{(i)},\dots,k_{N}^{(i)})$ indicates the class-agnostic assignment for all $N$ samples. Class-agnostic means that ${\mathbf{k}}^{(i)}$ is not aware of the exact class index, as there exist unseen tasks that the model has not learned from yet. Then we compute the matching score between the class-agnostic assignment ${\mathbf{k}}^{(i)}$ and the ground-truth class labels ${\mathbf{k}}_{\text{gt}}$ using Hungarian maximum matching algorithm $\mathcal{H}(\cdot,\cdot)$ Kuhn (1955). As a result, $\mathcal{H}({\mathbf{k}}^{(i)},{\mathbf{k}}_{\text{gt}})$ gives us an optimal matched mapping from ${\mathbf{k}}^{(i)}$ to ${\mathbf{k}}_{\text{gt}}$. After that, we calculate the accuracies of the best mapping with the ground-truth labels to populate the train-evaluation performance matrix ${\mathbf{A}}$, i.e.,

We perform similar computations for each task $i$ from $1$ to $T$ to fulfill all entries of the matrix ${\mathbf{A}}$. Finally, we compute the DMI as a quantification for stability, plasticity, and generalizability respectively by taking the mean of the entries in the corresponding regions as shown in Figure 2:

Note that we complete the performance matrix ${\mathbf{A}}$ with prior metrics in different ways. We compute the entries of matrix ${\mathbf{A}}$ by the overall clustering accuracies of both past and future tasks, as well as the current task. In this way, DMI consists of three separate scores to measure the stability, plasticity, and generalizability during the training process of the model. Therefore, it is expected to provide a disentangled quantification of the three properties, and reflect how the learned knowledge accumulates and evolves in a unified manner.

We establish our pipeline in a class-incremental learning (CIL) setting in SLU. Specifically, the model is continuously adapted to a sequence of different tasks, and incremental intent labels emerge sequentially across tasks. CIL is a challenging setting in real-world scenarios, as the model is agnostic to task labels during inference time Hsu et al. (2018).

We utilize a combined architecture for automatic speech recognition (ASR) and SLU, as it is shown that the auxiliary use of the ASR transcriptions can lead to better SLU performances Arora et al. (2022); Cha et al. (2021). Under this setting, we prepend intent tokens to the original transcription and separate them using a special token $\langle\textit{\_SEP}\rangle$. This operation extends the transcription and transforms the modeling into a sequence-to-sequence (seq2seq) problem. As a result, our architecture diverges from the conventional continual learning pipelines that consist of a feature extractor followed by a classifier. Instead, we employ a single ASR decoder that takes the encoder outputs and generates predicted tokens in a sequential manner. Assume that the input audio-text pair is denoted as $({\mathbf{a}},{\mathbf{t}})$. The goal of the ASR decoder is to find the most probable extended transcription sequence ${\mathbf{t}}$ given the audio input ${\mathbf{a}}$, i.e.,

The objective of knowledge distillation (KD) Hinton et al. (2015) aligns with continual learning by transferring knowledge from a teacher model to a student model. In continual learning, the teacher could refer to the model learned from previous tasks or a pretrained large network, while the student means the model that is learning currently. It is noteworthy that a small portion of previous data (typically smaller than $5\%$), together with its extracted features, is saved into one rehearsal memory for the future use of teacher models. We apply multiple types of KDs to improve the stability, plasticity, and generalizability of the SLU model. Figure 3 shows the multiple KD techniques that we have leveraged in our pipeline.

We base our experiments on the intent classification of Fluent Speech Commands (FSC) Lugosch et al. (2019) and Spoken Language Understanding Resource Package (SLURP) Bastianelli et al. (2020) datasets. FSC includes 30,043 English utterances, recorded at 16 kHz. It provides 248 different utterances that are mapped in 31 different intents (i.e., our classes). SLURP contains roughly 72K audio recordings of single-turn user interactions with a home assistant, annotated with scenarios, actions and entities. Overall, there are 18 unique scenarios and 69 intents. Following Cappellazzo et al. (2023), we divide the SLURP dataset into tasks using the scenario labels as splitting criterion, whereas for FSC we use the intents. We choose to divide the datasets into $T=6$ tasks, with each task containing different classes. Implementation details can be found in Appendix A.1.

By default, we establish the order of tasks based on the principle of prioritizing the most populated scenarios. This is expected to ensure the model to learn richer information in early tasks, thus mitigating the catastrophic forgetting effect. We will provide the overall evaluation results with the default setting in Section 4.2, and further investigate the effect of task orders in the later Section 4.3.

We evaluate the continual learning performance on both FSC and SLURP with metrics introduced in Section 2.2 including accuracy (Acc), backward transfer (BWT), forward transfer (FWT) Lopez-Paz and Ranzato (2017) and our proposed DMI. The accuracy is reported as last-ACC in Section 2.2. The experimental results are shown in Table 1.

The first row in the table is the naive fine-tuning result, without any continual learning algorithms applied. Therefore, it serves as the lower bound of our performance. The accuracy is relatively low and the BWT gives a substantial negative number, which offers us a general understanding of the extent to which the forgetting effect impacts the performance in the absence of continual learning techniques.

Recall that we mentioned in Section 3.3 that we are saving a portion of past data as the rehearsal memory for the future use of later tasks. In the implementation, we set the ratio as $1\%$. To retrieve samples from the rehearsal memory, we employ two sampling strategies: random sampling and herding-based sampling Rebuffi et al. (2017). Different from random sampling, herding-based sampling strategy selects the samples that are closest to a small subset of exemplars in their class. This is expected to make the sampling process more robust to the changes in data representations, thus improving the continual learning performance.

From Table 1, the results of random and herding-based sampling both yield significantly better performance than the fine-tuning experiment, demonstrating improved continual learning performance. We can also observe that herding-based sampling performs almost consistently better than random sampling.

Additionally, we incorporate various KD techniques as described in Section 3.3 to validate their effectiveness. As we have introduced previously, audio-KD and seq-KD are expected to increase the stability of the system by regularizing parameter changes at the encoder and decoder levels. By leveraging semantic information from a pretrained text encoder, sent-KD may increase the generalizability of continual learning. Table 1 presents the results of adding three KDs sequentially. We can observe that both BWT and the first number of DMI ($\text{DMI}_{stab}$) are increased after adding audio-KD and seq-KD, indicating that the stability of the model is improved. On the other hand, sent-KD boosts the generalizability by leveraging pretrained semantic information, reflected in the increase of FWT and the third number of DMI ($\text{DMI}_{gen}$). However, as the effect of stability-plasticity dilemma discussed in Section 2.1, the plasticity of the model ($\text{DMI}_{plas}$) might be decreased as a trade-off when KD techniques are added to regularize the model. The reason is that we are penalizing the parameter changes in the model with KDs from the teacher model of the previous task, thus potentially slowing down the learning process on the current task. Such a phenomenon is not reflected in prior metrics.

We also note that both BWT and FWT improve with the introduction of additional KD techniques. This is due to their entanglement with the plasticity as shown in Figure 1, therefore may not fully reflect the stability/generalizability of the model. As a comparison, our DMI metric provides a disentangled and unified evaluation of all three properties, and a more comprehensive view of how the model behavior changes and evolves during the continual learning process.

In addition to quantitative performance, we show the qualitative results of clustering with t-SNE Van der Maaten and Hinton (2008) in Figure 4. For visualization purposes, we select one class from each task to plot. Figure 4(a) to (c) depict how the class-agnostic clustering evolves during the learning process from the first task to the last. The greater the distance between clusters of different colors, the better the model’s performance. In the beginning, although the model has not encountered 5 out of the 6 classes from later tasks yet, it still exhibits some generalizability to distinguish unseen classes from seen intents. As the continual learning process progresses, the model starts to gain increasingly better capability to classify intents from each other. Finally, Figure 4(c) provides a clustering result with each of the classes clearly separated. This validates the increasing effectiveness and generalizability of our continual learning SLU model from a qualitative perspective.

Continual learning performance is affected by the ordering of tasks Bell and Lawrence (2022). In the case of SLU, since each transcription and its intent have different semantic meanings, the learning order of tasks may impact the overall performance, including stability, plasticity, and generalizability.

To validate this, we name the ordering scheme introduced in Section 4.1 as frequency order, as it is ranking tasks in terms of their frequencies. Alternatively, we choose to rank the tasks based on their semantic meanings in two opposite ways, namely close-semantic order and diverse-semantic order. Close-semantic order groups together data instances with similar semantic meanings within the same task. For example, utterances with intents “bring newspaper” and “bring shoes” are grouped together to establish a close semantic relationship within one task. Under the setting of diverse-semantic order, the intents are strategically arranged to ensure a mix of semantic meanings in close proximity.

The effect of different tasks orderings on metrics is presented in Table 2. By using the DMI metric, we observe that both close-semantic and diverse-semantic orders improve the generalizability while decreasing the stability and plasticity compared to the frequency order. The impact of diverse-semantic order is particularly more significant. The reason behind this may be that frequency order benefits stability and plasticity by exposing the model to a larger amount of data, while close-semantic order assists the model by grouping semantically similar classes together within a task. On the other hand, diverse-semantic order consists of a wider range of classes in each task, thereby equipping the model with a higher generalizability. However, as a trade-off, the complexity of individual tasks increases, resulting in decreased stability and plasticity of the model.

As a comparison, these changing tendencies are not effectively captured by prior metrics. From Table 2, BWT and FWT are mostly correlated with Acc, and a higher Acc would mostly lead to higher BWT and FWT scores. This is because these metrics are entangled with plasticity, and may not fully reflect the stability and generalizability. In contrast, our DMI metric provides a more sensitive measurement of stability, plasticity, and generalizability when the order of tasks changes. This makes our metric better suited for practical use-case scenarios.