Dual-CBA: Improving Online Continual Learning
via Dual Continual Bias Adaptors
from a Bi-level Optimization Perspective

In continual learning, the model is trained on a stream of tasks with evolving data distribution. Considering $N$ sequential tasks $\{\tau_{1},\tau_{2},\cdots,\tau_{N}\}$, each task can be represented as $\tau_{t}=\{(x^{t}_{i},y^{t}_{i})\}_{i=0}^{N_{t}}$, where $N_{t}$ is the total number of training samples in task $\tau_{t}$. For the online CL setting, the classification model $f_{\theta}$ trains each new sample only once, and the previously seen batches are almost inaccessible. Let $\tau_{t}$ represent the current learning task, and $\mathcal{M}$ denotes a small memory buffer that stores the samples of previous tasks $\{\tau_{1},\cdots,\tau_{t-1}\}$. Here we take a widely used rehearsal-based method, Experience Replay (ER) [14, 62], as an example. ER jointly trains current task $\tau_{t}$ with examples sampled from the buffer $\mathcal{M}$. The training objective function $\mathcal{L}$ is

where the training batch $\mathcal{B}^{trn}=\mathcal{B}^{t}\cup\mathcal{B}^{buf}$ consists of a batch of incoming new samples $\mathcal{B}^{t}\subset\tau_{t}$ and a batch of samples from the memory buffer $\mathcal{B}^{buf}\subset\mathcal{M}$, and $L$ denotes the cross-entropy loss. Note that the memory buffer $\mathcal{M}$ is updated by reservoir sampling after training each batch $\mathcal{B}_{t}$, which is a relatively balanced set containing samples of all seen tasks.

In this study, we focus on mitigating the issue of catastrophic forgetting of rehearsal-based models with an online CL scenario. As previously mentioned, existing rehearsal-based models often struggle with catastrophic distribution changes caused by dynamic data streams over time. To tackle this challenge, unlike prior methods that regard shifts in label or feature distributions, we suggest directly modeling the catastrophic distribution change for posterior probability $\mathbb{P}(Y|X)$, enabling the original classification model $f_{\theta}$ to learn a stable knowledge consolidation for all previous tasks.

The main methodology involves the design of a Continual Bias Adaptor (CBA) denoted as $g_{\phi}$, which serves two key purposes: 1) Dynamically augmenting the classification network $f_{\theta}$ to produce more diverse posterior distributions by adjusting the parameters of $g_{\phi}$ (where $\phi$ can be regarded as hyper-parameters), aiming to address the catastrophic posterior change. 2) Guiding the original classification model $f_{\theta}$ to fit an implicit posterior that tends to achieve a stable consolidation of knowledge from previously learned tasks. In summary, during the training stage, for a given training example $x^{trn}$, its posterior probability is modified by the augmented classification network $\mathcal{F}_{\theta,\phi}=g_{\phi}\circ f_{\theta}$ in an online CL manner, which can be formulated as

where $\circ$ is the function composition operator and $g_{\phi}$ is a lightweight network. The augmented classification network $\mathcal{F}_{\theta,\phi}$ is firstly updated to learn new knowledge from the training data $\mathcal{B}^{trn}=\mathcal{B}^{t}\cup\mathcal{B}^{buf}$ that minimizes the rehearsal-based empirical risk, i.e.,

where $\phi$ is a hyper-parameter of the optimal $\theta^{*}$. Note that different rehearsal-based loss functions can be used as the training loss $\mathcal{L}^{trn}$. Here we only take ER as an example for simplicity and more details can be found in Appendix D.

The ultimate objective of our method is to protect the original classification network $f_{\theta}$ from catastrophic distribution shift while achieving a stable knowledge consolidation across different tasks. To this end, we further keep tracking the performance of the classification network to prevent catastrophic forgetting, which requires that $f_{\theta^{*}(\phi)}$, obtained by minimizing the rehearsal-based empirical risk Eq. (3), maximizes the performance of all previously seen data. However, accessing all of this historical data is unfeasible in CL, we approximate it by the empirical risk over the memory buffer data, i.e.,

This objective function aims to find the optimal CBA such that the optimized classification network performs well on the memory buffer data, which acts as a stable consolidation of knowledge from the learned tasks.

Indeed, Eq. (3) and Eq. (4) formulate a bi-level learning framework and the main flowchart is illustrated in Fig. 3. In the inner loop Eq. (3), the classification network is updated to learn new knowledge and rehearse old knowledge from $\mathcal{B}^{trn}$ with the help of the CBA module. In the outer loop Eq. (4), the CBA $g_{\phi}$ are updated from $\mathcal{B}^{buf}$ to consolidate the previously learned knowledge against catastrophic posterior change. In subsequent sections, we elaborate on the proposed CBA module and bi-level optimization algorithm in detail:

(1) Update $\theta$. Given the CBA parameter $\phi^{k}$ at iteration step $k$, we update the classification network parameter $\theta^{k}$ in Eq. (3) by one-step stochastic gradient descent (SGD):

where $\alpha$ is the inner-loop learning rate.

(2) Update $\phi$. With the one-step updated $\theta^{k+1}$, a function of $\phi$, we further optimize $\phi$ in Eq. (4) as follows:

where $\beta$ is the outer-loop learning rate. Note that $\nabla_{\phi}\mathcal{L}^{buf}$ in Eq. (6) involves a second-order derivative, which can be easily implemented by automatic differentiation systems such as Pytorch [66]. The detailed calculations are provided in Appendix A. Additionally, to alleviate the calculation burden of this second-order derivation, we assume that $\phi$ only depends on the last linear layer of the classification network. Consequently, we only need to unroll the second-order derivation of the linear classification layer in Eq. (6). As the linear layer typically comprises a small number of parameters compared to the entire classification network, our proposed algorithm is more efficient than other bi-level optimization algorithms [67, 68, 64, 65]. Please refer to Appendix H for a comprehensive discussion on computation and GPU memory utilization. The complete training process is detailed in Alg. 1.

In this subsection, we delve into the architectural design of the CBA module $g_{\phi}$, which plays a vital role in adapting to catastrophic posterior changes in continual learning.

The primary reason is that the class-specific CBA cannot adapt to the abrupt change in the posterior distribution upon starting to learn a new task. To investigate this, we visualize the difference between the posterior probability $\hat{y}$ predicted by the classification network before and after the task transition timestamp in Fig. 4. We can see that the classification network $f_{\theta}$ produces a relatively high posterior probability for the 85th class, a new class introduced in task 9. However, when task 10 comes in, the probability of the 85th class is significantly lower than before. Since the posterior probability predicted by the classification network is the input of the class-specific CBA, the corruption of these inputs for task 9 hinders the class-specific CBA from appropriately adjusting its posterior probabilities.

The second reason is that the class-specific CBA cannot immediately adjust the posterior probability of the classes in the new task. When a new task comes in, the class-specific CBA fails to accurately accommodate the posterior probability of the new task because it has not yet encountered the incoming new classes. As shown in Fig. 4, the arrival of task 10 leads to a high posterior probability for its own classes, such as the 95th class. This indicates that the class-specific CBA fails to control the increasing probability of the new task.

To address the aforementioned challenges, instead of directly adjusting the posterior probability in a class-specific manner, we resort to modeling a stable relationship between new and old tasks to adapt to the posterior distribution shift while avoiding a significant stability gap upon starting to learn new tasks. Let $\mathcal{C}_{new}$ and $\mathcal{C}_{old}$ denote the class sets of the new task and old tasks, respectively. Our main idea is to adjust the posterior probability of the new task $\hat{y}_{new}=\sum_{c\in\mathcal{C}_{new}}p(y=c|x)$ and that of all the old tasks $\hat{y}_{old}=\sum_{c\in\mathcal{C}_{old}}p(y=c|x)$, given their relatively stable numerical relationship throughout the continual learning process as shown in Fig. 5. To achieve this, we design a novel class-agnostic CBA $g_{\nu}$ parameterized by $\nu$ as follows:

Note that the class-agnostic CBA treats the posterior probabilities of the classes in the new task as a whole, and does the same for the old task. Then, for a specific class, the adjustment of its posterior probability is formulated by averaging $\tilde{y}_{old}$ or $\tilde{y}_{new}$, and it can be formulated as

and the final adapted posterior probabilities are given by $\tilde{y}^{agn}=\left[p\big{(}y=1\big{|}\hat{y}\big{)},p\big{(}y=2\big{|}\hat{y}\big{)},\cdots,p\big{(}y=\left|\mathcal{C}_{t}\right|\big{|}\hat{y}\big{)}\right]^{T}$.

The proposed class-agnostic CBA module serves as a robust and transferable bias adaptor, effectively adapting to sudden changes in posterior distributions after training on a few pairs of new and old tasks. Specifically, we can analyze its adaptation mechanisms for both the preceding old task and the incoming new task. We assume that the task transition timestamp is $t+1$, and we have

• •

  For the incoming $(t+1)$-th new task, the class-agnostic CBA module has processed $t$ pairs of old and new tasks, and its current weights can produce a valuable initialization or experience learned from previous data. Therefore, the class-agnostic CBA can quickly adapt to new tasks.

• •

  For the $t$-th task, the class-agnostic CBA module naturally treats it as an old task at the $t+1$ timestamp. Similarly, CBA leverages the experience learned from previous old tasks to adjust its posterior distribution. This robust initialization makes class-agnostic CBA effectively adapt to the changed posterior distribution.

In summary, the proposed class-agnostic CBA can produce a robust initialization for both the $(t+1)$-th new task and the $t$-th old task, ensuring to adapt to the suddenly changed posterior distribution quickly. Consequently, the class-agnostic CBA can help the classification model learn the stable posterior distribution and mitigate the stability gap immediately upon starting to learn a new task.

Note that the proposed Dual-CBA module is only used in the training stage. In the test stage, the test sample $x^{tst}$ is predicted by $f_{\theta}$, that is $\hat{y}^{tst}=f_{\theta}(x^{tst})$. This indicates that our method does not introduce any calculation overhead in the test stage.

In this subsection, we further address the feature bias introduced during the training stage by the commonly used BN [18] of our proposed bi-level framework. Specifically, BN calculates batch statistics of each feature map for normalization during training, while using population statistics estimated by an exponential moving average (EMA) during testing. In our proposed bi-level optimization framework in the conference version, the population statistics of BN are only updated in the inner loop together with the classification network parameters $\theta$. Unfortunately, these population statistics estimated by EMA in the inner loop are seriously biased toward the current task because the training data are dominated by new tasks, leading to a biased feature extraction during testing. For simplicity, we take the population mean used for testing at task $t$ as an example. Specifically, we denote it as $\mu_{test}^{t}$, and it is updated using an EMA scheme as follows:

where $\mu^{t,k}_{train}$ is the feature mean of the $k$-th batch of training data $\mathcal{B}^{trn}$ at task $t$. Then, we have

where $\mu^{t,0}_{test}$ is initialized as the population mean calculated at the final of the last task $t-1$. It can be observed that $\mu^{t,0}_{test}$ encodes the feature mean of previous tasks and the corresponding weight is exponentially decreasing as current task training progresses. Additionally, the batch mean $\mu^{t,k}_{train}$ is dominated by the current new task data because the number of the memory buffer samples is much smaller than that of incoming new samples. Consequently, the population mean updated by EMA in the inner loop is seriously biased towards new tasks, where the population variance has the same tendency.

Note that this feature bias is not reflected in the posterior distribution during training, where the proposed Dual-CBA module cannot correct the biased feature. To achieve unbiased estimations of these population statistics, we should estimate them on the seen data from all tasks, which is unrealistic in continual learning. Therefore, from the bi-level optimization perspective, an intuitive idea is to update these population statistics by the balanced buffer data set used in the outer loop. Intrinsically, the memory buffer obtained via reservoir sampling is an excellent approximation of the entire data distribution for all seen tasks in an ideal case. Based on this principle, we propose a straightforward yet effective method to provide approximately unbiased population statistics in continual learning, termed Incremental Batch Normalization (IBN). Specifically, we stop the EMA updating of BN statistics in the inner loop optimization and estimate unbiased population statistics using the memory buffer in the outer loop optimization. For the implementation of IBN, we further refine the process by replacing the update of population statistics during the training stage with estimations based on balanced buffer data before the testing stage.

This procedure does not interfere with the optimization of our bi-level learning framework, which can effectively assimilate the feature discrepancy of the classification model during inference by estimating more accurate population statistics in BN for all seen tasks. Specifically, the population statistics estimated by IBN can be formulated as follows:

where $\mu^{t,i}_{buf}$ is the feature mean of the $k$-th batch of memory buffer samples $\mathcal{B}^{t}\in\mathcal{M}$ at task $t$, and $K$ is the total number of moving average. The implementation of IBN is simple as it only requires forward propagation of the small balanced buffer data without any back-propagation, and the population statistics can be updated by EMA automatically, which does not introduce too much calculation burden.

The following theorem reveals that the proposed bi-level optimization inherently establishes gradient alignment between the loss on the training set $\mathcal{B}^{trn}$ and the memory buffer $\mathcal{B}^{buf}$.

The detailed proof of Theorem 1 is shown in Appendix B. This theorem demonstrates that optimizing our proposed bi-level framework Eq. (3) and (4) encourages the angle between the gradients $\mathcal{G}^{buf}$ and $\mathcal{G}^{trn}$ to be as small as possible, which reveals two insights into our algorithm. On the one hand, this theorem indicates that our approach will push the gradients of the classification network on the training data (with the Dual-CBA module) to those on the buffer data (without the Dual-CBA module). This property potentially encourages the Dual-CBA module to mitigate the training bias present in the training data. That is why our algorithm can effectively mitigate the task-recency bias in the CL process. On the other hand, this theorem shows an alignment between the gradients of the classification network on the training data and buffer data, which regularizes the updating of the classification network on the new task and ensures it does not deviate from the previous updating direction too much. This gradient alignment can effectively alleviate forgetting in CL, which has been demonstrated in many previous gradient-alignment-based methods [70, 71]. However, these works calculate the classification network gradient on training data without accounting for task-recency bias, where the network may seriously bias to new tasks, resulting in an imprecise gradient alignment. In contrast, our proposed Dual-CBA method mitigates this bias, ensuring more accurate gradient alignment and improved performance.

We herein consider a convex model to delve into the insight of our proposed Dual-CBA model. Specifically, we reformulate Dual-CBA as a linear model where the origin and augmented classification networks are represented as:

where $X_{t}\in\mathbb{R}^{p\times n_{t}}$ is the input data and $\hat{Y}_{t},\tilde{Y}_{t}\in\mathbb{R}^{n_{t}\times c_{t}}$ denote outputs of the original and augmented classification networks, respectively. The parameters of the classification model are $\theta^{*}_{t}\in\mathbb{R}^{p\times c_{t}}$ and the Dual-CBA module is represented as $\phi^{*}\in\mathbb{R}^{c_{t}\times c_{t}}$, where we omit the notation of $\nu$ for clarity. Additionally, we assume the Gaussian feature and noise of this linear model following [72] with $e_{t}\sim\mathcal{N}(0,\sigma^{2}_{e})$ denoting the noise vector.

We consider the mean square error (MSE) loss as the convex objective function. In this case, the proposed bi-level optimization framework can be represented as follows:

Note that $\theta_{t}\in\mathbb{R}^{p\times c_{t}}$ is the weight matrix and $\theta^{*}_{t}(\phi)$ represents a function of $\phi\in\mathbb{R}^{c_{t}\times c_{t}}$. Thanks to the excellent properties of convex optimization, we can get the closed-form solution of this bi-level optimization framework, which can be summarized as follows:

The proof of this theorem can be found in the Appendix B. It can be observed that the optimal solution of $\phi^{*}_{t}$ in Eq. (17) actually measures the discrepancy between $\small{Y^{buf}_{t}}$ and $\small{AY^{trn}_{t}=(X^{buf}_{t})^{T}(X^{buf}_{t})^{\dagger}Y^{trn}_{t}}$. Intuitively, if we multiply these two terms by $(X^{buf}_{t})^{\dagger}$ on the left of the two sides of this equation, we can find that this discrepancy depends on the difference between $(X^{buf}_{t})^{\dagger}Y^{buf}_{t}$ and $(X^{trn}_{t})^{\dagger}Y^{trn}_{t}$, which are the optimal solutions of the linear model on the buffer data $\lVert(X^{buf}_{t})^{T}\theta-Y^{buf}_{t}\rVert^{2}$ and that on the training data $\lVert(X^{trn}_{t})^{T}\theta-Y^{trn}_{t}\rVert^{2}$, respectively. Therefore, the proposed Dual-CBA module $\phi^{*}_{t}$ captures the relationship between the optimal linear solutions on training data and buffer data, which can effectively feedback to the optimization of the classification network. Specifically, the optimal $\phi^{*}_{t}$ regularizes the optimal solution of $\theta^{*}_{t}$ by modifying the optimal linear solution on the training data $(X^{trn}_{t})^{\dagger}Y^{trn}_{t}$, ensuring the classification network pay more attention to the balanced buffer data $\{X^{buf}_{t},Y^{buf}_{t}\}$ rather than only overfit on the current training data $\{X^{trn}_{t},Y^{trn}_{t}\}$.

In semi-supervised continual learning, only a small portion of samples from each task is labeled with the proportion of these labeled data referred to as label ratio. We apply the FixMatch [76] to two rehearsal-based CL methods, namely ER and DER++. Similar to the fully supervised setting, our method Dual-CBA can also be easily applied to these baselines, and the comparison results across different datasets and various label ratios are shown in Table II. It can be observed that CBA only marginally improves the baselines in some cases, with overall performance showing modest gains. For example, when the label ratio is 0.2, ER-CBA even decreases the ACC and increases the FM of baseline ER across all three datasets. In contrast, our Dual-CBA consistently improves the performance of both baselines and surpasses CBA by a large margin, especially in terms of the FM metric. This is likely because the class-specific CBA requires more labeled data to adapt to changed posterior distribution for each new task, making it prone to overfitting in a semi-supervised CL setting with limited labeled samples. However, the strong transferability of the proposed class-agnostic CBA reduces the need for labeled data and enhances the performance of our Dual-CBA. Additionally, these results indicate that the proposed IBN is effective in a semi-supervised CL setting.

Following [43, 77], we adopt the Blurry-$K$ online CL setting. It simulates a practical situation where task boundaries are unclear, characterized by an overlap across all tasks. Specifically, a fraction ($K\%$) of the training data from one task may appear in other tasks. Here we take $K=10$ and $30$ as an illustration, and the comparison results on Split CIFAR-100 are summarized in Table III.

It can be observed that our Dual-CBA substantially improves the performance of the corresponding baselines and the original CBA module. For example, CBA only improves the ACC of ER by about 1.76% with 500 buffer samples under the Blurry-10 setting, while the proposed Dual-CBA can further enhance the performance of ER-CBA by about 2.91%. Additionally, our method significantly reduces the forgetting of the two baselines as shown in Table III. These results demonstrate that the proposed Dual-CBA can be flexibly applied to various rehearsal baseline models and help them adapt to the posterior distribution shift without being perturbed by unclear task boundaries, further verifying the effectiveness of the proposed improvements on CBA.

To further validate the generalization ability of the proposed Dual-CBA, we extend our method Dual-CBA to offline continual learning, where each task can be trained over multiple epochs to achieve more stable convergence. As shown in Table IV, all four baselines achieve better optimum solutions compared to the online context. Obviously, the proposed Dual-CBA significantly improves the performance of corresponding baselines and consistently outperforms the CBA module. These results verify the negative impact of the training bias on CL models and the strength of our method, which can also help the baseline models to adapt to the shifting posterior distribution in the offline setting, indicating the strong generalization ability of the proposed Dual-CBA.