GACL: Exemplar-Free Generalized Analytic Continual Learning

Existing CIL methods can be roughly divided into three categories: replay-based methods, regularization-based methods, and prototype-based methods.

The replay-based CIL methods such as iCaRL iCaRL_2017_CVPR , LUCIR LUCIR_2019_CVPR , PODNet PODNet_2020_ECCV , AANets AANets_2021_CVPR , FOSTER FOSTER_2022_ECCV , and OHO OHO_2023_AAAI , retain past training samples as exemplars and utilize them during the learning of new ones. However, storing original training samples presents a significant challenge, particularly in scenarios with strict data privacy requirements.

The regularization-based CIL aims to design a loss function that prevents the change of activations or important weights. Methods such as the Less-forgetting learning LessForgetting_2016_arXiv and the LwF LwF2018TPAMI introduce knowledge distillation KD_Hinton_arXiv2015 into their loss function to prevent the forgetting caused by activation drift. EWC EWC_2017_PNAS , EWC++, RWalk RWalk_2018_ECCV , and Rotate your Networks RN_2018_ICPR , introduce regularization that slows down learning on the weights important for old tasks by calculating the Fisher information matrix.

The prototype-based CIL maintains distinct prototypes for each category, which prevents overlapping representations of new and old categories. For example, the PASS PASS_2021_CVPR distinguishes prior categories by augmenting feature prototypes. The SSRE technique SSRE_2022_CVPR enhances the dissimilarity between old and new categories via selecting prototypes to incorporate with new samples into a distillation process. The FeTrIL FeTrIL_2023_WACV uses new representations to generate pseudo-features of old categories.

The ACL is a recently developed EFCIL branch inspired by the analytic learning PIL_2004_Neurocomputing ; BRMP_2022 ; ACnnL_2022_arXiv where the training of neural networks yields a closed-form solution using least squares. The ACIL ACIL2022NeurIPS first converts a continual learning problem to a batch recursive least-squares problem, eliminating the need to store samples by preserving the correlation matrix, and the RanPAC RanPAC_McDonnell_NeurIPS2023 applies this trick to pre-trained models. The GKEAL GKEAL2023CVPR focuses on the few-shot CIL scenarios by leveraging a Gaussian kernel projection. The DS-AL Zhuang_DSAL_AAAI2024 introduces an additional linear classifier to learn the residue of the ACIL to enhance the plasticity, while the REAL REAL_He_arXiv2024 introduces the representation enhancing distillation to improve the backbone’s generalization capabilities. The AFL AFL_zhuang2024 extends the ACL to federated learning, transitioning from temporal increment to spatial increment, and similar techniques are applied to the reinforcement learning locality_2024_iclr . The ACL is an emerging competitive CIL branch with a closed-form solution that leads to a valuable weight-invariant property, securing the equivalence between CIL and its joint learning. However, existing ACL methods are designed for the CIL scenario in which the categories of samples in each task must be entirely distinct. This restricts their applicability in real-world scenarios.

The GCIL simulates real-world incremental learning, as distributions of data category and size could be unknown in one task. The GCIL arouses problems such as intra- and inter-task forgettings and the class imbalance problem Siblurry2023ICCV . The key GCIL properties can be summarized as follows: (i) the number of classes across different tasks is not fixed; (ii) classes shown in prior tasks could reappear in later tasks; (iii) training samples are imbalanced across different classes in each task GCIL_2020_CVPR_Workshops (See Appendix B).

There are several GCIL settings. In the BlurryM setting BlurryM_2019_NeurIPS , $a\%$ of the classes are disjoint between tasks, while the remaining classes appear in every task. The i-Blurry-N-M CLIB_2022_ICLR setting has blurry task boundaries and requires the model to perform anytime inference. However, the i-Blurry scenario has a fixed number of classes in each task with the same proportion of new and old classes. The Si-Blurry Siblurry2023ICCV is the most complex and realistic GCIL setting satisfying all three GCIL properties since it has an ever-changing number of classes and is capable of effectively simulating newly emerging or disappearing data, highlighting the problem of uneven distribution in real-world scenarios.

To address the issue of the GCIL, gradient-based sample selection methods such as the GSS-IQP and the GSS-Greedy are proposed by BlurryM_2019_NeurIPS . The RM RM2021CVPR proposes a memory management strategy based on per-sample classification uncertainty and data augmentation, while the management in the CLIB CLIB_2022_ICLR eliminates samples based on a per-sample importance score. The DualPrompt dualprompt_2022 , as an EFCIL method, introduces the prompt-based learning to the CIL problem, and the MVP Siblurry2023ICCV proposes an instance-wise logit masking and contrastive visual prompt tuning loss.

We denote the complete set of available data as $\mathcal{D}$. When $\mathcal{D}$ is partitioned into a sequence of GCIL tasks, we assume that $\mathcal{D}_{k}^{\text{train}}\sim\{\bm{X}_{k}^{\text{train}},\bm{Y}_{k}^{\text{train}}\}$ is the set of training samples that are present in task $k$. The training dataset $\mathcal{D}_{k}^{\text{train}}$ consists of labeled samples, where $\bm{X}_{k}^{\text{train}}\in\mathbb{R}^{N_{k}\times c\times w\times h}$ represents $N_{k}$ input image samples with a shape of $c\times w\times h$. $\bm{Y}_{k}^{\text{train}}\in\mathbb{R}^{N_{k}\times d_{y_{k}}}$ represents $N_{k}$-stacked one-hot encoded label tensors with $d_{y_{k}}$ classes that have been seen from task $1$ to task $k$. $\mathcal{D}_{k}^{\text{test}}\sim\{\bm{X}_{k}^{\text{test}},\bm{Y}_{k}^{\text{test}}\}$ is the test dataset in task $k$. The goal of GCIL in task $k$ is to train networks using $\mathcal{D}_{k}^{\text{train}}$ and evaluate their performance on the test dataset $\mathcal{D}_{1:k}^{\text{test}}$. Here, $\mathcal{D}_{1:k}$ denotes the joint dataset spanning tasks $1$ to $k$.

In each GCIL task, classes may not appear exclusively. Hence, in any GCIL task $k$, we refer to classes that have appeared in previous tasks 1 to $k-1$ as the exposed classes of task $k$, while classes making their initial appearance are the unexposed classes of task $k$ as shown in Figure 1 (a). This distinction helps to characterize the evolving nature of class occurrences throughout different GCIL tasks.

In a task-wise GCIL scenario, we can involve all class labels in a set $\mathcal{S}$. In task $k$, the set of the exposed class labels is denoted as $\mathcal{S}_{\text{exposed, k}}\subseteq\mathcal{S}$, while the set of unexposed class labels is marked by $\mathcal{S}_{\text{unexposed, k}}\subseteq\mathcal{S}$, where $\mathcal{S}_{\text{exposed, k}}\cap\mathcal{S}_{\text{unexposed, k}}=\varnothing$. Note that $\mathcal{S}_{\text{exposed, k}}$ and $\mathcal{S}_{\text{unexposed, k}}$ may evolve from task $k-1$ to task $k$, that is

From the scope of exposed-unexposed classes, the $d_{y_{k}}$ can be represented as $d_{y_{k}}=\left|\mathcal{S}_{\text{exposed, k}}\right|+\left|\mathcal{S}_{\text{unexposed, k}}\right|=d_{y_{k-1}}+\left|\mathcal{S}_{\text{unexposed, k}}\right|$, where $\left|\cdot\right|$ denotes the cardinality of a set.

In task $k$, given training dataset $\mathcal{D}_{k}^{\text{train}}\sim\{\bm{X}_{k}^{\text{train}},\bm{Y}_{k}^{\text{train}}\}$, class labels $\bm{Y}_{k}^{\text{train}}$ can be partitioned due to the exposed-unexposed split as follows:

where $\bm{\bar{Y}}_{k}^{\text{train}}\in\mathbb{R}^{N_{k}\times d_{y_{k-1}}}$ is the exposed class label matrix and ${\tilde{\bm{Y}}_{k}}^{\text{train}}\in\mathbb{R}^{N_{k}\times(d_{y_{k}}-d_{y_{k-1}})}$ is the unexposed class label matrix. They correspond to segments displaying the appearance of exposed classes and unexposed classes.

The power of pre-trained models allows the GACL to adopt a frozen backbone from structures such as ViT ViT_2021_ICLR to extract the features of images shown in Figure 1 (b). Let

be the features extracted by the backbone, where $\bm{\Theta}_{\text{backbone}}$ indicates the backbone weight. Then, we use a buffer layer to project features, i.e.,

where $f_{\text{buffer}}$ indicates the operation of the buffer layer. Several options for the buffer layer exist, including a randomly initialized linear mapping in the ACIL ACIL2022NeurIPS or a kernel embedding projection in the GKEAL GKEAL2023CVPR . The selection of the buffer layer is not our focus. For convenience, we follow the ACIL, taking the random linear projection followed by a non-linear activation function as the buffer layer, i.e. $f_{\text{buffer}}(\bm{X}^{(\text{E})})=\operatorname{ReLU}(\bm{X}^{(\text{E})}\bm{W}_{\text{B}})$, where the elements of the buffer layer weight $\bm{W}_{\text{B}}$ are randomly sampled from a normal distribution.

Here, we derive the GACL by partitioning training samples into unexposed and exposed categories, as shown in Figure 1 (c). Let $\bm{X}_{1:k}^{\text{total}}$ and $\bm{Y}_{1:k}^{\text{total}}$ be the accumulated feature and label matrices in task $k$, which can be extended from the accumulated matrices $\bm{X}_{1:k-1}^{\text{total}}$ and $\bm{Y}_{1:k-1}^{\text{total}}$ in task $k-1$ as follows.

Subsequently, one could formulate the learning problem in task $k$ by a fully connected network (FCN) as the classifier

where $\lVert\cdot\rVert_{\text{F}}$ is Frobenius-norm, $\gamma\geq 0$ is the regularization term and $\bm{W}_{\text{FCN}}^{(k)}$ indicates the FCN layer weight. The optimal solution to (4) is

The goal of the GACL is then to obtain $\bm{\hat{W}}_{\text{FCN}}^{(k)}$ recursively from $\bm{\hat{W}}_{\text{FCN}}^{(k-1)}$ without directly involving historical samples (e.g., $\bm{X}_{1:k-1}^{\text{total}}$ and $\bm{Y}_{1:k-1}^{\text{total}}$). That is to solve

by recursively updating the previous-task weight $\bm{\hat{W}}_{\text{FCN}}^{(k)}$. To achieve this, we define an autocorrelation memory matrix as follows.

Accordingly, we summarize the recursive formulation of the proposed GACL in Theorem 3.1.

As indicated in Theorem 3.1, the weight $\bm{\hat{W}}_{\textup{FCN}}^{(k)}$ in task $k$ recursively obtained using the previous-task weight $\bm{\hat{W}}_{\textup{FCN}}^{(k-1)}$ is identical to its joint-learning counterpart formulated in (6). That is, the GACL maintains the same weight-invariant property in the GCIL scenario as other ACL methods.

The pseudo-code of the GACL is listed in Algorithm 1.

Exemplar-free. The recursive formulation is aided by $\bm{R}_{k}$ as indicated in (9). Note that this autocorrelation memory matrix records the inverse of inner products among the historical embedding matrices as shown in (7). Hence, the embeddings (e.g., $\bm{X}_{k}^{\textup{(B)}}$) are not reversible. Saving $\bm{R}_{k}$ instead of used samples is a safe alternative to preserve past knowledge. That is, our GACL is an exemplar-free technique without the need to keep any historical samples.

To more properly explain our GACL, as indicated in Figure 1 (c), the recursive solution in (8) can be rewritten as the sum of the unexposed-class contributed weight $\bm{\hat{W}}_{\textup{unexposed}}^{(k)}$ and the ECLG weight $\bm{\hat{W}}_{\textup{ECLG}}^{(k)}$, i.e.,

where

In the section, we conduct experiments on various benchmark datasets and compare the GACL with both EFCIL and replay-based state-of-the-art methods, including LwF LwF2018TPAMI , L2P L2P_2022_CVPR , DualPrompt dualprompt_2022 , ER ER_NEURIPS2019_ , EWC++ EWC_2017_PNAS , SLDA SLDA_2020_CVPR_Workshops , RM RM2021CVPR , MVP Siblurry2023ICCV , and MVP-R (MVP with exemplars).¹¹1The results for MVP and MVP-R are based on their official implementation, committed on October 26, 2024 (commit ID: ad8d1426a497545ba634521c00008c34ceece799).

As shown in Figure 2, we comprehensively compare the GACL with both EFCIL and replay-based methods.

The ECLG module is a core component that allows the GACL to obtain the weight-invariant property in the GCIL scenario. Here, we conduct an ablation study to justify the ECLG’s contributions under various blurry sample ratios $r_{B}$ with $\bm{r}_{\text{D}}=50\%$. Larger $\bm{r}_{\text{B}}$ indicates more complex data distributions in the Si-Blurry setting. As shown in Table 2, the GACL without ECLG exhibits poor performance with a visible gap for $\mathcal{A}_{\text{AUC}}$, $\mathcal{A}_{\text{Avg}}$, and $\mathcal{A}_{\text{Last}}$. For instance, on CIFAR-100 with $r_{B}$ = 10%, the ECLG contributes a 23.01% $\mathcal{A}_{\text{Last}}$ gain to the GACL.

As claimed in Theorem 3.1, the classifier without the ECLG module fails to absorb knowledge from joint classes in each task (i.e., classes that reappear), leading to substantial information loss under the GCIL setting. The GACL, equipped with the ECLG module, demonstrates competence in handling overlapping classes in realistic scenarios.

Here, we conduct a robust analysis by varying the disjoint class ratio $\bm{r}_{\text{D}}$ and the blurry sample ratio $\bm{r}_{\text{B}}$. The comparison happens among the GACL, the second-best EFCIL method SLDA, and the top-performing replay-based method MVP-R with a memory size of 500.

We evaluate our method under various $\bm{r}_{\text{D}}$, including extreme cases where each task shares classes ($\bm{r}_{\text{D}}=0\%$) and traditional CIL scenarios ($\bm{r}_{\text{D}}=100\%$). Table 3 illustrates that our GACL consistently outperforms the compared methods (e.g., leads the SLDA by 2%-10%) and produces near-identical $A_{\text{Last}}$ values with varying $\bm{r}_{\text{D}}$. This shows the accurate and robust traits of the GACL.

We also evaluate our method using various $\bm{r}_{\text{B}}$ values, as shown in Table 4. Similar patterns observed here align with those in Table 3, further demonstrating the robustness of the proposed GACL, which delivers exceptional performance across different GCIL settings.

2

Overall, the GACL exhibits various good characteristics as an exemplar-free GCIL technique. The major limitation here is the need for a well-trained backbone because the GACL does not update backbone weights. This could motivate the exploration of adjustable backbones to continuously improve their feature extraction abilities, thereby further enhancing GACL’s performance.