Task-Adaptive Saliency Guidance for Exemplar-free Class Incremental Learning

A variety of methods have been proposed for incremental learning in the past few years [7, 1]. Recent works can be coarsely grouped into three categories: replay-based, regularization-based, and parameter-isolation methods. Replay-based methods mitigate the task-recency bias by retaining training samples from previous tasks [40, 53]. In addition to replaying samples, BiC [45], PODNet [10], iCaRL [40] and LTCIL [30] apply a distillation loss to prevent forgetting and enhance model stability. GEM [33], AGEM [4], and MER [41] exploit past-task exemplars by modifying gradients on current training samples to match old samples. Rehearsal may cause models to overfit to stored samples. Regularization-based approaches such as LwF [24], EWC, R-EWC [18, 29], and DMC [54] offer ways to learn better representations while leaving enough plasticity for adaptation to new tasks. Parameter-isolation methods [35, 47] use models with different computational graphs for each task. With the help of growing models, new model branches mitigate catastrophic forgetting at the cost of more parameters and computational cost. It is also widely studied in other fields, such as semantic segmentation [3, 26, 46, 52] and object detection [13, 32].

As for saliency-guided incremental learning, LwM [9] regularizes the saliency activations of previous classes on current task data. However, there exist semantic gaps between current classes and old ones, which results in an inaccurate distillation target for preserving saliency activations on old samples. RRR [11] directly saves the Grad-CAM saliency activations of each sample in a replay buffer and applies distillation to memorize this old knowledge, which requires storing additional samples during incremental learning. Despite these initial works on saliency-guided incremental learning, saliency drift still remains problematic and leads to catastrophic forgetting.

Compared to conventional class incremental learning, exemplar-free class incremental learning is more appropriate for applications where training data is sensitive and may not be stored in perpetuity. DAFL [5] uses a GAN to generate synthetic samples from past tasks as an alternative to storing actual data. DeepInversion, which inverts trained networks using random noise to generate images, is another popular EFCIL method [50]. Always Be Dreaming further improves on DeepInversion for EFCIL [44]. SDC attempts to overcome the problems caused by semantic drift when training new tasks on old class samples [51]. It directly estimates prototypes of each learned class to use in a nearest class mean classifier. PASS [57] and IL2A [56] are prototype-based replay methods for efficient and effective EFCIL. Since these prototypes are features computed from past training samples, image exemplars need not be retained. SSRE [58] introduces a re-parameterization method to trade-off between old and new knowledge. Our task-Adaptive Saliency Supervision (TASS) method uses three new components to reduce saliency drift in EFCIL and is complementary to several of the approaches mentioned above.

Class-incremental learning aims to sequentially learn tasks consisting of disjoint classes of samples. Let $t\in\{1,2,..T\}$ denote the incremental learning tasks. The training data $D_{t}$ for each task contains classes $C_{t}$ with $N_{t}$ training samples $\{(x_{t}^{i},y_{t}^{i})\}_{i=1}^{N_{t}}$, where $x^{i}_{t}$ are images and $y_{t}^{i}\in C_{t}$ are their labels.

Most deep networks applied to class-incremental learning can be split into two components: a feature extractor $F_{\theta}$ and a common classifier $G_{\phi}$ which grows with each new task $t+1$ to include classes $C_{t+1}$. The feature extractor $F_{\theta}$ first maps the input $x$ to a deep feature vector $z=F_{\theta}(x)\in\mathbb{R}^{d}$, and then the unified classifier $G_{\phi}(z)\in\mathbb{R}^{|C_{t}|}$ is a probability distribution over classes $C_{t}$ used to make predictions on input $x$.

Class-incremental learning requires that the model be capable of correctly classifying all samples from previous tasks at any training task – that is, when learning task $t$, the model must not forget how to classify samples from classes from tasks $t^{\prime}<t$. Exemplar-free class-incremental learning additionally restricts models to learn each new task without access to samples from previous ones. This typically leads to learning objectives that minimize a loss function $\mathcal{L}$ defined on current training data $D_{t}$:

where $\mathcal{L}_{\text{ce}}$ is the standard cross-entropy classification loss and $\mathcal{L}_{t}^{\text{M}}$ is a method-specific loss that mitigates forgetting during incremental learning. Note that without $\mathcal{L}_{t}^{\text{M}}$, Eq. 1 reduces to fine-tuning on task $t$.

Simply distilling attention between tasks does not consider task-adaptive attention. A model generates low-level representations of each input image $x$ (saliency and boundary maps in our experiments). We use CSNet [6] to generate salient regions and boundary maps, as it is lightweight and efficient, but any model producing saliency maps can be used in our framework (we explore other options in the Supplementary Material). To introduce plasticity on attention at these intermediate layers in the backbone, we use the generated boundary maps as a type of adaptive supervision as shown in Figure 3. We add a penalty term on object boundaries to avoid drift into the background. We first use 0.5 as the threshold to binarize the generated boundary map and dilate the maps by:

where $A_{b}(x)$ is the generated boundary map of image $x$, which is converted from the saliency map with a Laplacian filter, and $d$ denotes the dilation radius applied on the boundary map for controlling the strictness of boundary-guided saliency.

Rather than use a decoder at each layer as done for the low-level saliency maps described above, the mid-level saliency maps of our model are generated using Grad-CAM [42]¹¹1https://github.com/jacobgil/pytorch-grad-cam at three stages of the CNN backbone (see Figure 2). We also experiment with several other methods for generating student saliency maps and report on them in the Supplementary Material. The dilated generated boundary map $B_{d}(x)$ is downsampled to match the feature map dimensions at these three stages in order to compare with the Grad-CAM generated saliency boundary maps. We use the binary cross entropy loss for supervision on dilated boundary regions. This loss is defined as:

where $S(x,j)$ denotes the saliency map of our model on image $x$ at pixel $j$, $B_{d}(x,j)$ is the dilated generated boundary map at pixel $j$, and $N$ is the number of pixels in $x$. We compute this loss only within dilated boundary regions (i.e. where $B_{d}(x,j)=1$). This loss helps the student saliency map avoid intersecting with the dilated teacher boundary region.

We propose to learn stable features from low-level stationary tasks shared across all incremental classification tasks during class-incremental learning. Low-level vision tasks like salient object detection require useful representations of input images. By learning these feature representations across tasks, the model can focus on key areas of input images and exploit learned, stable features with less representation drift since the low-level features change very little between tasks.

Saliency map prediction is relevant to image classification since the foreground largely determines the results, while the background is comparatively less important. When learning new tasks with new classes, the background of images of new classes may contain new visual concepts that introduce undesirable noise and lead to forgetting of essential previous knowledge. The effectiveness of saliency features for learning classification tasks was demonstrated by Saliency Guided Training [17]. Additional supervision of salient region boundaries can aid salient object detection tasks for both segmentation and localization [12, 21, 25, 34, 55]. The positive interaction between these two tasks brings richer attention to features relevant to the main classification task. It can provide positive guidance in the form of stationary knowledge across class-incremental tasks. Some examples are illustrated in Figure 5.

We incorporate low-level vision tasks into the network as an auxiliary supervision for enriching task-agnostic attention. The boundary map is computed with a Laplacian filter over the estimated saliency map. We add a decoder $D_{\psi}$ [22] after the backbone $F_{\theta}$ to predict low-level saliency and boundary maps for input images. The average L2 distance between the prediction and target is used as a low-level saliency distillation loss:

where $A(x)$ denotes the target low-level maps on input $x$, consisting of a saliency map $A_{s}(x)$ and a boundary map $A_{b}(x)$. $D_{\psi}(F_{\theta}(x))$ are combined saliency and boundary maps produced by the decoder, and $N$ is the number of pixels in the saliency maps.

Although we apply low-level distillation and dilated boundary supervision to maintain saliency representations across tasks, the model can still forget saliency on samples from previous tasks. To address this, we force the model to be able to recover correct saliency maps from injected saliency noise.

At each task there is no available training data from previous or future tasks, and therefore we cannot directly know saliency drift on these samples. Instead of supervising the model with ground-truth saliency drift signals, we introduce saliency noise on random feature channels. We use a random ellipse to approximate the potential saliency drift in future tasks and the model is trained to denoise within each stage. Therefore the model can learn to effectively reduce real saliency drift.

We generate elliptical noise using a very simple approach. There are six parameter dimensions: the center coordinate $(x,y)$, the major and minor axis lengths $(a,b)$, the rotation angle $\alpha$, and the mask weight $w$. A detailed explanation of this process is given in the Supplementary Material. With the help of dilated boundary supervision, each stage learns to eliminate this additional saliency noise and this aids generalization for future tasks and mitigates saliency forgetting in previous ones.

The overall learning objective combines the low-level multi-task learning, dilated boundary supervision, and random saliency noise injection modules:

Comparing this loss with Eq. 1, for TASS $\mathcal{L}_{t}^{\text{M}}=\mathcal{L}^{\mathrm{CIL}}_{t}+\mathcal{L}^{\text{lms}}_{t}$, thus incorporating saliency-aware supervision with the cross-entropy loss. The entire process is detailed in Algorithm 1.

We follow standard experimental protocols for EFCIL on three benchmark datasets.

We report three common metrics for class incremental learning: the average and last top-1 accuracy, as well as average forgetting for all classes learned up to task $t$. Denoting by $Acc_{i}$ the accuracy over all learned classes up to and including task $i$, the average accuracy is defined as $Avg=\frac{\sum_{i=1}^{T}Acc_{i}}{T}$, and the last accuracy is $Acc_{T}$. Letting $a_{m,n}$ denotes the accuracy of task $n$ after learning task $m$, the forgetting measure $f_{k}^{i}$ of task $i$ after learning task $k$ is computed as $f_{k}^{i}=\max_{t\in 1,2,...,k-1}(a_{t,i}-a_{k,i})$. The average forgetting $F_{k}$ is defined as $F_{k}=\frac{1}{k-1}\sum_{i=1}^{k-1}f_{k}^{i}$.

We compare TASS with the state-of-the-art on CIFAR-100 in Table 2. TASS outperforms all exemplar-free approaches. For exemplar-based methods like iCaRL [40], EEIL [2], and LUCIR [16], our method still has significantly better performance. On longer sequences (i.e. 10 and 20 tasks), our method significantly reduces forgetting when learning new classes compared to other EFCIL methods. TASS outperforms the best method SSRE by about 3% on the last task. This performance improvement can be also observed in terms of average forgetting.

As we see in Table 3 and Figure 4 for Tiny-ImageNet and ImageNet-Subset, although our method has similar top-1 accuracy on the first task in Figure 4, it has better performance at most intermediate tasks and also the final one. For longer sequences in Figure 4, the gap between our method and the best baseline is largely consistent, showing the effectiveness of our method at mitigating forgetting. The performance gain in Table 3 is larger on Tiny-ImageNet and ImageNet-Subset compared to CIFAR100, and this demonstrates that our method generalizes to datasets with larger images and object scales. It is worth mentioning that TASS also produces results with smaller variance. We believe this to be due to TASS reducing saliency drift to background regions, which may include random noise.

In this section we take a deeper look at the method we propose. If not specified, the results are produced using TASS integrated into SSRE [58].