Learnable Distribution Calibration for Few-Shot Class-Incremental Learning

FSCIL consists of base training and incremental learning stages, Fig. 3. During base training, it learns a representation model upon base classes ($\mathcal{C}^{(0)}$) with sufficient examples. During incremental learning, it generalizes the developed model to new classes using only a few samples. The incremental datasets are denoted as $\{\mathcal{D}^{(t)},t=0,1,2,...\}$, where $\mathcal{D}^{(t)}$ is the data of classes $\mathcal{C}^{(t)}$ for the $t$-th session. For $t_{1}\neq t_{2}$, we have $\mathcal{C}^{(t_{1})}\cap\mathcal{C}^{(t_{2})}=\varnothing$. In the $t$-th session, only the training data $\mathcal{D}^{(t)}$ for the class set $\mathcal{C}^{(t)}$ is available, while all old and new classes $\{\mathcal{C}^{(0)},...,\mathcal{C}^{(t)}\}$ require to be tested. That said, old classes $\{\mathcal{C}^{(0)},\cdots,\mathcal{C}^{(t-1)}\}$ shall not be forgotten while the new concepts shall be learned with few samples.

Fig. 3 shows LDC’s flowchart, where the network consists of a feature extractor ($\bm{\theta}_{b}$) and a PCU. The PCU contains a Gaussian sampler that initiates biased distributions, a learnable mapping function ($\bm{\theta}_{m}$) that generates the covariance matrix for each class, a recurrent calibration module ($\bm{\theta}_{r}$) that manipulates biased distributions, and a classifier ($\bm{\theta}_{c}$), Fig. 4. During base training, images are encoded to feature samples $\mathcal{X}^{(0)}$ by a feature extractor ($\bm{\theta}_{b}$), which are then fed into the PCU for classification prediction ($\mathcal{\hat{Y}}^{(0)}$). The feature extractor is trained under the supervision of ground-truth labels $\mathcal{Y}^{(0)}$. At the same time, PCU learns to generate feature samples $\mathcal{X}^{\prime(0)}$ (calibrated distributions) under the supervision of the real feature samples $\mathcal{X}^{(0)}$ (which represent real distributions). The base training procedure is formulated as:

where $\mathcal{L}_{\mathrm{c}}$ and $\mathcal{L}_{\mathrm{m}}$ respectively denote image classification loss and distribution matching loss. $\mathcal{L}_{\mathrm{c}}$ calculates the cross entropy between predictions and labels, and $\mathcal{L}_{\mathrm{m}}$ measures the distance between calibrated and real distributions. $g(\cdot)$ denotes the classification procedure.

In the $t$-th incremental session, the PCU trained upon base classes is utilized to recover old class distributions and generalized to new classes for distribution calibration. Old class samples (which is unavailable) are recovered in two steps: (1) Initializing feature samples using the Gaussian sampler; (2) Recurrently calibrating the feature samples so that their distributions approach the real ones. These two steps are also carried out for the new classes to augment training samples and calibrate the biased distributions. Few-shot samples are used to update classifier vectors and the stored covariance matrix, which are used to perform Gaussian sampling. The sampled features are then fed to the recurrent calibration module to generate calibrated feature samples for model training. The incremental learning procedure is formulated as:

where $\mathcal{X}^{(t)}$ denotes few-shot samples for the $t$-th session and $\mathcal{X}^{\prime(t)}$ denotes calibrated feature samples (calibrated distributions). $\hat{\mathcal{Y}}^{(t)}$ and $\mathcal{Y}^{(t)}$ denote the predicted and ground-truth labels of few-shot samples. $\hat{\mathcal{Y}}^{\prime(t)}$ and $\mathcal{Y}^{\prime(t)}$ denote the predicted and ground-truth labels of the calibrated samples, respectively. In order to handle new classes $\mathcal{C}^{(t)}$, $N^{(t)}-N^{(t-1)}$ classifier vectors are added to the classification layer as $\bm{\theta}_{c}=[\mathbf{w}_{0},...,\mathbf{w}_{N^{(t-1)}},...,\mathbf{w}_{N^{(t)}}]^{\intercal}$, where $N^{(t)}$ denotes the number of seen classes until the $t$-th session. The network is updated by optimizing Eq. 2.

PCU consists of a Gaussian sampler and a recurrent calibration module, both of which are parameterized, Fig. 4.

Gaussian Sampler. It is defined by multiplexing classifier vectors ($\bm{\theta}_{c}$) as mean vectors and storing a shared covariance matrix ($\bm{\Sigma}^{(t)}$) which is the mean covariance matrix for all classes. The classifier is calculates the cosine distance between sample features and classifier vectors for classification [46]. Classifier vectors are initialized by the mean feature vectors of the corresponding classes. The $i$-th classifier vector ($\mathbf{w}_{i}$) can be approximated as the prototype of the $i$-th class. The covariance matrix ($\bm{\Sigma}^{(t)}$) is dynamically updated as $\bm{\Sigma}^{(t)}=\bm{\Sigma}^{(t-1)}\cdot\frac{N^{(t-1)}}{N^{(t)}}+\mathrm{Cov}(\mathcal{X}^{(t)})\cdot\frac{N^{(t)}-N^{(t-1)}}{N^{(t)}}$. $\mathrm{Cov}(\cdot)$ calculates the covariance matrix. A learnable mapping function ($M$) is used to generate the covariance matrix for the $i$-th class, as $\bm{\Sigma}^{(t)}_{i}=M(\mathbf{w}_{i},\bm{\Sigma}^{(t)};\bm{\theta}_{m})$, which is realized by an expanding-concatenating operation and a Conv-ReLU-Conv stack. The $i$-th covariance matrix and classifier vector are fed to the sampler to generate a biased distribution ($\tilde{\mathcal{X}}_{i}^{(t)}$) for the $i$-th class. The sampling process is expressed as $\tilde{\mathcal{X}}_{i}^{(t)}=\{x|x\sim\mathcal{N}(\mathbf{w}_{i},\bm{\Sigma}^{(t)}_{i})\}$, where $x$ are the features sampled from the Gaussian distribution $\mathcal{N}(\mathbf{w}_{i},\bm{\Sigma}^{(t)}_{i})$. For the $t$-th session, we generate the biased distributions for all seen classes, which are calculated as $\tilde{\mathcal{X}}^{(t)}=\{\tilde{\mathcal{X}}_{0}^{(t)},\tilde{\mathcal{X}}_{1}^{(t)},\cdots,\tilde{\mathcal{X}}_{N^{(t)}}^{(t)}\}$.

Recurrent Calibration module. This is defined using convolutional layers, parameterized with $\bm{\theta}_{r}$, Fig. 4. For the $t$-th session, the recurrent calibration process is expressed as:

During recurrent calibration, the objective function is built upon previous optimization and prediction. This makes it possible to deconstruct a complex objective function ($e.g.$, the distribution matching loss) to series of simpler objective functions which are easier to realize.

Distribution Matching Loss. During base training, the shared covariance matrix and classifier vectors are used to sample biased distributions. Then those biased distributions are calibrated to obtain the calibrated distributions $\mathcal{X}^{\prime(0)}$. Given the real distributions $\mathcal{X}^{(0)}$, the loss is defined as:

where $\mathrm{KL}(\cdot)$ is defined as the KL divergence [47]. By minimizing the distribution matching loss $\mathcal{L}_{\mathrm{m}}$, PCU’s parameters are updated so that the calibrated distributions align with real distributions. PCU is generalized to new classes for calibration during incremental learning.

We managed to prove that recurrent sample calibration is theoretically plausible, by formulating it as a variational inference procedure during training [48]. Given real distributions $\mathcal{X}^{(t)}$, biased distributions $\tilde{\mathcal{X}}^{(t)}$ and the parameters $\bm{\theta}_{r}$ for recurrent distribution calibration. $(\bm{\theta}_{r},\tilde{\mathcal{X}}^{(t)})$ defines latent variable $\mathbf{z}$. The optimization of $\mathbf{z}$ is equivalent to finding a mean-field distribution [49] $q^{*}(\mathbf{z})$ which is to approximate the distribution $p(\mathbf{z}|\mathcal{X}^{(t)})$, which is formulated as:

where $\mathcal{Q}$ denotes the candidate probability distributions. According to the characteristics of mean-field distribution, $q(\mathbf{z})=\prod_{m=1}^{M}q_{m}(\mathbf{z}_{m})$ where $q_{m}(\mathbf{z}_{m})$ is a subset of $q(\mathbf{z})$. $\mathrm{KL}$ denotes the Kullback-Leibler divergence between distributions, which is calculated by:

$\mathbb{E}$ denotes the expectation function. $\mathrm{KL}$ is non-negative, thus we have:

The right side of the Eq. 7 is the lower bound of the left side, which is termed as evidence lower bound ($\mathrm{ELBO}$). We have:

According to Eq. 8, Eq. 5 is re-written as:

As $p(\mathcal{X}^{(t)})$ is independent to $q(\mathbf{z})$, Eq. 9 is equivalent to:

for the purpose of simplification, we use $\mathrm{ELBO}$ for $\mathrm{ELBO}(q,\mathcal{X}^{(t)})$, where $\mathrm{ELBO}$ is calculated as:

For $\forall j\in[1,M]$, the term $\prod_{m=1}^{M}q_{m}(\mathbf{z}_{m})$ is expressed as $q_{j}(\mathbf{z}_{j})\times\prod_{m\neq j}q_{m}(\mathbf{z}_{m})$, and the term $\sum_{m=1}^{M}\log q_{m}(\mathbf{z}_{m}))$ is expressed as $q_{j}(\mathbf{z}_{j})+\sum_{m\neq j}\log q_{m}(\mathbf{z}_{m}))$. Accordingly, Eq. 11 is re-written as:

where $\mathrm{c}$ denotes a constant. $\log\tilde{p}(\mathcal{X}^{(t)},\mathbf{z}_{j})$ is the non-normalized distribution of $\mathbf{z}_{j}$, calculated using:

where $\mathbf{z}_{\backslash j}$ are the latent variables other than $\mathbf{z}_{j}$. Assuming the $\mathbf{z}_{\backslash j}$ is fixed, we optimize $q_{j}(\mathbf{z}_{j})$ to maximize the $\mathrm{ELBO}$. According to Eqs. 12 and  13, the optimal $q^{*}_{j}(\mathbf{z}_{j})$ is proportional to the expectation of $\log p(\mathcal{X}^{(t)},\mathbf{z})$, as:

According to Eq. 14, by recurrently optimizing $q^{*}_{j}(\mathbf{z}_{j})$, which is achieved by calibrating $\tilde{\mathcal{X}}^{(t)}$ and updating $\bm{\theta}_{r}$, $\mathrm{ELBO}$ is maximized and converges to a local optimal solution. According to Eqs.  9 and  10, maximizing $\mathrm{ELBO}$ is equal to minimizing $\mathrm{KL}(q(\mathbf{z})||p(\mathbf{z}|\mathcal{X}^{(t)}))$, which infers that a feasible solution for $q^{*}(\mathbf{z})$ could be found so that the calibrated distributions $\mathcal{X}^{\prime(t)}$ in Eq. 3 approximate real ones ${\mathcal{X}}^{(t)}$.

Dataset. We evaluate LDC using three commonly used datasets, including CIFAR100, CUB200, and mini-ImageNet. The categories in the datasets are divided to base ones with adequate annotations and new ones with K-shot annotated images. For the FSCIL, the proposed LDC is trained upon base classes for the first session. New classes are gradually added to train LDC in T incremental sessions. In each incremental session, N-way new classes are added for model training. CIFAR100 and mini-ImageNet datasets consist of 100 classes, where 60 classes are set as base classes and 40 as new classes. Each new class has 5-shot annotated images (K = 5). New classes are then divided to eight sessions (T = 8), each of which has five classes (N = 5). CUB200 consists of 200 classes where 100 classes are set as base classes and the other 100 classes as new classes under the settings of K = 5, T = 10, N = 10.

Training and Evaluation. Following  [1, 6, 53], we adopt Resnet-18 [54] as the backbone for CUB200 and mini-ImageNet, and Resnet-20 for CIFAR100. The model is optimized using the SGD algorithm. Following FACT [53], we use normalization, horizontal flipping, random cropping, auto augmenting and random resizing for data augmentation. For the first session, we train the network using $D^{(0)}$ upon base classes, with a batch size 128 and an initial learning rate 0.1. When $t>0$, the network is trained upon $D^{(t)}$ with new classes and the learning rate is set to 0.01. The network is trained for 100 epochs during each session. During inference period, we conduct 10 experiments using random seeds and report averaged results to eliminate the randomness of the experiments. We adopt a simple baseline method which consists of a feature extractor and a distance-based classifier. Experiments are conducted using PyTorch 1.9.0 and run on Nvidia 2080Ti GPUs.

During inference, the model trained for all $T$ sessions are evaluated by classification on all the seen class $\{\mathcal{C}^{(0)},\mathcal{C}^{(1)},...,\mathcal{C}^{(t)}\}$ using the metric of $\mathrm{accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP+TN+FP+FN}}$, where $\mathrm{TP}$, $\mathrm{TN}$, $\mathrm{FP}$ and $\mathrm{FN}$ respectively denote the numbers of true positives, true negatives, false positives and false negatives. To better measure model forgetting, ‘$\mathrm{PD}$’ and ‘$\mathrm{PR}$’ are introduced to evaluate the model performance drop and performance retention respectively. ‘$\mathrm{PD}$’ is defined as $\mathrm{PD}=\mathrm{accuracy}^{(0)}-\mathrm{accuracy}^{(T)}$, where $\mathrm{accuracy}^{(0)}$ denotes the $0$-th session performance and $\mathrm{accuracy}^{(T)}$ denotes the $T$-th session performance. ‘$\mathrm{PR}$’ is defined as $\mathrm{PR}=\mathrm{accuracy}^{(T)}/\mathrm{accuracy}^{(0)}$.

We conduct ablation studies on CUB200 to validate PCU’s modules, including the Gaussian sampler, the recurrent calibration module and the distribution matching loss.

Gaussian Sampler. In Table I, the proposed approach outperforms the baseline method by 2.72% with ‘Gaussian Sampling’. The performance gain validates the plausibility of the Gaussian sampling process based on classifier weights and a single stored covariance matrix.

Recurrent Calibration. With recurrent calibration, LDC further improves the baseline method and Gaussian sampler by 9.72% and 7.00%, respectively. By recurrently calibrating biased distributions generated with the Gaussian sampler, generated distributions approach real distributions, which augments training data and benefits model learning.

Distribution Matching Loss. Four kinds of matching losses defined upon earth mover distance, KL divergence, JS divergence, and Hellinger distance are compared in Table II, where one can see that KL divergence performs the best.

Memory and Time Costs. As shown in Fig. 5, LDC’s memory costs for covariance matrix storage is much lower than Free-Lunch’s. And LDC’s time costs is comparable to Free-Lunch’s when the calibration iterations increase. It is attributed to the employing of convolution layers so that recurrent calibration costs negligible time.

Distribution Calibration. We utilize t-SNE to visualize real distributions and calibrated distributions generated using the proposed LDC methods to analyze the recurrent distribution calibration process better. Fig. 6 shows that the calibrated distributions generated by LDC approach real distributions when calibration proceeds. Consequently, training with calibrated distribution samples alleviates catastrophic forgetting and benefits estimations for new class distributions.

Outlier Cases. We compare the performance of LDC and Free-Lunch using normal and outlier cases to further validate the ability to calibrate biased distributions, Table III. For normal cases, few-shot samples are randomly sampled from new class distributions using the mini-ImageNet dataset. For outlier cases, few-shot samples are outlier samples in new class distributions. The larger performance gains ($i.e.$, $\Delta\mathrm{PD}$ and $\Delta\mathrm{PR}$) in outlier cases demonstrate that LDC achieves better calibration results than Free-Lunch when few-shot samples are extremely biased.

Scalability. We compare performance drops when the class number increases to validate LDC’s scalability. We fix the memory costs and calculate the classification accuracy for a fair comparison. From Fig. 7, one can see that when classes increase, LDC maintains the performance while Free-Lunch suffers significant performance drops. The results demonstrate that LDC has higher scalability w.r.t. class number.

Covariance matrix mapping strategy. We compare the convergence and generalization of our proposed covariance matrix mapping strategy and the strategy adopted in empirical distribution calibration [3]. As shown in Table IV the model with matrix mapping strategy converges at $206^{th}$ epoch on mini-ImageNet, while the compared model with empirical strategy converges at $312^{th}$ epoch. For generalization, we randomly select 20 unseen classes with few-shot samples for finetuning and test the classification accuracy. The results show that the model with matrix mapping strategy outperforms the counterpart model on both mini-ImageNet and CIFAR100 datasets. The results show that the proposed strategy (sharing mean covariance matrix and mapping the covariance matrix for each class) benefits learning of class distribution statistics, as well as improving the model generalization capability.

CUB200. In Table V, LDC achieves the best performance on all sessions and the lowest performance dropping rate. Benefited by PCU, LDC’s accuracy in the first session is higher than other methods, and it outperforms FSLL [55] and Fact [53] by 7.37% and 4.64% in the final session. CIFAR100. In Fig. 8(left), LDC outperforms the state-of-the-arts across all incremental sessions. Specifically, it improves SPPR [44] and Fact [53] by 10.83% and 1.98%, respectively. Mini-ImageNet. Compared to the baseline method (CEC), LDC improves CEC by 6.83% (54.46% v.s. 47.63%), which demonstrates the effectiveness of the proposed learnable distribution calibration mechanism. Compared to the state-of-the-art methods, LDC outperforms the Fact method by 3.97%, Fig 8(right).

LDC has great potential to be a plug-and-play module which improves model generalization capacities. To demonstrate such potential, we apply LDC to the regular setting of few-shot classification. The PCU trained using base classes with sufficient training samples is used to generate (augment) samples for novel classes. It improves the accuracy upon Free-Lunch for both 1-shot and 5-shot on mini-ImageNet and CUB datasets, demonstrating the general effectiveness for few-shot learning problems, Table VI.