Incremental Meta-Learning via Indirect Discriminant Alignment

We identify a learning task with a training set. This dataset, together with a function to be minimized (loss) and the set of functions to minimize it (models), defines an optimization problem whose solution is a trained model that “solves the task.” So, a learning task can be identified with both a training set, and a suitably trained model. In Sect. 2.1, we will introduce empirical cross-entropy as a loss, deep neural networks (DNNs) as a set of functions, and stochastic gradient descent (SGD) as an optimization scheme. A model consists of a feature representation, or embedding, obtained by processing each datum through a backbone, which is then used to test each hypothesis, or class, using a discriminant function. A discriminant is a function that maps a feature to the hypothesis space, where its value is used to render a decision. A discriminant vector is the collection of discriminant values associated to each hypothesis or class. A classifier is a function that outputs the minimizer (or maximizer) of a discriminant function, which corresponds to a predicted class or hypothesis. For instance, the Bayesian discriminant is the posterior density of the classes given the data. The corresponding discriminant vector is the collection of posterior probabilities for each hypothesis. Any sufficient statistic of a discriminant is also a discriminant, for instance its negative log or any other invertible function of it. The optimal Bayesian classifier is one one that returns the hypothesis with the maximum a-posteriori probability (MAP), or with the smallest negative log-posterior.

The main contribution of this paper can be more easily understood for the case of metric classifiers, where each class is represented by a prototype, or “center” (Fig. 1), and the discriminant compares features to prototypes, for instance using the Euclidean distance in the embedding space, although our method is not restricted to this case. Each new learning task has a set of classes that is possibly disjoint from those of old tasks. The goal of IML is to update the base model incrementally while solving each new task, without necessarily requiring access to data from old tasks, and despite each new task having only few samples per class. Simply imposing that all tasks use the same features would be too restrictive, as different tasks may require the embedding to change while preserving the old centers. On the other hand, we cannot compare discriminants or classifiers directly since they map to different hypothesis spaces.

Since we cannot compare classifiers directly, and we do not want to needlessly restrict the model’s freedom to evolve, the key idea is to align models for the old and new tasks indirectly, by imposing that their discriminants (in the metric case, the vector of distances to the class centers) be aligned relative to a minimal set of “anchor classes,” and otherwise leaving the embedding free to adapt to new tasks. The minimal anchor set is represented by the old centers. Thus, indirect discriminant alignment (IDA) is performed by mapping the data to old centers, through both the new and the old backbones, and minimizing the misalignment between the two resulting discriminant vectors. Misalignment can be measured in a number of ways, and the process can be conducted by only processing new data incrementally, resulting in a continuous improvement of the old model. The more general case, which applies to non-metric classifiers, is explained in Sect. 2.2. It results in the main contribution of our work, which is to propose what is, to the best of our knowledge, the first method for incremental few-shot meta-learning.

We tackle the case of few-shot learning since, in the presence of large amounts of data for the classes of interest, pre-training a large model and fine-tuning it for the task at hand already yields a strong baseline. This is not the case for few-shot learning, where the current state-of-the-art still lags far behind [5]. Our method directly generalizes several meta-learning algorithms [19, 35, 25], and is applicable to more yet.

In Sect. 2.4 we describe two implementations for performing incremental meta-learning and a number of baselines (Sect. 3.1), which form the basis for empirical evaluation in Sect. 3.3 on few-shot benchmark datasets outlined in Sect. 3.2. We also introduce DomainImageNet in Sect. 3.2 to measure the effect of meta-learning on incremental learning when new classes are both in- and out-of-domain. We highlight some limitations and failure cases in Sect. 3.4. We further discuss related work and future opportunities in Sect. 4.

A model for a classification task is a parametric approximation $p_{w}(y|x)$ of the posterior distribution of the class $y\in\{c_{1},\dots,c_{K}\}$ given the test datum $x$. For a given class of functions (architecture), the model may be identified with its parameters (weights) $w$. The model is trained by minimizing a loss function $L$ that depends on a dataset $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}$, which defines the task, so that $w_{0}\doteq\operatorname{arg\,min}_{w}L(w;\mathcal{D})$.

The challenge in extending incremental learning (3) to meta-learning (4) is that each task $\mathcal{D}_{\tau}$ in the latter has a different discriminant $f_{\tau}$ for a different set of classes $\mathcal{C}_{\tau}$. Thus, aligning the discriminants directly would be imposing alignment between different classes, which is not meaningful.

A naive solution would be to just align the features $\phi_{w}(x)$ on all inputs, for instance by minimizing their average distance

However, this would be needlessly restrictive: Completely different features can yield the same posterior density, and we want to exploit this flexibility for incremental learning. Moreover, we want our method to only process new data, rather than keep re-processing old data, which would defy the goal of incremental processing. To simplify the notation, we refer to $p_{w_{\tau}}$ as $p_{\rm old}$ and $p_{w_{\tau+1}}$ as $p_{\rm new}$, and so for the heads $f_{\rm old},f_{\rm new}$ and the backbones $\phi_{\rm old},\phi_{\rm new}$. Each can be trained on different tasks, or episodes, $\tau$.

The key idea of this paper is to enable aligning the old and new discriminants using “class anchors” from the old task $\tau$, while processing only data from the new task $\tau^{\prime}$. This is done through indirect discriminative alignment (IDA), illustrated in Fig. 1, which addresses the challenge that the tasks $\tau$ and $\tau^{\prime}$ may not share any classes. IDA uses the classes defined by the old discriminants as “anchors,” and imposes that the features processed through the old and new embeddings share the same discriminant relative to these anchor classes. For metric-based classifiers, the classes can be represented by points in latent (feature, or embedding) space, and the anchors are just an under-complete basis of this space, with the discriminant vector represented by the Euclidean distance to each anchor class representative. The under-complete alignment leaves the residual degrees of freedom free for continual learning. However, the method is more general, allowing any discriminant function.

To make the dependency on the anchor classes and episodes explicit, we write the model $p^{\tau}_{w}(y|x)=f^{\tau}_{w}(y|\phi_{w}(x))$. Indirect discriminative alignment of the new model to the old one is then performed by minimizing:

where $\mathcal{C}_{\rm old}$ is a set of classes obtained after training on the old training set and ${\tau^{\prime}}$ are tasks sampled from the new dataset $\mathcal{E}$.

Intuitively, we reuse the old class representatives $\mathcal{C}_{\rm old}$ and ask that the new features $\phi_{\rm new}$ remain compatible with the discriminant $f_{\rm old}$. Moreover, instead of sampling from the old dataset $\mathcal{D}$ – which we may no longer have access to – we sample from $x\sim\mathcal{E}$. In the case of metric classifiers, this can be interpreted as aligning the new features to a set of anchor points, which in particular are the old class representatives.

Note that $f$ can be any discriminant that can process data generated via a representation function $\phi$, where both $f$ and $\phi$ have their own parameters. Also, the choice of KL-divergence to measure the discrepancy between discriminant vectors is due to the fact that it yields a simple expression for most commonly used models, but IDA is not limited to it and any other divergence measure could be employed instead.

Given (14), incremental meta-learning consists of solving

where the first term corresponds to fine-tuning the base model on the new data, while the second term enforces indirect discriminant alignment relative to the anchors from old classes. In the next section, we describe our implementation and empirical evaluation.

The simplest implementation of our method eq. (7) is obtained by using a metric classifier as the base meta-learner. This choice limits us to each task having the same number of classes, a choice we will discuss and extend in App. F. We represent a metric-based classifier using a function $\psi_{w}$ that computes the class representatives, or prototypes, or “centers,” $c_{k}^{\tau}=\psi_{w}(\mathcal{D}_{\tau})_{k}$,²²2We overload the notation $c$ to indicate the classes in $\mathcal{C}$ and the class representation, which are the argument of $\chi$, since both represent the classes. and a function (metric) $\chi_{w}(z_{i},c_{k}^{\tau})$ that scores the fit of a datum, represented by the feature vector $z_{i}$, with an hypothesis corresponding to a class $c_{k}^{\tau}$. Each function can be fixed [35] or learned [25]. Note that the backbone $\phi_{w}$ is common to all tasks, whereas the metric changes with each few-shot task $\mathcal{D}_{\tau}$, since $c_{k}^{\tau}\in\mathcal{C}_{\tau}$ and $\mathcal{C}_{\tau}=\{c_{k}^{\tau}=\psi_{w}(\mathcal{D}_{\tau})_{k}\}_{k=1}^{K}$. According to this model, the optimal (Bayesian) discriminant for the task $\mathcal{D}_{\tau}$ is of the form:

where $z=\phi_{w}(x)$. Note that $\chi_{w}$ and $p(y|x)$ are equivalent discriminants: Maximizing the posterior is equivalent to minimizing the negative log, which yields a loss of the form

Our first implementation has a trained backbone $\phi_{w}$ but fixes the metric $\chi$ to be the $L_{2}$ distance and the class representatives to be the means:

The detailed computation of the loss eq. (2.4) is described in the appendix. After every training episode, we discard the data used for meta-training and only retain the class anchors $\mathcal{C}_{\tau}$. Our paragon (oracle), that will be described in eq. (10), does not retain any class anchors but trains a new meta-learner at every episode, utilizing all data seen thus far. Ideally, the final performance of the two should be similar, which would justify incremental processing of new datasets without the need to re-process old data, which was our working hypothesis and the basis of eq. (2). Indeed, this is what we observe in Sect. 3.3.

For meta-training, we sample few-shot tasks using episodic sampling i.e. each batch consists of $K$ classes sampled at random. We then sample $N_{s}$ samples as the support samples and $N_{q}$ samples as the query samples. The class representations are calculated only using the support samples, while the query samples are used to compute the loss. For training the base model, we sample few-shot tasks $\tau$ from the old dataset $\mathcal{D}$ and train the model using the loss function in eq. (4). To train the incremental model we sample a few-shot task $\tau^{\prime}$ from the new dataset $\mathcal{E}$. We then sample $K$ random class anchors (of the old dataset $\mathcal{D}$) from $C_{\tau}^{t}$, which are calculated and preserved after the previous training phase. During incremental phase(s), the network is trained by minimizing eq. (2.4).

To evaluate the method quantitatively, we need an upper-bound (oracle) represented by a model that performs meta-training using all the data as well as few other baselines to enable a fair comparison and ablation studies.

where $\mathcal{E}_{\tau^{\prime}}$ is task sampled from the new dataset and $\mathcal{C}_{t}$ and $\mathcal{C}_{t+1}$ are obtained by re-processing $\mathcal{D}_{\tau}$ (a task sampled from the old dataset) through the old and the new embeddings respectively. We expect this method to perform best, as it has access to old data. However, it is computationally more expensive than IDA as we need to re-process old data.

We test our algorithm on MiniImageNet [38], TieredImageNet [28] and another variant of ImageNet [29] which we call DomainImageNet. MiniImageNet consists of images of size $84\times 84$ sampled from $100$ classes of the ILSVRC dataset [29], with $600$ images per class. We used the data split outlined in [24], where $64$ classes are used for training, $16$ classes for validation and $20$ for testing. We further split the $64$ training classes randomly into $32$ for meta-training the base model and the remaining for training the incremental model; $16$ validation classes are only used for assessing generalization during meta-training for both the base and incremental models. For a fair measurement of performance on the old data, we also use a separate test set comprising 300 new images per class [10].

TieredImageNet is a larger subset of ILSVRC, with $779,165$ images of size $84\times 84$ representing $608$ classes that are hierarchically grouped into $34$. This dataset is split to ensure that sub-classes within the $34$ groups are not shared among training, validation and test sets. The result is $448,695$ images in $351$ classes for training, $124,261$ images in $97$ classes for validation, and $206,209$ images in $160$ classes for testing. For a fair comparison, we use the same training, validation and testing splits of [28] and use the classes at the lowest level of the hierarchy. Similar to MiniImageNet, we randomly pick $176$ classes from the training set for meta-training the base model and use the remaining $175$ classes for incremental meta-training. Here we also use a separate test set of about $1000$ images per class for measuring old task performance.

To investigate the role of domain gap in IML, we assemble DomainImageNet, along the format of MiniImageNet, with $32$ old meta-training classes, $32$ new meta-training classes, $16$ meta-validation classes and $40$ meta-test (unseen) classes. All classes are sampled from the ILSVRC dataset, but old, new and meta-test set have two subdivisions, one sampled from natural categories, the other sampled from man-made categories. $40$ unseen classes consist of $20$ classes each of natural and man-made categories. The domain split we use follows [41].

We test IML on each dataset using two common few-shot scenarios: 5-shot 5-way and 1-shot 5-way. We refer to the data used to train the base model as old classes, and that of the incremental model as new classes. We refer to unseen classes as classes that the model has not seen in any training. Final performance of the meta-learner is reported as the mean and $95\%$ confidence interval of the classification accuracy across $2000$ episodes or few-shot tasks.

Results of the different methods using PN as a meta-learner are shown in Table 1 for MiniImageNet, Table 3 for TieredImageNet and Table 4 for DomainImageNet. Further, we show results using ECM as a meta-learner in Table 2 for MiniImageNet. We also show the results using ECM on DomainImageNet for 5-shot 5-way in Table 5. All results for DomainImageNet are using natural objects as the old domain and man-made objects as the new domain. In the appendix, we show the results for 1-shot 5-way and also for all combinations of shots and meta-learners while using man-made objects as the old domain and natural objects as the new domain.

The implementation we chose, based on [35] and [25], and the tests we performed limit our assessment to tasks that share the same number of classes, $K=5$, as customary in the literature. While technically not a limitation as one could always build a set of models, each for a different number of classes, and indeed it is not uncommon to train and fine-tune different models for different “ways” as seen in the literature, we use the same model for all tests. It is nonetheless desirable to have a meta-learner that can handle an arbitrary number of classes, different for each training episode. While our general framework eq. (7) enables it, our simplest implementation described in eq. (2.4) does not. In appendix, section F, however, we describe a modified implementation that is not subject to this restriction. Since benchmarks in the literature most commonly refer to the cases $K=1,5$, we use the simpler model in our experiments.

Further, sampling $K$ classes among many has low probability of yielding hard tasks that can be informative of meta-learning. Even simple classifiers can easily tell 5 random classes from ImageNet apart. Hard task mining could be done by selecting tasks using a distance such as [1], by sampling a random class and picking the $4$ closest ones in Task2Vec space for a $5$-way setup.

Finally, in our experiments we have noticed that there is still a performance gap between IDA and the paragon. The performance is matched for the case of unseen classes, but there is room for improvement in tasks sampled from new/current task distribution across shots, datasets and methods.