Is Support Set Diversity Necessary for Meta-Learning?

The ability to learn from limited experience is a defining aspect of human intelligence. The domain of few-shot learning aims to evaluate this criterion within machine learning [10]. For few-shot learning problems, meta-learning [30] techniques have attracted increasing attention. Meta-learning methods typically construct few-shot classification tasks in an episodic manner by sampling support ($S$) and query ($Q$) examples from a fixed number of classes. The objective is then to deliver an algorithm that can perform well on the query points of a task by training on only a few support samples.

Conventional wisdom is that the performance of meta-learning methods will improve as we train on more diverse tasks. Therefore, during meta-training, we typically allow a task to be constructed from any possible pair of support and query sets. In this work, we question this notion, specifically investigating the effect that support set diversity has on meta-learning. Surprisingly, we find that reducing the total number of unique support sets not only does not result in adverse effects—in most cases, it in fact yields significant performance improvements.

The main contributions of this work are as follows: (1) We propose a simple modification to the meta-learning objective that applies to a broad set of methods, in which we aim to optimize a biased objective by restricting the number of support sets used to construct tasks. (2) On multiple datasets and model architectures we empirically demonstrate the performance improvement of the learned algorithm when trained with our proposed objective. (3) Finally, we explore a framework to understand this surprising phenomenon and attempt to provide some insights into the observed gains. Our work delivers a simple, competitive baseline for few-shot learning and, more generally, investigates the role that task construction plays in meta-learning.

Meta-learning formulations typically rely on episodic training, wherein an algorithm is learned to adapt to a task, given its support set, so as to minimize the loss incurred on the query set. Meta-learning methods differ in terms of the algorithms they learn: Gradient-based meta-learning methods [11, 12, 23, 27, 18] consider learning gradient-based algorithms, which are typically parameterized by an initialization (possibly together with some pre-conditioning matrix). Recently, last-layer meta-learning methods  [18, 22, 3, 26] have gained popularity. These methods don’t learn an optimal model parameter initialization for the task distribution but instead learn a feature extractor for the covariates in the support and query sets. The extracted support set features along with their labels are used to learn the parameters of the last layer, e.g., in a non-parametric fashion [26] or by finding the unique solution for convex problems  [18, 2].

Several recent works [6, 7, 21] on few-shot learning have proposed methods that don’t directly optimize the original meta-learning objective. Instead they chose to minimize a transfer-learning based loss where a deep network is trained in a supervised manner on all the $(x,y)$ pairs present in the dataset. In contrast to episodic training, they show that a model trained without any notion of a task can be used to construct an algorithm whose performance matches or improves upon meta-learning methods across many benchmarks. Similar to these works, we propose to optimize an alternate objective for meta-learning; however, our work differs by retaining the episodic structure from the meta-learning framework but restricting the support set (and by consequence task) diversity.

In addition to using other objectives for few-shot learning, recent work aims to improve meta-learning by explicitly looking at the task structure and their relationships. Among these, Yin et al. [32] propose to handle the lack of mutual exclusiveness among different tasks through an information-theoretic regularized objective. Collins et al. [8] propose a minimax objective to make the learned algorithm task-robust, which improves the performance of the algorithm on the most difficult task in the face of a task pool with diverse difficulty. Liu et al. [20] propose to augment the set of possible tasks by augmenting the pre-defined set of classes that generate the tasks with varying degrees of rotated inputs as new classes. In addition, several popular meta-learning methods [26, 18], in order to improve the meta-test performance, change the number of ways or shots of the sampled meta-training tasks, thus increasing the complexity and diversity of the tasks. In contrast to these works, we look at the structure and diversity of tasks specifically through the lens of support set diversity, and show that, surprisingly, reducing diversity (by fixing support set) not only maintains—but in many cases significantly improves—the performance of meta-learning. Our results indicate that this simple modification to meta-learning is an effective method worthy of future study.

We begin with some notation that we will use to re-formulate the meta-learning objective. This will allow us to introduce the concept of a support pool in a manner amenable with current meta-learning methods.

Notation. In a standard offline meta supervised learning problem, we are given a dataset $\mathcal{D}\subseteq\mathcal{X}\times[N]$ with examples from $[N]\coloneqq 1,\ldots,N$ different classes. A task $(C,S,Q)$ is constructed by first choosing a subset of classes $C\subseteq[N]$, and conditioning on $C$, a support set $S$ and a query set $Q$ are sampled i.i.d. from $\mathcal{D}^{C}\coloneqq\{(x,y)\in\mathcal{D}:y\in C\}$. The goal is to learn the parameters of an algorithm $w\in\operatorname{\mathbb{R}}^{d}$ so that when given a support set $S$, the algorithm will output a model that achieves a low loss value on the query set. We denote this loss value as a function of the algorithm parameter, support, and query set: $\ell:\operatorname{\mathbb{R}}^{d}\times\left\{S\right\}\times\left\{Q\right\}\to\operatorname{\mathbb{R}}.$ To identify the optimal parameter of the algorithm $w$, we optimize the objective in [A] below:

$\displaystyle\begin{split}\textrm{[A]:}\,\min_{w}\operatorname{\mathbb{E}}_{C}\operatorname{\mathbb{E}}_{(S,Q)\;\mid\;C}\ell(w,S,Q)\quad\displaystyle\equiv\quad\textrm{[B]:}\,\min_{w}\operatorname{\mathbb{E}}_{S_{p}\sim\textrm{Unif}(\mathcal{P})}\operatorname{\mathbb{E}}_{C}\operatorname{\mathbb{E}}_{Q\;\mid\;C}\ell(w;S_{p}^{C},Q)\end{split}$

(1)

Reformulating the meta-learning objective. We now give another view of this objective, which will motivate our proposed modified objective. Instead of sampling from the set of classes $C$ when constructing a task, we consider first sampling uniformly from $\mathcal{P}$, a collection of support pools. A support pool $S_{p}\in\mathcal{P}$ is a smaller subset of the entire dataset $D$ that contains some examples from every class in $D$, i.e. $\left\{y:(x,y)\in S_{p}\right\}=[N]$. With $S_{p}$, we sample a subset of classes $C\subseteq[N]$ as before and take all the examples from $S_{p}$ that are of the classes in $C$. This set of examples is now our support set which we denote by $S_{p}^{C}=\left\{(x,y)\in S_{p}:y\in C\right\}$. With the classes $C$ determined, we can then sample the query set $Q$ of examples from class $C$ the same way as described previously. In this view, the marginal distribution of the support query set pair $(S_{p}^{C},Q)$ is exactly the same as that of $(S,Q)$ discussed before. As a result, we can rewrite our meta-learning objective as in (1)[B].

This reformulation prompts us to first check how big the collection of support pools $\mathcal{P}$ typically is. In the most unconstrained form, $\mathcal{P}$ can contain all possible subsets of a dataset where we have at least one sample from every class $(\forall c\in[N])$. However, most meta-learning methods only train on episodic tasks where the support $S$ is of a specific configuration, i.e., the support of each task is comprised of exactly $n$ ways (classes) with each class having $k$ shots (examples). For instance, when training on miniImagenet [31] $5$-way $5$-shot ($n=5,k=5$) tasks, all tasks that are sampled consist of exactly $5$ shots from each of the $5$ classes. Hence, it also makes sense to limit $\mathcal{P}$ to only support pools that obey a certain structure. In this work, we consider $\mathcal{P}$ to consist only of subsets of $\mathcal{D}$ that have exactly $k$ samples from each class i.e. $\mathcal{P}=\{S_{p}:{\footnotesize\sum}_{(x,y)\in S_{p}}\mathbbm{1}(y=c)=k,\;\forall c\in[N]\}$. The meta-training set of miniImagenet [24] has 64 classes with 600 examples each, resulting in $|\mathcal{P}|=\binom{600}{5}^{64}\approx 3\times 10^{755}$ support pools. This makes us wonder: Is this astronomically large number of support pools necessary or possibly redundant to achieve good meta-test performance? Can we use a much smaller number of support pools instead?

Fixing the support pool. Motivated from the questions above, we take the most extreme reduction approach: before training begins, we randomly construct one support pool $S_{p,0}$ and let $\mathcal{P}=\left\{S_{p,0}\right\}$ (just this single support pool) and solve for the corresponding optimization problem:

$\displaystyle\min_{w}\operatorname{\mathbb{E}}_{C}\operatorname{\mathbb{E}}_{Q\;\mid\;C}\ell(w;S_{p,0}^{C},Q).$

(2)

Note that the expectation over $S_{p}$ is removed because there is only a single possible realization of the random support pool. In terms of the total number of different possible support sets that the algorithm being learned can see, we have also achieved a reduction factor of $\binom{600}{5}^{5}$ times for the aforementioned miniImagenet example. In the rest of the paper, we first show the performance obtained by optimizing this reduced objective (2) for some of the best-performing meta-learning methods on multiple meta-learning problems and/or with different model architectures. We then make an initial attempt at understanding the effectiveness of using this fixed support pool objective.

We now aim to understand the differences in generalization performance between the algorithm learned by optimizing (2), i.e. our proposed objective (which we denote fix-ml), and the algorithm obtained by optimizing the original meta-learning objective (1) (which we denote ml).

For every fix-ml experiment, we randomly sample $k$ samples (shots) from each class in the dataset to form the support pool $S_{p,0}$, which is then fixed throughout training. Here, different runs of fix-ml on the exact same problem and meta-learning method can use different (but fixed) support pools. Later in this section, we will discuss how this randomness affects meta-test performance over multiple fix-ml runs. For an unbiased estimation and a fair comparison, when we evaluate a fix-ml learned algorithm on the meta-validation or meta-test set, we compute the loss and accuracy using the objective given in (1) (i.e., an average over all possible support pools), just as we would for the ml learned algorithm.

To fully evaluate the performance differences, we compare fix-ml and ml along three axes:

1. 1.

   Meta-learning methods: We focus our initial experiments on protoypical-networks (PN) [26] which uses a non-parametric last layer solver, as Protonets is a highly competitive ml approach, and also has the benefit of training faster and with less memory than competitors such as SVM [18], Ridge Regression (RR) [3]. However, we also experiment with SVM and RR-based solvers to confirm that the trends observed with Protonets generalize to these methods. As initialization-based methods are harder to optimize for highly over-parameterized models, we defer such evaluations to future work.

2. 2.

   Datasets: Our study mainly uses two of the most widely-used benchmarks in few-shot learning: (i) miniImagenet (mini) [31], which consists of 100 classes of $84\times 84$ images, split into 64 train, 16 validation and 20 test, and (ii) CIFAR-FS (cifar) [3] with an identical split but with $32\times 32$ images. Additionally for the SVM method we elect to compare on the more difficult FC-100 (FC) dataset, as the SVM-based solver has achieved significantly superior performance on this benchmark [18].

3. 3.

   Architectures: We conduct our main experiments using Resnet-12, as many meta-learning methods [2, 22] have relied on this architecture to achieve good performance. Additionally, we experiment with another wider but shallower over-parameterized architecture, WideResNet-16-10 [33] (also used by [25]), and Conv64, which was used by some initial works in meta-learning [11, 26].

To understand the improved meta-test performance of the fix-ml-learned algorithms, we first analyze their performance on the meta-training set. Evaluating the fix-ml-learned algorithm’s performance averaged over all possible support pools (equivalent to the standard ml training loss) is important because, by the generalization theory using uniform convergence bounds (e.g. VC dimension, Rademacher complexity [1]), the test loss tracks training loss more closely with more training samples. In the case of meta-learning, the “samples” are tasks in the form of $(S,Q)$ pairs. Therefore, we see that an algorithm’s ml-training loss which considers all possible $(S,Q)$ pairs, should track the algorithm’s loss on the meta-test set more reliably than the fix-ml training loss which uses only a single support pool (inducing much fewer $(S,Q)$ pairs).

Naively, one might think that because fix-ml is optimized on a single fixed support pool, the training loss on any other support pool would be much worse. To verify this idea, we re-evaluate the fix-ml-learned algorithms 1) over $10$ other fixed support pools by replacing $S_{p,0}$ in (2) with $S_{p}\neq S_{p,0}$; 2) on the original meta-learning objective (1)[B] by averaging over all possible support pools $S_{p}$ (because of the large number of possible support pools, we estimate this average using a very large number of sampled tasks). We conduct this evaluation for Protonet on cifar-5w5s and cifar-5w1s tasks, for all the trained algorithm snapshots over the entire $60$ training epochs. Here, the random class set $C$ in (1)[B] is set to always contain exactly $5$ classes ($|C|=5$) in order to match the meta-test scenario. The results are shown in Figure 3(b),(c). As we see in both plots, despite optimizing for one fixed support pool (blue curve), the training losses of other support pools (green curves) and the ml training loss (red curve) are all decreasing consistently in the same trend as the blue curve throughout the entire optimization trajectory. To understand this consistent decrease, we note that the gradient of fix-ml objective is a biased gradient of the original ml objective. When the gradient bias is not too large, stochastic gradient descent using a biased gradient can still lead to a reduction in objective loss and convergence [4], which is what we observe for the ml objective when using biased stochastic fix-ml gradients.

However, this does not give us a complete picture for meta-training. Even though ml training loss is considerably reduced for the fix-ml-learned algorithm, we need to compare this value against ml-training loss of the corresponding ml-learned algorithm to understand the meta-test performance difference. In Figure 3(c), we compare the end-of-training fix-ml solutions with the corresponding ml solutions using the ml training objective (1)[B] on 4 different dataset/task configurations. We see that fix-ml still consistently achieves higher ml-training losses than their ml counterparts. With a relatively lower ml test loss but a much higher ml training loss, the fix-ml solution must have a smaller generalization gap than that of the ml solution. Since there isn’t a single best way to understand neural networks’ generalization gap, we look at some of the recently proposed metrics:

1) Sharpness of the training loss landscape: A popular hypothesis (dating back to [13]) is that the flatness of the training loss landscape is correlated with (and possibly leads to) a smaller generalization gap [17, 5]. Several sharpness metrics based on this hypothesis were called into question by [9], showing that a functionally equivalent neural network with differently scaled parameters could have arbitrarily different sharpness according to these metrics. However, it was argued in [16] that even though sharp minima with similar test performance as flat minima do exist, stochastic gradient descent will not converge to them. While further understanding sharpness’s relationship with generalization gap is still an ongoing research topic, we still consider two sharpness analyses here.

1a) Sharpness visualization using 1d interpolation plots was first proposed by [17] and also used by [15] to compare the flatness of two different solutions’ training loss landscape when explaining the two solutions’ generalization gap differences. For two models with the same neural architecture but different weights, training and test loss are evaluated at model parameters that are different linear interpolations of the two models’ weights. In our case, we interpolate between the parameters of end-of-training fix-ml-learned algorithm and ml-learned algorithm from Protonet for miniImagenet 5w5s. This result is shown in Figure 3(a),(b). In 3(a), we see that fix-ml is located at a relatively “sharper” minima than ml despite having a smaller generalization gap. This is inconsistent with observations in [17, 15] possibly because in our case, the loss landscape between meta-train and meta-test are not necessarily shifted versions of each other (confirmed also in Figure 3(b)). Instead, we see that the trends of these two graphs align well with each other. Therefore, we cannot solely rely on 1d visualization-based sharpness analysis to explain the generalization gap.

1b) In addition to visualizations, quantitative measures of the training loss landscape sharpness has also been used. Specifically, one notion of the loss landscape sharpness is in the curvature of the loss function. This is typically measured by the maximum eigenvalue [19] or the trace $(\textrm{tr}(\mathbf{H}))$ of the Hessian matrix ($\mathbf{H}$) [16] for the parameters at the end of training . However, directly computing $\mathbf{H}$ accurately is both computationally expensive and memory-wise “impossible" for over-parameterized models like Resnet-12. Thus, $\mathbf{H}$ is typically approximated with the Fisher information matrix ($\mathbf{F}$) at the same parameter [29]. $\textrm{tr}(\mathbf{F})$ can be easily estimated without constructing the entire matrix $\mathbf{F}$. However, $\mathbf{F}$ is only equal to $\mathbf{H}$ when the parameters are the true parameters of the training set (in practice, achieve training loss extremely close to $0$). On the other hand, this approximation is very crude when the training loss is much larger than $0$ in the case of fix-ml (Figure 3(c)). Hence, $\textrm{tr}(\bm{H})$ for fix-ml’s parameters cannot be accurately estimated using $\mathrm{tr}(F)$. As a result, for our analysis, this approach is not readily applicable.

2) Another quantitative measure which approximates the Takeuchi Information Criterion (TIC) [28] was empirically shown to correlate more with the generalization gap as compared to measures of the loss landscape’s flatness [29]. Specifically, Thomas et al. (2020) proposes the quantity $\frac{\mathrm{tr}(\bm{C})}{\mathrm{tr}(\bm{F})}$, where $\bm{C}$ is the uncentered covariance matrix of the gradient with respect to the training data distribution. To evaluate how well this measure tracks the generalization gap difference between fix-ml and ml, we evaluate this approximate TIC value for three pairs of ml and fix-ml-learned algorithms in Figure 3(d). We see that for all three datasets, ml has both higher $\frac{\mathrm{tr}(\bm{C})}{\mathrm{tr}(\bm{F})}$ and larger generalization gap than its fix-ml counterpart, indicating some possible correlation between these two. Thus it would be interesting to understand what intuitive property of the solutions is captured by this quantity and why optimizing the fix-ml objective leads to solutions with lower $\frac{\mathrm{tr}(\bm{C})}{\mathrm{tr}(\bm{F})}$.

In this paper, we have proposed a simple yet effective modification of the meta-learning objective. We begin by re-formulating the traditional meta-learning objective, which motivates our approach of reducing the support set diversity. Then, we experimentally show that for a variety of meta-learning methods, datasets, and model architectures, the algorithms learned using our objective result in superior generalization performance relative to the original objective. We analyze this improvement by understanding the optimization loss and the generalization gap separately. While the optimization loss trade-off is easier to reason about using our framework, the generalization gap is still not fully understood based on several initial analyses, which we hope would pave the way for future work in this area.