Few-Shot Learning with a Strong Teacher

On the surface, our approach is reminiscent of knowledge distillation [30]. However, there are several notable differences from other FSL methods that are based on knowledge distillation [31, 32, 33, 34]. For instance, Tian et al. [34] studied how distillation with self-ensemble improves the pre-trained features for the downstream FSL task. In contrast, we focus on how to fundamentally improve meta-learning based FSL methods by leveraging distillation in the meta-training process. That is, our approach is complementary to their efforts as we apply distillation at a different training stage (meta-training vs. feature pre-training)⁵⁵5Nowadays, nearly all meta-learning based FSL methods are initialized with pre-trained features learned in a conventional way with the base class data [19, 23, 35, 34], and our approach is built upon them.. Wang et al. [33] studied how to generate extra support set examples such that the generated classifier by the few-shot learner could perform similarly to the target classifier. In contrast, we do not introduce the extra data generation task, making our method more easily applicable. We also note that, our method is different from self-distillation [36, 37, 38, 39] — our teacher is a classifier trained with ample data while our student is a few-shot learner. Please see section 2 for more details and discussions.

FSL generally focuses on the $C$-way classification scenario in which the few-shot learner receives few labeled examples for $C$ new classes and needs to generate a classifier to recognize them ($C$ is usually set between $5$ to $30$). FSL can be studied from roughly three aspects from data, models, and algorithms [40]. We describe non-meta-learning based FSL methods in this subsection, and describe meta-learning based methods in the next subsection.

From the aspect of data, FSL can be resolved if one has sufficient data for the $C$ novel classes. Data augmentation approaches hallucinate additional examples of novel classes based on their few examples [41, 42, 43]. The hallucinated examples are used together with the original ones to train a better classifier for the novel classes. Since there are more training examples, it prevents the model from over-fitting.

From the aspect of models, FSL can be resolved if one can extract features that faithfully capture the semantic and discriminative information of each example to support nearest-neighbor classifiers. Feature or metric learning approaches learn a feature extractor or metric that encourages examples from the same classes to be closer and examples from different classes faraway [18, 44, 45]. The resulting features or metrics are used in a linear or nearest-neighbor classifier that is trained on the examples of the novel classes [46, 47, 9]. Another direction to resolve FSL from the aspect of models is to construct generative models, which are learned to generate visual content conditioned on stochastic binary variables that indicate object identities. Recognizing novel classes involves adding new variables to the model and estimating the parameters associated with it based on few labeled data [48, 49, 50].

From the aspect of algorithms, one could alter the optimization strategy to search for a better model given limited data. Hypothesis transfer reuses a well-trained model from a related dataset. Through fine-tuning it, we can leverage the knowledge learned from the related dataset to avoid over-fitting to few-shot data [11, 51]. [52, 53, 54] proposed special regularization strategies used during fine-tuning.

Here we mainly discuss how meta-learning helps FSL. Meta-learning constructs simulated few-shot tasks, called episodes [10, 15], from the base class data to mimic future few-shot tasks, and trains a few-shot learner to maximize the classification accuracy on the episodes. It is expected that the few-shot learner could be generalized to the few-shot tasks with novel classes. Various strategies have been investigated to implement the few-shot learner. For example, based on the meta-trained feature extractor, the few-shot learner classifies the query instances with (adapted) embeddings [10, 13, 3, 28, 55, 56, 57]. Based on the meta-trained model initialization [7, 58, 59, 60, 61, 62], the few-shot learner can adapt it to a good classifier for each few-shot task with a small number of gradient descent steps. The optimization strategy for a few-shot learner to fit the few-shot examples can also be meta-trained and shared across episodes [63, 64, 65], e.g., the learning rate [66, 15]. An image generator could also be meta-trained to augment the support set [67]. We note that meta-learning can explicitly take characteristics of few-shot tasks into account. For instance, the number of novel classes $C$ and the number of labeled examples per class (i.e., shots) can be specified in the episodes.

Meta-learning based FSL has been applied in various fields, such as unsupervised learning [68], human motion prediction [20], domain adaptation [69], image generation [70], and object detection [71, 72]. Some novel directions have also been explored recently. For example, [73, 74] leveraged the graph propagation to improve meta-learning; [75, 76] constructed a generalized FSL objective towards adaptive calibration strategies between base and novel classes; [77] investigated specialized data augmentations; [78] studied how to identify invariant features in meta-learning. Auxiliary self-supervised tasks are also incorporated to improve the generalization ability of the resulting few-shot learners or classifiers [79, 80, 81].

Most of the meta-learning approaches rely on the classification loss on the query sets to drive the meta-training of the few-shot learner. [16, 21, 82] were the first to propose the notion of constructing target classifiers to guide few-shot learners. Specifically, they measured the Euclidean distance between the target classifier and the classifier generated by the meta-learner as the loss. However, due to the large number of parameters, they only managed to meta-train the few-shot learner to generate the linear classifiers (i.e., the final fully-connected layers).

Some recent methods emphasize the importance of pre-trained features [44, 64, 23, 19, 34, 35]. In particular, the parameters of the backbone model are initialized by optimizing a classification objective over the base class data, which results in more discriminative features. Fine-tuning over the backbone leads to strong FSL baselines [11, 51]. The influence of the number of shots (e.g., between meta-training and meta-testing) is discussed in [83, 84, 28].

We empirically find that the effectiveness of meta-learning diminishes sharply with increasing shots. Specifically, compared to the simple classifiers built upon the pre-trained embeddings, the episodic meta-training could not get further improvement. We propose a plug-and-play module LastShot for meta-learning based FSL to encourage the few-shot learner to generate classifiers which perform like strong classifiers. This helps the resulting few-shot learner outperform other methods across different shot numbers.

As mentioned in subsection 1.1, our approach is reminiscent of knowledge distillation [30] as we leverage target classifiers trained on ample examples to supervise the meta-training of the few-shot learner. In this subsection, we therefore review knowledge distillation and contrast our usage of it to other existing methods for few-shot learning [31, 32, 33, 34].

Knowledge distillation (KD) [30] is a training strategy that leverages the dark knowledge from one (teacher) model to facilitate the learning progress of another (student) model. One popular way to extract the dark knowledge is by the instance-wise prediction, e.g., the posterior probability on a data instance [30, 85, 86], which provides the relative comparison among classes, in contrast to the conventional one-hot label. By matching the predictions of a “student” model to those of a “teacher” model, the student model obtains richer learning signals and becomes more generalizable. The dark knowledge for KD can be in the form of soft labels [30], hidden layer activation [87, 88], and comparison relationships [89]. Knowledge distillation has also been exploited for model interpretability [90] and incremental learning [91, 92], etc.

Besides [34, 33] mentioned in subsection 1.1, KD was also employed in two recent works of FSL [31, 32], but again based on very different motivations and manners from our LastShot. [32] applied KD mainly to construct a portable few-shot learner; [31] applied self-supervised learning [93, 94] to train robust features and use them to guide the few-shot learner. Both of them only studied a single FSL method, e.g., [7]. In contrast, our approach is more general, compatible with many meta-learning based FSL methods and requiring no self-supervised learning steps. Wang et al. [33] investigated constructing a data generator to create extra support set examples; the generator is learned via KD, from a many-shot target classifier. In LastShot, we take advantage of a plug-and-play loss term to distill knowledge from the target classifier more easily.

Self-distillation is a special case of KD which distills knowledge from the model itself, i.e., the teacher can be the previous generation of the model along the training progress [36, 95, 37]. Tian et al. [34] applied self-distillation in the model pre-training stage to obtain more discriminative features. Our LastShot works in an orthogonal direction to directly improve the meta-learning based FSL methods. Different from self-distillation, the target classifier in LastShot is not the previous few-shot learner or feature extractor, but a stronger classifier trained with ample base class data.

In FSL, a few-shot learner is first provided with a training set $\mathcal{D}_{\text{base}}=\{({\bm{x}}_{1},y_{1}),\dots,({\bm{x}}_{N},y_{N})\}$ that contains $N$ labeled images from $B$ base classes; i.e., $y_{n}\in\{1,\cdots,B\}$. Here, each class is supposed to have ample training examples. The few-shot learner is then provided with a support set $\mathcal{D}_{\text{support}}^{\prime}$ of labeled images from $C$ novel classes $\{B+1,\cdots,B+C\}$, in which each novel class has $K$ examples. The goal of the few-shot learner is to generate a classifier that can accurately classify among the $C$ novel classes. This FSL setting is referred to as the $C$-way $K$-shot setting.

The core concept of meta-learning is to use the base class data $\mathcal{D}_{\text{base}}$ to simulate the FSL scenario for meta-training the few-shot learner, before it receives the few-shot data from the novel classes. Namely, meta-learning samples multiple FSL tasks $\mathcal{T}$ from $\mathcal{D}_{\text{base}}$, each contains a $C$-way $K$-shot support set $\mathcal{D}_{\text{support}}$ whose class labels are from $\{1,\cdots,B\}$. Meta-learning then inputs each support set to the few-shot learner $\mathcal{A}$, whose goal is to output/generate a classifier that can classify the $C$ input classes well. To evaluate the quality of $\mathcal{A}$, meta-learning approaches usually sample a corresponding query set $\mathcal{D}_{\text{query}}$ for each $\mathcal{D}_{\text{support}}$, whose examples are from the same $C$ classes. The quality of $\mathcal{A}$ is measured by the outputted classifier’s accuracy on $\mathcal{D}_{\text{query}}$.

Putting all these notations together, we can now formulate the meta-learning problem as follows

where each $\mathcal{T}$ is a tuple $(\mathcal{D}_{\text{support}},\mathcal{D}_{\text{query}})$; $\mathcal{A}(\mathcal{D}_{\text{support}})$ is the $C$-way classifier outputted by $\mathcal{A}$, after taking the support set $\mathcal{D}_{\text{support}}$ as input. $\ell$ is a loss function on the prediction; e.g., the cross-entropy loss. As shown in Equation 1, the query set $\mathcal{D}_{\text{query}}$ is the major supervision that drives the meta-training of the few-shot learner $\mathcal{A}$.

After $\mathcal{A}$ is properly meta-trained, we then apply it to the true $\mathcal{D}_{\text{support}}^{\prime}$ of the novel classes to construct the classifier.

Let us now introduce our approach. We define each simulated FSL task $\mathcal{T}=(\mathcal{D}_{\text{support}},\mathcal{D}_{\text{query}},h^{\star})\sim\mathcal{D}_{\text{base}}$ as a tuple that contains both the query set and the target classifier beside the support set. The target classifier $h^{\star}$ is constructed similarly as described in subsubsection 3.2.2, i.e., using all the data in $\mathcal{D}_{\text{base}}$ that are from the same classes as in $\mathcal{D}_{\text{support}}$. We will provide more details in subsection 3.4. We propose a new meta-learning objective to meta-train the few-shot learner $\mathcal{A}$ with this new definition of $\mathcal{T}$,

This objective is as general as Equation 1 but is able to benefit from the target classifiers $h^{\star}$. Specifically, Equation 5 involves two loss terms on the query data. One is exactly the same as in Equation 1, comparing $\mathcal{A}(\mathcal{D}_{\text{support}})({\bm{x}})$ to $y$. The other term compares $\mathcal{A}(\mathcal{D}_{\text{support}})({\bm{x}})$ to $h^{\star}({\bm{x}})$ via a loss function $\ell_{\textsc{LastShot}}$, which can be interpreted as a relaxed version of Equation 4: to match the predictions of the two classifiers, not their parameters. This relaxed version effectively overcomes the two drawbacks of Equation 4. On the one hand, we are now able to employ a stronger target classifier (e.g., the entire neural network model) to teach the few-shot learner, since the comparison between them is in the lower-dimensional space govern by the number of ways $C$, not by the number of parameters of $h^{\star}$. On the other hand, comparing the predictions allows us to disentangle the model architectures of the few-shot learner, its generated classifier, and $h^{\star}$. In other words, the same $h^{\star}$ associated with each $\mathcal{D}_{\text{support}}$ can now be applied to train different kinds of few-shot learners, making Equation 5 a widely applicable objective for meta-learning. Indeed, wherever Equation 1 can be applied, Equation 5 should be equally applicable.

We realize the loss $\ell_{\textsc{LastShot}}$ in Equation 5 by the Kullback–Leibler (KL) divergence, inspired by [38]. That is, instead of comparing the hard one-hot predictions, we compare the soft predictions that encode the target classifier’s confidence as well as the learned semantic relationship among classes. More specifically, we take the logits $\bm{\psi}({\bm{x}})\in\mathbb{R}^{C}$ and $\bm{\psi}^{\star}({\bm{x}})\in\mathbb{R}^{C}$ produced by $h=\mathcal{A}(\mathcal{D}_{\text{support}})$ and $h^{\star}$ on a query example ${\bm{x}}$, smooth the latter with a temperature $\tau>0$ [100], and normalize them with the $\mathbf{softmax}$ function before computing the KL divergence,

We note that the KL divergence is not the only way to realize $\ell_{\textsc{LastShot}}$. One can apply other measures that can characterize the difference of predictions between two classifiers.

So far, we assume that the target classifier $h^{\star}$ is well-specified and can be easily prepared for each sampled FSL task $\mathcal{T}$. In this subsection, we investigate several ways to construct it. We note that, a target classifier $h^{\star}$ for each $\mathcal{T}$ should be a $C$-way classifier; the $C$ classes are the same as those contained in $\mathcal{D}_{\text{support}}$ and $\mathcal{D}_{\text{query}}$. That is, $h^{\star}$ would be different across the simulated FSL tasks.

Besides different ways of constructing $h^{\star}$, we also investigate how to query $h^{\star}$; i.e., how to obtain $\bm{\psi}^{\star}({\bm{x}})$ for Equation 6. The basic way is to input the query example ${\bm{x}}$ directly. Motivated by recent studies [22, 101], where strengthening the teacher or weakening the student (e.g., adding noise) could further improve the distillation or self-training performance, we explore (a) varying the size of the input image for ${\bm{x}}$ or (b) applying autoaugment [102] to ${\bm{x}}$. We note that both operations are applied only in the meta-training phase.

So far in section 3, we focus on how to meta-train the few-shot learner $\mathcal{A}$ that can output a classifier $h=\mathcal{A}(\mathcal{D}_{\text{support}})$ given the support set $\mathcal{D}_{\text{support}}$ of a few-shot task $\mathcal{T}$. The few-shot task $\mathcal{T}$ during meta-training is sampled/simulated from ${D}_{\text{base}}$, and one can flexibly define it as $\mathcal{T}=(\mathcal{D}_{\text{support}},\mathcal{D}_{\text{query}})$, $\mathcal{T}=(\mathcal{D}_{\text{support}},h^{\star})$, or $\mathcal{T}=(\mathcal{D}_{\text{support}},\mathcal{D}_{\text{query}},h^{\star})$ (cf. subsection 3.2, subsubsection 3.2.2, subsection 3.3, respectively).

During meta-testing, in which the trained few-shot learner $\mathcal{A}$ is to be evaluated on novel classes (cf. subsection 3.1), we follow the standard evaluation protocol (cf. subsection 4.2): each meta-testing task $\mathcal{T}^{\prime}$ is a tuple $(\mathcal{D}_{\text{support}}^{\prime},\mathcal{D}_{\text{query}}^{\prime})$. The accuracy on a task is defined as

where 1 is an indicator function whose value is 1 if the input is true; otherwise, 0. In other word, during meta-testing, we do not need a target classifier for each meta-testing task. We construct the target classifiers only to facilitate meta-training, and this is possible because the base class data ${D}_{\text{base}}$ contains sufficient examples.

We evaluate our approach LastShot using multiple benchmark datasets, including miniImageNet [10], tieredImageNet [26], CUB [29], CIFAR-FS [27], and FC-100 [28]. There are 100 classes in miniImageNet with 600 images per class. Following the split of [15], the meta-training/validation/testing sets contain 64/16/20 (non-overlapping) classes, respectively. In tieredImageNet [26], the class numbers in the three sets are 351/97/160, respectively, again without overlapping. For CUB, we follow the split of [23] and set 100 classes for meta-training. Two 50-class splits from the remaining classes are used for meta-validation and meta-testing. CIFAR-FS [27] and FC-100 [28] are two variants of the CIFAR-100 dataset [103] for few-shot learning but with different partitions. The classes in CIFAR-FS are randomly split into 64/16/20 for meta-training/validation/testing, while FC-100 takes the split of 60/20/20 classes following the super-classes. There are 600 images in each class in CIFAR-100. All images are resized to 84 by 84 for the first three datasets and are resized to 32 by 32 for the latter two CIFAR variants in advance. The detailed statistics are listed in Table I.

We follow the evaluation in the literature [64, 23, 19, 35], where 10,000 $C$-way $K$-shot tasks are sampled from the meta-testing set. In each task, the query set contains 15 instances in each class. Mean accuracy (in %) as well as the 95% confidence interval are reported. For every meta-training task, we also sample 15 instances per class to construct the query set, unless stated otherwise.

LastShot is a plug-and-play approach that is compatible with most of the existing meta-learning FSL methods. We apply LastShot to four representative meta-learning methods: ProtoNet [13], FEAT [23], ProtoMAML [24], and MetaOptNet [25]. The first two are embedding-based methods to learn good feature extractors; the latter two are similar to MAML that learn in a two-stage inner/outer-loop manner.

We first evaluate LastShot with 5-Way 1-Shot and 5-Way 5-Shot tasks on miniImageNet (Table II), tieredImageNet (Table III), CUB (Table IV), CIFAR-FS (Table V), and FC-100 (Table V). We highlight the results when a baseline method is combined with LastShot via a cyan background in the tables, and make the best performance in each setting in bold. We denote the two implementations of the target classifiers, i.e., the nearest centroid classifier and the logistic regression, by NC and LR, respectively.

We find that LastShot can consistently improve each baseline method on all the datasets by 1$\sim$3%. For example, with LastShot, the 1-Shot classification accuracy of FEAT improves from 77.97% to 80.20% in Table IV. By further investigating the four baseline methods, we find that MetaOptNet and FEAT are stronger than ProtoNet and ProtoMAML. In general, LastShot can lead to a larger boost for the relatively weaker baselines, but it can still improve MetaOptNet and FEAT. The target classifiers based on the nearest centroid and LR have similar performance. Overall, LastShot variants achieve the state-of-the-art performance on miniImageNet and CUB, while obtaining competitive results on tieredImageNet. These verify that a strong teacher can provide better supervision in meta-learning.

Last but not the least, comparing to Model Regression [16] and RFS [34], in which the former only uses the target classifier to guide the few-shot learner in the last fully-connected layer while the latter uses distillation to strengthen the pre-trained features, our LastShot outperforms them on miniImageNet, CUB, and CIFAR-FS, and performs on par with them on other datasets. The promising performance further demonstrates the effectiveness of our approach.

Besides the benchmark results, we further investigate the scenarios when there are more shots in the support set for the novel classes (i.e., during meta-testing). For instance, $K=50$ is much larger than the usual few-shot learning scenario (where $K$=1 or $K$=5), but is still smaller than usual many-shot datasets like ImageNet [110] for training a complex deep neural network. We consider $K=\{1,5,10,20,30,50\}$. For meta-training on the base class data, we also consider $K=\{1,5,10,20,30,50\}$. However, instead of setting the same $K$ for meta-training and meta-testing, we pick the best $K$ in meta-training for each $K$ of meta-testing, according to the accuracy on the meta-validation set. We note that, for a larger $K$ in meta-training, the query set size may be shrunk due to the memory constraint. For example, for $K=50$, the number of query examples per class becomes $2$. For meta-testing, since no gradients need to be computed for the few-shot learner $\mathcal{A}$, we can still keep the query set size intact, i.e., $15$ examples per class.

The results are shown in Table VI and Table VII: LastShot can consistently improve the corresponding meta-learning method at all the different shot settings. From the further analysis in Figure 1, we see that the strength of conventional meta-learning decreases when $K$ increases. For instance, the improvement by the meta-learned method ProtoNet over the plainly-trained method “PT-EMB” diminishes when the number of shots becomes large. PT-EMB even outperforms ProtoNet when $K>5$. (Note that, both methods use exactly the same prediction rule.) In Figure 3, we see a similar trend when comparing the best-performing meta-learning method FEAT and the best-performing non-meta-learning method SimpleShot. SimpleShot outperforms FEAT when $K=50$.

While one may argue that this is not surprising given that meta-learning is designed for small $K$, the fact that there seems to exist a turning point that one must switch from meta-learning methods to non-meta-learning ones is notoriously annoying in practice, especially that such turning point is quite unstable (it is around 5 for ProtoNet but 50 for FEAT). It is desired to design learning mechanisms that can seamlessly combine the advantages of these two methods.

We claim that LastShot is one of such mechanisms. On the one hand, LastShot is itself a meta-learning method, drawing few-shot tasks to train the few-shot learner. On the other hand, LastShot explicitly uses the non-meta-learning based methods with a larger $K$ to construct the target classifier to teach the few-shot learner in the meta-training process. The results on miniImageNet and tieredImageNet (in Table VI and Table VII, respectively) further demonstrate this. Our LastShot combined with FEAT consistently improves FEAT and outperforms SimpleShot at every $K$ values, making LastShot the best-performing methods to be applied across a wide spectrum of $K$ values.

We generate synthetic, one-dimensional few-shot regression tasks with inputs $x$ sampled uniformly from $[-5,5]$. Every few-shot task is characterized by a true sine function parameterized by $(a,v,b)$:

In this sine curve, the amplitude $a$ is sampled uniformly from $[0,2]$, the frequency $v$ is sampled uniformly from $[2,4]$, the bias $b$ is sampled uniformly from $[0,2\pi]$, and $\epsilon$ is sampled from $0.3\times\mathcal{N}(0,1)$, where $\epsilon$ is an additional noise. Given $(a,v,b)$, we can then sample the support and query sets by first sampling $x\sim[-5,5]$ and then obtaining the corresponding $y$ using Equation 11. We note that, the sine curves defined in Equation 11 have higher frequencies than those studied in [7]. Thus, it requires more support set examples to learn a regression function.

The goal is to build a regression function using the support set, such that it can predict the real-valued labels well for the corresponding query set. We randomly sample $1,000$ meta-testing tasks (i.e., $1,000$ sin curves) for evaluation. From each, we randomly sample $K$ pairs of $(x,y)$ according to Equation 11 to form the few-shot support set $\mathcal{D}^{\prime}_{\text{support}}$, and another non-overlapping 100 pairs as the query set $\mathcal{D}^{\prime}_{\text{query}}$. (In our experiments, we consider $K\in\{5,50\}$.) The performance is measured by the average Mean Square Error (MSE) and 95% confidence intervals over the $1,000$ tasks.

We investigate training a meta-learning based few-shot learner $\mathcal{A}$ to output the regressor. Specifically, we model the regression function by a three-layer fully-connected neural network $f_{\bm{\theta}}(x)$ as the feature extractor. The hidden and output dimensions are all set to $100$. We can learn such a feature extractor using nearly all the existing meta-learning methods. For example, with MAML [7], we will add a linear regressor $w$ on top. When embedding-based methods like ProtoNet [13] are used, it is equivalent to learning a non-parametric regression

where $\mathbf{Softmax}(\cdot)$ is taken over $K$ examples in $\mathcal{D}_{\text{support}}$.

We train $\mathcal{A}$ with at most $1,280,000$ few-shot tasks, each is associated with a specific $(a,v,b)$, a support set $\mathcal{D}_{\text{support}}$, and a query set $\mathcal{D}_{\text{query}}$. We pack every $32$ tasks into a mini-batch, and train $\mathcal{A}$ for $40,000$ iterations. We apply stochastic gradient descent (SGD) with a momentum of $0.9$ as the optimizer. We set the initial learning rate as $0.001$, and multiply it by $0.5$ every $160,000$ tasks. We perform early stopping using another $1,000$ validation tasks.

To apply LastShot for few-shot regression, we must (a) construct the target regressor $h^{\star}$ for each few-shot regression task, and (b) define the suitable $\ell_{\textsc{LastShot}}$ for supervision.

For the former, to efficiently construct $h^{\star}$ without sampling extra data from each few-shot task, we discretize the ranges of $a\in[0,2]$, $v\in[2,4]$, and $b\in[0,2\pi]$ each by $0.1$ to construct a set of “anchor” tasks. For each of them, we can sample many-shot examples (e.g., $1,000$ in our experiments) to construct a many-shot regressor. Here we train a neural network regressor using the regularized least square loss (i.e., the loss used in ridge regression) for each anchor task, whose hyper-parameter is tuned by cross-validation. Then during meta-training, given a few-shot task and its associated $(a,v,b)$, we retrieve the anchor task at the same grid to be the target model $h^{\star}$ — the rationale here is that sine curves with similar $(a,v,b)$ are shaped similarly.

With the target regressor $h^{\star}$, we realize $\ell_{\textsc{LastShot}}$ (cf. Equation 5) by a weighted square loss [116],

in which $\mathbf{Softmax}$ is taken over the examples in $\mathcal{D}_{\text{query}}$. The $\mathbf{Softmax}$ measures how well the target regressor $h^{\star}$ performs on a query example — we measure this because the retrieved anchor regressor does not exactly match the few-shot task at hand. The better $h^{\star}$ performs (i.e., a smaller error between $h^{\star}(x)$ and $y$), the larger the score is after $\mathbf{Softmax}$. We then use these $\mathbf{Softmax}$ scores to perform weight average over the error between the few-shot regressor $A(\mathcal{D}_{\text{support}})(x)$ and the target regressor $h^{\star}(x)$ for each query example $x$. Namely, the few-shot learner pays more attention to those query examples that $h^{\star}(x)$ truly performs well on.

We compare LastShot with ProtoNet [13], MAML [7], MetaOptNet [25], and FEAT [23] in Table XV. Specifically for MetaOptNet, we implement its top-layer classifier as ridge regression and tune the hyper-parameter. We make the following observations. First, predicting a sine curve is difficult given few-shot examples⁷⁷7One-dimensional sine functions have an infinite VC dimension., and all methods perform better when there are more shots. Second, FEAT and MetaOptNet, being more advanced meta-learning methods, outperform MAML and ProtoNet. Third and most importantly, LastShot can notably improve each of the baseline methods.

We further visualize the few-shot learners’ performance on three few-shot tasks whose true, oracle curve are $y=\sin(2x)$, $y=\sin(3x+\pi)$, and $y=2\sin(2x+\pi)$ in Figure 4. Specifically, we consider both $5$ and $50$ shots cases, and plot the oracle curve (cyan), predicted curve without LastShot (blue), and predicted curve with LastShot (red). We see that the predicted curves with LastShot can better match the oracle ones. It is worth noting that for each of these three curves, there is more than one period in the range. Therefore, it is difficult to fit the curve when there are merely five examples: there can be multiple feasible solutions. We surmise that this is why ProtoNet creates a nearly horizontal curve. By learning with a stronger teacher via LastShot, ProtoNet can generate more faithful curves in both 5-shot and 50-shot scenarios.