Few-shot Image Classification: Just Use a Library of Pre-trained Feature Extractors and a Simple Classifier

We begin by asking: what if we eschew advanced meta-learning methods, and instead simply use a very high-quality deep CNN pulled from a library, trained on the ILSVRC2012 data set, as the basis for a few-shot classifier?

Specifically, we are given a high-quality, pre-trained deep CNN, from a library of pre-trained networks; we take the CNN as-is, but remove the topmost layers used for classification. This results in a function that takes an image, and returns an embedding. We then use that embedding to build a classifier in an elementary fashion: we feed the embedding into a multi-layer perceptron with a single hidden layer; a softmax is used to produce the final classification. Given a few-shot classification problem, two weight matrices $\textbf{W}_{1}$ and $\textbf{W}_{2}$ are learned; the first connecting the embedding to the hidden layer, the second connecting the hidden layer to the softmax. To prevent over-fitting during training, simple $L_{2}$ regularization is used on the weight matrices.

To evaluate this very simple few-shot learner, we first identify nine, high-quality deep CNNs with published models, trained on ILSVRC2012: ResNet18, ResNet34, ResNet50, ResNet101, ResNet152 (all of the ResNet implementations are the ones from the original ResNet designers (He et al., 2016)), DenseNet121, DenseNet161, DenseNet169, and DenseNet201 (all of the DenseNet implementations are also from the original designers (Huang et al., 2017)).

Our goal is to produce an “out-of-the-box” few-shot learner that can be used on any (very small) training set $\mathcal{D}_{\mathit{few}}$ without additional data or knowledge of the underlying distribution. We are very careful not to allow validation or parameter tuning on testing data domains, so all parameters and settings need to be chosen apriori. If it is possible to build such a few-shot learner, it would be the most widely applicable: simply produce a few training images for each class, apply the learner. Thus, we perform an extensive hyper-parameter search, solely using the Caltech-UCSD Birds 200 set (Welinder et al., 2010) as a validation data set, and then use the best hyperparameters from that data set in all of our experiments. Hyperparameters considered were learning rate, number of training epochs, regularization penalty weight, the number of neurons in the MLP hidden layer, and whether to drop the hidden layer altogether. A separate hyper-parameter search was used for 5-way, 20-way, and 40-way classification.

We then test the resulting few-shot learner—one learner per deep CNN—on eight different data sets, FGVC-Aircraft (Maji et al., 2013), FC100 (Oreshkin et al., 2018), Omniglot (Lake et al., 2015), Traffic Sign (Houben et al., 2013), FGCVx Fungi (Schroeder & Cui, 2018), Quick Draw (Jongejan et al., 2016), and VGG Flower (Nilsback & Zisserman, 2008). To evaluate a few-shot learner on a data set, for an “$m$-way $n$-shot” classification problem, we randomly select $m$ different classes, and then randomly select $n$ images from each class for training; the remainder are used for testing. As this is “out-of-the-box” learning, no validation is allowed.

We performed this evaluation for $m$ in $\{5,20,40\}$ and $n$ in $\{1,5\}$. Due to space constraints, the full results are presented as supplementary material, and we give only a synopsis here (Table 2 and Table 3). In Table 1, we show the best and worst accuracy achieved across the 9 learners, for each of the 8 data sets, for $m$ in $\{5,20,40\}$ and $n=1$.

To give the reader an idea of how this accuracy compares to the state-of-the-art, we compare these results with a number of few-shot learners from the literature. We compare against Baseline and Baseline++ (Chen et al., 2019), MAML (Finn et al., 2017), MatchingNet (Vinyals et al., 2016), ProtoNet (Snell et al., 2017), RelationNet (Sung et al., 2018), Meta-transfer (Sun et al., 2019), FEAT (Ye et al., 2020), and SUR (Dvornik et al., 2020). When a deep CNN classifier must be chosen for any of these methods, we use a ResNet18. For methods that require a pre-trained CNN (FEAT, Meta-transfer, and SUR), we use the ResNet18 trained by the ResNet designers (He et al., 2016). Lest the reader be concerned that we chose the worst option (ResNet18), we point out that of the library-based few-shot learners, on the 5-way, 5-shot problem ResNet18 gave the best accuracy out of all of the ResNet-based learners for two of the data sets (see Table 4). Further, these competitive methods tend to be quite expensive to train—MAML, for example, requires running gradient descent over a gradient descent—and for such a method, the shallower ResNet18 is a much more reasonable choice than the deeper models (even using a ResNet18, we could not successfully train first-order MAML for 5-shot, 40-way classification, due to memory constraints).

For the competitive methods (other than SUR) we follow the same procedure as was used for the library-based few-shot classifiers: any training that is necessary is performed on the ILSVRC2012 data set, and hyperparameter validation is performed using the Caltech-UCSD Birds 200 data set. Each method is then used without further tuning on the remaining eight data sets. To evaluate SUR on data set $X$, we use feature extractors trained on the data sets in {Omniglot, Aircraft, Birds, Texture, Quickdraw, Flowers, Fungi,and ILSVRC} $-$ $X$. Traffic Sign and FC100 datasets are reserved for testing only.

A comparison of each of these competitive methods with the the best and worst-performing library-based learners on the 5-way, 5-shot learning problem is shown in Table 2; a comparison on 20-way, 5-shot learning in Table 3. A more complete set of results is in the supplementary material.

There are a few key takeaways from these results. The best library-based learner always beat every one of the other methods tested, with the only exception of SUR when testing on Omniglot data set. For the other data sets, the gap only grows as the number of ways increases. In fact, for the 20-way problem, the worst library-based learner always beat all of the other methods tested (except SUR on Omniglot). The gap can be quite dramatic, especially on the 20-way problem. The best non-transfer based few-shot learner (MAML, Proto Nets, Relation Nets, and Matching Nets fall in this category) was far worse than even the worst library-based learner: 39% accuracy for Proto Nets vs. 59% accuracy for a classifier based on RestNet18 on Texture, 48% accuracy for Relation Nets vs. 57% accuracy for classifier based on ResNet101 on Quick Draw. There have been a large number of non-transfer-based methods proposed in the literature (with a lot of focus on improving MAML in particular (Finn et al., 2017, 2018; Kim et al., 2018; Nichol & Schulman, 2018; Lee & Choi, 2018)) but the gap between MAML and the library-based classifiers is very large.

We also note that of the rest non-library methods, Meta-transfer, Baseline, and FEAT were generally the best. We note that Meta-transfer, Baseline, and FEAT use the pre-trained ResNet18 without modification. This tends to support the hypothesis at the heart of this paper: starting with a state-of-the-art feature extractor, trained by experts, may be the most important decision in few-shot learning.

There is not a clear pattern to which of the library-based classifiers tends to perform best, and which tends to perform worst. Consider the complete set of results, over all of the library-based few-shot learners, for the 5-way, 5-shot problem, shown in Table 4. For “out-of-the-box” use, where no validation data are available, it is very difficult to see any sort of pattern that might help with picking a particular library-based classifier. The DenseNet variations sometimes do better than ResNet (on the Aircraft data set, for example), but sometimes they do worse than the ResNet variations (on the FC100 data set, for example). And within a family, it is unclear which library-based CNN to use. As mentioned before, ResNet18 provides the best ResNet-based few-shot learner for two of the data sets, but it forms the basis of the worst ResNet-based learner on another two.

Given the relatively high variance in accuracies obtained using the library deep CNNs across various data sets, it is natural to ask: Is it perhaps possible to use all of these library feature extractors in concert with one another, to develop a few-shot learner which consistently does as well as (or even better then) the best library CNN?

Given the lack of training data in few-shot learning, the first idea that one might consider is some simple variant of ensembling: given a few-shot learning problem, simply train a separate neural network on top of each of the deep CNNs, and then use a majority vote at classification time (hard ensembling) or we can average the class weights at classification time (soft ensembling).

Another option is to take all of the library deep CNNs together, and view them as a single feature extractor. Using the nine models considered thus far in this paper in this way results in 13,984 features. We then train an MLP on top of this, using $L_{2}$ regularization. Again using the Caltech-UCSD Birds 200 data set for validation, we perform hyperparameter search to build a few-shot learner for $5$-way, $20$-way, and $40$-way classification problems.

We test both of these options over the eight test sets, and give a synopsis of the results in Table 5. We find that across all 24 test cases (eight data sets, three classification tasks), the best single learner was never able to beat the best method that used all nine deep CNNs. All of the tested methods had similar performance on the 5-way classification task, though the best single learner was generally a bit worse than the other methods.

Where the difference becomes more obvious is on the classification tasks with more categories. For the $20$-way and $40$-way problems, the two ensemble-based methods had a small but consistent drop in accuracy, and building a single network on top of all nine deep CNNs is clearly the best. This may be somewhat surprising; for the 40-way problem, hyperparameter search on the Caltech-UCSD Birds 200 data set settled on a single network with 1024 neurons in a hidden layer; this means that more than 14 million parameters must be learned over just 200 images. Clearly, the neural network is massively over-parameterized, and yet the accuracies obtained are remarkable, with over 90% accuracy on two of the data sets.

We turn our attention to asking: why does using a full library of pre-trained feature extractors seem to work so well? One reason is that fine-tuning appears to be very easy with a library of pre-trained features. Consider the following simple experiment, designed to test whether the number of training points has much effect on the learned model.

We choose a large number of 40-way, problems, over all eight of our benchmark data sets, and train two classifiers for each. Both classifiers use all of the 13,984 features provided by the nine library feature extractors. However, the first classifier is trained as a one-shot classifier, and the second classifier is trained using all of the available data in the data set. Our goal is to see whether the sets of learned weights have a strong correspondence across the two learners; if they do, it is evidence that the number of training points has a relatively small effect. Note that in a neural network with a hidden layer consisting of hundreds or thousands of neurons, there are likely to be a large number of learned weights that are of equal quality; in fact, simply permuting the neurons in the hidden layer results in a model that is functionally identical, but that has very different weight matrices. Thus, we do not use a hidden layer in either classifier, and instead use a softmax directly on top of an output layer that linearly combines the input features. Both the one-shot and full-data classifiers were learned without regularization.

Over each of the eight benchmark data sets, for each feature, we take the $L_{1}$ norm of all of the 40 weights associated with the feature; this serves as an indication of the importance of the feature. We compute the Pearson correlation coefficient for each of the 13,984 norms obtained using the models learned for the 1-shot and full data classifiers. These correlations are shown in Table 7.

What we see is a remarkably high correlation between the two sets of learned weights; above 80% correlation in every case, except for the Traffic Sign data set. However, even in the case of the Traffic Sign data set, there was a weak correlation (the Traffic Sign data is a bit of an outlier in other ways as well, as we will see subsequently).

This would seem to indicate that there is a strong signal with only one data point per class, as the learned weights do not differ much when they are learned using the whole data set. This may be one explanation for the surprising accuracy of the full library few-shot learner. Of course, the strong correlation may gloss over significant differences, and more data does make a significant difference (consider the difference between the one-shot accuracies in Table 1 and the five-shot accuracies in Table 2.). But even one image per class seems to give a lot of information.

Another reason for the accuracy obtained by the full library method may be that different problem domains seem to utilize different sets of features, and different feature extractors. Having a large library ensures that some features relevant to any task are always present.

To investigate this, we construct a large number of 40-way training tasks over each of the various data sets, and for each, we learn a network without a hidden layer on top of all 13,984 features obtained using the library of deep CNNs. Again, we compute the $L_{1}$ norm of the set of weights associated with each of the features. This time, however, for each problem we then consider the features whose norms are in the top 20%; these could be considered the features most important to solving the classification task.

For each of the (data set, data set) pairs, we compute the average Jaccard similarity of these top-feature sets. Since each set consists of 20% of the features, if each set of features chosen was completely random, for $n$ features in all, we would expect a Jaccard similarity of $\frac{0.04n}{.2n+.2n-0.04n}$ $=$ $\frac{0.04}{.4-0.04}$ $=$ $0.111$. Anything greater indicates the sets of features selected are positively correlated; lower indicates a negative correlation. Results are in Figure 1. For each of the nine CNNs, we also compute the fraction of each CNN’s features that are in the top 20% when a full library classifier is constructed. These percentages are in Figure 2.

The data in these two plots, along with the previous results, tells a consistent story: there appears to be little correspondence between data sets in terms of the set of features that are chosen as important across data sets. The largest Jaccard value in Figure 1 is less than 0.5 (observed between FC100 and Texture). This shows, in our opinion, a relatively weak correspondence. The Traffic Sign data set, had an average Jaccard of $0.108$ across the other eight data sets, which is even lower than the $0.111$ that would be expected under a purely random feature selection regime. One might speculate that the lack of correspondence across data sets is evidence for the hypothesis that different tasks tend to use different features, which would explain why it is so effective to use an entire library of deep CNNs for few-shot learning.

Also note that in Figure 2, we tend to see that different deep CNNs contribute “important” features at very different rates, depending on the particular few-shot classification task. This also seems to be evidence that diversity is important. In general, the DenseNets’ features are preferred over the ResNets’ features, but this is not universal, and there is a lot of variation. It may not be an accident that the three data sets where the selection of features from library CNNs shows the most diversity in Figure 2 (Traffic Sign, Quick Draw, and Omniglot) are the three data sets where the classifier built on top of all nine library CNNs has the largest advantage compared to the few-shot classifiers built on top of the single, “Big Transfer”-based deep CNN.

For all the datasets, we resize each image into 256 x 256 then crop 224 $\times$ 224 from the center (except quick draw which is already aligned to the center of the drawing’s bounding box, so we directly resize quick draw images to 224 $\times$ 224).

In this work, for all the methods the training process is performed solely on ILSVRC dataset. For our library based methods, this is followed by hyperparameter validation on the CUB birds dataset. After that, each method is tested on the remaining eight datasets without further tuning.

To be more specific, for the library based methods we only use the pre-trained (on ILSVRC dataset) models. While for the meta-learning based methods, we randomly split ILSVRC into a base (training) set of 900 classes for meta-training and a validation set of the remaining 100 classes.

In order to evaluate a few-shot learner on a data set, for an “$n$-way $m$-shot” classification problem, we randomly select $n$ different classes, and then randomly select $m$ images from each class for training (equivalent to ‘support’ images in meta-learning literature). We then randomly select $k$ images for rest of the images from each class for testing (equivalent to ‘query’ images in meta-learning literature). We perform this evaluation for $n$ in $\{5,20,40\}$ and $m$ in $\{1,5\}$.

For the library based methods, the query size $k$ is set to 15 (except FGCVx Fungi dataset). For the meta-learning based methods, due to GPU memory constraints, for each class in a task we used 15 query images for 5-way, 10 query

For the FC100 data set, there is a small portion overlaps with ILSVRC2012 data set but we still think that the FC100 as a testing set is interesting, as (1) all of the few-shot learners benefitted from the overlapping classes, and (2) this shows how the methods work in the case that the test classes are close to something that the few-shot learner has seen before.