Few-shot Classification via Ensemble Learning with Multi-Order Statistics

According to how the base set is used, FSC methods can be roughly categorized into two groups, meta-learning-based Zhang et al. (2020a) and transfer-learning-based Chen et al. (2019); Liu et al. (2020). Meta-learning creates a set of episodes to simulate the real FSC test scenarios and simultaneously accumulate meta-knowledge for fast adaptation. Typical meta-knowledge includes optimization factors such as initialization parameters Finn et al. (2017) and task-agnostic comparing ingredients of feature embedding and metric Snell et al. (2017); Wertheimer et al. (2021). Recent literature on transfer learning Tian et al. (2020); Chen et al. (2019) questioned the efficiency of the episodic training in meta-learning, and alternatively used all base samples to learn an off-the-shelf feature extractor and rebuilt a classifier for novel classes. Feature representations play an important role in this regime Tian et al. (2020). To this end, regularization techniques such as negative-margin softmax loss and manifold mixup Liu et al. (2020); Mangla et al. (2020) have been adopted to enhance the generalization ability of cross-entropy loss. Moreover, self-supervised Zhang et al. (2020b); Rajasegaran et al. (2020) and self-distillation Ma et al. (2019); Zhou et al. (2021) methods have also shown promising performance in transfer-learning. To this end, supervised learning tasks can be assisted by several self-supervised proxy tasks such as rotation prediction and instance discrimination Zhang et al. (2020b), or by adding an auxiliary task of generating features during the pre-training Xu et al. (2021b). When knowledge distillation is adopted, a high-quality backbone network can be evolved through multiple generations by a born-again strategy Rajasegaran et al. (2020). All these methods suggest the importance of obtaining generalization representations, and we will leverage ensemble learning to achieve this goal.

Ensemble learning builds several different individual learners based on the same training data and then combines them to improve the generalization ability of the learning system over any single learner. This learning scheme has shown promising performance on traditional classification tasks with deep learning on large-scale labeled datasets. Recently, ensemble learning for FSC methods has been presented. For example, Dvornik et al. (2019) combined an ensemble of prototypical networks through deep mutual learning under a meta-learning setting. Bendou et al. (2022) reduced the capacity of each backbone in the ensemble and pre-trained them one by one with the same routine. However, the size of the ensemble learner increased for inference in the former work, while the latter required extra time to pre-train many learning individuals. Therefore, it still lacks efficient designs for learning individuals in FSC ensemble learning. Moreover, these works did not involve any theoretical analysis of the underlying mechanism of ensemble learning in FSC. In this paper, we investigate why ensemble learning works well in FSC under the transfer-learning setting. Based on the analysis, we propose an efficient learning method using a shared backbone network with multiple branches to generate learning individuals.

Convolutional neural network models progressively learn high-level features through multiple convolution layers. A pooling layer is often added at the end of the network to output the final feature representation. To this end, Global Average Pooling (GAP) is the most popular option, however, it cannot fully exploit the merits of convolutional features because it only calculates the $1^{st}$-order feature statistics. Global Covariance Pooling (GCP) such as Deep$\rm O^{2}$P explores the $2^{nd}$-order statistic by normalizing the covariance matrix of the convolutional features, which has achieved impressive performance gains over the classical GAP in various computer vision tasks. Further research shows that using richer statistics may lead to further possible improvement. For example, Kernel Pooling Cui et al. (2017) generates high-order feature representations in a compact form. However, a certain order statistic can only describe partial characteristics of the feature vector from the view of the characteristic function of random variables. For example, the first- and second-order statistics can completely represent their statistical characteristic only for the Gaussian distribution. Therefore, higher-order statistics are still needed for the non-Gaussian distributions, which are more ubiquitous in many real-world applications. This motivates us to calculate multi-order statistics to retain more information on features.

Under the standard setting of few-shot classification, three sets of data with disjoint labels are available, i.e., the base set $S_{b}$, the validation set $S_{val}$ and the novel set $S_{n}$. In the context of transfer-learning, $S_{b}$ is used for pre-training a model to well classify the novel classes in $S_{n}$, with the hyper-parameters tuned on $S_{val}$. Let $S_{b}=\{(x_{i},y_{i})\}_{i=1}^{N_{b}}$ denotes the source domain with ${N_{b}}$ labelled samples and $S_{n}$ denotes the target domain labelled with $K$ samples in each episode, where $N_{b}>>K$. Let the label function of $S_{b}$ and $S_{n}$ be $f_{b}$ and $f_{n}$, respectively. During the pre-training, a learner $h$ is obtained to approximate the optimal mapping function $h^{*}$ based on all ${N_{b}}$ training samples in $S_{b}$ from all possible hypotheses $\mathcal{H}$. When ensemble learning is introduced into the pre-training, several learners denoted as $\{h_{o}\}_{o=1}^{O}$ can be obtained. With the ensemble technique of weighted averaging, the final learner $\overline{h}$ is produced as:

where $\alpha_{o}$ is the weight parameter. There is a domain shift between the base and novel classes Tseng et al. (2020), and we use the $L_{1}$ distance Kifer et al. (2004) to measure the domain divergence between $S_{b}$ and $S_{n}$:

where $\eta_{b}(x)$ and $\eta_{n}(x)$ is the density functions of $S_{b}$ and $S_{n}$ respectively.

The proof is provided in the Supplementary Material.

miniImageNet contains 600 images over 100 classes, which are divided into 64, 16 and 20 respectively for base, validation and novel sets. tiredImageNet consists of 779, 165 images belonging to 608 classes, which are further grouped into 34 higher-level categories with 10 to 30 classes per category. These categories are partitioned into 20 categories (351 classes), 6 categories (97 classes) and 8 categories (160 classes) respectively for base, validation and novel sets. CIFAR-FS is derived from CIFAR100 and consists of 100 classes with 600 images per class. The total classes are split into 64, 16 and 20 for base, validation and novel sets. Caltech-UCSD Bird-200-2011(CUB) has a total number of 11,788 images over 200 bird species. These species are divided into 100, 50, and 50 for the base, validation and novel sets, respectively.

In the experiments, we primarily used ResNet12 architecture with 4 residual blocks. Each block had 3 convolutional layers with 3×3 kernels. The number of kernels for the 4 blocks was 64, 160, 320, and 640, respectively. A max-pooling layer was added at the end of the first three blocks. The last block was branched with three pooling layers, which respectively modeled different statistical representations of the images. We opted for the SGD optimizer with a momentum of 0.9 and a weight decay of 5e-4. The learning rate was initialized to be 0.025. We trained the network for 130 epochs with a batch size of 32 in all the experiments. For miniImageNet, tiredImageNet and CIFAR-FS, the learning rate was reduced by a factor of 0.2 at the 70-$th$ and 100-$th$ epoch. For CUB, the learning rate was reduced by a factor of 0.2 for every 15 epochs after the 75-$th$ epoch. We randomly sampled 2,000 episodes from $S_{n}$ with 15 query samples per class for both 5-way 1-shot and 5-shot evaluations, to produce the mean classification accuracy as well as the 95% confidence interval.

The effectiveness of our method is attributed to the ensemble of different branches equipped with multi-order statistics. In this section, we conducted ablation studies to analyze the effect of the $1^{st}$-order, $2^{nd}$-order and, $3^{rd}$-order statistical pooling and their combination on the miniImageNet, CIFAR-FS and CUB datasets. Above methods are respectively denoted as B_1, B_2, B_3,and ELMOS. Their accuracies under 5-way 1-shot and 5-shot tasks on three datasets are shown in Table 1. From the results, we can see that: (1) On all three datasets, the test accuracy of $B_{1}$ and $B_{3}$ is higher than $B_{2}$ under the 1-shot task, but the test accuracy of $B_{2}$ is higher than $B_{1}$ and $B_{3}$ under the 5-shot task. The above phenomenon shows that different order statistics provide different information about the images. (2) The test accuracy of ELMOS is higher than $B_{1}$, $B_{2}$ and $B_{3}$ under both 1-shot and 5-shot tasks, which illustrates that different order statistics complement each other. Combing them can bring more useful information for classification, resulting in higher classification performance.

For each individual in the ensemble learning, the optimization is cooperatively accomplished by the Classification-Based (CB) loss and Similarity-Based (SB) loss Scott et al. (2021). Hence, we conducted ablation experiments to analyze the contribution of each loss on three benchmark datasets: miniImageNet, CIFAR-FS and CUB. Subsequently, we pre-trained the model respectively with CB and SB loss alone and their combination, resulting in three methods denoted as CB, SB and CB&SB. The test accuracies under different methods are shown in Figure 3. The test results show that the accuracy of CB&SB is higher than CB and SB, which implies that both classification-based and similarity-based losses play important roles in our method.

Our method is most related to EASY Bendou et al. (2022), which is also a FSC ensemble learning method in context of transfer learning. The comparison of results between them on CIFAR-FS and CUB datasets is shown in Table 4. From the results, we can see that our method beats EASY by a very large margin under both 1-shot and 5-shot tasks. Please note that our method is more efficient that EASY, because EASY needs to pre-train multiple individual networks, which spends much more pre-training time than our method.

We compare the performance of our method with several state-of-the-art methods. These methods are either meta-learning based or transfer-learning based. The comparison of results is shown in Table 2 and Table 3. From Table 2, we can see the performance of our method ranks at the top under both 1-shot and 5-shot tasks on CUB. Specifically, our method exceeds the second-best model DeepBDC by 1.2% and 1.0% respectively in 1-shot and 5-shot settings. From Table 3, we can see that our method beats state-of-the-art methods under both 5-way 1-shot and 5-way 5-shot tasks on the dataset of miniImageNet, tiredImagegNet, and CIFAR-FS. Specifically, on miniImageNet, PAL and IE behave the second best respectively in 1-shot and 5-shot settings. Our method beats them by 0.93% and 1.39%. On tiredImageNet, our method outperforms the second-best MFS by 0.21% and 0.39% respectively in 1-shot and 5-shot settings. On CIFAR-FS, our method achieves 0.31% and 0.13% improvement over IE for 1-shot and 5-shot respectively. In brief, our method consistently outperforms the state-of-the-art FSC methods under both 1-shot and 5-shot tasks on multiple datasets. The promising results are achieved because of the generalization representation obtained by ensemble learning with multi-order on the base set.