A Simple Task-aware Contrastive Local Descriptor Selection Strategy for Few-shot Learning between inter class and intra class

In this paper, we follow the same setting as previous methods[14, 4, 30, 19]. Given a support set $S$, a query set $Q$, and an auxiliary set $A$, where the label space of the auxiliary set $A$ is disjoint from $S$ and is used to learn transferable knowledge. The support set $S$ contains $C$ classes, each with $K$ labeled samples, while the samples in the query set $Q$ are unlabeled and share the same label space as $S$. We are given a support set consisting of $n$ classes, each with $k$ samples, and a query image, and the task is to classify the query image into one of the $n$ support classes. This constitutes the n-way k-shot few-shot classification problem. Under this setting, we introduce a meta-training mechanism[25] called the episodic training mechanism. We randomly sample from the auxiliary set $A$ to construct an n-way k-shot task. Each task consists of a support set $A_{S}$ and a query set $A_{Q}$. During the training phase, we construct tens of thousands of tasks to learn transferable knowledge.

We obtain a three-dimensional feature representation $f_{\theta}(X)\in\mathbb{R}^{h\times w\times d}$ for the image $X$ through the embedding module $f_{\theta}(\cdot)$, which is considered as a set of $d$-dimensional local descriptors (LDs):

Where $l_{i}$ denotes the $i$-th deep local descriptor (LD). Similar to other descriptor-based Methods [14, 4, 16, 30, 19], we consider it as a set of $m$ $d$-dimensional descriptors, and $m=h\times w$.

In each episode, each support class has $k$ images. We denote the descriptor set from category $c$ as $\mathcal{L}_{c}^{S}$, where there are $n$ classes in total, and represent the descriptor representation for each query image as $l^{q}$. When using shallower embedding modules (e.g., Conv-4), each support category is represented in its original form. When using deeper embedding modules (e.g., ResNet-12), each support category is represented by the empirical mean of its support descriptors.

As mentioned above, $X_{S}$ represents an image in the support class, fed into the embedding module $f_{\theta}$ to obtain local descriptors $\mathcal{L}^{S}=f_{\theta}(X_{S})\in\mathbb{R}^{m\times d}$, where $m=h\times w$. Here, $l^{s}$ denotes a supporting local descriptor in $\mathcal{L}^{S}$, $\hat{\mathcal{L}}^{S}$ represents the set of remaining support descriptors in $\mathcal{L}_{S}$ excluding the current $l^{s}$, and $\bar{\mathcal{L}}^{S}$ represents the set of local descriptors from the remaining support classes. Thus, we obtain $m$ $d$-dimensional local descriptors (LDs) for an image in the support class. Under the n-way k-shot setting, there are a total of $nkm$ $d$-dimensional support LDs. Previous methods [19] only considered the average similarity between each LD and the remaining LDs within the same class as the discriminative score. However, our goal is not only to maintain discriminative relationships within the same class but also across other classes. For each $l^{s}$, we calculate its average similarity with all other LDs within the same support class, referred to as intra-class similarity, and then calculate its average similarity with LDs from the remaining support classes, referred to as inter-class similarity. We seek support LDs with high intra-class similarity and low inter-class similarity. High intra-class similarity indicates strong representational capabilities of the support LD for its corresponding class, while low inter-class similarity signifies poorer representational capabilities of the support LD for other classes. Support LDs exhibiting these characteristics suggest discriminative capabilities, potentially incorporating discriminative background features to enhance classification results. Therefore, the calculations for intra-class and inter-class similarities are as follows:

Where $\hat{\mathcal{L}}^{S}$ represents the set of remaining support descriptors in $\mathcal{L}_{S}$ excluding the current $l^{s}$ (in the case of 1-shot, it corresponds to the remaining local descriptors of the current image). $\bar{\mathcal{L}}^{S}$ denotes the set of local descriptors from the remaining support classes, $\text{SIM}_{intra}$ denotes the intra-class similarity score, and $\text{SIM}_{inter}$ denotes the inter-class similarity score. Furthermore, we normalize these two similarity scores and subsequently calculate their discriminative scores:

Where $\mathcal{D}_{intra}$ denotes the discriminative score of the local descriptor $l^{s}$ within its own class, and $\mathcal{D}_{inter}$ represents its discriminative score across classes.

Based on the above results, we can calculate the two discriminative scores $\mathcal{D}_{intra}$ and $\mathcal{D}_{inter}$ for each support descriptor using a comparative approach. Subsequently, an optimized Contrastive Discriminative Score ($\mathcal{CDS}$) can be computed:

We can observe that $\mathcal{CDS}$ aligns well with our initial idea, indicating that the current local descriptor exhibits high similarity with other local descriptors within the same class and low similarity with local descriptors from other classes, where $\sigma$ is a sigmoid function. Furthermore, in Fig. 2, based on the descending order of $\mathcal{CDS}$, we select the top $K$ support descriptors with the contrastive discriminative scores for each class, forming a discriminative support descriptor set:

The value of $K$ will be discussed in Ablation Studies 3.4. We will form a set $\mathcal{L}_{\mathcal{CDS}}$ with the support descriptors selected after screening.

Given a query image $X_{q}$ embedded as $\mathcal{L}^{Q}=f_{\theta}(X_{q})\in\mathbb{R}^{m\times d}$. $l^{q}_{i}$ denotes a query descriptor in $\mathcal{L}^{Q}$. Previous works[30, 19] employed $k$-NN to select $k$ support descriptors from each support class. However, we have observed that, after computing the discriminative support descriptor set $\mathcal{L}^{\mathcal{CDS}}$, it is not necessary to use $k$-NN for selecting $k$ support LDs from $\mathcal{L}^{\mathcal{CDS}}$. We directly compute the sum of similarities between each query descriptor $l_{i}^{q}$ and the discriminative support descriptor set $\mathcal{L}_{c}^{\mathcal{CDS}}$ for each support class $c$:

Where $c\in\{1,2,\ldots,k\}$ denotes a support class, and $l_{c}$ is one discriminative support LD from the discriminative support descriptor set of support class $c$. Similarly, the discriminative score for each query descriptor $l_{i}^{q}$ is calculated as:

Previous works [14, 16, 4, 30, 19] have employed methods that directly select query descriptors by using a fixed threshold $\mathcal{V}$ and the top-$k$ query descriptors with the highest similarity. However, both of these methods suffer from poor generalization, as they may overlook some discriminative LDs. Thus, inspired by [4, 30, 19, 8], we employ a network $\mathcal{F}_{q}$ consisting of two fully connected layers as an MLP to adaptively predict the threshold $\mathcal{V}^{l_{i}^{q}}$ for each query descriptor. Finally, we utilize the predicted threshold $\mathcal{V}$ to learn a query descriptor weights map $\mathcal{M}_{q}$. We feed the discriminative support descriptor set $\mathcal{L}_{c}^{\mathcal{CDS}}$ and the query descriptor $l_{i}^{q}$ into $\mathcal{F}_{q}$, ultimately predicting the threshold $\mathcal{V}^{l_{i}^{q}}$:

Where $i\in\{1,2,\ldots,m\}$ denotes a query LD, and $c\in\{1,2,\ldots,n\}$ denotes a support class. The final calculation for the query descriptor weights map $\mathcal{M}_{q}$ is as follows:

Where, when $\lambda$ is sufficiently large and $\mathcal{D}^{{l_{i}^{q}}}>\mathcal{V}^{l_{i}^{q}}$, the values of $\mathcal{M}_{q}$ approximates $1$. Conversely, the values of $\mathcal{M}_{q}$ approximates $0$.

Therefore, we can utilize $\mathcal{M}_{q}$ to select query LDs. The calculation for the similarity scores between each query image $X_{q}$ and each support class $c$ is as follows:

The cross-entropy loss is used to meta-train the network:

miniImageNet[25] is a subset of ImageNet [3]. It is divided into a training set with $64$ classes, a validation set with $16$ classes, and a test set with $20$ classes. Each class consists of $600$ image samples, each of size $84\times 84$ pixels.

tieredImageNet[20] is another subset of ImageNet. It comprises $608$ classes, with each class containing $1281$ images. These $608$ classes are divided into $351$ for training, $97$ for validation, and $160$ for testing.

CUB-200[26] is a fine-grained dataset that consists of $11788$ bird images, encompassing $200$ different bird species. We partition it into $100$ classes for training, $50$ classes for validation, and $50$ classes for testing. For fine-grained datasets, we resized the images in them to the same size as miniImageNet, which is $84\times 84$ pixels.

Model architecture. We use Conv-4 and ResNet-12 as feature extraction networks $f_{\theta}$, similar to previous work [14, 4, 30, 19]. Conv-4 consists of 4 convolutional blocks, each containing a convolutional layer, batch normalization layer, and Leaky ReLU layer. ResNet-12 is composed of 4 residual blocks, with each block consisting of 3 convolutional layers with $3\times 3$ kernels, 3 batch normalization layers, 3 Leaky ReLU layers, and a $2\times 2$ max-pooling layer. Conv-4 and ResNet-12 generate feature maps of size $19\times 19\times 64$ and $5\times 5\times 640$ for $84\times 84$ images, respectively. These feature maps are then mapped through a transformation layer $f_{\phi}$, which consists of a $1\times 1$ convolutional layer, a batch normalization layer, and a LeakyReLU layer. Finally, $\mathcal{F}_{q}$ is implemented with two fully connected layers.

Training and evaluation details. During the meta-training phase, we followed the settings in [16, 30, 19]. For Conv-4, we set the learning rate to $1e-3$ and decay $0.1$ every $10$ epoch, training for a total of 30 epochs using the Adam optimizer. For ResNet-12, we pre-trained it first and then conducted meta-training for 40 epochs using momentum SGD with an initial learning rate of $5e-4$ and decay $0.1$ every $10$ epochs. During the test, as in [16, 30, 19], we randomly constructed $10000$ episodes from the test set to calculate the classification accuracy. This process was repeated five times, and we reported the average accuracy along with a $95\%$ confidence interval.

We choose $7$ generic few-shot learning state-of-the-art baselines[25, 23, 24, 28, 27, 12], as well as $5$ SOTA baselines based on LDs[31, 14, 16, 4, 30]. For fine-grained datasets, we also selected $9$ state-of-the-art baselines[23, 14, 22, 12, 27, 31, 28, 16, 30].

Results on miniImageNet dataset. As shown in Table 1, the performance of our method in the 5-way 1-shot and 5-shot settings exceeds that of all current LDs-based methods [31, 14, 16, 4, 30]. Compared to the baseline DN4, our method exhibits significant improvement. In the 5-way 1-shot and 5-shot settings, using Conv-4 as the backbone, it achieved improvements of $5.90\%$ and $4.87\%$, respectively. Compared to the state-of-the-art (SOTA), our method also improved by $1\%$ and $1.21\%$, respectively. When using ResNet-12 as the backbone, improvements of $3.18\%$ and $4.02\%$ were achieved, surpassing SOTA by $1.27\%$ and $0.89\%$, respectively.

Results on tieredImageNet dataset. As shown in Table 1, our method outperforms the current state-of-the-art LDs-based methods as well. In the 5-way 1-shot and 5-shot settings, when using Conv-4 as the backbone, our method improves by $0.79\%$ and $0.08\%$, respectively, compared to the state-of-the-art method based on LDs. When using ResNet-12 as the backbone, our method improves by $1.14\%$ and $0.89\%$, respectively, compared to the state-of-the-art method based on LDs.

Results on fine-grained CUB-200 dataset. As shown in Table 2, our method also achieves state-of-the-art performance on fine-grained datasets. In the 5-way 1-shot and 5-shot settings, when using Conv-4 as the backbone, our method improves by $0.26\%$ and $1.68\%$, respectively, compared to the state-of-the-art method based on LDs. When using ResNet-12 as the backbone, our method improves by $1.02\%$ and $1.19\%$, respectively, compared to the state-of-the-art method based on LDs.

Influence of Top $K$ in support LDs selection. In subsection 2.2, we selected $K$ ($K$ as a percentage) LDs based on $\mathcal{CDS}$ for each support class to form a discriminative LDs set. As shown in Table 3, we conducted experiments on the miniImagenet and CUB-200 datasets under the 5-way 5-shot setting. When using Conv-4 as the backbone, we set $K$ to $1\%,2\%,5\%,10\%,30\%$ respectively. When using ResNet-12 as the backbone, we set $K$ to $3\%,5\%,10\%,25\%,30\%$ respectively. Through experiments, we found that when using Conv-4 as the backbone, the performance is best when $K=2\%$ and $K=10\%$ on both datasets. When using ResNet-12 as the backbone, the performance is best when $K=2\%$ and $K=25\%$ on both datasets. The experimental results indicate that compared to general datasets, fine-grained datasets require more discriminative LDs. Similarly, under the 5-way 1-shot setting, the best performance is achieved with Conv-4 as the backbone when $K=5\%$ and $K=10\%$, and with ResNet-12 as the backbone, the best performance is achieved when $K=5\%$ and $K=10\%$.

Comparison with methods using backbones with different parameters. As shown in Table 4, we selected three baselines[13, 17, 15] using ResNet-18 as the backbone, three baselines[12, 32, 18] using WRN-28-10 as the backbone, and baselines[7] using ViT-Small and Swin-Tiny as the backbone. These methods are not LDs-based baselines. Compared to the baselines using ResNet-18 as the backbone, our method outperforms the best-performing method by $4.41\%$ and $2.86\%$ in the 1-shot and 5-shot settings, respectively. When compared to baselines using WRN-28-10 as the backbone[12, 32, 18], our method achieves a $0.70\%$ improvement in the 1-shot setting and is only $0.17\%$ lower than OM[18] in the 5-shot setting, despite WRN-28-10 having three times the parameters of ResNet-12. Compared to FewTURE[7] with ViT-Small as the backbone, our method achieves improvements of $0.51\%$ and $0.61\%$, and is only $1.26\%$ lower than FewTURE with Swin-Tiny as the backbone in the 5-shot setting. However, Swin-Tiny has 2.3 times the parameters of ResNet-12. Additionally, FewTURE’s ViT-Small and Swin-Tiny were trained using 4 and 8 Nvidia A100 40GB GPUs, respectively, making their GPU requirements relatively high.