Class Similarity Transition: Decoupling Class Similarities and Imbalance from Generalized Few-shot Segmentation

Our idea of class similarity originates from the idea that by giving an image containing both base and novel classes, the model only trained on base classes would wrongly classify a pixel of the novel class as the similar base class. After learning $D_{train}$, given an image containing both novel and base classes, pixel $j$ of novel class $y_{n}$ will be wrongly classified as base class $y_{b}$ by the model, i.e.

Note that conditional probability $p(y_{b}|\mathbf{x}(j))$ is the wrong classification probability given by the classifier after training on $D_{train}$. Thus, with $p(y_{b}|\mathbf{x}(j))$ and $\mathbf{S}(\mathbf{x})$ obtained, our network consists of two branches, as shown in Fig. 2:

• •

  Classification Branch. The classification branch is a classifier that takes in the feature vector and outputs classification logits. The implementation of classifiers can be in any form, e.g. cosine similarity-based classifiers, Euclidean distance-based clustering, etc. In our implementation, we simply choose a linear classifier as a vanilla classification network does. Thus we have classification logits:

  $$\text{Logits}_{cl}=\text{cat}([W^{b}_{f},W^{n}_{f}])\circ\phi(\mathbf{x})$$

  (4)

  in which cat represents concatenates operation, $W^{b}_{f},W^{n}_{f}$ represents the parameter of the linear classifier of base classes and novel classes to learn during few-shot learning.

• •

  Transition Branch. The transition branch takes in the feature vector and outputs the similarity transition logits. Given similarity transition matrix $\mathbf{S}(\mathbf{x})$ and classifier weights $W^{b}_{t}$ after training on $D_{train}$. We can obtain the similarity transition logits (abbreviated as transition logits):

  $$\text{Logits}_{tr}=\mathbf{S}(\mathbf{x})\circ\text{Softmax}(W^{b}_{t}\circ\phi(\mathbf{x}))$$

  (5)

  To obtain the similarity transition matrix $\mathbf{S}(\mathbf{x})$ we adopt a Multilayer Perceptron (MLP) $\mathbf{S}(\mathbf{x})=g(\mathbf{x})$ to estimate the transition matrix. Also, to leverage the feature extraction functionality provided by the backbone, in our implementation, we take the feature vector map $\mathbf{V}$ as input rather than raw image $\mathbf{x}$ as input. Finally, the output vector of MLP is resized to the shape of the similarity transition matrix.

Natural datasets exhibit long-tailed distribution. Simply adopting cross-entropy as Eq. (6) in which $z_{k}$ is the $k$-th logits of the output of the model to ensure high confidence on predictions will eventually lead the model biased towards the dominant classes, i.e. base classes.

This is because, utilizing traditional cross-entropy, the great number of base class samples causes the models to incur a higher punishment when misclassifying base classes as novel classes, leading to larger gradients [32] to mitigate that. On the contrary, misclassifying novel classes as base classes will not result in a great punishment. Consequently, the optimized model is more inclined to predict all samples as base classes, as shown in our experiments.

A proven effective way to mitigate the disproportional punishment of base classes and novel classes is rectifying the logits before cross-entropy [17, 3], i.e., amplifying the loss on novel classes. Given the class distribution $n_{k}$ for $k$-th class, we use such class distribution to rectify the logits. Following LDAM [3], we adopt rectified cross-entropy as:

The setting of GFSS prevents the supporting images from providing explicit supervision, thus during few-shot learning when continually updating the parameters, the lack of supervision of the base class will eventually result in the model forgetting the knowledge of base classes obtained from the training phase. As shown in Fig. 3, after removing the Knowledge Distillation, DIaM [8] wrongly classifies the base class to the novel class, e.g., Agric land type 1 to Agric land type 2, Sea to River, and Building type 1 to the background.

The previous method DIaM [8] adopts Knowledge Distillation to overcome the forgetting

In our proposed method, we preserve the knowledge of base classes in a different way by extending the Transition Matrix from $|\mathcal{C}^{n}|\times(1+|\mathcal{C}^{b}|)$ to $(1+|\mathcal{C}^{b}|+|\mathcal{C}^{n}|)\times(1+|\mathcal{C}^{b}|)$, as shown in Eq. (9), where $\mathbf{S}^{\text{base2novel}}(\mathbf{x})$ is the aforementioned $\mathbf{S}(\mathbf{x})$. Each element in $\mathbf{S}^{\text{base2base}}(\mathbf{x})$ represents the transition probability from base classes to base classes. Under an extreme situation in which the diagonal elements of $\mathbf{S}^{\text{base2base}}(\mathbf{x})$ are close to 1 while others are close to 0, the learned knowledge is preserved after outer product. The detailed exemplification is shown in Sec. 4.3.

Learning a classifier from $D_{support}$ for epochs leads to the classifier overfitting the support set, as shown in Fig. 4(a). To prevent the overfitting, [2] introduces a transductive normalization term. We adapt the proposed method into our framework as

$\pi$ is agnostic during few-shot learning, so we estimate the $\pi$ by the model’s prediction, i.e.

Comparison to state-of-the-art GFSS. Tab. 1 reports the comparison of our method and current SOTA methods in GFSS. To make a fair comparison, we use ResNet-101 as the backbone for all methods compared. It can be seen that: A) Compared with other methods, our method with ResNet-101 backbone, i.e. Ours-RN, outperforms existing SOTA. And the main improvement comes from the novel classes. B) The great majority of the novel class improvement comes from River and Agric land type 2, this is because similar base class Sea helps the model classify River. Similar to Agric land type 2.

Further on, to test our method’s effectiveness on different backbones, we implement various backbones as shown in Tab. 3. The experiment result indicates that our method is equally valid for different backbones. Even for a backbone strong enough, our method can still improve the performance. We report the best result of ConvNext-L [16] as backbone which achieves 34.57 weighted mIoU. In the OpenEarthMap Land Cover Mapping Few-Shot Challenge, the result ranked No.2 till we finished the paper.

In our method, we do not leverage the knowledge distillation to perverse the knowledge learned from base classes. Instead, we achieve the preservation by extending the transition matrix with base-to-base parts. To verify our claim and show the effectiveness of base-to-base parts, we visualize the transition matrix, as shown in Fig. 5.

As Fig. 5 indicates, the diagonal elements of the base-to-base parts, $\mathbf{S}^{\text{base2base}}(\mathbf{x})$, are dominant in the corresponding columns which suggests that after the outer product with classification probabilities given by $W^{b}_{t}$, the logit on the corresponding position is much greater than others. After the softmax, the base classes have greater logits. Thus, our proposed method succeeds in preserving the knowledge from base classes.

Besides, the first column in the transition matrix represents the similarity transition probabilities of the background class in the training set to base and novel classes during the few-shot learning. It can be seen that the transition probabilities from background to novel classes stand high while background to base classes are low. This is because, during the training phase, potential novel classes are treated as background whereas base classes are not. Thus, having high transition probabilities from background to novel classes, the transition branches further help the model distinguish the classes that were previously considered as background in the training set.

As we claim optimizing the classifier on the support set may result in overfitting. We do a quantitative study to verify our claim, as shown in Fig. 4. A) Obviously, as shown in Fig. 4(a), as the optimization keeps, the classifier suffers from overfitting, i.e., the mIoU on the support set kept increasing while that of query images started decreasing as it reached the highest score at about 450 epoch. B) Compared to $k=1$ and $k=4$, $\mathcal{L}_{\pi}$ significantly delays the onset of overfitting from about 450 epochs to about 620 epochs. Moreover, the classifier achieves higher performance on query images. The performance without $\mathcal{L}_{\pi}$ is below 4, whereas with it, the performance exceeds 5. C) Compared $k=4$ to $k=1$, the classifier keeps gaining improvement on the query image and does not reach the pike yet, suggesting our method still has ample untapped potential, enabling higher performance to be achieved.

To validate our proposed method, we do various ablation studies from different perspectives.