Self-Supervised Side-Tuning for Few-Shot Segmentation

While matching unseen objects across modalities appears effortless for human vision, it is extremely challenging for computer vision. When new categories are added or replaced in tasks such as classification, detection and segmentation, it takes not only time to collect and label a large amount of data, but also time to retrain the corresponding model. So it is very necessary to study how to design the model so that it can quickly learn the corresponding features from few number of samples, which is the few-shot learning problem. This is especially significant for segmentation tasks, because it takes much more time to finely assign pixel level labels to a single image than bounding box anotations or image-level tags. In this case, few-shot segmentation method can effectively reduce the overall cost and improve the flexibility of segmentation models.

Few-shot segmentation task aims at minimize generalization error across different episodes in which few samples and corresponding labels (we call it support setting) of an unseen class are provided to segment other images (we call it query images) of the same class. Existing methods generally follow the steps below. First, the query and support images are respectively projected into the same feature space, and then the support mask information is applied to select the features of the corresponding category in the support setting. Guided by the learnable similarity metric function, the approximate spatial position of the corresponding object in the query image is located and finally refined by the decoder.

However, we find that the existing methods are not good enough even to segment the support image itself under the guidance of corresponding support setting in many cases, which is the simplest special case in the few-shot segmentation task. The representative texture or spatial features extracted from the support image and mask are not discriminative due to the previously unseen and sparse number of categories in each episode. In the absence of such valid semantic information, it is difficult to identify the support image itself, let alone objects of the same category in other scenarios (query images).

To make up for this, we applied a self-supervised side-tuning framework for few-shot segmentation to provide personalized prior parameters for each new task. Specifically, before query-support similarity comparision in each episode, we use the support setting to construct an online self-supervised segmentation task, in which the initial support feature is used as the inputs to both branches. The resulting loss gradient provides information about how to segment itself better and dynamically adjust the spatial texture characteristics of the corresponding class, thus guiding the query image to locate the spatial position with more sufficient semantics. Moreover, the self-supervised module we designed is exactly the subtask of the few-shot segmentation mentioned above. While using the self-supervised module to guide the segmentation, we also directly add the resulting loss to the final main segmentation loss to promote the update of whole model. This subtask loss is equivalent to a data enhancement to few samples and then constructing more training content with the same number of training iterations. Our experimental results on both Pascal-5i and COCO datasets achieve the state of the art. Furthermore, the visual analysis of SSM modules and subtask loss also proves their superiority.

Our main contributions are as follows:

1.We design a self-supervised side-tuning suitable for few-shot segmentation to help the model dynamically adjust its inductive bias in different episodes.

2.The self-segmentation subtask is introduced into the training process to assist the network to learn better feature representation and similarity function.

3.We achieved the state of the art in both 1-shot and 5-shot few-shot segmentation task.

Few-shot learning In the computer vision community, one-shot learning has recently received a lot of attention and substantial progress has been made based on deep learning. Generally, these methods can be devided into three categories. Metric-based method aims at learning the best feature representation for new categories with few samples and finding a suitable comparision method such as cosine or Euclidean distance. Unlike the above methods that focus on the projection and metric of two branches, optimization-based method optimizes network weights quickly based on the backpropagation of gradient. The model-agnostic meta-learning (MAML) and Reptile algorithm are proposed to obtain the best global initial seeds such that a small number of gradient updates will lead to fast learning on a new task. Model-based method mainly utilizes the internal architecture of the network (such as memory module, meta-learner, etc.) to realize the rapid parameter update in new categories. In the latest study, A, B introduce machine learning methods such as SVM and ridge regression into the inner loop of the base learner, and C directly replace the inner loop with neural network structure in an encoded-decode form. This method is widely influenced by meta-learning and has achieved the state-of-art performance in one-shot classification task at present. Our model is inspired by the ideas of model-based methods.

Semantic Segmentation Semantic segmentation is an important task in computer vision and FCNs have greatly promoted the development of the field. Among them, DeepLabV3 and PSPNet propose different global contextual modules, which pay more attention to the scale change and global information in the segmentation process. A and B consider the full-image dependencies from all pixels based on Non-local Networks, which show superior performance in terms of reasoning. We draw ideas from these methods and apply them to one-shot segmentation problem.

Few-shot Semantic Segmentation While the work on one-shot learning is quite extensive, the research on one-shot segmentation has been established only recently, including one-shot image segmentation and one-shot video segmentation. shaban2017one first proposes the definition and task of one-shot segmentation. Following this, A solves the problem with sparse pixelwise annotations, and then extends their method to interactive image segmentation and video object segmentation. B generalizes the few-shot semantic segmentation problem from 1-way (class) to N-way (classes). C introduces an attention mechanism to effectively fuse information from multiple support examples and propose an iterative optimization module to refine the predicted results. D process both support and query images within one network with similarity guidance and adaptive masked proxies respectively. E and F leverage the annotations of the support images as supervision in different ways. Different from them, we pay more attention to the contextual dependencies reasoning between the query and support images.

We define a input triple $Tr_{i}=(Q_{i},S_{i},T_{S}^{i})$, a label $T_{Q}^{i}$ and a relation function $F:A_{i}=F(Q_{i},S_{i},T_{R}^{i};\theta)$, where $Q_{i}$ and $S_{i}$ are the query and support images containing objects of the same class i correspondingly, $T_{S}^{i}$ and $T_{Q}^{i}$ are the pixel-wise labels corresponding to the $i^{th}$ class objects in $S_{i}$ and $Q_{i}$, $A_{i}^{j}$ is the actual segmentation result, and $\theta$ is all parameters to be optimized in function F. Our task is to randomly sample triples from the dataset, train and optimize $\theta$, thus minimizing the loss function $L$:

We expect that the relationship function $F$ can segment the same class object region in another target image each time it sees a reference image of a new class. This is the embodiment of the meaning of one-shot segmentation. It should be mentioned that the classes sampled by the test set are not present in the training set, that is, $U_{train}\bigcap U_{test}=\O$. The relation function $F$ in this problem is implemented by the model detailed in Section.