FGN: Fully Guided Network for Few-Shot Instance Segmentation

Suppose for a set of base classes $\mathcal{C}^{\text{base}}$, we have a large set of images annotated with object instances, denoted by $\mathcal{D}^{\text{base}}$. Now let us consider a different set of semantic classes $\mathcal{C}^{\text{novel}}$ (called novel classes), which do not overlap with the base classes, i.e., $\mathcal{C}^{\text{base}}\cap\mathcal{C}^{\text{novel}}=\phi$. For these novel classes, we only have a very limited number of annotated instances $\mathcal{D}^{\text{novel}}$, referred to as support set. In practice, this is usually due to difficulties in collecting images or acquiring instance-level annotations. The task of few-shot instance segmentation (FSIS) is to segment, from any given query image $\mathbf{I}^{q}$, all the object instances belonging to the novel classes. Note that when $|\mathcal{C}^{\text{novel}}|=N$ ($|\cdot|$ represents the cardinality of a set throughout this paper) and there are $K$ annotated instances for each novel class, we call it an $N$-way $K$-shot instance segmentation task.

In this paper, we conjoin the few-shot learning paradigm with general instance segmentation to address the FSIS problem. Following the spirit of few-shot classification [29, 27], we simulate a quantity of $N$-way $K$-shot instance segmentation tasks $\mathcal{T}=\{(\mathcal{S}_{i},\mathbf{x}_{i})\}_{i=1}^{|\mathcal{T}|}$ by randomly sampling support sets and queries from $\mathcal{D}^{\text{base}}$ (of the base classes $\mathcal{C}^{\text{base}}$), where the $i$-th task is formed by sampling a support set $\mathcal{S}_{i}$ and a query image $\mathbf{x}_{i}$. By the use of these simulated tasks $\mathcal{T}$, we learn a conditional instance segmentation model $f_{\theta}(\mathbf{x}|\mathcal{S})$ parameterized by $\theta$, which performs instance segmentation on the query image $\mathbf{x}$ conditioned on the support set $\mathcal{S}$. The learned model $f_{\theta}(\mathbf{x}|\mathcal{S})$ can then be applied to the target task, i.e., $N$-way $K$-shot instance segmentation over the novel classes $\mathcal{C}^{\text{novel}}$ (simply letting $\mathcal{S}=\mathcal{D}^{\text{novel}}$ and $\mathbf{x}=\mathbf{I}^{q}$). It is worth pointing out that, instead of straightforwardly learning $f_{\theta}(\mathbf{x})$, our strategy is to learn a conditional model $f_{\theta}(\mathbf{x}|\mathcal{S})$, which can be viewed as to utilize the support set $\mathcal{S}$ to guide the instance segmentation of $\mathbf{x}$. The presence of guidance plays a critical role for the model trained on the base classes $\mathcal{C}^{\text{base}}$ to generalize well to the novel classes $\mathcal{C}^{\text{novel}}$.

Central to any FSIS approach is how to effectively encode and utilize the support set to guide the basic instance segmentation network (mostly typically Mask R-CNN [12]). Previous works fulfill such guidance by incorporating a single guidance module at a certain location in Mask R-CNN, which may undesirably enforce components for different tasks to share the same guidance [29], or neglect certain components [27]. We present the Fully Guided Network (FGN) in this paper, which is distinct from previous works [29, 27] in that, components for different tasks in Mask R-CNN are guided by the support set differently to achieve full guidance.

An overview of the proposed FGN is demonstrated in Fig. 2. Generally, FGN introduces guidance into Mask R-CNN at three key components, i.e., the RPN, the detection branch (including classification and bbox regression) and the mask branches, leading to the Attention-Guided RPN (AG-RPN), the Relation-Guided Detector (RG-DET) and the Attention-Guided FCN (AG-FCN), respectively. In the proposed FGN, the given support set $\mathcal{S}$ (containing $K$ annotated instances for each of the $N$ classes) and the query image $\mathbf{x}$ are encoded by a shared backbone $\varphi$ (ResNet101 [13] in our implementation) to give the feature maps $\mathbf{F}_{n}^{k},\ \mathbf{Y}$ $\in\mathbb{R}^{H\times W\times C}$ respectively. $\mathbf{F}_{n}^{k}$ encodes the support set, which is used by AG-RPN to guide the proposal generation from $\mathbf{Y}$ in the first stage. Then, in the second stage, for each proposal [also called Region-of-Interest (RoI)] with the aligned feature maps $\mathbf{z}_{j}\in\mathbb{R}^{h\times w\times C}$, the aligned $\hat{\mathbf{F}}_{n}^{k}\in\mathbb{R}^{h\times w\times C}$ is utilized by RG-DET to guide the classification and bbox heads, and by AG-FCN to guide the mask head. Another key contribution of our work is to design novel and effective guidance mechanisms for these modules, which are detailed as below.

Attention-Guided RPN. Mask R-CNN relies on RPN to obtain class-agnostic proposals of potential objects for subsequent processing. Under the problem setting of FSIS, RPN has to be trained on the base classes $\mathcal{C}^{\text{base}}$ and tested on a solely different set of novel classes $\mathcal{C}^{\text{novel}}$. In this case, RPN may generate a lot of undesired proposals but miss the ones of concern, especially when $\mathcal{C}^{\text{novel}}$ departs far from $\mathcal{C}^{\text{base}}$, or the number of novel classes is small, which will largely degrade overall performance. To tackle this issue, our idea is to introduce guidance from the support set into RPN such that it can focus on the classes of concern and generate class-aware proposals, which we call Attention-Guided RPN (AG-RPN).

The structure of AG-RPN is depicted in Fig. 3. The feature maps $\mathbf{F}_{n}^{k}\in\mathbb{R}^{H\times W\times C}$ with $n=1,...,N,k=1,...,K$, which encode the support set, undergo the global average pooling (GAP) and the averaging operation over each individual class, given by

with $\{\mathbf{a}_{1},...,\mathbf{a}_{N}\}\in\mathbb{R}^{C\times 1}$ being the class-attentive vectors associated with the $N$ novel classes. Each $\mathbf{a}_{n}$ is then taken to weight the feature maps of the query image $\mathbf{Y}\in\mathbb{R}^{H\times W\times C}$ as below

which means taking $\mathbf{a}_{n}$ to perform element-wise multiplication along the channel dimension at every spatial location in $\mathbf{Y}$. Each $\tilde{\mathbf{Y}}_{n}$ is fed into the basic RPN for proposal generation independently and the results are then aggregated to yield the final proposals. The aggregation procedure can be described as follows: For each particular anchor, an objectness score can be acquired through the RPN over every $\tilde{\mathbf{Y}}_{n}$, and the softmax results over the $N$ scores are taken as the class-aware confidence of the anchor. Anchor refinement is conducted by the regression corresponding to the top matching score during inference. The final proposals are picked up from the anchors by thresholding their confidence and performing non-maximal suppression.

Relation-Guided Detector. The guidance on the detector branch in Mask R-CNN (including the classification and bbox regression heads) is imposed in an implicit way in previous works [21, 30], which just simply modulate the feature extraction in the first or second stage by the use of support set. In this paper, we propose a different guidance mechanism for the detector (actually the classification branch), termed as Relation-Guided Detector (RG-DET). RG-DET achieves guidance by explicitly comparing the features extracted from the support set and the RoI, inspired by the Relation Network (RN) [28] originally proposed for few-shot classification. We favor RN mainly because it is characterized by that, both the feature embedding and the similarity measure are learnable, compared to other competitors like [29, 27, 16].

Unfortunately, RN cannot be directly deployed to our task because there exists an essential difference between our problem here and the general few-shot classification, that is, the rejection of background class. RG-DET operates on individual RoIs output by AG-RPN, which may inevitably contain background RoIs belonging to neither of the novel classes in the support set. By contrast, recall that few-shot classification methods (including RN) always classifies the query to be one of the classes indicated by the support set. Taking into account the background rejection issue, the structure of RG-DET is illustrated in Fig. 4.

For a particular RoI, its aligned feature maps $\mathbf{z}_{j}\in\mathbb{R}^{h\times w\times C}$ are concatenated with the $N$ aligned feature maps $\hat{\mathbf{F}}_{n}=\left(\frac{1}{K}\sum_{k}\hat{\mathbf{F}}_{n}^{k}\right)\in\mathbb{R}^{h\times w\times C}$ extracted from the support set (as shown in Fig. 4), followed by a stack of conv and fc layers (termed as MLP), to give the matching scores (the cls branch) and the object box (the bbox reg branch). The matching score between $\mathbf{z}_{j}$ and the $i$-th feature maps $\hat{\mathbf{F}}_{n}$ is represented by a doublet $(c_{i}^{+},c_{i}^{-})$, where $c_{i}^{+}$ and $c_{i}^{-}$ stand for the confidence of matching the $i$-th class and the background respectively. To enable background rejection, we need to derive an $(N\!+\!1)$-length matching vector $\mathbf{c}=(c_{1},...,c_{N},c_{N+1})$ from the $2N$ original scores, with $c_{i},i=1,...,N$ reflecting the confidence of the $i$-th class and $c_{N+1}$ the background. For this purpose, we set $c_{i}=c_{i}^{+}$ and $c_{N+1}=c_{i^{*}}^{-}$ with $i^{*}=\operatorname*{arg\,max}_{i}\left\{c_{i}^{+}\right\}$, which physically means we depend on the best-matched class (the most reliable one) to estimate the confidence of background $c_{N+1}$. A softmax operation is then performed over the matching vector $\mathbf{c}$, yielding the final classification score.

The bbox regression branch shares the concatenation and the first conv layer with the classification branch, but has a separate MLP layer as shown in Fig. 4.

Attention-Guided FCN. As illustrated in Fig. 5, the Attention-Guided FCN (AG-FCN) introduces guidance into the FCN-based mask head. AG-FCN basically follows the guidance scheme for few-shot semantic segmentation [26], except for two modifications. First, an operation of masked pooling [32] is performed on the aligned feature vectors $\hat{\mathbf{F}}_{n}^{k}\in\mathbb{R}^{h\times w\times C}$ before computing the class-attentive vectors $\{\mathbf{b}_{1},...,\mathbf{b}_{N}\}\in\mathbb{R}^{C\times 1}$ as described in Eq. (1). Masked pooling on $\hat{\mathbf{F}}_{n}^{k}$ means pooling $\hat{\mathbf{F}}_{n}^{k}$ within the binary mask $\hat{\mathbf{m}}_{n}^{k}\in\mathbb{R}^{h\times w\times C}$, which is obtained by performing RoIAlign over the original instance mask ${\mathbf{m}}_{n}^{k}\in\mathbb{R}^{H\times W\times C}$. Second, a selector is used to pick up the one $\mathbf{b}_{n^{*}}$ from $\{\mathbf{b}_{1},...,\mathbf{b}_{N}\}$, where $n^{*}$ is chosen to be the ground truth class for training, and the one with the highest classfication score for testing. Note that $\tilde{\mathbf{z}}_{j}=\mathbf{z}_{j}\otimes\mathbf{b}_{n^{*}}$ where the operator $\otimes$ is identical to that in Eq. (2).

FGN is a two-stage structure since it is based on Mask R-CNN. Hence, our pipeline for training is basically similar to Mask R-CNN (including the loss functions). But differently, following the common practice in [2, 15, 30], our training includes two steps. For the first step, we purely take $\mathcal{D}^{\text{base}}$ of the base classes $\mathcal{C}^{\text{base}}$ as the training data. And for the second step, we take data from both the base classes and the novel classes, i.e., $\mathcal{C}^{\text{base}}\cup\mathcal{C}^{\text{novel}}$, to further fine-tune the model. More precisely, the second-step training data consist of the whole support set $\mathcal{D}^{\text{novel}}$ (containing $NK$ instances) and $3K$ instances for each class in $\mathcal{C}^{\text{base}}$ randomly sampled from $\mathcal{D}^{\text{base}}$, which contain totally $(N+3|\mathcal{C}^{\text{base}}|)K$ instances. Our training requires randomly sampling the training set to simulate the target FSIS tasks (constructing the episodes), which will be detailed in Section 4.1.

We adopt two commonly-used datasets for our experiments, i.e., Microsoft COCO 2017 [18] and PASCAL VOC 2012 [6] (termed as COCO and VOC respectively). COCO has $80$ object classes, consisting of a training set (trainset) with $118,287$ images and a validation set (valset) with $4,952$ images. VOC covers $20$ classes that are a subset of COCO’s $80$ classes, with a trainset of $1,464$ images (annotated with instance masks) and a valset of $1,449$ images.

General Settings. According to the problem definition in Section 3.1, our evaluation requires the following basic settings: 1) Setting the base classes $\mathcal{C}^{\text{base}}$ and the novel classes $\mathcal{C}^{\text{novel}}$, and accordingly the training set $\mathcal{D}^{\text{base}}$ and the query set $\mathcal{D}^{\text{novel}}$ (testing set): As our main setting, we adopt a challenging cross-dataset setting to better compare the generalization ability of various models, inspired by previous works [15, 30]. Specifically, we set the $20$ classes at the intersection of COCO and VOC to be $\mathcal{C}^{\text{novel}}$ and the rest $60$ classes covered by COCO but not VOC to be $\mathcal{C}^{\text{base}}$. Further, we take from COCO’s trainset the subset belonging to $\mathcal{C}^{\text{base}}$ as the training set $\mathcal{D}^{\text{base}}$, and take VOC’s valset (belonging to the $20$ novel classes $\mathcal{C}^{\text{novel}}$) to construct the testing set (see details later). We refer to this main experimental setting as COCO2VOC. Additionally, we also consider another setting termed as VOC2VOC, which only uses the VOC dataset. More precisely, we randomly sample $15$ out of $20$ classes covered by VOC to be the base classes $\mathcal{C}^{\text{base}}$ and the rest $5$ are taken as $\mathcal{C}^{\text{novel}}$. The training set $\mathcal{D}^{\text{base}}$ and the query set $\mathcal{D}^{\text{novel}}$ are constructed respectively from VOC’s trainset and valset. 2) Specifying the numbers of $N$ and $K$: We consider three different settings (a) $N=1,K=1$ (termed as 1way-1shot); (b) $N=3,K=1$ (termed as 3way-1shot); (c) $N=3,K=3$ (termed as 3way-3shot).

Methods for Comparison. To our knowledge, there exist only two FSIS methods in the literature so far, i.e., Siamese MRCNN [21] and Meta R-CNN [30], which are included in our comparison. Similar to our FGN, Siamese MRCNN and Meta R-CNN also achieve FSIS by introducing guidance into Mask R-CNN (but using different guidance mechanisms), for which we use the source codes released by the authors for our experiments. Besides, we also build a baseline for comparison, termed as MRCNN-FT, which is basically a Mask R-CNN trained with the strategy detailed in Section 3.3.

Implementation Details. We follow the training strategy in Section 3.3 and the settings of $\{\mathcal{C}^{\text{base}},\mathcal{D}^{\text{base}},\mathcal{C}^{\text{novel}},\mathcal{D}^{\text{novel}},N,K\}$ above in Section 4.1 to train our FGN model. We use ResNet101 [13] as the backbone for our model. The initial learning rates of SGD for training the first-stage AG-RPN and the second-stage RG-DET and AG-FCN are set to $0.01$ and $0.001$ respectively. We train for $60,000$ steps and a $10$-times learning rate decay is applied to the second-half steps.

To construct the simulated tasks $\mathcal{T}=\{(\mathcal{S}_{i},\mathbf{x}_{i})\}_{i=1}^{|\mathcal{T}|}$ (typically called “episodes”) for training, we basically follow the sampling strategy proposed in [29]. Note that, we crop the local patches extended by $20$ pixels around ground truth boxes of instances to form the support set, rather than using holistic images. And for testing, the tasks $\{(\mathcal{D}^{\text{novel}}_{i},\mathbf{I}^{q}_{i})\}_{i}$ are constructed to ensure every novel class in every image in the testing set is tested for once. Specifically, for each image $\mathbf{I}^{q}_{i}$, we collect all the classes it covers. Then, for each class we randomly sample other $N-1$ classes and pick up instances accordingly to form an $N$-way $K$-shot episode. We report the average performance over all the testing tasks.

We present the main results under the settings of COCO2VOC and VOC2VOC and related analysis respectively in the following.

COCO2VOC. The FSIS performance obtained by the various methods under the COCO2VOC setting is comparatively reported in Table 1, where we use $\text{mAP}_{\text{50}}$ as the quantitative performance measure. As can be observed that, our FGN can generally outperform the two state-of-the-art methods Siamese MRCNN [21] and Meta R-CNN [30] to a large margin for the three settings of $N$ and $K$. Siamese MRCNN  [21] performs comparatively to ours in case of 1way-1shot, but degrades heavily under the other two settings. This is probably because that, the guidance in this approach follows the Siamese Network mechanism which is originally designed for pairwise input. Meta R-CNN [30] does not perform well either, probably because this method relies much on the finetuning procedure in training, which cannot acquire sufficient data for finetuning when $N$ and $K$ are small like in our settings. As expected, the baseline MRCNN-FT performs very poorly, which suggests that the strategy of naively finetuning a model pretained from base classes with data from novel classes is inappropriate for FSIS.

In addition to segmentation, we also compare the various methods on the task of few-shot object detection, as shown in Table 1. Our FGN can also outperform the other methods consistently for all the settings. One can further observe that, there is an obvious performance drop from detection to segmentation for all the methods, which may indicate that FSIS cannot be achieved by trivial extension of few-shot object detection methods. We also provide some exemplary results obtained by various methods for visual comparison in Fig. 6.

While the proposed FGN can outperform the state-of-the-art as stated above, one may be concerned with a fact that, the performance of various methods (including ours) generally looks limited, significantly worse than conventional instance segmentation. We argue this is likely due to the intrinsic challenges of the FSIS problem, especially in case of low numbers of ways and shots like ours. To justify this point, we further carry out another experiment where the settings of $\mathcal{C}^{\text{base}}$ and $\mathcal{D}^{\text{base}}$ are identical to those in COCO2VOC, but the novel classes $\mathcal{C}^{\text{novel}}\subset\mathcal{C}^{\text{base}}$ and the testing tasks are sampled from COCO’s validation set (the data used for testing are different). Such case where $\mathcal{C}^{\text{novel}}\subset\mathcal{C}^{\text{base}}$ does not coincide with the problem definition of FSIS but general instance segmentation. Also, MRCNN-FT is a Mask R-CNN trained by the common strategy described in Section 3.3, which is shared by all the compared methods (including ours). As shown in Table 2, under the setting of general instance segmentation, even the standard Mask R-CNN trained in the same fashion as commonly required by FSIS approaches can only achieve limited performance. This may reflect that, the FSIS problem setting is inherently challenging, and the training strategy adopted by these FSIS methods (including our FGN) is effective in this sense. It is worth noticing that, it is not meaningful to make comparison among the various methods under this experimental setting.

VOC2VOC. In addition to our main setting of COCO2VOC, we also evaluate under the VOC2VOC setting. The results obtained by various methods in terms of $\text{mAP}_{\text{50}}$ are listed in Table 3. Although VOC2VOC shares the same validation set as COCO2VOC, it has a far smaller training set ($\sim 1.4\text{K}$ in contrast to $\sim 118\text{K}$ images). As a result, the performance of VOC2VOC is worse than that of COCO2VOC for all the methods. In this case, our FGN can still achieve the best overall performance among the compared methods for both segmentation and detection.

We perform ablation study to further reveal the merits of our FGN. All the following experiments are conducted under the COCO2VOC 3way-3shot setting.

Full Guidance. One key reason of FGN’s effectiveness is that we carefully design three guidance mechanisms, i.e., AG-RPN (P), RG-DET (D) and AG-FCN (S) to achieve full guidance. To verify the contributions of these modules, we construct several variants by disabling one or more modules from the full FGN model.

The results obtained by these variants in terms of $\text{mAP}_{\text{50}}$ for segmentation and detection are comparatively reported in Table 4. It can be seen from the degraded performance of these variants that, each module contributes to some extent on both tasks.

AG-RPN. We compare our AG-RPN with the basic RPN in Mask R-CNN and a variant termed as AG-RPN-v1 by evaluating separately the quality of the proposals generated. AG-RPN-v1 follows the design in  [21] to achieve guidance. As can be observed from Table 5 that, AG-RPN (ours) obtains the best performance in terms of $\text{AR}_{\text{50}}$.

AG-FCN. We construct two variants of AG-FCN (ours) for comparison, termed as AG-FCN-v1 and AG-FCN-v2. AG-FCN-v1 is the FCN guidance mechanism suggested in [32] for the task of semantic segmentation. AG-FCN-v2 tiles the channel attention vectors $\mathbf{b}_{n^{*}}$ to be of the same size as $\mathbf{z}_{j}$ and then concatenates them together (see Fig. 5). We also include the basic FCN used by Mask R-CNN (without guidance) for comparison. As can be seen from Table 6, AG-FCN (ours) performs the best among all the variants.