Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation

Self-reasoning Scheme. Following the paradigm of advanced FSS approaches Zhang et al. (2019), we freeze the backbone network to boost generalization capabilities. Taking ResNet He et al. (2016) for example, the intermediate feature maps after each convolutional block of the support branch are represented as $\{{{\bf{F}}_{\rm{s}}^{b}}\}_{b=1}^{B}$^***$B$ denotes the total number of convolutional blocks contained in the backbone network, which is equal to 4 for ResNet.. We use the high-level features generated by the last block, i.e., block4, to perform the self-reasoning scheme:

$${\bf{\hat{M}}}_{\rm{s}}^{{\rm{aux}}}{\rm{=}}{f_{{\rm{seg}}}}({{\bf{F}}_{\rm{s}}^{4}}),$$

(1)

where ${f_{{\rm{seg}}}}(\cdot)$ is a lightweight segmentation decoder with residual connections, including three convolutions with 256 filters, one convolution with 2 filters, and an $\arg\max(\cdot)$ operation for producing coarse prediction masks ${\bf{\hat{M}}}_{\rm{s}}^{{\rm{aux}}}$$\in$${\{{0,1}\}^{H\times W}}$. $H$ and $W$ denote the height and weight respectively, and their product $H$$\times$$W$ is the minimum resolution of all feature maps. Unlike the widely adopted prototype-guided segmentation framework, such a mask-agnostic self-reasoning scheme is more efficient and can also provide reliable auxiliary information.
Proxy Acquisition. Given the coarse prediction ${\bf{\hat{M}}}_{\rm{s}}^{{\rm{aux}}}$ and down-sampled ground-truth mask ${{\bf{M}}_{\rm{s}}}$ of the support image, we first evaluate their differences to derive the corresponding mask (i.e., valid region) ${{\bf{M}}_{m\in{\mathcal{P}_{1}}}}$ of each proxy where ${\mathcal{P}_{1}}$$=$$\{{\alpha,\beta,\gamma,\delta}\}$ denotes the index set of proxies:

$$\left\{\begin{array}[]{l}{\bf{M}}_{\alpha}^{\left({x,y}\right)}=\mathds{1}[{{\bf{\hat{M}}}_{\rm{s}}^{{\rm{aux;}}\left({x,y}\right)}={\bf{M}}_{\rm{s}}^{\left({x,y}\right)}=1}]\\ {{\bf{M}}_{\beta}}{\kern 12.05553pt}={{\bf{M}}_{\rm{s}}}{\rm{-}}{{\bf{M}}_{\alpha}}\\ {\bf{M}}_{\gamma}^{\left({x,y}\right)}=\mathds{1}[{{\bf{\hat{M}}}_{\rm{s}}^{{\rm{aux;}}\left({x,y}\right)}={\bf{M}}_{\rm{s}}^{\left({x,y}\right)}=0}]\\ {{\bf{M}}_{\delta}}{\kern 12.91663pt}={\bf{G}}{\rm{-}}{{\bf{M}}_{\rm{s}}}{\rm{-}}{{\bf{M}}_{\gamma}}\end{array}\right.,$$

(2)

where $({x,y})$ indexes the spatial location of the mask, $\mathds{1}(\cdot)$ is an indicator function that outputs $1$ if the condition is satisfied or $0$ otherwise, and ${\bf{G}}$$\in$$\mathbb{R}{{}^{H\times W}}$ represents an all $1$’s matrix with the same shape as ${{\bf{M}}_{m}}$. ${{\bf{M}}_{\alpha}}$ and ${{\bf{M}}_{\beta}}$ aggregate the main and auxiliary foreground information respectively, generally corresponding to the body and boundary regions of objects; while ${{\bf{M}}_{\gamma}}$ and ${{\bf{M}}_{\delta}}$ gather the primary and secondary background information respectively, which typically correspond to the regions of the generalized background and distractor objects. As described above, these masks are generated according to the segmentation results of the model, therefore the main (primary) and auxiliary (secondary) status may change when encountering complex tasks. The extreme case is ${{\bf{M}}_{\beta}}$$=$${{\bf{M}}_{\rm{s}}}$, but even so, the guidance information is still complete. To better understand the effect of each mask, we present several examples in Figure 3.
Then, we conduct masked average pooling operations Zhang et al. (2020) on the support features ${{\bf{F}}_{\rm{s}}}$$\in$$\mathbb{R}{{}^{C\times H\times W}}$ and masks ${{\bf{M}}_{m}}$ computed above to encode the set of proxies $\{{{{\bf{p}}_{m}}}\}$:

$${{\bf{p}}_{m}}=\frac{{\sum\nolimits_{x,y}{{\bf{F}}_{\rm{s}}^{(x,y)}\cdot}{\bf{M}}_{m}^{(x,y)}}}{{\sum\nolimits_{x,y}{{\bf{M}}_{m}^{(x,y)}}}},$$

(3)

where ${{\bf{p}}_{m}}$$\in$$\mathbb{R}{{}^{C}}$ represents a specific proxy in the set, playing some roles in conquering the typical challenges of FSS tasks.

Prototype Acquisition. Similar to the acquisition method of proxies (Eq. (3)), the set of prototypes ${\{{{{\bf{p}}_{n}}}\}_{n\in{\mathcal{P}_{2}}}}$ are also generated using the masked average pooling operation where ${\mathcal{P}_{2}}$$=$$\left\{{{\rm{f}},{\rm{b}}}\right\}$ represents the index set of prototypes:

$${{\bf{p}}_{n}}=\frac{{\sum\nolimits_{x,y}{{\bf{F}}_{\rm{s}}^{(x,y)}\cdot\mathds{1}[{{\bf{M}}_{\rm{s}}^{(x,y)}{\rm{=}}j}]}}}{{\sum\nolimits_{x,y}{\mathds{1}[{{\bf{M}}_{\rm{s}}^{(x,y)}{\rm{=}}j}]}}},$$

(4)

where $j$$=$$1$ if the foreground prototype ${{\bf{p}}_{\rm{f}}}$ is evaluated and $j$$=$$0$ if the background one ${{\bf{p}}_{\rm{b}}}$ is calculated.

Feature Matching. Dense feature comparison is a common approach in the FSS literature that compares the query feature maps ${{\bf{F}}_{\rm{q}}}$ with the global feature vector (i.e., prototype) at all spatial locations. In view of its effectiveness, we follow the same technical route and extend it to a dual-prototype setting (see Figure 3(b)), which can be defined as:

$${\bf{F}}_{{\rm{q;}}n}^{{\rm{tmp}}}{\rm{=}}\bigoplus\left({{{\bf{F}}_{\rm{q}}},{\zeta_{H\times W}}({{{\bf{p}}_{n}}})}\right),$$

(5)

where ${\zeta_{h\times w}}(\cdot)$ expands the given vector to a spatial size $h$$\times$$w$ and $\bigoplus$ represents the concatenation operation along channel dimensions. The superscript “tmp” denotes temporary, and the subscript $n$$\in$${\mathcal{P}_{2}}$ indicates the index of the prototype.
Feature Activation. Given the proxies ${\{{{{\bf{p}}_{m}}}\}_{m\in{\mathcal{P}_{1}}}}$ and prototypes ${\{{{{\bf{p}}_{n}}}\}_{n\in{\mathcal{P}_{2}}}}$ computed by Eqs. (3-4), we evaluate the cosine similarity between each of them and query features at each spatial location respectively:

$${\bf{A}}_{l}^{\left({x,y}\right)}=\frac{{{\bf{p}}_{l}^{\mathsf{T}}\cdot{\bf{F}}_{\rm{q}}^{\left({x,y}\right)}}}{{\left\|{{{\bf{p}}_{l}}}\right\|\cdot\left\|{{\bf{F}}_{\rm{q}}^{\left({x,y}\right)}}\right\|}},\;\;\;l\in{\mathcal{P}_{1}}\cup{\mathcal{P}_{2}},$$

(6)

where ${{\bf{A}}_{l}}$$\in$$\mathbb{R}^{H\times W}$ denotes the activation map of the $l$-th item. Figure 4 presents some examples to illustrate the region of interest for each feature vector ${{\bf{p}}_{l}}$. Actually, there is an alternative approach to propagate information about feature vectors ${\{{{{\bf{p}}_{l}}}\}_{l\in{\mathcal{P}_{1}}\cup{\mathcal{P}_{2}}}}$ to the query branch, that is, to perform the same operation as Eq. (5). Such a scheme, however, introduces additional computational overhead and performance degradation, as discussed in § 5.3.

Parallel Decoder Structure. Most previous works abandon the prototype ${{\bf{p}}_{\rm{b}}}$ due to the complexity of background regions in support images. We argue that specifically designing the network to capture foreground and background features separately could have a positive effect on performance, whereas a simple fusion strategy would be counterproductive. To this end, a unique parallel decoder structure (PDS) is devised in which proxies/prototypes with similar attributes are integrated to boost the discrimination power:

$${\bf{F}}_{{\rm{q;}}n}^{{\rm{refined}}}{\rm{=}}\bigoplus\left({{\bf{F}}_{{\rm{q;}}n}^{{\rm{tmp}}},{{\bf{A}}_{n}},{{\left\{{{{\bf{A}}_{m}}}\right\}}_{m\in{\mathcal{P}_{{\rm{tmp}}}}}}}\right),$$

(7)

where ${\mathcal{P}_{{\rm{tmp}}}}$ is a temporary set of vector indexes, representing $\{{\alpha,\beta}\}$ when foreground features ${\bf{F}}_{{\rm{q;f}}}^{{\rm{refined}}}$ are evaluated and $\{{\gamma,\delta}\}$ when background ones ${\bf{F}}_{{\rm{q;b}}}^{{\rm{refined}}}$ are computed. These two refined query features are then fed into the corresponding decoder network respectively, and the output single-channel probability maps are concatenated to generate the final segmentation result.

To guarantee the richness of the information provided by proxies while preventing extreme situations (§ 4.1), we introduce additional constraints on the prediction results of self-reasoning schemes for both support and query branches. The total loss for each episode can be written as:

$${\mathcal{L}_{{\rm{total}}}}{\rm{=}}{\mathcal{L}_{{\rm{main}}}}+{\lambda_{1}}{\mathcal{L}_{{\rm{aux1}}}}+{\lambda_{2}}{\mathcal{L}_{{\rm{aux2}}}},$$

(8)

where $\mathcal{L}$ represents the binary cross-entropy (BCE) loss function. ${\lambda_{1}}$ and ${\lambda_{2}}$, the balancing coefficients, are set to $1.0$ in all experiments.

Datasets. We evaluate the proposed approach on two standard FSS benchmarks: PASCAL-$5^{i}$Shaban et al. (2017) and COCO-$20^{i}$ Nguyen and Todorovic (2019). The former is created from PASCAL VOC 2012 Everingham et al. (2010) with additional mask annotations from SDS Hariharan et al. (2011), consisting of $20$ semantic categories evenly divided into $4$ folds: $\{{{5^{i}}}\}_{i=0}^{3}$, while the latter, built from MS COCO Lin et al. (2014), is composed of 80 semantic categories divided into $4$ folds: $\{{{{20}^{i}}}\}_{i=0}^{3}$. Models are trained on $3$ folds and tested on the remaining one in a cross-validation manner. We randomly sample $1,000$ episodes from the unseen fold $i$ for evaluation.
Evaluation Metrics. Following previous FSS approaches Li et al. (2021); Tian et al. (2022), we adopt mean intersection-over-union (mIoU) and foreground-background IoU (FB-IoU) as our evaluation metrics, in which the mIoU metric is primarily used since it better reflects the overall performance of different categories.
Implementation Details. Two different backbone networks (i.e., VGG16 Simonyan and Zisserman (2014) and ResNet50 He et al. (2016)) are adopted for a fair comparison with existing FSS methods. Following Tian et al. (2022), we pre-train these backbone networks on ILSVRC Russakovsky et al. (2015) and freeze them during training to boost the generalization capability. The setup of pre-processing technology is the same as Tian et al. (2022). The SGD optimizer is utilized with a learning rate of $0.005$ for $200$ epochs on PASCAL-$5^{i}$ and $50$ epochs on COCO-$20^{i}$. All experiments are performed on NVIDIA RTX 2080Ti GPUs with the PyTorch framework. We report the average results of $5$-runs with different random seeds to reduce the influence of selected support-query image pairs on performance.

We evaluate the proposed DCP framework with state-of-the-art methods on two standard FSS datasets. Table 1 presents the 1-shot and 5-shot results assessed under the mIoU metric on PASCAL-$5^{i}$. It can be observed that our models outperform existing approaches by a sizeable margin, especially under the 5-shot setting, indicating that more appropriate and reliable information is supplied for query image segmentation. Table 3 summarizes the performance of different methods with ResNet50 backbone on COCO-$20^{i}$, in which the proposed approach sets new state-of-the-arts with a small number of learnable parameters. Specifically, our DCP achieves $2.89\%$ (1-shot) and $3.78\%$ (5-shot) of mIoU improvements over the previous best approach (i.e., PFENet), respectively. The FB-IoU evaluation metric is also included for further comparison, as shown in Table 3. Once again, the proposed DCP substantially surpasses recent methods.
For qualitative analysis, we first visualize the segmentation results of DCP and baseline approach, as illustrated in Figure 5. It can be found that the false positives caused by irrelevant objects are significantly reduced (see the first two rows). Meanwhile, more accurate segmentation boundaries are obtained, benefiting from the superiority of ${{\bf{p}}_{\beta}}$ (see the last row). Note that for more details of the baseline approach, please refer to Table 4 in the ablation studies (§ 5.3). Then we randomly sample $2,000$ episodes from $20$ categories of PASCAL VOC 2012 to draw the confusion matrix, as shown in Figure 6. The proposed DCP effectively reduces the semantic confusion between object categories and exhibits better segmentation performance.

The following ablation studies are performed with VGG16 backbone on PASCAL-$5^{i}$ unless otherwise stated.
Proxy & Prototype. As discussed in § 4.2, the derived proxies, along with foreground/background prototypes, tend to propagate pertinent information for the query branch in the form of activation maps. In Table 4, we evaluate the impact of each of them on segmentation performance under 1-shot settings. Overall, the following observations could be made: (i) By introducing the foreground activation map ${{\bf{M}}_{\rm{f}}}$, the performance of baseline approach is greatly improved from $58.52\%$ to $59.64\%_{\bf{+1.12}}$, demonstrating the importance of foreground cues for guidance. Furthermore, we argue that such a simple yet powerful variant could serve as a new baseline approach for future FSS works. (ii) The background information provided by mask annotations needs to be used with caution since the performance might even deteriorate without PDS (see the third row). (iii) Our proposed proxies show a superior capability compared to the prototypes in facilitating query image segmentation ($60.12\%\,$vs.$\,60.63\%$ mIoU), which is the so-called “beyond the prototype”. (iv) There is a complementary relationship between the prototype and proxy, the combination of which could further boost performance. (v) Surprisingly, background-related proxies could be better integrated with prototypes (see rows 6-7). One possible reason for this phenomenon is that the interference of irrelevant objects may have a greater influence on the FSS model among many factors, while ${{\bf{M}}_{\delta}}$ can effectively reduce false positives.
Parallel Decoder Structure. After obtaining the activation maps (Eq. (6)), we first attempt to leverage a simple fusion scheme to guide the query branch, i.e., concatenating ${\bf{F}}_{{\rm{q;f}}}^{{\rm{tmp}}}$ and ${\{{{{\bf{A}}_{l}}}\}_{l\in{P_{1}}\cup{P_{2}}}}$ and feeding the refined features to a single decoder network. Unfortunately, such a fusion scheme does not offer performance gains, which we attribute to the information confusion between the expanded foreground prototype ${\zeta_{H\times W}}\left({{{\bf{p}}_{\rm{f}}}}\right)$ and background-related activation maps (${{\bf{M}}_{\gamma}}$ and ${{\bf{M}}_{\delta}}$). To this end, we devise a unique parallel decoder structure (PDS) where proxies/prototypes with similar attributes are integrated, yielding a superior result compared to the previous scheme (see rows 3-4 of Table 4).
Efficiency. We also evaluate the efficiency of different fusion strategies between derived proxies ${\{{{{\bf{p}}_{m}}}\}_{m\in{P_{1}}}}$ and query features ${{\bf{F}}_{\rm{q}}}$, as presented in Table 5. Please note that the first row represents the previously discussed scheme (see Eq. (7)), while the second row denotes the scheme that concatenates query features with the expanded proxies, similar to Eq. (6). The experiment results indicate that the first fusion strategy exhibits remarkable segmentation performance with fewer parameters ($0.26$M) and faster speed ($16+\,$FPS), demonstrating the effectiveness of activation maps for guidance.