Box-supervised Instance Segmentation with Level Set Evolution

The existing instance segmentation methods can be roughly divided into two categories. The first group [13, 22, 9, 55] performs segmentation on the regions extracted from the detection results. Another category [4, 5, 43, 51, 48] directly segments each instance in a fully convolutional manner without resorting to the detection results. However, all these methods rely on the expensive pixel-wise mask annotations.

Box-supervised instance segmentation, which only employs the bounding box annotations to obtain pixel-level mask prediction, has recently been receiving increasing attention. Khoreva et al. [20] proposed to predict the mask with box annotations under the deep learning framework, which heavily depends on the region proposals generated by the unsupervised segmentation methods like GrabCut [40] and MCG [39]. Based on Mask R-CNN [13], Hsu et al. [16] formulated the box-supervised instance segmentation into a multiple instance learning (MIL) problem by making use of the neighbouring pixel-pairwise affinity regularization. BoxInst [44] uses the color-pairwise affinity with box constraint under an efficient RoI-free CondInst framework [43]. Despite the promising performance, the pairwise affinity relationship is built on either partial or all neighbouring pixel pairs with the oversimplified assumption that spatial pixel or color pairs are encouraged to share the same label. This inevitably introduces noises, especially from the nearby background or similar objects. Besides, the recent methods like BBAM [26] and DiscoBox [25] focus on the generation of proxy mask labels, which often require multiple training stages or networks to achieve promising performance. Unlike the above methods, our proposed level set-based approach is learned implicitly in an end-to-end manner, which is able to iteratively align the instance boundaries by optimizing the energy function within the box region.

As a classical variational approach, the level set methods [1, 37] have been widely used in image segmentation, which can be categorized into two major groups: region-based methods [7, 45, 36] and edge-based methods [6, 33]. The key idea of level set is to represent the implicit curve by an energy function in a higher dimension, which is iteratively optimized by using gradient descent. Some works [15, 21, 17, 49, 54] have been proposed to embed the level set into the deep network in an end-to-end manner and achieve promising segmentation results. Wang et al. [49] predicted the evolution parameters and evolved the predicted contour by incorporating the user clicks on the boundary points. The energy function is based on the edge-based level set method in  [6]. Levelset R-CNN [15] performs the Chan-Vese level set evolution with the deep features based on Mask R-CNN [13], where the original image is not used in the optimization. Yuan et al. [54] built a piecewise-constant function to parse each constant sub-region corresponding to a different instance based on the Mumford–Shah model [36], which achieves instance segmentation by a fully convolutional network. The above methods perform level set evolution between deep features and ground-truth mask in a fully supervised manner, which train the network to predict different sub-regions and get object boundaries. Our proposed approach performs level set evolution only using the box-based annotations without the pixel-wise mask supervision.

Kim et al. [21] performed level set evolution in an unsupervised manner, which is mostly related to our proposed approach. To achieve $N$-class semantic segmentation, it employs the global multi-phase Mumford–Shah function [36] that only evolves on the low-level features of the input image. Our method is based on the Chan-Vese functional [7], which is constrained within the local bounding box with the enriched information from both input image and high-level deep features. Moreover, the initialization of level set is generated automatically for robust curve evolution.

We first give a brief review of the level set methods [36, 7, 45], which formulate the image segmentation as a consecutive energy minimization problem. In the Mumford-Shah level set model [36], the segmentation of a given image $I$ is obtained by finding a parametric contour $C$, which partition the image plane $\Omega\subset\mathbb{R}^{2}$ into $N$ disjoint regions ${\Omega_{1}},\cdots,{\Omega_{N}}$. The Mumford-Shah energy functional $\mathcal{F}^{MS}(u,C)$ can be defined as below:

Chan and Vese [7] later simplified the Mumford-Shah functional as a variational level set, which has been explored aplenty [46, 31, 34, 52]. Specially, it can be derived as follows,

Our proposed method exploits the level set evolution with Chan-Vese energy-based model [7] to achieve high-quality instance segmentation using the box annotations only.

To obtain the accurate boundary for each instance, we learn a series of level sets $\phi(x,y)$ by minimizing the following energy function:

The energy function $\mathcal{F}$ can be optimized with gradient back-propagation during training. With the time step $t\geqslant 0$, the derivative of energy function $\mathcal{F}$ upon $\phi$ can be written as follows:

In addition to the normalized input image, we take into account the high-level deep features $I_{f}$, which embed the image semantic information, to obtain more robust results. To this end, we make full use of the unified and high-resolution mask feature $F_{mask}$ from all FPN levels in SOLOv2, which is further fed into a convolution layer to extract the high-level features $I_{f}$. Besides, the features $I_{f}$ are enhanced by the tree filter [41, 27], which employs minimal spanning tree to model long-range dependencies and preserve the object structure. The overall energy function for level set evolution can be formulated as follows:

where $\lambda_{1}$ and $\lambda_{2}$ are weights to balance the two kinds of features. $c_{{u_{1}}}$, $c_{{u_{2}}}$ and $c_{{f_{1}}}$, $c_{{f_{2}}}$ are the mean values for input terms $I_{u}$ and $I_{f}$, respectively.

In particular, we utilize the coordinate projection of ground-truth box to $x$-axis and $y$-axis and calculate the projection difference between the predicted mask map and the ground-truth box. Such a simple scheme limits the predicted initialization boundary within the bounding box, providing a good initial state for curve evolution. Let $m^{b}\in{\{0,1\}^{H\times W}}{\rm{}}$ denote the binary region by assigning one to the locations in the ground-truth box, and zero otherwise. The mask score predictions ${m^{p}}\in{(0,1)^{H\times W}}$ for each instance can be regarded as the foreground probabilities. The box projection function ${\mathcal{F}(\phi_{0})_{box}}$ is defined as below:

where ${m^{p}_{x}}$, $m^{b}_{x}$ and ${m^{p}_{y}}$, $m^{b}_{y}$ denote the $x$-axis projection and $y$-axis projection for mask prediction $m^{p}$ and binary ground-truth region $m^{b}$, respectively. $\mathcal{P}_{{dice}}$ represents the projection operation measured by 1-D dice coefficient [35].

Loss Function. The loss function $L$ to train our proposed network consists of two items, including $L_{cate}$ for category classification and $L_{inst}$ for instance segmentation with box annotations:

where $L_{cate}$ is the Focal Loss [29]. For $L_{inst}$, we employ the presented differentiable level set energy as the optimization objective:

where $N_{pos}$ indicates the number of positive samples, and $p_{i,j}^{*}$ denotes the category probability at target location $(i,j)$. $\mathbbm{1}$ represents the indicator function, which ensures only the positive instance mask samples perform the level set evolution. $\mathbbm{1}$ is set to one if $p_{i,j}^{*}>0$, and zero otherwise. $\alpha$ is the weight parameter, which is set to $3.0$ empirically in our implementation.

Pascal VOC [11]. Pascal VOC consists of 20 categories. As in [16, 44, 26], the augmented Pascal VOC 2012 [12] dataset is used, which contains 10, 582 images for training and 1, 449 validation images for evaluation.

The models are trained with the AdamW [32] optimizer on 8 NVIDIA V100 GPUs. The training schedules of “1$\times$” and “3$\times$” are the same as mmdetection framework [8] with 12 epochs and 36 epochs, respectively. ResNet [14] is employed as the backbone, which is initialized with the ImageNet [10] pre-training weights. For COCO, the initial learning rate is $10^{-4}$ with 16 images per mini-bath. For Pascal VOC, the initial learning rate is $5\times 10^{-5}$ with 8 images per mini-bath. The scale jitter is used where the shorter image side is randomly sampled from 640 to 800 pixels on COCO and Pascal VOC datasets for fair comparison. For iSAID and LiTS, all the models on each dataset are trained with the same settings. COCO-style mask AP (%) is adopted for performance evaluation. Following [26, 25], we also report the average precision (AP) at four IoU thresholds (including 0.25, 0.50, 0.70 and 0.75) for the comparison on Pascal VOC dataset. The non-negative weight $\gamma$ in Eq. 3 is set to $10^{-4}$ by default.

We compare our proposed method against the state-of-the-art instance segmentation approaches, including box-supervised and fully mask-supervised methods in different scenarios.

Most box-supervised methods are evaluated on the Pascal VOC dataset. Table 1 reports the comparison results. Our method outperforms BoxInst [44] by 2.0% and 1.8% AP with ResNet-50 and ResNet-101 backbones, respectively, achieving the best performance. For AP₂₅ and AP₅₀, our method can obtain 77.9% and 66.3% accuracy, largely outperforming the recent DiscoBox [25] by 5.1% and 4.1%. The high IoU threshold-based AP metrics can reflect the segmentation performance with accurate boundary, which is in line with the practical application. Our approach achieves 38.7% AP₇₅ with ResNet-101, which outperforms BoxInst [44] and DiscoBox [25] by 1.7% and 1.2%, respectively.

Table 2 shows the main results on COCO test-dev split. Both fully mask-supervised and box-supervised methods are compared in the evaluation. Our method outperforms BBTP [16] by 12.3% AP with the same backbone. In contrast to the recent box-supervised methods, our method outperforms BBAM [26] and BoxCaseg [47] by 7.7% AP and 2.5% AP using ResNet-101. It achieves 33.4% AP and 35.4% AP, which is higher than BoxInst [44] by 0.2% and 0.4% with ResNet-101 and ResNet-101-DCN backbones, respectively. Our approach achieves 16.8% AP_S on small objects, which is slightly lower than BoxInst [44] by 0.3%. This is because small objects lack rich features for level set evolution to distinguish the foreground object and background within the bounding box. However, our method obtains the best results for large objects, largely outperforming BoxInst [44] by 2.4% AP_L using the same ResNet-101-DCN. Our method even performs better than some recent fully mask-supervised methods, such as YOLACT [4] and PolarMask [51]. This shows that our method narrows the performance gap between mask-supervised and box-supervised instance segmentation. Fig. 2 visualizes some instance segmentation results on COCO and Pascal VOC datasets.

We then compare our method with other deep variational-based instance segmentation approaches. DeepSnake [38] is based on the classical snake method [19]. Levelset R-CNN [15] and DVIS-700 [54] are also built on level set function. These methods are all fully supervised by the mask annotations. As shown in Table 3, our method achieves comparable results to the fully supervised variational-based methods, and even outperforms DeepSnake [38] and Levelset R-CNN [15].

To further validate the robust performance of our method in more complicated scenarios, we conduct experiments on remote sensing and medical image datasets. In remote sensing, the objects of the same class are densely-distributed. For medical images, the background is highly similar to the foreground. The previous pixel relationship model-based methods are built on the neighbouring pixel pairs. They are easily affected by the noisy context. Our level set-based method drives the curve to fit the object boundary under the guidance of level set minimization, which is more robust. Table 4 and Table 5 show the mask AP results on iSAID and LiTS datasets, respectively. It can be clearly seen that our approach outperforms BoxInst [44] by 2.3% AP on iSAID and 3.8% AP on LiTS. Fig. 3 and Fig. 4 show several examples of instance segmentation on iSAID and LiTS, respectively. One can see that our method is effective in various scenarios.

The ablation study is conducted on Pascal VOC dataset to examine the effectiveness of each module in our proposed framework.