A Weakly Supervised Amodal Segmenter with Boundary Uncertainty Estimation

In this paper, we seek to address the problem of weakly supervised amodal instance segmentation (WAIS). Our goal is to segment both visible and occluded (amodal) parts of object instances in images. The weak supervision in training provides only ground-truth visible (modal) instance segmentations. Important applications of amodal segmentation include autonomous driving and robot path planning, where identifying the whole spatial extents of partially occluded objects is critical. Considering this problem under weak supervision is also important because human annotators often cannot provide reliable ground truth. For example, different annotators are likely to have very different and sometimes poor guesses of occluded object parts.

There is scant prior work on WAIS. Following recent PCNet [37], our training consists of two stages. First, we use data augmentation to train a common image segmenter – UNet [33] – on manipulated training images to predict their amodal segmentations. As input to UNet, we use the available ground-truth modal segmentation and information about where the data augmentation generated the occlusion in the training image. In the second training stage, UNet’s amodal segmentations are taken as a pseudo-ground truth for learning a standard instance segmenter – Mask-RCNN [13], as in [37]. On test images with occlusions, Mask-RCNN trained on the pseudo-ground truth is expected to output correct amodal instance segmentation.

Our contributions are aimed at advancing the first training stage, and include: (1) a new way to exploit the weak supervision for training of UNet; and (2) enabling UNet to estimate uncertainty of the predicted amodal segmentation, and enforcing the training of UNet to explicitly minimize this uncertainty.

Our first contribution is motivated by the following limitation of PCNet [37]. For manipulating training images, as illustrated in Fig. 1, PCNet randomly places an occluder object onto an occludee object based on their ground-truth modal segmentation masks, and in this way artificially generates the occluded mask of the occludee. Then, as three inputs to UNet, PCNet uses the manipulated training image, the occluded mask of the occludee, and the occluder’s mask. However, using the occluder’s mask as input to UNet puts the restrictive constraint that the occluder itself cannot be occluded by another object. To address this limitation, PCNet estimates an object ordering graph in the image, and for the occluder selects a union of all objects estimated as closer to the camera than the occludee (i.e., a union of multiple occluders). Our novelty is in replacing the occluder’s mask with the occlusion boundary in the input to UNet, as depicted in Fig. 1. Thus, our occluders are allowed to be themselves partially occluded by some other objects. This reduces complexity of estimating the pseudo-ground-truth amodal segmentation, as we do not need to estimate the object ordering graph.

Our second contribution is aimed at accounting for shape priors. As shown in Fig. 4, we implicitly capture a “shape prior” through learning to estimate an uncertainty map for the predicted amodal segmentation. In our experiments, we observe that the estimated uncertainty typically takes low (high) values over regions far away (close) to the occlusion boundary. This suggests that our uncertainty map is capable of representing a spatial distribution of object shapes, and hence can be used for regularizing our learning. Our regularization uses the estimated uncertainty map to appropriately modulate a difference between the predicted amodal segmentation and the original ground-truth mask of the occludee (before the occlusion), such that lower loss is incurred on regions with high uncertainty.

Our two contributions are incorporated in the new Amodal Segmenter with Boundary Uncertainty estimation (ASBU). ASBU is evaluated on the COCOA [38] and KINS [32] datasets on three tasks: amodal instance segmentation, amodal completion, and ordering recovery. ASBU significantly outperforms the state of the art in all three tasks.

In the following, Sec. 2 reviews previous work; Sec. 3 specifies ASBU; Sec. 4 formalizes our uncertainty estimation and uncertainty weighted loss; Sec. 5 presents our implementation details and experimental results; and Sec. 6 concludes the paper.

This section reviews closely related work.

Instance Segmentation is aimed at labeling pixels with object instance labels, and can be addressed with bounding-box-based and bounding-box-free methods. In the former [13, 25, 5, 30, 2], for every detected bounding box, a foreground object is segmented. In the latter [21, 23, 3, 10], first, a semantic segmentation is obtained, and then pixels of the same semantic class are clustered into instances based on visual cues such as object center or inner sign distance function. All of these approaches segment only visible object parts and thus are not suitable for our problem.

Amodal Instance Segmentation infers visible and occluded object parts, under full supervision in training. For the ground truth, prior work uses amodal segmentation of either real images [24, 38, 32, 11, 16] or synthetic data [9, 15, 19]. However, the existing quality of synthetic data introduces a domain gap between training on synthetic images and testing on real images, resulting in a considerable performance difference between the two domains.

Amodal Instance Completion differs from amodal instance segmentation since the goal is to complete occluded parts of an object given its modal mask, whereas in amodal instance segmentation the modal mask is not provided. Prior work typically uses the Gestalt principles and makes certain assumptions about shape convexity and length. For example, amodal instance completion has been addressed by using Euler spiral, cubic Bezier curves, shape primitives, and variational auto-encoder in [20, 26, 35, 29]. Our first stage of training for generating the amodal pseudo-ground truth is based on amodal instance completion. We evaluate ASBU on the task of amodal instance completion.

WAIS provides access only to modal-mask annotations in training. Recent work [37] begins by converting modal-mask annotations of training images into pseudo amodal masks in a self-supervised manner, as illustrated in Fig. 1. However, in a complex scene, the occluder can also be occluded by another object, as shown in Fig. 2, which requires [37] to construct an object ordering graph. This increases complexity of their first stage of training of UNet, and is not even suitable for addressing cases of entangled partial occlusions when the occluder-occludee relationship of a pair of objects is not unique, as illustrated in Fig. 3. We overcome this limitation by replacing the occluder’s mask with the occlusion boundary for our input. Unlike [37], we effectively remove from our input any information about the occluder’s spatial extent. Consequently, we do not need to estimate the object ordering graph as in [37].

Recently, [36] proposes to learn shape priors for each category by using modal object bounding boxes. Then the amodal object bounding box is obtained by aligning modal box with learned shape priors. This approach only works well with low-deformation object categories such as car and motorbike of KINS [32] so that we can robustly learn object shape priors. In contrast, ASBU can handle many types of object category as in COCOA [38].

Uncertainty Estimation in Segmentation has a long track record in the literature. Prior work typically estimates aleatoric uncertainty (data uncertainty) [22] and epistemic uncertainty (weight uncertainty) [17], where the former estimates noise in observations and the latter accounts for a distribution of model parameters. For example, in [22], UNet [34] is extended with a variational auto-encoder for aleatoric uncertainty estimation. In [17], estimation of a distribution of the SegNet parameters [1] replaces the common fixed-point parameter estimation. Shape priors have also been studied in the following related work [19, 20, 26, 35].

The section specifies our ASBU. Fig. 4 shows that ASBU uses two distinct sets of inputs for training of UNet to jointly predict the amodal segmentation mask and the associated uncertainty map. These predictions incur an uncertainty-weighted loss function, specified such that our training minimizes both uncertainty and errors in the predicted amodal segmentation.

Our data manipulation of training images, first, randomly samples two objects as occludee and occluder, then, randomly samples a relative position between their modal masks such that the occluder’s mask partially occludes the occludee’s mask, and finally prepares the following two sets of input data:

1. 1.

   (set 1 and set 2) Manipulated training image, where the occluder’s image is superimposed onto the occludee’s image with matting for realistic appearance;

2. 2.

   (set 1 and set 2) Occlusion boundary mask, estimated as an intersection of the morphologically enlarged masks of the occludee and occluder;

3. 3.
   1. (set 1)

      Occluded mask of the occludee, where pixels of the ground-truth modal mask of the occludee covered by the occluder are zero.

   2. (set 2)

      Ground-truth modal mask of the occluder.

Importantly, while both input sets are used in training, ASBU is not aware if the the input segmentation mask comes from the occludee or from the occluder. In this way, ASBU is trained to identify when and how to perform amodal completion of the input segmentation mask. Specifically, for set 1 at the input, ASBU is supposed to learn to extend the input segmentation mask in the region with zero pixels, which is delineated by the input occlusion boundary, because this region is likely to represent the manipulated occlusion. On the other hand, for set 2 at the input, ASBU is supposed to learn not to extend the input segmentation mask in the zero-valued region.

Our uncertainty estimation is aimed at implicitly capturing a “shape prior” of training object instances, which is used for regularizing our learning. Thus, our learning has the following two objectives:

1. 1.

   Minimizing uncertainty of the predicted amodal segmentation, so when uncertainty is estimated as large it captures a truly large variability in plausible shapes;

2. 2.

   Penalizing a difference between the predicted amodal segmentation and the ground truth in an adaptive manner, such that this loss is appropriately reduced when the shape is estimated to come from a highly variable distribution – i.e., when the estimated uncertainty is high.

To this end, our ASBU extends UNet to output a $H{\times}W{\times}2$ feature map which has two channels, one for amodal segmentation prediction and the another for the uncertainty prediction, where $H$ and $W$ denote the height and width of the input image. The predicted values for amodal segmentation, $\{m_{i}:i=1,\dots,N\}$, $N=H\cdot W$, are output by the sigmoid function, so they range in $m_{i}\in[0,1]$. The predicted uncertainty values, $\{u_{i}:i=1,\dots,N\}$, are output by the softplus function, $\text{softplus}(z)=\log(1+\exp(z))$, so they are positive $u_{i}\in\mathbbm{R}^{+}$.

For learning to jointly predict $\{m_{i}\}$ and $\{u_{i}\}$, we specify a new uncertainty-weighted segmentation loss. The prediction of $\{m_{i}\}$ can be supervised by the corresponding original modal mask as it was before the data manipulation $\{m_{i}^{*}\}$. On the other hand, the prediction of $\{u_{i}\}$ remains unsupervised, since there is no ground-truth annotation of uncertainty in our training data. We integrate the mentioned supervised and unsupervised training strategies in the following loss:

	$\displaystyle L=$	$\displaystyle\displaystyle\frac{1}{N}\sum_{i=1}^{N}\mathbbm{1}(m_{i}^{C}=0)L_{i}+\lambda\mathbbm{1}(m_{i}^{C}=1)L_{i},$
	$\displaystyle L_{i}=$	$\displaystyle\displaystyle\frac{1}{2}\left[\left(\frac{m^{*}_{i}-m_{i}}{u_{i}}\right)^{2}+u_{i}^{2}\right],$		(1)

where $m^{C}$ is the mask of the occluder, $\lambda$ is a positive constant, $\mathbbm{1}(\cdot)$ is the indicator function, and $N$ is the number of pixels in the image. We empirically estimate that $\lambda=5$ gives the best results, i.e., we put more weight on predictions inside the occluder mask.

Our loss in (1) is inspired by the Mumford-Shah energy [31], whose minimization is a classical framework for image segmentation. The Mumford-Shah energy has two terms – namely, data term and regularization term. By minimizing a sum of these two terms, the predicted segmentation is encouraged to simultaneously minimize energy and complexity (e.g., favor solutions with a high log-likelihood and smooth boundaries).

Similarly, our loss $L_{i}$ in (1) also has two terms. The first term can be interpreted as the data term for minimizing the energy of a weighted difference between the predicted amodal segmentation $m_{i}$ and ground truth $m_{i}^{*}$. The weighting is inversely proportional to the estimated uncertainty for pixels $i$, such that a lower loss is backpropagated in our training for pixels with higher uncertainty. Since $u_{i}$ is typically higher on object boundaries than over other object parts, the specified weighting effectively accounts for shape variability. The second term in $L_{i}$ can be interpreted as the regularization term for penalizing large $u_{i}$ values. In our experiments, we observe that this regularization favors zero $u_{i}$ values over object regions that are not close to the boundary. Consequently, the data term in $L_{i}$ for such regions introduces a large loss for any errors in the amodal segmentation $m_{i}$, because $u_{i}\approx 0$.

Our loss formulation fundamentally differs from other recent approaches aimed at estimating uncertainty for object segmentation. For example, in [18], segmentation and its uncertainty are assumed as governed by a Gaussian distribution with the following loss function:

$$L_{i}^{\text{Gaussian}}=\frac{1}{2}\left[\frac{(m^{*}_{i}-m_{i})^{2}}{u^{2}_{i}}+\log u^{2}_{i}\right]$$

(2)

In contrast, we do not explicitly specify any probability distribution of box locations. Also, unlike our $L_{i}$ in (1), $L_{i}^{\text{Gaussian}}$ in (2) minimizes uncertainty only when $u>1$. In [29], uncertainty is modeled in the latent space of their variational auto-encoder, and the amodal mask is predicted by sampling multiple latent codes from a Gaussian distribution. In contrast, we predict and regularize our uncertainty map directly in the spatial domain.

Datasets: We evaluate ASBU on COCOA [38] and KINS [32], which are two benchmark amodal instance segmentation datasets with real images derived from the MSCOCO [28] and KITTI [12] datasets, respectively. COCOA consists of 2500, 1323, and 1250 images for training, validation, and test, respectively. There are 2140 object categories that can be divided into two superclasses: stuff (e.g., sky, grass, sea) and things (e.g., dog, cat, human). The content of images is mostly dense with multiple objects occluding one another in cluttered scenes. On the other hand, KINS is a large-scale traffic dataset, which consists of 7474 images for training and 7517 images for testing. There are 7 object categories in KINS including: cyclist, pedestrian, car, tram, truck, van, and miscellaneous vehicles. Scenes in KINS images are less cluttered than in COCOA, and if objects are occluded the occlusion is by mostly one other object. We further randomly divide the KINS training set into training and validation sets with 6000 and 1474 images, respectively. Both datasets provide the ground-truth amodal masks, which we use only for evaluation.

Evaluation Tasks and Metrics: We evaluate ASBU  on three tasks: ordering recovery, amodal completion, and amodal instance segmentation.

For ordering recovery, we estimate the following relationships. Let $({\bf m}_{j},{\bf m}_{j}^{A})$ and $({\bf m}_{k},{\bf m}_{k}^{A})$ denote two pairs of input modal and output amodal masks of two adjacent objects $j$ and $k$, respectively, where $|{\bf m}_{j}^{A}-{\bf m}_{j}|$ and $|{\bf m}_{k}^{A}-{\bf m}_{k}|$ are the extended areas after amodal segmentation of $j$ and $k$. Then, we specify the ordering of $j$ and $k$ as

$$O(j,k)=\left\{\begin{array}[]{rl}0,&\text{ if }\left|{\bf m}_{j}^{A}-{\bf m}_{j}\right|=\left|{\bf m}_{k}^{A}-{\bf m}_{k}\right|=0\\ 1,&\text{ if }\left|{\bf m}_{j}^{A}-{\bf m}_{j}\right|<\left|{\bf m}_{k}^{A}-{\bf m}_{k}\right|\\ -1,&\text{ otherwise },\end{array}\right.$$

(3)

where $O(j,k)=1$ indicates that $j$ occludes $k$. If $j$ and $k$ are not adjacent, we set $O(j,k)=0$. We evaluate our performance on the task of ordering recovery in terms of the average pairwise accuracy, O-Acc, between our predicted ordering relationships $O(j,k)$ and the ground truth relationships $O^{*}(j,k)$, for all pairs $(j,k)$ of adjacent objects.

For amodal completion, we compute the mean intersection-over-union, mIOU, between the predicted and ground-truth amodal masks, as well as invisible mIoU, inv-mIoU, for the predicted and ground-truth occluded regions.

For amodal instance segmentation, we report the common metrics suggested by COCO, including average precision AP for thresholds 50%, 75%, 95%, and average recall AR for top 1, 10, 100 predictions, among others.

We evaluate the following baseline and ablations:

• •

  PCNet-m: our strong baseline from [37].

• •

  Boundary$\to$PCNet-m: in the input to PCNet-m we replace the occluder mask with the occlusion boundary; this tests only our contribution 1 (see Fig. 4), as PCNet-m does not estimate uncertainty.

• •

  Uncertainty$\to$PCNet-m: PCNet-m is extended to predict uncertainty and trained with our uncertainty weighted loss, given by (1), while the input uses the occluder mask as in [37]; this tests our contribution 2 (see Fig. 4).

• •

  uBCE$\to$ASBU: the uncertainty weighted loss, given by (1), is replaced with the following uncertainty weighted binary cross-entropy (uBCE) loss for training our ASBU; this tests our proposed data term in (1):

  $$L_{i}^{\text{uBCE}}{=}\frac{1}{2}[-\frac{m_{i}^{*}\log m_{i}{+}(1{-}m_{i}^{*})\log(1{-}m_{i})}{u_{i}^{2}}{+}u_{i}^{2}].$$

  (4)

• •

  ASBU: our full approach illustrated in Fig. 4.

We have specified a new amodal segmenter with boundary uncertainty estimation (ASBU) for weakly supervised amodal instance segmentation. To address the lack of ground-truth amodal masks, we have trained ASBU on manipulated images to produce pseudo-ground truth amodal masks, and then learned a common instance segmenter, Mask-RCNN, on our pseudo-ground truth. We have made two contributions. First, we have replaced the occluder mask used in prior work [37] for input with the occlusion boundary, and consequently removed the need for one step in [37] – that of estimating the object ordering graph. Second, we have enabled ASBU to estimate uncertainty of the predicted amodal segmentation and proposed a new loss function that uses the estimated uncertainty to regularize learning of ASBU. Our evaluation on the tasks of amodal completion, ordering recovery, and amodal instance segmentation, on the COCOA dataset, demonstrates that ASBU outperforms the state of the art. Specifically, in comparison with a strong baseline PCNet-m [37], our performance improves by $3.5\%$ and $4.5\%$ in mean intersection-over-union (mIoU) for amodal completion and average pairwise accuracy (O-Acc) for ordering recovery, respectively. In amodal instance segmentation, Mask-RCNN trained on our pseudo amodal masks has nearly the same performance as Mask-RCNN trained on the ground-truth amodal masks with a performance gap of $0.1$ in average precision (AP) on the COCOA-test dataset.

Acknowledgement. This work was supported in part by DARPA MCS Award N66001-19-2-4035.