Unseen Object Instance Segmentation with Fully Test-time
RGB-D Embeddings Adaptation

To modulate features during test time, a learning objective based on the model’s predictions of test data is typically required. Recent works [8, 27] have demonstrated the effectiveness of using the entropy objective based on discriminative outputs (e.g., classification probabilities) [8, 27]. However, unlike the standard recognition model, UOIS is conducted in an open-world setting, and thus does not explicitly train a discriminative layer that generate logits and probabilities for the direct calculation of entropy. To address this problem, we propose a novel non-parametric entropy objective (NEO). Specifically, NEO leverages the unsupervised clustering to obtain centroids that present pseudo instance labels, then calculates non-parametric classification probabilities to construct the entropy objective.

Unsupervised Clustering Given a bunch of RGB-D embeddings on a feature map, we aim to cluster all pixels into groups to segment unseen objects. But the number of unseen objects is uncertain, which prevents the usage of clustering algorithms with a known number of clusters such as $k$-means or spectral clustering. Thus, we follow UCN [2] to use the mean shift [28] clustering algorithm with the von Mises-Fisher (vMF) distribution [29], which automatically discovers the number of objects and generates a segmentation mask for each object. After the unsupervised clustering, we can calculate the centroid of each cluster. The $i$-th cluster’s centroid vector $\mathbf{c}_{i}$ is obtained by averaging all feature map vectors $\mathbf{v}_{x,y}$ which belong to the $i$-th cluster $\mathbf{C}_{i}$ as

$\displaystyle\mathbf{c}_{i}=\mathrm{avg}(\mathbf{v}_{x,y})~{}\mathrm{for~{}all}~{}\mathbf{v}_{x,y}\in\mathbf{C}_{i},~{}i=1,2,...,n$

(1)

where $x$ and $y$ denote locations on the feature map along the $x$-axis and $y$-axis. After performing the average operation for each cluster, we obtain the set of all centroids $\mathbf{c}$ as

$\displaystyle\mathbf{c}=\{\mathbf{c}_{1},\mathbf{c}_{2},...,\mathbf{c}_{n}\},$

(2)

where $n$ is the number of objects, which is estimated by the unsupervised clustering algorithm.

Non-parametric Classification Probability Different from the classification with standard supervised learning, UOIS segments objects in the instance-level, thus having an uncertain number of “classes”, which prevents the usage of the parametric classifier. Inspired by recent self-supervised learning methods with instance discrimination [30, 31], we propose to calculate classification probabilities in a non-parametric way.

Without specifying the number of classes, we conduct non-parametric classification by using the metric between the candidate feature vector $\mathbf{v}$ and the $i$-th cluster’s centroid vector $\mathbf{c}_{i}$ as

$\displaystyle P(i|\mathbf{v,c})=\frac{\exp(s(\mathbf{v},\mathbf{c}_{i}))}{\sum_{i}\exp(s(\mathbf{v},\mathbf{c}_{i}))},~{}s(\mathbf{v}_{1},\mathbf{v}_{2})=\frac{\mathbf{v}_{1}^{\top}\mathbf{v}_{2}}{\|\mathbf{v}_{1}\|\|\mathbf{v}_{2}\|},{\color[rgb]{0,0,0}~{}i\in N_{k},}$

(3)

where $s(\cdot,\cdot)$ is the cosine similarity to measure how well $\mathbf{v}$ matches the $i$-th cluster/object, $N_{k}$ is the set of numbers to indicate candidate $\mathbf{v}$’s corresponding cluster and $k$ nearest clusters.

Entropy Objective As an unsupervised objective, the Shannon entropy [7] is widely used and demonstrated to be effective for assessing the output error and shift [8]. With the proposed non-parametric classification probability, we can calculate the Shannon entropy as

$\displaystyle L_{neo}=-\sum_{i}P(i|\mathbf{v,c})\log_{2}P(i|\mathbf{v,c}),$

(4)

where $P(i|\mathbf{v,c})$ is the probability that $\mathbf{v}$ is recognized as the $i$-th cluster given a collection of centroids $\mathbf{c}$.

Compared to the “perfectly” generated RGB-D data, real-world RGB-D images have inevitable noise, such as the black holes in the depth images. The fusion of RGB-D features generally makes the network more stable to such noise [2, 32]. Therefore, while testing on realistic RGB-D images, the privileged [33] multimodal (i.e., RGB-D fused) features can be utilized to enhance the unimodal (i.e., RGB or depth) networks. Based on this motivation, we propose to distill the knowledge from the multimodal features to the unimodal ones to encourage the test-time knowledge transfer for feature enhancement. This cross-modality knowledge distillation (CKD) constructs another objective for test-time adaptation. In CKD, the full multimodal network (two-stream CNN) is set as the teacher, while the student is the smaller partial unimodal network (one-stream CNN).

We use soft targets (i.e., probabilities of the input belonging to the classes) as the distilled knowledge in CKD. For each candidate feature vector $\mathbf{v}$ on the teacher feature map, the $i$-th soft target is

$\displaystyle P(i|\mathbf{v,c,}~{}T)=\frac{\exp(s(\mathbf{v},\mathbf{c}_{i})/T)}{\sum_{i}\exp(s(\mathbf{v},\mathbf{c}_{i})/T)},$

(5)

where $T$ is the temperature factor to control the importance of each soft target. Due to the absence of a parametric classifier, here we use the non-parametric probability as similar with Equation 3.

For the individual RGB and depth modality (i.e., the student), we use the same spatial cluster assignment as the teacher, but they hold different representations $\mathbf{v}^{rgb}$ and $\mathbf{v}^{d}$, thus producing different cluster centroids $\mathbf{c}^{rgb}_{i}$ and $\mathbf{c}^{d}_{i}$. Then, the soft targets of RGB and depth modality, i.e., $P(i|\mathbf{v}^{rgb},\mathbf{c}^{rgb},~{}T)$ and $P(i|\mathbf{v}^{d},\mathbf{c}^{d},~{}T)$, can be obtained similarly to Equation 5. Finally, we formulate the cross-modality knowledge distillation objective as

	$\displaystyle L_{ckd}=$	$\displaystyle\frac{1}{2}(KL(P(i|\mathbf{v,c,}~{}T),P(i|\mathbf{v}^{rgb},\mathbf{c}^{rgb},~{}T))+$
		$\displaystyle KL(P(i|\mathbf{v,c,}~{}T),P(i|\mathbf{v}^{d},\mathbf{c}^{d},~{}T))),$		(6)

where $KL(\cdot,\cdot)$ denotes the Kullback-Leibler divergence loss. The gradients propagated through the multimodal distribution are stopped. By optimizing Equation 6, the response knowledge of the multimodal teacher network is distilled into the unimodal student network.

By minimizing the above entropy objective $L_{neo}$ and distillation objective $L_{ckd}$, we can adapt our model fully in test time. However, tuning all parameters like that in the training phase is inefficient and could easily cause model instability [8]. As the channel of the feature map can be considered as a feature detector [35, 36], re-calibrating channel responses has been widely studied and utilized in network pruning [37], multimodal fusion [38, 39], representation learning [40, 41], etc. Besides, previous works [42, 43, 44, 45] on unsupervised domain adaptation (UDA) share a similar idea, i.e., utilizing statistics of the BN layer as domain-related knowledge. However, the UDA methods require the use of source or target data to conduct the adaptation in advance, thus becoming difficult to be applied during test time or online. Differently, we use the channel-wise affine transformation provided by the BatchNorm (BN) [11] layer to stabilize the test-time adaptation and aim for an effective fusion of RGB-D embeddings.

In genral, the BN layer is widely used in deep learning to eliminate internal covariate shift and improve generalization. It performs a linear transformation followed by the convolutional or fully-connected layers. We denote by $\mathbf{x}_{m,l,k}$ the $k$-th channel for the $l$-th layer feature maps of $m$-th modality (RGB or depth in our setting), then the transformation of the BN layer can be written as

$\displaystyle\mathbf{x}^{\prime}_{m,l,k}=\gamma_{m,l,k}\frac{\mathbf{x}_{m,l,k}-\mu_{m,l,k}}{\sqrt{\sigma^{2}_{m,l,k}+\epsilon}}+\beta_{m,l,k},$

(7)

where the scaling and shift factors $\gamma_{m,l,k}$ and $\beta_{m,l,k}$ are adjustable affine parameters, the normalization statistics $\mu_{m,l,k}$ and $\sigma_{m,l,k}$ are updated with momentum.

Thus, by adapting the scaling and shift factors $\gamma_{m,l,k}$, $\beta_{m,l,k}$ and updating the statistics $\mu_{m,l,k}$, $\sigma_{m,l,k}$ of the BN layer, the proposed method does not introduce new parameters. Given the simulation-trained model, our method is independent of re-training on the large-scale synthetic data. Meanwhile, the fully test-time adaptation re-calibrates channel-wise responses of RGB and depth feature maps, thus providing better weighted fused RGB-D embeddings for unseen object instance segmentation. Finally, the overall objective for test-time adaptation can be formulated as

$\displaystyle L_{total}=\lambda_{1}L_{neo}+\lambda_{2}L_{ckd},$

(8)

where the two terms $L_{neo}$ and $L_{ckd}$ are weighted by the balancing parameters $\lambda_{1}$ and $\lambda_{2}$.

Datasets The model is pre-trained on the synthetic Tabletop Object Dataset (TOD) [1], which consists of 40k synthetic scenes of cluttered objects in tabletop environments. The proposed method is evaluated and compared on two real-world datasets, named OSD [12] and OCID [13]. OSD consists of 111 images in tabletop environments with averaged 3.3 objects per image. OCID has 2,346 images on both tabletop and floor with averaged 7.5 objects per image. It is worth noting that OSD has manually labeled segmentation masks while OCID generates semi-automatically labeled annotations, which are easily influenced by the noise of depth images.

Evaluation Metrics We follow previous works [1, 2, 4] to use the object precision/recall/F-measure (Overlap P/R/F) metrics to evaluate the object segmentation performance, and the Boundary P/R/F to evaluate how sharp the predicted boundary matches against the ground truth boundary. Besides, [email protected] is used to denote the percentage of segmented objects with Overlap F-measure $\geq 75\%$ [2]. Though the IoU is a standard metric in segmentation tasks, it is highly correlated to the overlap F-measure, thus is not reported in the UOIS task [3]. All P/R/F and [email protected] measures are reported in the range of $[0,100]$.

In this section, we first compare the proposed method with several state-of-the-art methods. As shown in Table I, our method outperforms all competitors on both OSD and OCID datasets. Specifically, FTEA+¹¹1FTEA+ denotes adopting the zoom-in refinement strategy in [2]. achieves 89.5 and 92.3 Overlap F-measure, 73.8 and 86.7 Boundary F-measure, 88.3 and 91.1 [email protected], on OSD and OCID respectively, which validates the effectiveness of the proposed approach. Additionally, when we use the end-to-end model without extra zoom-in refinement, i.e., FTEA in the next-to-last line in Table I, the improvement of our method is consistent. Compared to UCN [2], the proposed FTEA improves the Overlap F-measure by 2.4 and 1.0, the Boundary F-measure by 4.6 and 0.3, the [email protected] by 5.1 and 1.8, on OSD and OCID respectively.

Non-parametric Entropy Objective This objective is crucial for UOIS to conduct test-time adaptation. As shown in Table II, when equipped with the non-parametric entropy objective $L_{neo}$, the model’s performances on two real-world datasets are overall improved. Especially, $L_{neo}$ significantly improves the overlap and boundary recall, which is desirable for discovering potential objects in unseen environments.

Cross-modality Knowledge Distillation The proposed cross-modality knowledge distillation provides another effective learning objective $L_{ckd}$ for test-time feature enhancement. Table II shows that $L_{ckd}$ works well on improving the overlap and boundary precision of segmentation results. Additionally, as shown in the last row in Table II, the cross-modality knowledge distillation $L_{ckd}$ can be effectively combined with the former entropy objective $L_{neo}$, thus further improving the performance on most evaluation metrics for unseen object instance segmentation.

Test-time Adaptation Since the test-time adaptation (TTA) needs to be enabled with at least one proposed learning objective, i.e., NEO or CKD, the ablation of TTA is implicitly included in the first row in Table II. From Table II we can observe that whether TTA is in conjunction with the $L_{neo}$ or $L_{ckd}$, the segmentation results on both OSD and OCID can be significantly improved.

Adaptation Consumption Besides the performance, we further investigate the computation consumption of the proposed method. The adaptation time only accounts for single backward pass of BN parameters, thus is way faster than the inference with multiple zoom-in operations [2]. Table III shows the averaged time consumption of the adaptation process and inference, i.e., averaged 0.04s v.s. 1.24s per iteration. Based on the setting of only adapting 500 iterations as stated in Section IV-E, the total adaptation consumption for a new scenario is $\sim$20s (0.04s per frame), which is efficient for practical application.

Qualitative Results We visualize some qualitative results with and without the proposed FTEA in Figure 3. We can observe in Figure 3(a) that the proposed adaptation process mitigates the under-segmentation problem of two close objects. Besides, different from the smooth depth images in synthetic training data, realistic depth images are generally noisy, especially on the object’s boundary. This problem makes outputs of boundaries blurred and causes over-segmentation around the boundary, as illustrated in Figure 3(b). The last row in Figure 3(b) shows that, with the proposed adaptation process in FTEA, this problem can be largely alleviated.

Robustness to the Sampling Order Additionally, we train our models for multiple runs on different random sampling orders. Figure 3 illustrates the cumulative means of performance with $95\%$ confidence intervals across 20 repeated runs on OCID and OSD. It can be observed that the overall performances of the model are stable, demonstrating the robustness of the proposed method. Note that the experiments on OCID also use random samples (not just the sampling order) since we conduct the adaptation with only 500 test-time images out of 2,346 images in OCID.

Modalities for Adaptation Table IV shows results with different adaptation modalities on OSD. First, segmentation performances can be improved whether RGB or depth modality is tuned. Second, by simultaneously tuning RGB and depth modalities, we can obtain better-fused RGB-D embeddings. Thus the performances become significantly better (see the last row in Table IV) due to the effective use of multimodal data.

Modulation Parameters We also conduct analysis on the modulation parameters, i.e., the linear BN and the nonlinear convolution parameters. As shown in Table IV, using BN as modulation parameters achieves the best results. Since the nonlinear high dimensional convolution layers are difficult to optimize, tuning all model parameters could lead to a negative transfer.

How Many BN Layers to Use For brevity, we split all layers into four main building blocks as in ResNet [46]. Figure 4 illustrates the segmentation performances on OSD when we gradually increase the number of BN layers for adaptation. We can observe that the Overlap F-measure, Boundary F-measure, and [email protected] are consistently improved with more BN layers, demonstrating the effectiveness of the proposed method. On the other hand, the improvement is not plateaued, which implies that the adaptation process can be further enhanced by considering other effective parameters in addition to the BN layers of the model.

For fair comparisons, we follow previous work [2] to use a 34-layer, stride-8 ResNet (ResNet34-8s) as the backbone, and the full resolution $640\times 480$ feature map with embedding dimensions $C=64$ is obtained by bilinearly upsampling. To avoid noisy outliers, we set $k=1$ in Equation 3 (i.e., select two clusters, the nearest cluster and the corresponding cluster) for the calculation of the non-parametric entropy objective $L_{neo}$. The temperature factor $T$ in cross-modality knowledge distillation is 1. The weight factor for the overall loss is set as $\lambda_{1}=\lambda_{2}=1$. During test time, our model is adapted with the SGD optimizer. We use batchsize=1 as in the typical inference phase. For the first 100 iterations (images), the learning rate is linearly warmed up to the base value $lr=0.005$, then decayed with a cosine scheduler for another 400 iterations (images). We do not set the “epoch” number since there is no concept of “dataset” in the online test-time adaptation. We use the same learning schedule for the OSD and OCID. The data in test-time adaptation are not shuffled unless otherwise stated. All experiments are conducted on a single NVIDIA 2080Ti GPU with PyTorch.