MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation

Let us denote the set of unlabeled shapes as $\mathcal{X}_{u}$. Each shape instance $X\in\mathcal{X}_{u}$ can be rendered from a viewpoint $i$ into color, normal and depth images denoted as $V^{i}$. We use a 2D CNN backbone $\mathbf{\Phi}$ which maps each view into pixel-wise embeddings $\{\mathbf{\Phi}(V^{i})_{p}\}\in\mathbb{R}^{D}$, where $p$ is an index of a pixel and $D$ is the dimensionality of the embedding space. We pre-train the network ${\mathbf{\Phi}}$ using the following self-supervised loss:

where $\mathcal{R}({X})$ is the set of 2D renderings of shape $X$, and $V^{i}$ and $V^{j}$ are sampled renderings in different views from $\mathcal{R}({X})$. The self-supervision loss $\ell_{\textsf{ssl}}$ is applied to the pair of sampled views $V^{i}$ and $V^{j}$.

Since $V^{i}$ and $V^{j}$ originate from the 3D mesh, each foreground pixel in the rendered images corresponds to a 3D point on the surface of a 3D object. We find matching pixels from $V^{i}$ and $V^{j}$ when their corresponding points in 3D lie within a small threshold radius. We use the obtained dense correspondences in the self-supervision task. Specifically, we train the network to minimize the distance between pixel embeddings that correspond to the same points in 3D space and maximize the distance between unmatched pixel embeddings. This encourages the network to learn pixel embeddings to be invariant to views, which is a non-trivial task, as two rendered views of the same shape may look quite different, consisting of different contexts and scales. We use InfoNCE [33] as the self-supervision loss. Given two rendered images ($V^{i}$ and $V^{j}$) from the same shape $X$ and pairs of matching pixels $p$ and $q$, the InfoNCE loss is defined as:

where $\mathbf{\Phi}(V^{i})_{p}$ is the embedding of pixel $p$ in view $i$, $M$ is the set of paired pixels between two views that correspond to the same points in 3D space, and the temperature is set to $\tau=0.07$ in our experiments. We use two views that have at least $15\%$ overlap. The output $\mathbf{\Phi}(V^{i})_{p}$ and $\mathbf{\Phi}(V^{j})_{q}$ of the embedding module are normalized to a unit hyper-sphere. Pairs of matching pixels are treated as positive pairs. The above loss also requires sampling of negative pairs. Given the matching pixels $(p,q)\in M$ from $V^{i}$ and $V^{j}$ respectively, for each pixel $p$ from the first view, the rest of the pixels $k\neq q$ appearing in $M$ and belonging to the second view, yield the negative pixel pairs $(p,k)$.

In the fine-grained shape segmentation stage, the network learns to predict pixel level segmentation labels. Once the embedding module is pre-trained using the self-supervised approach, it is further fine-tuned in the segmentation stage, using a small labeled shape set $\mathcal{X}_{l}$ to compute a supervised loss, as follows:

and $\mathbf{\Theta}$ is the segmentation module, $\lambda$ is a hyper-parameter set to $0.001$, and $\ell_{\textsf{sl}}$ is the semantic segmentation loss implemented using cross-entropy loss applied to each view of the shape separately. $\mathcal{R}(X,Y)$ is the set of renderings for shape $X$ and its 3D label map $Y$. $L^{i}$ represents the projected labels from the 3D shape for view $V^{i}$. Since the labeled set is much smaller than the unlabeled set, the network could overfit to the small set. To avoid this over-fitting during the fine-tuning stage, we use the self-supervision loss ${\cal L}_{\textsf{ssl}}$ as an auxiliary loss along with supervision loss ${\cal L}_{\textsf{sl}}$ as is shown in Eq. 3. In Table 5 we show that incorporating this regularization improves the performance.

During inference we render multiple overlapping views of the 3D shape and segment each view. The per-pixel labels are then projected onto the surface. We use ray-tracing to encode the triangle index of the mesh for each pixel. To aggregate labels from different views for each triangle, one option is to use majority-voting. An illustration of the process is shown in Fig. 2. However, not all views should contribute equally towards the final label for each triangle as some views may not be suitable to recognize a particular part of the shape. We instead define a weighted voting scheme based on the average entropy of the probability distribution predicted by the network for a view. Specifically, a weight $W^{(i)}=(1-\sum_{p\in F^{(i)}}H^{(i,p)}/|F^{(i)}|)^{\gamma}$ is given to the view $i$, where $F^{(i)}$ is its set of foreground pixels, $H^{(i,p)}$ is the entropy of the probability distribution predicted by the network at pixel $p$, and $\gamma$ is a hyperparameter set to $20$ in our experiments. More weight is given to the view with less entropy. Consequently, for each triangle $t$ on the mesh of the 3D shape, the label is predicted as: $l_{t}=\operatorname*{arg\,max}_{c\in C}\sum_{i\in I,p\in t}W^{(i)}P^{(i,p)}$, where $I$ is the set of views where the triangle $t$ is visible, $P^{(i,p)}$ is the probability distribution of classes at a pixel $p\in t$ in view $i$, and $C$ is the set of segmentation classes. In cases where no labels are projected to a triangle due to occlusion, we assign a label to the triangle by nearest neighbor search.

The embedding $\mathbf{\Phi}$ is implemented as the DeepLabV3+ network [7] originally proposed for image segmentation with ResNet-50 backbone. We add extra channels in the first layer to incorporate depth and normal maps. Specifically, it takes a K-channel image ($V^{i}$) as input of size $H\times W\times K$ and outputs $\mathbf{\Phi}(V^{i})$ per pixel features of size $H\times W\times 64$, where the size of pixel embedding is $64$. In the second stage, we add a segmentation head (a 2D convolutional layer with a softmax) on top of the pixel embedding network to produce per-pixel semantic labels. Additional architecture details are provided in the supplementary material.

To generate the dataset for the self-supervision stage, we start by placing a virtual camera at 2 unit radius around the origin-centered and unit normalized mesh. We then render a fixed number of images by placing the camera at uniform positions and adding random small perturbations in the viewing angle and scale. In practice, we use approximately $90$ rendered images per shape to cover most of the surface area of the shapes. We also render depth and normal maps for each view. Normal maps are represented in a global coordinate system. Depth maps are normalized within each view. We use ray tracing to record the triangle index to which each pixel corresponds to and also the point-of-hit for each pixel. This helps in identifying correspondences between two views of the same shape. More information is provided in the supplementary material.

We compare our method against the following baselines:

• •

  (2D) Scratch. In this baseline, we train our 2D networks ($\mathbf{\Phi}$ and segmentation head $\mathbf{\Theta}$) directly using the small set of labeled examples, without pre-training, and apply the same multi-view aggregation.

• •

  (2D) ImageNet. We create a baseline in which the backbone ResNet in our embedding network $\mathbf{\Phi}$ is initialized with ImageNet pre-trained weights and the entire network is fine-tuned using few-labeled examples. The first layer of ResNet trained on Imagenet is adapted to take the extra channels (normal and depth maps) following the method proposed in ShapePFCN [26].

• •

  (2D) DenseCL. To compare with the 2D dense contrastive learning method we also create a baseline using DenseCL [45]. We pre-train this method on our unlabeled dataset $\mathcal{X}_{u}$ using their original codebase. Once this network is trained, we initialize the backbone network with the pre-trained weights and fine-tune the entire architecture using the available labeled set. Further details about training these baselines are in the Supplement.

• •

  (3D) Scratch. In this baseline, we train a 3D ResNet based on sparse convolutions (Minkowski Engine [10]) that takes uniformly sampled points and their normals from the surface and predicts semantic labels for each point. We train this network directly using the small set of labeled examples.

• •

  (3D) PointContrast. To compare with the 3D self-supervision methods, we pre-train the above 3D ResNet (Minkowski Engine) on $\mathcal{X}_{u}$ using the approach proposed in PointContrast [47]. The pre-trained network is later fine-tuned by adding a segmentation head (a 3D convolution layer and softmax) on top to predict per-point semantic labels. We use the codebase provided by authors to train the network. Details are provided in the Supplement.

• •

  (3D) Weak supervision via learned deformation. We use the approach by Wang et al. [43], which uses learning-based deformation and transfer of labels from the labeled set to unlabeled shapes. We use our labeled and unlabeled set to train this method using the code provided by the authors.

Our self-supervision is based on enforcing consistency in pixel embeddings across views for pixels that corresponds to the same point in 3D. In Figure 3 we visualize correspondences using our learned embeddings between human subjects from the RenderPeople dataset in different costumes and poses. The smoothness and consistency in correspondences implies that the network can be fine-tuned with few labeled examples and still perform well on unseen examples.

To evaluate our method on the textured dataset, we use the RenderPeople dataset. We use the same set of 2D and 3D baselines as described in §4.3. Note that, we provide color and normal with point cloud input to 3D Scratch and 3D PointContrast. In addition to the settings described in §4.2, we analyze the effect of different inputs given to the network, i.e. when only RGB images are used as input for self-supervision and fine-tuning, and when both RGB images and geometry information (normal + depth maps) are available for self-supervision and fine-tuning. We train all baselines in these two settings, except 3D baselines that take geometry by construction. The results are shown in Table 4.