3DRM: Pair-wise Relation Module for 3D Object Detection

Traditional networks for 3D object detection mainly leverage geometric features of objects to regress the bounding box and conduct classification. Different from that, our Relation Module is aimed for learning the pair-wise objects relationships to extract relation features, which fills a gap in features of 3D data representations. The goal of this paper is to apply the proposed 3DRM to the existing popular detection pipelines with point clouds as input and improve the final performance.

In our method, we leverage two detection frameworks to generate object candidates and extract their features: proposal-based methods and voting-based methods. With features of objects extracted by the backbones as input, our 3DRM can build up pair-wise object relations and extract the comprehensive relation features. As a result, the relation feature will be concatenated with input feature to help the task of detection (Section 3.2). Strategies about application of our Relation Module on different backbones are demonstrated in Section 3.3. Designs for loss function about different backbones are illustrated in Section 3.4.

Different from existing work on 3D object detection which extract contextual information by taking the entire scene as input, our method learns the object-level relational context features and infers attributes between object pairs. We argue that the relation between object pairs is beneficial to object reasoning for 3D scene understanding. Motivated by the Relation Networks proposed in [17], we adapt the relation module to 3D object detection task. Unlike the strategy that applies relation prediction on the cells of feature map, we perform the relation prediction on individual objects, so object relations are explicitly obtained. On the other hand, our goal of relation reasoning is not to aggregate the global context feature for predicting the attributes of the entire scene. Since we aim to detect individual objects, our relation reasoning module is essentially learning the relation-related feature for individual objects.

The architecture of 3DRM is shown in Figure 2. The input of our relation module is the object candidates with their features $\textbf{o}_{i}\in\mathbb{R}^{d}$ generated by backbone methods. We propose to use a deep network to extract relational features and predict the relations of a pair of objects. Specifically, for each object $\textbf{o}_{i}$, we randomly choose $k$ objects $\textbf{o}_{j}$ in the same scene, which then compose several pairs. In Eq. (1), the pairwise function $g_{\theta}$ aims to exploit the semantic or spatial relations between $\textbf{o}_{i}$ and $\textbf{o}_{j}$, and then $f_{\phi}$ fuses the relations followed by an element-wise sum for all $\textbf{o}_{j}$. As a result, $\textbf{r}_{i}$ is the learned relational feature of $\textbf{o}_{i}$.

where both $i$ and $j$ are the indexes of the objects in the same scene; $g_{\theta}$ and $f_{\phi}$ are the functions implemented by MLPs.

At the same time, we also predict the relationships of each pairs of objects. The output of the pairwise function $g_{\theta}$ will be sent to the classification MLPs to predict the relation label $l_{{rn}}$. Note that $h_{\varphi}$ is the function implemented by MLPs.

We design different classifiers for different relations. There are four types of relations which include semantic and spatial information: group, same as, support and hang on. Relations of different components within an object are omitted since we argue that pair-wise object relations are more significant to indoor object detection. The reason why we choose these four types of relations is that these relations are typical in indoor scenes and sufficient for attaining useful relation features for detection intuitively. For the sake of efficiency and practicability, relations like facing or contain are not under consideration in this paper because indoor 3D object detection is usually aimed for representative objects like chairs and tables instead of bottles of wine in the cabinet. In what follows, we describe how to compute the relation labels of each pair.

Semantic relations. Generally, there are many objects belonging to the same category in the same scene. For example, in most cases, couples of chairs often simultaneously exist in a conference room. We argue that semantic information is beneficial to detection tasks. In this paper, semantic information covers two relations: group and same as.

Group relations learn the potential connection between objects that have the same categorical class label and learn the diversity of the different types of objects. We try to capture the relations between objects in terms of the semantic class-specific properties. Distinguishing semantic class relations from various objects benefits the classification, which is equal to answer the question that whether two objects have the same categorical label or not.

Same as relations indicate that a pair of objects may belong to the same instance even if they cover different parts of the object. This type of relation gets the candidate objects belonging to same instance closer while keeps other objects away. Actually this is an exploration of the instance’s intrinsic property.

Spatial relations. Objects in the same scene are potentially connected in the field of space, especially for those with 3D representations. Spatial information indeed implies abundant and useful information, which is helpful for better understanding of the scene with our relation network. We divide spatial relations into two canonical relations: support and hang on. The algorithm for spatial relation computation is illustrated in Algorithm 1.

Support relations describe the spatially adjacent relations between the objects. In other words, it emphasizes that objects are functionally close and connected with this relation. Automatically extracting relations of support is quite challenging due to the noisy and partial occlusion of real 3D scans. We define that two objects have relations of support only when they are close enough on the z-axis and IoU between their projection in the horizontal plane is large enough. Furthermore, if object A is on top of object B and ground projection of object A against the one of object B is higher than a certain threshold, we argue that object A and object B have support relations. For example, a flower vase standing on the table describes the relation of support.

Specifically, when deciding the support relations of two proposals, we first check if 1) their relative height $\Psi_{z}(p_{i},p_{j})$ between the lower surface of one object and the upper surface of the other object is smaller than the threshold $\tau_{z}$, and 2) the overlapping ratio in xy-plane $\Omega_{xy}(p_{i},p_{j})$ surpasses the threshold $\tau_{xy}$. If so, they will have relations of support.

where $p_{i}$ and $p_{j}$ are two objects, $\nu^{z}(p_{i})$ and $\nu^{z}(p_{j})$ denote their position on z-axis for points of them. $\Psi_{z}(p_{i},p_{j})$ is their minimum distance on the z-axis.

where $\delta_{xy}(\cdot)$ computes the intersection area of projection for two objects. $\beta_{xy}(\cdot)$ denotes the size of projection area. $\Omega_{xy}(p_{i},p_{j})$ indicates the larger overlapping ratio of the IoU towards these two projection areas.

Hang on relations imply that two proposals are horizontally adjacent and one hangs on the other. Similar to relations of support, if object A is horizontally close to object B and perpendicular projection (parallel to xz-plane or yz-plane) of object A against the object B is higher than a certain threshold, we argue that object A and object B have hang on relations. For example, one is classified as wall and the other is an object that can hang on the wall, like board, lamp, curtain, etc. The relations of these kinds of object pairs can be regulated as hang on. Such compact spatial relations are helpful for detection and understanding of the scene.

In order to apply our 3DRM to existing popular detection pipelines and verify its effectiveness and generalization, we design specific strategies to plug our 3DRM into two mainstream methods: proposal-based method and voting-based method. The detection pipeline including our Relation Module is shown in Figure 3. It is composed of two main parts, backbones for processing raw point cloud, and Relation Module for reasoning pair-wise object relations. Both the integrated and relation features are concatenated together to perform the classification and regression towards numerous candidate bounding boxes. Followed by 3D non-maximum suppression (NMS), the pipeline outputs classified and qualified 3D bounding box. We introduce how we apply 3DRM to different 3D detection frameworks as following.

Proposal-based methods. Lots of detection methods establish a baseline system by introducing region proposals as object candidates and classifying the objects as well as regressing the bounding box which we called proposal-based methods. These methods have achieved promising results but ignore the relation information between object candidates, so we aim to equip these methods with our 3DRM to improve the performance. Since there is no widely used framework for proposal-based methods, we design the whole backbone by ourselves. We choose over-segmentation method described in [44] for raw proposal generation, object hypothesis generation module for filtering low quality proposals, PointCNN [28] as feature extractor for point clouds and contextual features for enriching features of objects. Above all is the baseline for proposal-based methods in this paper.

With 3D point clouds as input, we first perform an over-segmentation on the input point cloud as described in [44]. The over-segmented patches are then merged recursively by a bottom-up fashion. The output is a binary hierarchy in which each node is a potential object (segment). We take the node segments as the initial proposals. To improve the quality of raw proposals, we first start from an object hypothesis generation module by filtering proposals with low objectness. This is achieved by using a deep neural network based on PointCNN and MLPs, predicting the objectness labels of the proposals.

With selected and reliable proposals, each proposal can be recognized as a candidate object. Moreover, for improving the baseline performance of detection, like many of the previous works on object detection [12, 35], we add the context information by exploring the points around the object candidate with radius $R$ to extract enriched contextual feature.

Both of the original geometry features and enriched features, which extract corresponding information from the object itself and its surroundings, consist of the integrated feature for each object candidate. After that, we apply our 3DRM to predict the pair-wise object relations and extract relation features.

Then we perform concatenation of multiple features learning from different aspects including geometric features, context features and relation features. After the concatenation, we feed the concatenated features into MLPs to predict the categorical label and regress the bounding of detection box. Finally, a 3D non-maximum suppression (NMS) is used to remove the redundant proposal candidates and obtain the final 3D objects with their bounding boxes.

Voting-based methods. VoteNet [37] is an end-to-end 3D object detection network based on a synergy of deep point set networks and Hough voting. Similar to classical Hough voting, VoteNet generates votes that lie close to objects centers, and then these votes are grouped and aggregated as clusters to generate box proposals. Each cluster can be regarded as an object candicate which can also be utilized to capture the relation information by our 3DRM.

Specifically, with $N\times 3$ point cloud as input, VoteNet first subsamples $M\times(3+C)$ seed points by a backbone network. Note that $C$ is the extended feature dimensions and $M$ is the sample number. Each seed point then goes through a voting module, predicts the offset to its object center and thus becomes a vote point for potential clusters. Furthermore, all the votes will be grouped into $K$ clusters each with dimension $(3+C)$.

At this stage, clusters with their features are sent to our 3DRM to extract the enriched $C_{r}$-dimensinal relation feature vector. Similar to proposal-based methods, we take the concated features $K\times(3+C+C_{r})$ as the input for the following detection modules. In this way, the inference of the final 3D bounding boxes and the object classes will consider the compatibility with the relations, which makes the final prediction more reliable. In the following steps, We keep the same proposal and classify module as VoteNet to generate final 3D bounding boxes. More details will be described in the Section3.4.

The loss for our 3DRM is simply formulated as $\mathcal{L}_{{rn}}$ using the binary cross entropy. In this way, it is judged independently whether an object pair should have a certain relation. As for the detection pipeline, different backbones lead to diverse designs for the final loss function.

Proposal-based methods. The network can be trained in an end-to-end manner with a multi-task loss including a semantic classification loss of the object candicate, a 3D bounding box regression loss and a classification loss of relations. We weigh the losses with the parameter $\lambda_{1},\lambda_{2},\lambda_{3}$ to make sure they are in similar scales. In our experiments, we set $\lambda_{1}=1.0,\lambda_{2}=10,\lambda_{3}=0.5$.

Voting-based methods. The network is trained in an end-to-end manner with a multi-task loss including a voting loss, an objectness loss, a 3D bounding box estimation loss, a semantic classification loss and a relation loss. It is worthy to note that the objectness loss is designed to help the proposal module to generate good enough proposals. The component losses except the voting loss are weighted by $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$. In our experiments, we set $\lambda_{1}=0.5,\lambda_{2}=1.0,\lambda_{3}=0.1,\lambda_{4}=0.1,$.

We leverage a widely used dataset that provides 3D point clouds of indoor scenes: Stanford large-scale 3D Indoor Spaces Dataset S3DIS [2] for proposal-based pipeline. S3DIS is from real scans of indoor environments which contains 3D scans from Matterport scanners in 6 areas including 271 rooms. The objects in this dataset are divided into 13 categories. We perform a k-fold cross validation across areas [50].

Both of ScanNetV2 [9] and SUN RGB-D [48] are leveraged to evaluate the voting-based pipeline. ScanNetV2 is an RGB-D video indoor scene dataset with richly annotated 3D reconstructed meshes. It contains about 1.5K scans annotated with both semantic segmentation and object instance labels for 18 categories. Since it doesn’t provide reconstructed point clouds and oriented bounding boxes, we sample the reconstructed meshes and predict axis-aligned bounding boxes in the same way as VoteNet.

SUN RGB-D is a large single-view RGB-D dataset for scene understanding. It contains about 10K RGB-D images captured by four different sensors with accurately annotated oriented bounding boxes for 37 object categories. Note that since it doesn’t provide point cloud data, we first convert the depth images to point clouds using known camera parameters.

We use average precision as our evaluation metric of the detected object bounding boxes against the ground truth bounding boxes. We use two $IoU$ thresholds as 0.5 and 0.25 respectively in our experiments. The mean average precision (mAP) is the macro-average on average precision across all test categories.

In this section, we denote the detection framework proposed in Section 3.3 as baseline named OSegNet which utilizes proposals generated by over-segmentation and PointCNN [28] as backbones. Applying our 3DRM to OSegNet is denoted as OSegNet+RM. We first compare our method with OSegNet on 3D object detection. We also compare our method to state-of-the-art methods and analyze the difference and gap. After that, we conduct the ablation studies to evaluate the impact of each component in our approach. Last, we demonstrate the qualitative results of our method.

Comparison to baseline and State-of-the-art methods. We evaluate our method against several prior works and our baseline OSegNet which detects 3D object in indoor scenes with point cloud as input:

• •

  Sliding PointCNN [28]: A baseline which contains a PointCNN backbone and detect objects in a 3D sliding window fashion.

• •

  PointNet [39]: A method that first predicts the category of all points and then uses a breadth-first search to group nearby points with the same category.

• •

  SGPN [52]: A semantic instance segmentation approach for point clouds by using an embedding learning network for point pairs.

• •

  VDRAE [44]: A variational auto-encoder that detects 3D objects in indoor scene by using a hierarchical structure.

Table 1 reports the average precision on the S3DIS dataset using 6-fold cross validation across six areas with [email protected]. Compared to the baseline OSegNet, our method OSegNet+RM obtains 54%, 36%, 26% and 29% increase on chair, table, sofa and mAP respectively, which proves the efficiency of our Relation Module. Note that, limited to the performance of over-segmentation method for proposal generation, there are very few proposals with good quality for board objects, resulting in low performance on this category. While all methods are learning-based methods and there is a lack of valid proposals on board category, our method still achieves the best performance. Especially, our method get a huge improvement (33% increase) compared to the state-of-the-art on chair category and 4% increase on mAP, thanks to our 3DRM proposed in Section 3.2 and illustrated in Figure 1. Moreover, without relation prediction and using only the relation features, OSegNet+RM- still surpasses OSegNet by a large margin, proving the effectiveness of the relation features extracted by our method.

Ablation study. We evaluate the impact of each component of our approach to investigate the efficiency of different relations and relation features. Note that, all experiments for the ablation studies are trained on Area2$\sim$Area6 and tested on Area1. Board category is eliminated from comparison due to poor quality of proposals generated by OSegNet.

Specifically, we first analyze the contribution of different relations to our 3DRM. The ablation results are shown in Table 2. While our method OSegNet+RM(all) which predicts all relations at one time achieves the best performance on table and sofa with mAP of 0.62 on three categories. OSegNet+RM using only group relations achieves the highest AP 0.72 on category of chair. We argue that, on S3DIS dataset, spatial structures of objects like table and sofa are various and complex to distinguish, leading to the dependence of both semantic and spatial relations. Shapes of chairs are relatively fixed, which relies more on semantic relations like group.

As for the selection mode of object pairs, we also conduct the ablation study about random mode and nearest mode. Experiments are trained on Area2$\sim$Area6 and tested on Area1. Table 3 illustrates that random selection outperforms selecting object pairs according to the nearest euclidean distance on all terms. Specifically, random selection surpasses the nearest selection by 5% on mAP. The improvement shows that object pairs randomly selected provide more information for the network to learn, while the pairs selected nearestly can be regarded as a kind of local information.

Visualization of relation prediction. Figure 4 visualizes the relations between objects in the same room. Both of semantic and spatial relations provide rich context information to help detection. Different objects belonging to the same categories implies that they should have similar shapes. If objects belonging to the same instance have overlapping bounding boxes, this helps the network to regress the box correctly. For spatial relations, such explicit information provides extra knowledge to accomplish the task of detection.

Qualitative examples. Figure 5 shows several qualitative results of object detection on S3DIS dataset. The proposed Relation Module leverages multiple features from relations to help detection module classify the objects in 3D bounding boxes. We visualize the result of OSegNet (first column and fourth column), our method OSegNet+RM (second column and fifth column) and ground truth (third column and last column). It is demonstrated that our method is capable of detecting objects in cluttered scenes and regressed bounding boxes are more accurate and distinct than OSegNet.

We first compare our method with the baseline VoteNet on 3D object detection in 3D point clouds. Results justify the effectiveness and practicality of the proposed 3DRM. After that, we conduct extensive ablation studies to evaluate the impact of each component in our approach. Lastly, we demonstrate the qualitative results of our method.

Comparison to VoteNet. We evaluate our method against VoteNet in 3D object detection. Quantitative results on ScanNet and SUN RGB-D are summarized in Table 4. We apply the proposed Relation Module to the representative VoteNet and denote the network as VoteNet+RM as our method. Note that we take the performance VoteNet+RM with semantic relations only as the final results since it performs the best on these two datasets. Our method significantly outperforms VoteNet by not only 1.1% and 3.8% on ScanNet, but also 1.4% and 1.4% on SUN RGB-D in terms of mAP with IoU=0.25 and IoU=0.5 respectively. Note that, our method increase the performance of VoteNet by $3.8\%$ on [email protected] which illustrates that our 3DRM can not only mitigate ambiguity but also increase accuracy of the detection. Furthermore, we argue that this benefits from the enriched relation features from our 3DRM, which provides comprehensive understanding to the object and its surrounding environment. More quantitative results are shown in appendix.

Ablation study. Extensive ablation studies are performed to verify the increased accuracy of our approach. Note that we combine group and same as relations as semantic relations and support and hang on as spatial relations. To prove the efficiency of the proposed 3DRM, we first study how the semantic and spatial relations help the task of 3D object detection for different categories. Applying our 3DRM to VoteNet without predicting relation labels denoted as VoteNet+RM- is also considered. Results on ScanNet are shown in Table 5.

The results show that VoteNet+RM with only semantic relations achieves the best performance with an increase of 3.34% in terms of [email protected]. The reason is that objects of most categories are sensitive to semantic relations, and can obtain more context information from semantic relations. For example, objects, such as windows, sofas, fridges, toilets and so on, have simple and clear spatial structure, and thus need various objects and context to help understanding. Objects, such as counters, pictures, doors, baths and cabs, are usually placed in a complex environment where semantic information is rich enough and spatial information are critical to them since their structures are more complicated. Moreover, some categories of objects benefit from both semantic and spatial relations like shower, chair, desk, etc. Note that applying our 3DRM to VoteNet without predicting relation labels and only use the relation features also outperforms VoteNet, which proves that relation features extracted by our 3DRM do help detect 3D objects better.

As for the mode of selecting object pairs, we compare two ways of selection: random mode and nearest mode. Comparison results are illustrated in Table 6, It is clear that random selection of object pairs achieves higher performance than selecting several nearest objects to form relation pairs. This is because random selection can provide various object pairs distributed in the whole scene and thus enrich the information around objects to improve the detection quality.

Qualitative results and discussion. The qualitative results on ScanNet are shown in Figure 6. Ours method detects the objects more accurately and robustly, which is beneficial from our 3DRM. It is noteworthy that detection results of VoteNet are confused with other objects and ambiguous in some areas with noisy point cloud, while ours can classify and locate the objects precisely and clearly without redundant bounding boxes. We argue that this is attributed to the pair-wise relation reasoning. Details are shown in the second row of Figure 6 where red rectangles on the left refer to the ambiguous and wrong detections on chairs by VoteNet. Green rectangles on the right demonstrate the accurate detections by ours.

Figure 7 shows the detection results on ScanNetV2 val set. From the comparison of Ours and VoteNet, we can detect the objects accurately and robustly with less ambiguity. Specifically, in some cluttered areas, ours can distinguish different objects and regress the bounding boxes precisely. There are usually many chairs in scenes like offices and it is quite common to misunderstand the chairs as other categories due to noise and their various appearance. Our 3DRM can help alleviate this problem by using relation reasoning and thus achieve better detection results.

Complexity and computational efficiency. Our 3DRM is consisted of semantic relations and spatial relations including group, same as, support and hang on. Among these, relations like group, same as are easy and fast to compute since we only need to compare semantic or instance labels. The computational time for semantic relations can be ignored. As for spatial relations, Algorithm 1 can illustrate the complexity. Note that although we use loops for simplication in the algorithm, we actually use matrix multiplication for further acceleration in experiments. Moreover, the computational time and complexity grows as the number of object pairs is larger. Numerically, we test the computational efficiency of spatial relations for 2048 proposals in ScanNet dataset. We sample 8 object pairs for each proposal, which means $8*2048$ object pairs in total. The total time is around $0.047s$ and the average time is approximately $3\times 10^{-6}s$ for each object pair, which proves the high efficiency of our 3DRM.

With regard to the number of object pairs for each object which may have influence on the complexity of relation computation, we perform experiments by comparing the contributions of different pair numbers to the detection with VoteNet+RM on ScanNet dataset. Results are shown in Figure 8. Explicitly, sampling 8 or 14 object pairs for an object achieves the most improvement considering both [email protected] and [email protected]. Since these contributions of these two numbers of pairs are almost the same, we build 8 object pairs for each one for the balance of computational efficiency and performance.

Improvement on different pipelines. We have evaluated the improved performance of detection on mAP for both OSegNet and VoteNet to justify the generalization of our 3DRM on detection pipelines with different basic performance. Explicitly, applying 3DRM to OSegNet obtains 29% improvement on [email protected] on S3DIS while VoteNet gets 3.8% on ScanNet and 1.4% on SUN RGB-D. OSegNet is an intuitively simple baseline implemented mainly with over-segmentation for proposal generation and PointCNN as backbones. Initial proposals generated by over-segmentation in OSegNet are relatively less organized and accurate compared to VoteNet which relies on deep hough voting. The network architecture of OSegNet is much simpler than VoteNet. Less organized proposals and simpler architecture result in lower basic performance for OSegNet. However, this actually demonstrates the generalization and effectiveness of our 3DRM by being able to help attain a huge improvement on detection for simple pipelines, and comparable improvement even for comprehensive pipelines.