Investigating Attention Mechanism in 3D Point Cloud Object Detection

Non-local [39] block is spatial attention that uses a weighted sum of features to represent a pixel. Particularly, the weights are estimated as the long-range dependencies (i.e., inner produces) between the pixels, by which the learned feature maps can be further enhanced with both local and global information.

Criss-cross attention [14] is another spatial attention module, which exploits the criss-cross pixels to obtain the contextual information of a certain point effectively. Compared to the Non-local block, the criss-cross attention saves memory and achieves comparable performance in the meantime.

Squeeze and Excitation (SE) block [13] is channel attention that intends to refine the features by exploiting the inter-dependencies between channels. At first, the SE block squeezes the spatial information and generates the channel-wise descriptors; then, it learns the weights for different channels by exciting the descriptors with convolutions and activations.

Convolutional Block Attention Module (CBAM) [41] is mixed attention that consists of two sequential attention blocks. To be specific, a channel attention block is leveraged to capture the channel-wise information based on the inter-channel relationship of features. In practice, it utilizes the sum of space-wise average-pooled and max-pooled descriptors to encode a channel attention map. In addition, the spatial attention block exploits the inter-spatial relationship of features for complementary context. Differently, the spatial attention block applies the average-pooling and max-pooling operations along the channels, then concatenates the pooled features to generate a spatial attention map.

Moreover, Dual-Attention [8] is another mixed attention structure exploiting both channel-wise and spatial-wise long-range dependencies of features to enhance the discriminant ability. Notably, this structure utilizes a position attention module and a channel attention module in parallel. To be more specific, the position attention module can model the dependencies between positions in a feature map following a similar manner of the Non-local [39] block. In contrast, the channel attention module directly estimates the dependencies between the channels without any convolution. Finally, the two modules are aggregated using simple element-wise summation to generate the final feature map.

Attentional ShapeContextNet (A-SCN) [43] introduces a self-attention-based module to exploit the shape context-driven features in the 3D point cloud. By comparing the query and key matrices, the attention map is estimated as the point-wise similarities. The output is then calculated as a matrix product between the attention map and the value matrix, together with an additional skip connection from the value matrix.

Point-Attention [7] module also follows the basic structure of self-attention to captures more shape-related features and long-range correlations from the point space of local point graphs. Additionally, it applies a skip-connection to strengthen the relationship between the input and output.

Channel Affinity Attention [28] estimates the attention map between the channels by calculating the channel-wise affinities in a self-attention structure. Specifically, it utilizes a compact channel-wise comparator block and a channel affinity estimator block to compute the similarity matrix and affinity matrix.

Recently, inspiring by the success of transformers [16] in the 2D image, researchers also realize the transformer-based networks for point cloud analysis, in order to heavily exploit the attention mechanism as the basic point feature learning module. For example, Offset-Attention [10] is proposed to estimate the offsets between the input and attention features, which are calculated from a self-attention structure. Mainly, Offset-Attention leverages the robustness of relative coordinates in transformations and the effectiveness of the Laplacian matrix in graph convolution.

Moreover, Point Transformer [51] is designed to take advantage of the local geometric relations between the center point and its neighbors. Using basic MLP operations, the Point Transformer block can effectively aggregate a local feature for each point based on the learned attention weights for its neighbors. With the help of rich local and geometric context, this method achieves outstanding performances in both point cloud classification and segmentation tasks.

We evaluate the performances of different attention modules on two datasets, SUN RGB-D [34] and ScanNetV2 [5], which are both captured from the indoor scenes in real-world using RBG-D cameras. To be more specific:

• •

  SUN RGB-D: There are 5,285 training and 5,050 testing RGB-D images in the dataset, where each object is precisely annotated with a bounding box and one of 35 semantic classes. According to the provided camera parameters, the original data and annotated bounding boxes can be projected as 3D point clouds. Following a widely used experimental setting [23], we only use the 3D coordinates as input and report the average precision (AP) and recall of the ten most common classes, together with the overall metrics of mean average precision (mAP) and average recall (AR).

• •

  ScanNetV2: The original dataset contains the reconstructed meshes from 18 object categories, where 1,201 samples are for training and 312 samples are in the validation set. The point cloud data is sampled from the vertices of reconstructed meshes. Besides the usage in segmentation, we input the 3D coordinates and predict the bounding box and category of each object in the same evaluating protocol as [23].

In general, all attention modules in our work are adopted and slightly modified from the available official implementations. For 2D attentions, we regard the spatial space of 2D image as the point space of point cloud, following relation of $H\times W=N\times 1;$ where $H$ and $W$ are the height and width of an image while $N$ is the number of points. Moreover, to achieve a stable training process in 3D cases, we replace the original $1\times 1$ convolutions in 2D attentions with the MLP operations if necessary. For fair comparisons, the reduction factor in all attention modules is empirically set as 8. As for the recent Point Transformer [51] whose code is not released yet, we reproduce the structure according to the related descriptions in the paper.

The implementations are realized by PyTorch and Python platforms on a single Tesla-P100 GPU using CUDA and Linux operating system. All the experiments adopt the similar training settings such as the learning rate of 0.001, the batch size of 8, and a total of 180 training epochs. Following the default configurations in [23], in SUN RGB-D [34] dataset, the number of input points is 20,000; while in ScanNetV2 [5] dataset, the input size is 40,000. Coupled with this paper, we will release the source code of all deployed attention modules.

By applying different attention modules in our attentional backbone, we conduct the experiments of 3D point cloud object detection on SUN RGB-D [34] dataset and ScanNetV2 [5] dataset, respectively.

Table 1 and 2 present the detailed detection results in SUN RGB-D [34] dataset under the metrics of average precision (AP), mean average precision (mAP), recall and average recall (AR). Although we can notice the different effects by using different attention modules, as shown in Table 1, the SE [20] method realizes the best overall result (59.6% mAP) among all tested attention modules and significantly exceeds the baseline’s result (57.5% mAP) by 2.1%. In terms of each category’s AP, the SE [20] method achieves the highest values in four out of ten object categories. Meanwhile, the results in Table 2 can also verify the outstanding performance of SE [13] under the metrics of recall. In comparison to the complicated self-attention methods, a compact attention structure like in SE [20] or CBAM [41] is able to benefit the point cloud detection task effectively and efficiently. Moreover, we observe that the channel-related information plays a crucial role in the attention mechanism of the point cloud, since the effectiveness of SE [13], CBAM [41] and CAA [28] are more prominent than the spatial attention modules.

In Table 3 and 4, we further compare the performances of different attention modules under a challenging condition of more input points and detected object categories using ScanNetV2 [5] dataset. Even though SE [13] and CBAM [41] can still provide relatively good performances evaluated under either precision- or recall-related metrics, the Point Transformer [51] achieves the best overall result (59.1% mAP) among all the ten tested attention modules. Mainly, the advantages of Point Transformer [51] can be concluded from two sides: on the one hand, it incorporates more local context for each point rather than a single feature representation learned from a shared MLP; on the other hand, the attention map is estimated from the geometric relations in 3D space, while most of the rest methods only calculate the dependencies from the feature space.

In addition to the average precision and recall evaluated under an IoU threshold of 0.25 in Section 4.3, we provide the overall evaluations, mean average precision (mAP) and average recall (AR), under an IoU threshold of 0.5.

In general, Table 5 shows the similar results as in Table 1 and 2, where the SE [13] and CBAM [41] methods achieve better mAP and AR scores under both the IoU thresholds of 0.25 and 0.5 in SUN RGB-D dataset. Moreover, it is worth noting that when the IoU threshold is set at 0.5, Point Transformer [51] achieves the improvements of 4.3% mAP and 3.6% AR compared to the baseline results, since it can integrate more local information for the object detection task in dense point cloud scenes in ScanNetV2 [5] dataset.

Apart from VoteNet [23], we conduct more experiments by utilizing our attentional backbone in BoxNet [23] and MLCVNet [42]. In general, the attentions show similar effects with them as when tested with VoteNet. More experimental data can be found in the supplementary material.

Table 6 presents some reference data regarding a model’s complexity. By adding different attention modules in the backbone of VoteNet [23], the model size of whole network may increase more or less depending on the number of parameters used in each attention module. Although some attentions, e.g., Point Transformer [51] and CAA [28], can achieve relatively higher performances, they require more computational resources such as longer training time or larger memory consumption. In terms of the inference time, all tested models can perform at a similar level. However, as Point Transformer [51] needs an additional local neighbor searching operation compared to other methods, the efficiency of its inference process will be affected. Alternatively, we may integrate more global perception in both spatial and channel domains as in CBAM [41] and SE [13], to better balance the effectiveness and efficiency.

Recall that in the VoteNet pipeline, the most critical usage of its backbone is to generate the votes (yellow points) that are expected to approach the centroids (red points) of detected objects. Therefore, the generated votes can intuitively indicate the quality of the backbone’s output.

To this end, in Figure 4, we compare the votes that are generated from different attentional backbones. In the sub-figure of the baseline, we can see that the votes can be easily attached to the most significant object’s (the middle one) centroid, while there are fewer votes around the centroids of two smaller objects. As for the sub-figure of the SE method, it can be clearly observed that more votes have been centralized at the centroids of two smaller objects (especially for the left object), providing more confident estimations on the detected objects bounding boxes. To further visualize the effects of different attention modules, in the supplementary material, we also compare the point features learned from our attentional backbone.