A Novel Patch Convolutional Neural Network for View-based 3D Model Retrieval

Given a 3D model $S$, we first generate a set of 2D greyscale images $X_{S}=\{x_{1},x_{2},...,x_{N}\}$, where $x_{i}$ represents the $i^{th}$ view image and $N$ denotes the number of view images. In this work, we follow the camera setup in the MVCNN (su2015multi, ), which generates 12 rendered 2D view images by setting 12 virtual cameras around the model every 30 degrees. The generated view images are arranged in order. To extract patch features, ResNet34 pre-trained on the ImageNet dataset is employed as the backbone, but the last average pooling layer of ResNet34 is removed. The reason why ResNet34 is utilized as the backbone, is that overfitting often occurs due to lack of sufficient training samples, when ResNet50 or ResNet101 is employed as the backbone. Also when ResNet18 is employed, it cannot well represent each view. Thus, to balance the performance and the number of training samples, ResNet34 is utilized as the backbone. In the experiments, to match the input of ResNet34, each 2D image is first scaled to $224\times 224$, and then these multi-view images are simultaneously fed into the backbone, thus, patch-level features can be extracted for each view. Each patch feature contains information of the corresponding region in the image. Each view image can be represented with $7\times 7$ patch features whose dimensions are $D$-dimension, and the patch features can be denoted as $p_{j}\in\mathbb{R}^{D}$. Thus, we can define $P_{S}=\{p_{1},p_{2},...,p_{M}\}$ as the set of patch features from all multi-view images to denote the 3D model $S$, where $p_{j}$ denotes the $j^{th}$ patch and $M$ represents the number of all patches of all multi-view images. Note that $7\times 7$ patches and the feature dimension of each patch depend on the structure of ResNet34. Since ResNet34 is employed, thus, in our experiments, $N=12$, $D=512$, and $M$ = $7\times 7\times N$.

In the existing view-based 3D model retrieval methods, the relationships between view-level features are fully exploited, but this is not enough. Due to viewpoint variations, view features are very different from each other, in this way, it leads to the difficulty of directly mining relationships between view-level features. Although these views are very different, there might exist more relevant patches within all views. For example, if the patch feature represents the airplane wing in a particular view, we can find other patch features that represent airplane wings in different views through the k-NN algorithm, and combining features from all views can compensate for the information loss in a particular view. Through these more relevant patches, more relational information between views can be learned. Thus, it is more reasonable to utilize patch-level features, rather than view-level features, to capture long-range associations among all multi-view images, which is the key to obtain robust 3D model features.

Inspired by DGCNN (wang2019dynamic, ) and MHBN (yu2018multi, ), we propose a novel neural network module named PatchConv. Instead of directly working on view features, PatchConv exploits patch features and patch coordinates to construct multiple local neighborhood graphs in the patch feature space. The convolution-like operations are then utilized to extract relational information from the edges connecting neighboring patches in the graphs. Through this operation, we can obtain new patch-level features containing information of neighboring patches, which can make patch features more robust and capture long-range associations among all multi-view images. Supposing that there is a patch $p_{i}$, the $k$-nearest neighbors algorithm ($k$-NN) is employed to obtain the $k$-nearest patches among all patch features $P_{S}=\{p_{1},p_{2},...,p_{M}\}\in\mathbb{R}^{M\times D}$. Moreover, a graph $G(V,E)$ is further constructed, where $V=\{1,2,...,k\}$ and $E\subseteq|V|\times|V|$ denote the set of vertices and edges, respectively. In the simplest case, we construct $G$ as the k-nearest neighbors graph, and $E$ represents the edges of the form $(i,j_{1}),...,(i,j_{k})$ from $p_{i}$ to its $k$-nearest patches $p_{j}\in\mathcal{N}(i)$, which denotes the k-nearest neighbors of patch $i$. In PatchConv, we adopt the form $(p_{i},p_{j}-p_{i})$ to represent the features of edges connecting patch $p_{i}$ and its neighboring patches, which combine both the patch feature (captured by $p_{i}$) and local information of neighboring patches (captured by $p_{j}-p_{i}$). Due to viewpoint variations, the position of the model in different views will change, which leads to the differences between patch features in the same position of views. Considering that each patch has its particular position in the views, more information about spatial transformation, which is caused by viewpoint variation, can be obtained by utilizing patch coordinates. Thus, we further introduce $C=\{c_{1},c_{2},...,c_{M}\}\in\mathbb{R}^{M\times 3}$ to represent patch coordinates, which contain spatial information of patches in the original space. Concretely, $c_{i}=(x,y,z)$ denotes the coordinate of the $i^{th}$ patch, and $(x,y,z)$ represents the $x^{th}$ row and the $y^{th}$ column of the $z^{th}$ view image. Thus, the new edge feature can be represented by $(p_{i},p_{j}-p_{i},c_{i},c_{j}-c_{i})$, which contains not only the information of patches in the feature space but also the coordinates of patches in the original space. The PatchConv operation can be described as

where $h$ : $\mathbb{R}^{2\times(D+3)}\to\mathbb{R}^{D+3}$ is a parameterized nonlinear function. Here we implement function $h$ with 1 $\times$ 1 convolution, which extracts the relationships between the central patch and neighboring patches. Finally, we adopt max-pooling as the aggregation operation to generate new patch features, which can capture long-range associations among multi-view images according to the neighborhood information of patches.

Now, each view image is represented by $7\times 7$ patch features, which contain information of neighboring patches. The average pooling layer with kernel size $7\times 7$ is employed to generate view-level features $F_{S}=\{f_{1},f_{2},...,f_{N}\}$, where $f_{i}$ represents the $i^{th}$ view feature and $N$ denotes the number of view features. Since multi-view images are sequential, features of adjacent views are relatively similar. To further mine the relationships between adjacent view images, a 1-dimensional convolution with kernel size $3$ is utilized. In the aggregation stage of view features, max-pooling is a simple strategy to pool the view-wise features into a global model feature. The max-pooling operation only keeps the maximum activation, while the non-maximum values are ignored, which leads to sub-optimal performance. To better utilize discriminative information of different views, we design an attention mechanism to automatically assign different weights to all view images, and its pipeline is shown in Fig. 2. Concretely, the view-pooling feature $g$ is first obtained through max-pooling the view-wise features, which keeps the most prominent feature within all multi-view images. Then, we estimate the similarity score between each view feature and the view-pooling feature with cosine similarity, which is also called self-attention. The cosine similarity can be formulated as

where $f_{j}$ represents the feature of the $j^{th}$ view image. $D$ is the feature dimension of $f_{j}$ and $g$, and $s_{j}(f_{j},g)$ denotes the cosine similarity between $g$ and $f_{j}$. According to the similarity $s_{j}$, the weight $\alpha_{j}$ of the $j^{th}$ view is calculated as follows:

where $N$ is the number of all multi-view 2D images. Moreover, the weights of all multi-view 2D images are further indicated by $\alpha=\{\alpha_{1},...,\alpha_{N}\}$. Finally, based on the self-attention weight of each view, we can obtain the $j^{th}$ weighted view feature $f^{\prime}_{j}$ by

where $f^{\prime}_{j}$ is the specific view feature of the self-attention weight embedded into the view feature. The fusion feature $g^{\prime}$ can be obtained by

where $g^{\prime}$ is the fusion feature, which is the weighted sum of the view features.

In the 3D model retrieval task, the softmax loss (krizhevsky2012imagenet, ) is usually utilized as the basic loss to increase the feature discriminability. Recently, some loss functions, such as center loss (wen2016discriminative, ) and triplet-center loss (he2018triplet, ), have been employed in view-based 3D model retrieval methods to ensure that 3D model features from the same classes are pulled closer and that 3D model features from different classes are pushed away.

We note that only the 3D model feature is utilized when computing the loss values in these works, while the view features are ignored, which leads to the loss of view information. Although a single view feature cannot effectively represent the whole 3D model, it contains abundant information of the 3D model in a particular view, which can make the 3D model features easier to distinguish from other 3D models. Since each view feature can also describe the 3D model from other perspectives, we hope that these multi-view images can also be accurately recognized. To tackle this problem, a novel discrimination loss function is designed, which consists of softmax loss values generated by the fusion classifier and the specific classifier, to simultaneously classify the 3D model and each view image. To prove its effectiveness, only the simplest softmax loss is employed in the fusion classifier and specific classifier. The discrimination loss function can be defined as follows:

where $L_{Dis}$ is the discrimination loss function. $L_{model}$ indicates the loss value generated by the fusion classifier when the model feature is utilized, and $L_{views}$ denotes the loss value generated by the specific classifier when all multi-view features are used. $\beta$ and $\gamma$ are the hyperparameters used to control the contributions of $L_{model}$ and $L_{views}$.

In the calculation of $L_{views}$, the mean of losses of all multi-view features is first used, and we name it as the average view loss (AVL), which is shown as follows:

where $L_{views}^{j}$ represents the loss value of the $j^{th}$ view image. In the experiments, we observe that some images are difficult to classify, while some images are easy to recognize. If the network can correctly recognize these difficult images, it is easier for it to classify the 3D model. Thus, we let the network focus on recognizing these difficult images. In detail, the view weight $\alpha$ in Eq.(3) is first taken as the loss weight $\alpha^{\prime}$, and then, the loss weight $\alpha^{\prime}_{j}$ of the $j^{th}$ view image is set to $(\frac{2}{N}-\alpha^{\prime}_{j})$, where N denotes the number of views. In other words, when the view weight of the $j^{th}$ image is small, the image feature is very different from the view-pooling feature, and it is difficult for the specific classifier to recognize it. Therefore, we hope that the specific classifier can pay more attention to difficult views by giving their loss values larger weights. In this way, if the difficult images can be classified, the model can be easily recognized. Thus, the weighted value of $L_{views}$ can be calculated by

We name the weighted calculation of $L_{views}$ as the weighted view loss (WVL)

To evaluate the performance of the proposed PCNN, we have conducted 3D model retrieval experiments on two public 3D model datasets including ModelNet40 (wu20153d, ), and ModelNet10 (wu20153d, ). ModelNet40 contains a total of 12,311 models from 40 categories, and ModelNet10 contains a total of 4,899 models from 10 categories. In our experiments, we strictly follow the training/testing splits in MVCNN (su2015multi, ). In PCNN, ResNet34 is utilized as the backbone. When converting the 3D model to 2D images, we strictly follow the experimental setup of MVCNN (su2015multi, ), and 12 virtual cameras around the model are placed every 30 degrees to generate 12 2D images. To make the network learn spatial relationships among views, the generated views are sequential. To match the input of ResNet34, each 2D image is scaled to 224 $\times$ 224. In PatchConv, $k$ of the $k$-nearest neighbors algorithm is set to 12, which means that each new patch feature aggregates the information of the 12 nearest neighbor patches. In the implementation of PatchConv, the dimension of patch features is 515 after introducing patch coordinates, and the dimension of edge features is 1,030. $1\times 1$ convolution is employed to extract relational information from edge features, whose input dimension is 1,030, and the output dimension is 515, followed by batch normalization and LeakyReLU activation function. The parameter of LeakyReLU is set to 0.2. In the optimization of PCNN, Adam (Kingma2014Adam, ) is employed as the network optimizer, where the momentum is set to 0.9. To maintain the training stability, we clip the gradients into the range [-0.01, 0.01]. The learning rate is set to $4\times 10^{-5}$. We train the network for 30 epochs with a mini-batch size of 16. In the calculation of discrimination loss, since both the fusion classifier and specific classifier are very important, hyperparameters $\beta$ and $\gamma$ are empirically set to 0.5 and 0.5, respectively. Note that $\alpha$ doesn’t need to be optimized, and it can be automatically calculated by the cosine similarity between the view-pooling feature $g$ and the feature of $j^{th}$ view image, and we also rerank the retrieval list according to classification results. Besides, in our experiments, two public evaluation metrics including Mean average precision (mAP) and precision-recall (PR) curve (feng2018gvcnn, ; gao20113d, ) is utilized.

In this section, we conduct a comprehensive comparison and evaluation of PCNN. Specifically, we first compare PCNN with state-of-the-art methods on two widely used datasets, the ModelNet40 and ModelNet10 datasets. Then, we present some retrieval examples for the ModelNet40 dataset. Finally, the PR curves are also given. Their results are shown in Table 1, Table 2, Fig.3, and Fig. 4, from which we make four key observations.

I) As shown in Table 1, we compare our method with competing view-based 3D retrieval methods on the ModelNet40 dataset, including SPH (kazhdan2003rotation, ), LFD (Chen2003On, ), 3D ShapeNets (wu20153d, ), MVCNN (su2015multi, ), GIFT (bai2016gift, ), GVCNN (feng2018gvcnn, ), SeqViews2SeqLabels (han2018seqviews2seqlabels, ), View N-gram (he2019view, ), 3D2SeqViews (han20193d2seqviews, ), MDPCNN (Gao2020Multiple, ), IMVCNN (He2020Improved, ) and MLVCNN(jiang2019mlvcnn, ). When compared with these state-of-the-art methods, PCNN can obtain the best performance on both datasets. For example, in Table 1, when comparing PCNN with traditional methods, its performance is 60.37%, 52.77%, and 44.47% higher than that of SPH, LFD, and 3D ShapeNets, respectively. Similarly, when comparing PCNN with deep learning methods, its performance is 13.47%, 11.73%, 7.97%, 4.58%, 2.91%, 6.01%, 3.57% and 0.83% higher than that of MVCNN, GIFT, GVCNN, SeqViews2SeqLabels, 3D2SeqViews, MDPCNN, IMVCNN and MLVCNN, respectively.

II) As shown in Table 2, we compare our method PCNN with competing view-based 3D retrieval methods on the ModelNet10 dataset, including SPH (kazhdan2003rotation, ), LFD (Chen2003On, ), 3D ShapeNets (wu20153d, ), GIFT (bai2016gift, ), SeqViews2SeqLabels (han2018seqviews2seqlabels, ), 3D2SeqViews (han20193d2seqviews, ), PANORAMA-ENN (sfikas2018ensemble, ), View N-gram (he2019view, ) and IMVCNN (He2020Improved, ). When compared with state-of-the-art methods, PCNN also can obtain the best performance on both datasets. For example, in Table 2, when comparing PCNN with traditional methods, its performance is 52.18%, 46.41%, and 27.97% higher than that of SPH, LFD, and 3D ShapeNets, respectively. Similarly, when comparing PCNN with deep learning methods, its performance is 5.11%, 4.80%, 4.11%, 2.95%, 3.43%, and 3.23% higher than that of GIFT, SeqViews2SeqLabels, 3D2SeqViews, PANORAMA-ENN, View N-gram and IMVCNN, respectively.

III) In these competitors, the latent relationships among different views are explored in different ways, for example, the view-pooling operation is proposed in the MVCNN to generate the model feature, and the GVCNN distinguishes different views to mine relationships among view images. However, these issues are separately discussed, and the patch information of each view image is ignored. In our proposed PCNN, the patch convolution layer is designed to capture long-range associations among all multi-view images, and an adaptive weighted view layer is further embedded into the PCNN to automatically assign a weight to each view. Finally, a discrimination loss function is employed to extract the discriminative model feature. Most importantly, these three modules are integrated into a unified framework. Thus, the performance of PCNN is much better than those of MVCNN and GVCNN, and the improvements are 13.47% and 7.97%, respectively. Moreover, we can observe the same conclusions in Table 2 when the ModelNet10 dataset is utilized.

IV) In Fig. 3, the retrieval examples of PCNN and MVCNN are given, where the results of the search error are indicated with red boxes. We observe that these 3D models are very similar and that it is even difficult for people to identify them. For example, when the desk is the query (the first row in Fig.3), it is often confused with the table. Similarly, when the sofa is used as the query (the second row in Fig.3), it is often confused with the bed. Since PatchConv in PCNN is used to capture long-range associations among all multi-view images and a discrimination loss function is designed, the feature discriminability is improved. Thus, when PCNN is employed, the extracted features have very good discrimination. Although there are few retrieval errors in Top10, the number of red boxes is very small. In contrast, when MVCNN is utilized, the feature discriminability needs to be improved, and there are many retrieval errors in Top10. Thus, the proposed PCNN is very effective and efficient. Besides, the precision-recall (PR) curves of different modules on the MonelNet40 dataset are also given in Fig.4.

In this section, we will assess the benifits of PatchConv and AWV. The network architecture of MVCNN is first employed, and then different backbones including VGG-m and ResNet34 are utilized in the MVCNN, and they are named MVCNN-VGG-m and MVCNN-ResNet34, respectively. To verify the validity of PatchConv and AWV, PatchConv, AWV, and the combination of both are separately embedded into the MVCNN. Finally, the performance is evaluated on the ModelNet40 dataset. The results are shown in Table 3. Note that when PatchConv, AWV, and the combination of both are embedded into MVCNN-ResNet34, they are named MVCNN-ResNet34 (PatchConv), MVCNN-ResNet34 (AWV), and PCNN (PatchConv+AWV), respectively. Note that only $L_{model}$, which is generated by the fusion classifier, is used.

In Table 3, we observe that MVCNN-ResNet34 can outperform MVCNN-VGG-m, whose improvement reaches 1.5%. In other words, when ResNet34 is utilized as the backbone of MVCNN, its retrieval accuracy is slightly better than that of VGG-m. Based on MVCNN-ResNet34, PatchConv is utilized to mine the intrinsic relationships among all different views. From Table 3, we can see that the performance of MVCNN-ResNet34 (PatchConv) is much higher than that of MVCNN-ResNet34, and the improvement achieves 4.58%. Thus, PatchConv is very effective in mining the intrinsic relationships among different views according to patch features and patch coordinates. Also, an adaptive weighted view layer is embedded within MVCNN-ResNet34. From Table 3, we can also observe that the performance of the MVCNN-ResNet34 (AWV) is much higher than that of MVCNN-ResNet34, and the improvement reaches 7.46%. Thus, the AWV is very efficient to automatically assign a weight to each view by the similarity between the view feature and the view-pooling feature. Besides, when both PatchConv and AWV are embedded into the MVCNN-ResNet34, the performance of PCNN (PatchConv+AWV) can be further improved.

In PatchConv, the patch coordinates are also used. In fact, this is inspired by methods based on point clouds. The point cloud is another representation of 3D models. Related methods take the coordinates of points as input and project coordinates into high-dimensional space. In the process of multi-level feature extraction, these methods combine coordinates and low-level features to generate high-level features with a multilayer perceptron. In this work, the position of the object in different images will change due to view variations, which leads to the differences between patch features in the same position. With patch coordinates, we can obtain more information about spatial transformation caused by view variation. As shown in Table 3, the performance of PCNN (EdgeConv+AWV) is 92.14%, in which patch coordinates are not utilized. The performance of PCNN (PatchConv+AWV) is 92.65%, which is 0.51% higher than that of PCNN (EdgeConv+AWV).

To verify the validity of the discrimination loss, the model loss (ML for short) is first used in MVCNN-ResNet34 and PCNN, and they are marked as MVCNN-ResNet34 (ML) and PCNN (ML). The specific view loss is then added into MVCNN-ResNet34 (ML) and PCNN (ML), and their corresponding names are MVCNN-ResNet34 (ML+AVL) and PCNN (ML+AVL), where AVL indicates the average view loss in Eq.(7). Besides, when the weighted view loss (WVL) shown in Eq.(8) is utilized, we refer to it as PCNN (discrimination loss). The results are shown in Table 4. From the findings, we can observe that when AVL is used in MVCNN-ResNet34, the mAPs of MVCNN-ResNet34 (ML) and MVCNN-ResNet34 (ML+AVL) are 81.7% and 88.39%, respectively, and the improvement achieved is 6.69%. Similarly, when employed in PCNN, the mAPs of PCNN (ML) and PCNN (ML+AVL) are 92.65% and 93.24%, respectively. When the discrimination loss is further used, its performance can be further improved. This result, therefore, proves the effectiveness of discrimination loss. Also, when comparing PCNN (discrimination loss) with MVCNN-ResNet34, the improvement can reach 11.97%. This further proves the effectiveness of PatchConv, AWV and discriminative loss. These modules make the network able to capture the intrinsic relationships among different views, automatically fuse all multi-view images and extract more discriminative model features.

To further prove the effectiveness of discrimination loss, we also visualize the results of MVCNN-ResNet34, MVCNN-ResNet34 (ML+AVL), PCNN (ML), and PCNN (discrimination loss) on the ModelNet10 dataset, and their visualization results are shown in Fig. 5. The reason why we choose the ModelNet10 dataset is that the number of categories in ModelNet40 is 40, thus, it is very difficult to clearly show the results. From Fig. 5(b) and Fig. 5(d), we can observe that points from the same category are aggregative, but points from different categories are dispersed. Thus, these model features can be distinguished easily.

In this section, we assess the convergence of the proposed PCNN on the ModelNet40 dataset. Moreover, we compare it with state-of-the-art approaches, and their convergence curves are shown in Fig. 6. From the results, we can see that although PatchConv and AWV are embedded into MVCNN-ResNet34, the convergence speed is also slightly quicker than those of MVCNN-ResNet34 and MVCNN. Thus, the modules we proposed are very efficient. Further, when the view loss is added into PCNN (ML), its convergence speed is slightly slower than those of PCNN (ML), MVCNN-ResNet34, and MVCNN. The reason for this result is that we classify the 3D model correctly and that all multi-view images are required to be accurately recognized in PCNN (ML+AVL) and PCNN (discrimination loss). However, PCNN (ML+AVL) and PCNN (discrimination loss) can quickly converge after only 20 epochs, which further proves the effectiveness of PCNN.