2nd Place Solution for IJCAI-PRICAI 2020 3D AI Challenge: 3D Object Reconstruction from A Single Image

Voxel representations are a straightforward generalization of pixels to 3D case. Unfortunately, the memory cost for voxel representations grows cubically with resolution, therefore, it can only generate $32^{3}$ or $64^{3}$ voxels. While it’s possible to reduce memory cost by using special data structures such as octrees, this structure leads to complex algorithms and is still limited to relatively small $256^{3}$ voxel grids.

Point-based representations are widely used both in computer graphics and robotics. Pointnet Qi et al. (2017) pioneered point clouds as a representation for discriminative deep learning tasks. They achieved permutation invariance by applying a global pooling operation. As a result, a category of techniques pioneered by PointNet that express surfaces as point clouds, and techniques pioneered by Atlas that adopt 2D-to-3D mapping process Groueix et al. (2018). But these methods do not guarantee watertight results and hardly scale beyond a hundred vertices

Implicit functions representations model shapes as learnable indicator functions rather than samples, as in the case of voxel methods. The resulting networks treat reconstruction as a classification problem and are universal approximators whose reconstruction precision is proportional to the network complexity. However, at inference time, these methods still require the execution of an expensive iso-surfacing operation whose performance effected by the desired resolution. Compared with other methods, implicit functions representations can achieve higher visual quality.

Overview. Fig. 1 shows an overview of our method. For single-view 3D reconstruction, Actually, we did not follow the idea from AtlaNet to first train an autoencoder by only using point cloud data, then fix the parameters of the decoder when training SVR. We replace the original AtlasNet’s decoder with FoldingNet’s decoder. In our experiments, we adopted a more radical approach by only training the whole network to minimize the mean CD loss between the predicted and ground truth point clouds.

Backbone. We used ResNet He et al. (2016) as our image encoder to obtain 1024D features from $224\times 224$ images. In addition, all relu layers in ResNet were replaced with leakly-relu layers.

Data pre-processing. For data pre-processing, since there are multi-view images and the scales of point clouds are different, We use the official code to rotate the point cloud to the related image’s view and then normalize the point cloud. After normalization, a trade-off is between performance and computation is down-sampling the original point cloud to 2048 points. Also, we add a small Gaussian noise to each point to improve the robustness of our network. For image augmentation, we only keep resizing images to 224, which is different from AtlasNet.

Decoder design. To satisfy the submission requirements, the number of points in the predicted point cloud is 2048. so that the primitives of the decoder are eight, and the number of each decoder’s layer is 3. The input latent vector of the $Decoder2Dto3D$ is obtained from the image encoder. We replicate it $n$ times and concatenate the latent $n$-by-1024 matrix with an $n$-by-2 matrix that contains the n sampled points from $[0,1]^{2}$. The result of the concatenation is a matrix of size $n$-by-514. The matrix is processed row-wise by a 3-layer perception, and the output is a matrix of size$n$-by-3 output and feeds it into a 3-layer perception. For the $Decoder3Dto3D$, it is the same as the $Decoder2Dto3D$. The only difference is the input of its is an $n$-by-515 matrix.

Sampling method. During the training stage, random sample points from $[0,1]^{2}$ can improve the robustness of the whole network. AtlasNet noted that sampling points regularly on a grid on the learned parameterization yields better performance than sampling points randomly. For inference, all results used this regular sampling.

End-to-end training. We briefly describe how the whole network is built on top of the AtlaNet and FoldingNet for end-to-end training. Given a 2D input image I, our goal is to learn making a prediction for the ground-truth S. Let the prediction of the final reconstruction output be $\widetilde{\emph{S}}$. Both of them are point sets. The learning process tries to minimize the $Chamferloss$ between the S and $\widetilde{\emph{S}}$:

the term $\min_{y\in\widetilde{S}}\|x-y\|$ enforces that any 3D point x in the ground-truth has a matching 3D point y in the reconstruction point cloud, and the term $\min_{x\in S}\|x-y\|$ enforces the matching vice versa. The $\frac{1}{|S|}$ and $\frac{1}{\widetilde{|S|}}$ are all mean operation to avoid too many points cause the huge loss. In FoldingNet, they used max operation instead of mean operation but if there is an outlier, it will make loss huge and then, and then the next predicted point cloud still has an outlier so repeatedly. It is not easy to make the network converge.

Evaluation criteria. We evaluated our generated shape outputs by comparing them to ground truth shapes using two criteria. First, we compared point sets of the output and ground-truth shapes using CD. But CD only reflects the distance between the two point clouds as a whole. Let the precision of a reconstructed model be the percentage of reconstructed points that have a ground truth point within distance $\tau$. So we introduce F-score to measure the accuracy of each point. Let the recall of a reconstructed model be the percentage of ground-truth points that have a reconstructed point withi distance $\tau$. For evaluating above two metrics, the overall score in trackA:

In trackB:

Ablative analysis. During the competition, several operations on the ground truth point clouds affect the final result.

Normalization method. In Table 1, different normalization methods impact results. Sphere normalization means to find the farthest point from the origin and then use its distance from the origin to normalize all points. In this way, for compact objects such as desk or nightstand, the points on their surfaces are not very far away from a specific point, which means all points could keep their relative position. However, for some specific objects like a chandelier, it has at least one outlier point, which makes the normalized length is too considerable for the other points. After normalization, the whole point cloud could not keep its original shape because of more points closer to the origin point. Due to the essence of AtlasNet is to learn the mapping function from 2D points to 3D points, the network is sensitive to the correspondence between 2D points and 3D points. In summary, the key to getting more accurate results is to keep the whole point cloud shape no matter what to do any augmentation.

Projection. In Table 3, whether to use multi-view images to rotate original point clouds or not could bring a huge difference. In Table 4, different scale factors lead to distinguishing results. Using multi-view images to rotate original point clouds is similar to calculate the relative projection matrix. Using different scale factors is like to guess the focal length. To sum up, it is not only lead by the reason from Table 1 but also relative to the projection transformation. If the camera matrix and projection matrix are known, we can reconstruct point clouds from single RGB images. As a result, our network also estimated the camera matrix and projection matrix implicitly.

Sampling method. Fig. 2 shows the point cloud before and after sampling mesh with lloyd’s algorithmWilliams (2018). After sampling, the density of point clouds increases. For the whole point cloud, their surfaces are more continuous. It seems that it’s not difficult to learn the mapping function. However, Table 2 shows that the network trained by the sampled point cloud is not better than the network trained by the original point cloud. In this way, we found that Chamfer distance is sensitive to the location of the given points. In this challenge, the points in the ground-truth point cloud are all vertices, which hopes all predicted points could be close to the edge or corner rather than distribute continuous surfaces. For visualization purposes, the more point clouds distribute on the surface, the more comfortable people observe. As a result, Chamfer distance maybe not enough to measure the quality of the generated point cloud. It needs to introduce some criteria for visual quality or generated mesh.

Evaluation on trackA server. We report quantitative results for shape generation from single images in Table 5, where each approach is trained on all categories and results are averaged over all categories. According to the ablative analysis, we combine all tricks to get the best results.

Acknowledgements Our work is supported by Em-Data Technology Co., Ltd. We thank Jianfei Gao, Fengliang Qi, Yuan Gao, Changyong Shu, Yunjie Xu, Qi Liu, Yao Xiao for valuable discussions.