View Transformation Robustness for Multi-View 3D Object Reconstruction With Reconstruction Error-Guided View Selection

Deep learning-based multi-view 3D object reconstruction methods have achieved remarkable performance. Early methods focus on the fusion of multi-view features, such as [19, 20, 21, 22], where the multi-view features are reduced into a fixed size of feature maps. Later methods tried recurrent neural network fusion methods [23, 24, 25, 26], regarding input views as a sequence. However, RNN fusion methods are not invariant to input order permutations and cannot handle a large number of views efficiently. Attention-based fusion methods [27, 28, 29, 16] are also proposed to fuse multi-views according to the attention maps estimated from the attention subnet. Recent methods utilize transformer networks for more complicated fusion between the views, such as [30, 31, 32, 33, 34, 17, 35]. LRGT [17] proposed long-range grouping attention for grouping tokens from all views with separate attention operations. However, none of these methods have ever paid attention to the multi-view 3D object reconstruction methods’ robustness to the input view transformations.

The issue of view transformation and rotation robustness has been studied in various areas, such as point cloud classification and multi-view image classification. Instead of using rotation invariant descriptors [36, 37] as inputs or designing rotation equivariant networks [38, 39], ART-Point [3] improved rotation robustness by training the point classifier on inputs with adversarial rotations. Considering that the camera viewpoints are often fixed for all shapes in multi-view 3D shape classification, Hamdi et al. [40] proposed the Multi-View Transformation Network (MVTN), which uses differentiable rendering to determine optimal view-points for 3D shape recognition. Recent methods have started to focus on novel view synthesis for boosting the model viewpoint robustness. ViewFool [41] found adversarial viewpoints that mislead 3D recognition models with an entropic regularize. VIAT [42] proposed to improve the viewpoint robustness of multi-view image classification with the inner diverse adversarial viewpoints and the outer viewpoint invariant classifier training. Overall, synthesizing novel viewpoints and selecting the most optimal views are the keys for these methods to improve the models’ view transformation or rotation robustness. Note that none of these methods have focused on the view transformation robustness issue for multi-view 3D object reconstruction.

A multi-view 3D object reconstruction model $\mathit{f}$ usually consists of the following parts [23]: First, an encoder to extract the feature representation for reconstruction from the image set ($I_{1}$,$\dots$,$I_{n}$) of Object $\mathit{I}$; Second, multiple 2D image features or 3D volume features are fed into a merger to fuse features from different views; Third, a decoder to predict the corresponding voxel-base 3D shape $\mathit{V}$ from the features map. The overall prediction process is formulated as: $V=f(I_{1},\cdots,I_{n})$. We adopt two typical kinds of multi-view 3D object reconstruction models, CNNs-based Pix2Vox++ [16] and transformer-based LRGT [17]. Denote the 3D voxel ground truth as $V^{gt}$, and the reconstruction loss is $Loss(V,V^{gt})$ (details in Supp.). Following [16] and [17], we use binary cross entropy loss in Pix2Vox++ while Dice loss [43] in LRGT. The model is first trained on the training set of ‘Aligned’ data.

The view selection module first projects the 3D reconstruction error $\mathit{E}=\left|V-V^{gt}\right|$ to a series of viewpoints, and then selects viewpoints containing the most reconstruction errors to be fed into the view synthesis module (see Figure. 3). Specifically, We denote $v=[\psi,\theta,\phi]\in\mathbb{R}^{3}$ as the viewpoint parameters, where $[\psi,\theta,\phi]$ is the camera rotation along the z-y-x axes using the Tait-Bryan angles, and roll $\theta$ is set as zero. Then, our target is to estimate the $v=[\psi,\phi]$ from the 3D reconstruction error $\mathit{E}$, which indicates the areas with bad reconstruction results of an object. Naturally, if we could provide more information (views) about the areas with large reconstruction errors and try to rectify these errors, the performance of the reconstruction model could be boosted. Particularly, if a camera view can cover the reconstruction error regions more clearly, it is supposed to aid the robustness of the 3D reconstruction model.

To reduce computation, we first divide yaw $\psi$ and pitch $\phi$ into discrete values with $K$ degree interval to obtain a set of dense discrete 3D viewpoints. The yaw is divided into $\left\{\left[-180^{\circ}\!+\!(i\!-\!1)\!\times K^{\circ},-180^{\circ}+i\!\times K^{\circ}\right]\right\}_{i=1}^{n_{\psi}}$ and the pitch is divided into $\left\{\left[-90^{\circ}\!+\!(i\!-\!1)\!\times\!K^{\circ},-90^{\circ}\!+\!i\!\times\!K^{\circ}\right]\right\}_{i=1}^{n_{\phi}}$, where $n_{\psi}=360^{\circ}/K^{\circ}$and $n_{\phi}=180^{\circ}/K^{\circ}$ refer the number of interval for the yaw $\psi$ and the pitch $\phi$, respectively. Then we traverse each interval and use the median value of each interval as the viewpoint position for that interval: $\psi=-180^{\circ}+(i-\frac{1}{2})\times K^{\circ}$ and $\phi=-90^{\circ}+(i-\frac{1}{2})\times K^{\circ}$. As $K$ gets smaller, the traversed viewpoints $v=[\psi,\phi]$ are more continuous. By rotating the voxel grid $E$ by the corresponding degree $v$, we set the viewpoint direction to the negative direction along the X-axis, denoted as $E_{rot}=Rot(E,v)$, where $Rot$ is the rotation operation.

We then perform the orthographic projection on $E_{rot}$ to each discrete viewpoint to obtain the reconstruction error projection maps $P$, conducted as follows:

where the $x^{\ast}=\min_{E_{rot}(x,y,z)\neq 0}X$, $E_{rot}(x,y,z)$ represents the integer 3D coordinate and the $X$ represents the range of $x$. Intuitively, we get the first occupancy value along each line of sight, ie., the surface of the error voxel.

After we get the error projection map $P$ under each viewpoint $v$, we obtain the sum of the pixels of each projection map: $sum_{v}=\sum_{i=1}^{S}P_{i}$, where $S$ is the number of a projection map pixels and $P_{i}$ is the $i$-th pixel of the projection map $P$. Then, we select the maximum $n$ views $\{v_{s}\}$ based on the $sum_{v}$. Because larger $sum_{v}$ means the viewpoint covers more reconstruction error regions. Rather than directly using the selected viewpoints for reconstruction, which will decrease the diversity of inputting viewpoints, we model each selected viewpoint $v_{s}$ as a Gaussian distribution and sample views among the distribution: $v\sim\mathcal{N}(v_{s},\sigma)$, where $\sigma$ refers the standard deviation, which is set to $K/6$ in our experiments. Finally, we can obtain the $n$ corresponding viewpoints $v=[\psi,\phi]$ from the 3D reconstruction error $E$ for each object for $n$-view reconstruction.

Viewpoint pool. During experiments, we found that we construct a viewpoint set for each object under each category, the learned viewpoint distribution will degenerate with the increase of training iterations, that is, the diversity of selected viewpoints will decrease in the later training period, resulting in the phenomenon of overfitting in iterative training. A simple solution is to construct multiple viewpoint sets with multiple iterations for each object of each category. However, it will be very time-consuming. To solve this problem, according to [3], it is observed that the selected viewpoint distributions of objects within the same category are highly similar, which has a strong transferability. Based on this, we construct a viewpoint pool by saving the viewpoints selected on each object by category:

where $v_{i,k}$ refers to the viewpoint selected on object $i$ of category $k$. We will save the viewpoints corresponding to all $n_{k}$ objects in the category $k$ and traverse all $M$ categories to construct the viewpoint pool. In the process of iterative optimization, it is only necessary to sample the viewpoint pool according to the category and convert it into the corresponding view. Due to the transferability of the viewpoint distribution of the same type of object, high reconstruction loss can also be induced.

With the camera viewpoint $v=[\psi,\phi]$ selected from the view selection module, we synthesize novel images of an object from the Stable Diffusion model Zero-1-to-3 [15], which is a viewpoint-conditioned diffusion approach and can generate an image under the novel viewpoint by providing an image of the object and the viewpoint transformation matrix. Given a dataset of paired images and their relative camera extrinsic ($\hat{I}$, $I$, $\hat{v}$, $v$), the model is trained by solving for the following objective:

where $\varepsilon$ denote the encoder, $\epsilon_{\theta}$ is a denoiser U-Net, $t$ is the diffusion time step, and $z$ is the latent representation of the input image encoded by the encoder. After the model is trained, we can generate the novel image from any viewpoint by performing iterative denoising from a Gaussian noise image conditioned on $c(\hat{I},\hat{v},v)$.

Use Zero-1-to-3 for direct 3D reconstruction. Directly using Zero-1-to-3 for 3D reconstruction reports bad performance as in Figure 4. It suffers drastically from the view transformations and the 3D reconstruction inference process is very time-consuming (see running time in the Supplemental). Thus, we utilize Zero-1-to-3 for novel view synthesis and rely on the new views to boost the view transformation robustness of existing 3D reconstruction models, instead of using it for direct 3D reconstruction.

The training adopts an iterative optimization scheme containing the inner maximization and the outer minimization. Specifically, in the first iteration, we evaluate the pre-trained 3D reconstruction model to select the viewpoints for inner maximization, and then re-train the model on new view images generated by the view synthesis module from the previously selected viewpoints before obtaining a robust model for outer minimization. The process will be repeated until the model converges to the most robust state. Our goal is formulated as: $\min_{W}\sum_{i=1}^{N}\mathbb{E}_{v_{i}}\left[\max_{v_{i}}L(W,\mathcal{R}(v_{i}),y_{i})\right]$, where $W$ denotes the parameters of the reconstruction model $f$, $L$ is a reconstruction loss function, $\mathcal{R}(v_{i})$ is the rendered images of the $i$-th object given the viewpoint $v_{i}$ and the $y_{i}$ is the GT of the corresponding object. We then detail how we improve the inefficiency of the iterative optimization scheme with the random update strategy.

Random update strategy. A random update strategy is adopted to reduce the training time. Instead of generating new views for all objects in each epoch, we randomly select 5% objects and generate new views by Zero-1-to-3 for them, while the remaining 95% objects are still trained with the views from the ‘Align’ training set. At the same time, to make up for the diversity of viewpoints and improve the effectiveness of training, the newly generated view images will be added to the original view images set after each fine-tuning epoch, so that each epoch of iterative training can fully and effectively obtain the prior information left by the previous epoch, and help the 3D reconstruction model learn more efficiently. Note that only the multi-view 3D reconstruction model is needed at inference.

Dataset generation. Since no existing dataset is suitable for evaluating view transformation robustness, we generate a new dataset ShapeNet-VTR, based on ShapeNet [18], which consists of larger viewpoint ranges than the original version via rendering new views from the 3D CAD models in ShapeNet. ShapeNet-VTR consists of 13 categories and 39239 objects in total. The objects in each category are divided according to 7$:$1$:$2 for training, validation, and testing, respectively. ShapeNet-VTR consists of 3 sets: ‘Aligned’, ‘Hemispherical’, and ‘Spherical’, sharing the same object dividing way but differing in the viewpoint range as shown in Figure 5. ‘Aligned’ set is rendered at a fixed $\phi$ of $60^{\circ}$ and every $15^{\circ}$ in $\psi$; In the ‘Hemispherical’ set, the range of views is set as $\phi$ $\in$ $[0^{\circ},90^{\circ}]$, $\psi$ $\in$ $[-180^{\circ},180^{\circ}]$, and thus images are randomly rendered in the upper hemisphere. In ‘Spherical’, the range of views is $\phi$ $\in$ $[-90^{\circ},90^{\circ}]$, $\psi$ $\in$ $[-180^{\circ},180^{\circ}]$, and thus images are randomly rendered in the spherical space. In each set, each object contains 24 views but from different camera angle ranges. We train the 3D reconstruction models on the training set of ‘Aligned’ at first and use the models’ performance on ‘Hemispherical’ and ‘Spherical’ sets to evaluate the robustness of 3D object reconstruction models to view transformations.

Implementation details. We utilize two 3D reconstruction models, CNN-based Pix2Vox++ [16] and Transformer-based LRGT [17], respectively. The input view number is 3 and the image resolution is $256\times 256\times 3$ and the size of the voxel output is $32\times 32\times 32$. We use a threshold of 0.4 to obtain the occupancy voxel grid and set the interval of degree $\mathit{K}=30^{\circ}$ in the experiments.

Comparison methods. We compare with SOTA 3D object reconstruction methods Pix2Vox++ [16], GARNet [29], UMIFormer [35], and LRGT [17] (without view transformation robustness approaches, denoted as None-VTR) and four view transformation robustness comparison methods (denoted as VTR). Note that there are no existing VTR methods for multi-view 3D object reconstruction, and we propose reasonable baselines or adopt methods from other areas for comparison:

• •

  Nature: Data augmentation with the most common viewpoint renderings from training objects’ natural states (e.g. cars are usually viewed in the side view).

• •

  Random: Standard data augmentation with random viewpoints, and the view range is set as $\phi$ $\in$ $[-90^{\circ},90^{\circ}]$, $\psi$ $\in$ $[-180^{\circ},180^{\circ}]$.

• •

  VIAT [42]: By regarding viewpoint transformation as an attack, VIAT solves the inner maximization problem to parameterize the Gaussian Mixture distribution of adversarial viewpoints with trainable parameters. In our experiment setting, VIAT keeps the roll $\theta$ fixed and learns the parameters of the Gaussian Mixture distribution of yaw $\psi$ and pitch $\phi$ to get an adversarial viewpoint. Next, we select the other two adversarial viewpoints at azimuth intervals of $120^{\circ}$ and $240^{\circ}$, starting from the first adversarial viewpoint.

• •

  MVTN [44]: A viewpoint selection algorithm that uses differentiable rendering to determine optimal viewpoints for 3D shape recognition through the gradient descent algorithm. We changed its downstream task to multi-view 3D reconstruction. We changed the 3D representation from point clouds to voxels and set the number of the selected viewpoints to 3 as the view amount of the input.

Besides, we also evaluate the method’s single-view reconstruction robustness, comparing with single-view reconstruction state-of-the-art methods [45], which does not incorporate prior knowledge of point clouds or depth maps to assist in reconstruction as ours.

Metrics. The evaluation metrics include the mean Intersection-over-Union (mIoU) and F_score (see details in Supp.), whose higher values indicate better performance.