Adaptive Joint Optimization for 3D Reconstruction with Differentiable Rendering

In this section, we provide a brief overview of texture and geometry optimization for 3D reconstruction. The first research line focuses on geometry refinement. Zollhöfer et al. [16] optimized the geometry of reconstructed 3D model encoded in a truncated signed distance field (SDF). Choe et al. [17] exploited shading cues captured from infrared cameras to improve the quality of a 3D mesh. Romanoni et al. [18] refined the surface geometry by optimizing composite energy in a variational manner, meanwhile updating the semantic labels based on Markov Random Field (MRF). Jiang et al. [19] combined both facial priors and SFS to enhance the fine geometric details of a portrait model. Schmitt et al. [20] recovered geometry details via jointly correcting camera pose and estimated material properties.

Another research line is about texture optimization. Zhou et al. [6] employed local warping for texture images to rectify complex distortions of geometry. Bi et al. [5] used patch-based synthesis to correct the misalignment of multi-view reference images. Fu et al. [21] proposed a global-to-local non-rigid optimization method to achieve better texture mapping results. Li et al. developed a fast texture mapping scheme to reduce misalignment at texture boundaries. Lee et al. adopted a texture-fution method with an SDF voxel grid to optimize the texture of a real-time scanning model. Huang et al. [22] designed a misalignment-tolerant metric based on generalized adversarial networks (GANs) to produce photorealistic textures.

Different from the above-mentioned works, some recent research has attempted to optimize texture and geometry jointly. Wang et al. [23] used planar primitives to partition a model and jointly optimized plane parameters, camera poses, texture and geometry using photometric consistency and planar constraints. However, this method relies on plane priors, and is not suitable for complex nonplanar objects. Maier et al. [10] proposed an SFS-based method to simultaneously optimize texture, camera pose, geometry, and lighting, but SFS is an ill-posed problem with potentially ambiguous solutions. Such ambiguity will lead to inferior results with texture-copy artifacts. Fu et al. [4] suggested directly optimizing the 3D models and avoiding the SFS process, which successfully alleviates the problem of texture-coping. However, they used different optimization schemes and objectives for texture, geometry, and camera pose optimizations, which makes the system complicated and less robust, especially to large reconstruction errors. Unlike Fu et al. [4], we propose a unified framework to jointly optimize texture, geometry, and camera poses via differentiable rendering. Experiments showed that our unified framework is more robust to different levels of reconstruction errors and achieves better results than existing methods.

Triangle meshes, a well-established representation of a surface, have been used in almost every graphics pipeline due to their efficiency and flexibility in terms of vertex transformations and texturing. However, traditional graphics engines do not produce usable gradients for optimization purposes. To solve this problem, some early work [12, 24] approximated gradients on mesh vertices, and recent methods [25, 13] propose fully-differentiable formulations. The property of differentiable renderer bridges the connection between a traditional optimization pipeline and advanced deep learning techniques. We adopt a soft-rasterizer renderer [13] in this work, which allows us to optimize the parameters of a 3D model, including texture, geometry, and camera poses, by enforcing weakly supervised losses between the rendered images and the corresponding RGB-D inputs.

GANs [15] have achieved impressive progress in areas such as super-resolution [26], image restoration [27], image translation [28], and 3D topological representation [29]. The main principle of a GAN is to run a zero-sum game between a generator and a discriminator in an adversarial manner. Unlike previous photometric approaches such as L1 or L2, which will lead to over-smooth results, adversarial loss can ensure high-quality image generation with high-frequency details. However, it is difficult to include adversarial loss in a traditional optimization pipeline [4, 7] due to the unstable process caused by entangled objectives. Huang et al. [22] first propose to combine texture optimization and GANs to solve the misalignment problem through view transformation-based warping. In this work, we combine GANs with differentiable rendering to achieve effective joint texture and geometry optimization for 3D reconstruction.

The inputs of our method include a set of RGB-D images scanned by a consumer range camera, corresponding estimated camera poses and object silhouettes for each frame, and an initial imperfect 3D model reconstructed using an existing reconstructing method like BundleFusion [30], KinectFution [31], or COLMAP [32]. Let ${M}$ indicate the reconstructed 3D mesh, and ${T}$ represent the texture map to be optimized. $\boldsymbol{I}_{A}$, $\boldsymbol{D}_{A}$, and $\boldsymbol{S}_{A}$ respectively denote the color image, depth map, and silhouette map under view ${A}$, and its corresponding camera pose, indicated as $\boldsymbol{P}_{A}$, is composed of a rotation matrix and a translation vector, which can transform each vertex, $\boldsymbol{v}_{i}$, of a mesh, ${M}$, from the world coordinate system to the camera coordinate system by $\left[\widetilde{\boldsymbol{v}}_{i}\right]_{\text{world}}=\boldsymbol{P}_{A}\left[\widetilde{\boldsymbol{v}}_{i}\right]_{\text{camera}}$, where $\left[\widetilde{\boldsymbol{v}}_{i}\right]_{X}$ indicates the homogeneous coordinate of the vertex $\boldsymbol{v}_{i}$ in the $X$ coordinate system. Now the problem is how to produce the optimized 3D reconstruction, based on known information.

Considering that 3D reconstruction errors come from mixed factors regarding both texture and geometry, we jointly optimize the vertex position, camera pose, and texture map with a unified framework based on differentiable rendering. As depicted in Sec. 3.3, the 3D mesh, ${M}$, with texture, ${T}$, is rendered with different camera poses, $\boldsymbol{P}$. Then a series of image-level losses measuring the differences between the rendered frames and the RGB-D inputs are computed, to update ${M}$, ${T}$, and $\boldsymbol{P}$. To achieve fast and stable convergence, we do not naïvely update ${M}$, ${T}$, and $\boldsymbol{P}$ together in every iteration. Instead, we designed a strategy to adaptively interleave the optimization of ${M}$, ${T}$, and $\boldsymbol{P}$ according to the convergence rate in Sec. 3.4. The overall pipeline is shown in Figure 2.

To enable the effective and efficient joint optimization of the different components of a 3D model, a unified framework supporting the optimization of texture, geometry, and camera pose is essential. However, this task is not easy, since texture, geometry, and camera pose are in different domains. Specialized schemes and objectives have often been designed for the optimization of each component in previous work. Inspired by the fact that the rendering process is a function combining texture, geometry, and camera pose, we propose a novel unified optimization framework based on differentiable rendering. Next, we describe the details of this framework including the multi-view rendering process and common objectives.

With our unified optimization framework, one naïve joint optimization strategy is to update the texture color $T^{t}$, mesh vertices $M^{t}$, and camera parameters $\boldsymbol{P}^{t}$ together in every single iteration $t$. However, we found this naïve strategy results in slow and unstable convergence. To solve this problem, we developed a joint optimization strategy to adaptively interleave updates of different components based on the common objective’s convergence rate (Eq. 6). With this strategy, the updating iterations for each component will be dynamically adjusted in different stages, leading to more stable and efficient optimization. In this way, the optimization of different components will still be beneficial to each other across iterations.

In addition to convergence acceleration, the interleaving strategy brings another benefit: it allows the addition of specialized objectives for a component in its own iteration. Specifically, we add Laplacian loss in the geometry iterations to constrain the mesh smoothness, while adding adversarial loss in the texture iterations to increase photorealism. Next, we introduce specific losses used in geometry and texture iterations, and then our adaptive interleaving strategy.

To quantitatively evaluate the performance of our method, we first tested it on a synthetic dataset for which both geometry and texture ground truth are known. Three representative baselines were compared with our method: G2LTex [21], in which only texture is optimized via a texture mapping strategy; Adversarial Texture Optimization (ATO) [22], in which only texture is optimized with adversarial loss; and JointTG [4], in which camera pose, geometry, and texture are jointly optimized. Here, we ignored Intrinsic3d [10], since its model refinement is based on a signed distance field (SDF) volume instead of a given noisy textured mesh, although this method was used in experiments with real data.

To demonstrate the performance of the proposed method on scanned objects and scenes in real scenarios, we compared our method to state-of-the-art baselines including two texture optimization methods (G2LTex [21] and ATO [22]) and two joint optimization methods (Intrisic3D [10] and JointTG [4]) on real datasets. The datasets contain 61 models from our works, and that of Zhou et al. [6], Maier et al. [10], and Huang et al. [22]. Figure 6 and Figure 7 display the visual comparisons of the rendered images and the geometry details of the optimized 3D models. The corresponding quantitative evaluations using image quality metrics for different datasets are listed in the Table I. Here, the Hausdorff distance metric is not given, because no ground truth mesh was available for the real data. Our method performed consistently better than the other methods.

Qualitative comparisons also demonstrated the superior performance of our method in recovering both fine-scale geometry and high-fidelity texture. As shown in Figure 7, G2LTex [21] based on texture mapping is sensitive to camera-poses and geometry noises, and finds it hard to obtain the correct texture when noise exists. ATO [22] obtains better texture by fusing texture information from multiple views and using adversarial loss. Still, it fails to recover geometric details, since only the texture is optimized. Intrinsic3d [10] and JointTG [4] can refine the geometry details to some extent. However, Intrinsic3d, based on SFS, is sensitive to lighting and has texture-copy artifacts (shown in Figure 10b), especially when the views are sparse. The texture and mesh generated by JointTG are visually harmonious, but it fails to recover both geometric and texture details as accurately as our method. Therefore, its rendered results look blurrier. More importantly, no previous methods could recover imperfect geometry shapes from some rough initial models, because of their limited geometric deforming capabilities (see the first example of Figure 7). Our method can handle geometric deformation problems, due to the IoU loss supported by our framework, as shown in Figure 10a.

To further compare the performance of different methods on real data, we report the average running time on different datasets in Table II. Since our method is implemented based on differentiable rendering, the running time of our method is linearly related to the model complexity and the number of scanned frames. As shown in Table II, the compared methods were more sensitive to the model complexity and the number of frames than ours. Whereas our method took a similar time to other methods in the low-complexity models (Dolls and Chairs), our method was faster in the high-complexity models (Intrinsic3d and Scene3d).

We investigated the effectiveness of each major component in our method using the synthetic dataset. Firstly, to validate the effect of joint optimization, we conducted an experiment by removing camera pose correction, geometry refinement, or adversarial loss of texture optimization from our joint optimization framework. Without the camera pose correction, the rendered image was misaligned with the RGB-D inputs. Therefore, the reconstruction objectives were wrongly calculated and caused a severe performance drop, as shown in Table IV and Figure 8. Removing geometry refinement was less significant than removing the camera-pose correction, since the influence of geometry errors is more local. Still, the performance was worse than that of the complete method. Compared to the case without adversarial loss, our complete method can produce more realistic texture details. This ablation study showed that the camera pose, geometry, and texture affect each other, and thus a joint optimization is necessary.

To illustrate the effectiveness and superiority of our adaptive interleaving strategy, we compared it to a hybrid strategy in which texture color, mesh vertices, and camera parameters are updated together in each single iteration. The hybrid strategy was implemented based on common loss, since the Laplacian and adversarial loss cannot be added into the hybrid strategy. To be fair, we compared it to an interleaving strategy (common) without the Laplacian and adversarial loss. To further investigate the performance of our adaptive algorithm, we conducted experiments by removing the adaptive strategy and replacing it with fixed internal iteration numbers: 40 steps and 200 steps. The qualitative comparison and convergence curves of common loss are shown in Figure 9 and Figure 11, respectively. The common loss was averaged under a batch view in each step for all strategies, hence there are slight vibration on the curves. The hybrid strategy led to an unstable convergence, and found it hard to reach a lower convergence position, while other interleaving strategies can overcome this problem by optimizing texture, geometry, and poses in an iterative manner. However, the internal iteration number of interleaving strategies is an important hyper-parameter, since the use of a small number of iterations may result in inadequate optimization (Figure 11 (red)), while a large one can be unnecessarily time-consuming (Figure 11 (blue)). Thus, it is important to dynamically select a suitable internal iteration number via the adaptive algorithm.

We report the quantitative comparison among different strategies in Table III. These results show that our adaptive interleaving strategy had a similar performance as the 200-step interleaving strategy, but was much more time-efficient. Compared to the hybrid strategy and the 40-step interleaving strategy, our strategy was more effective and produced better optimized results. To sum up, the proposed adaptive interleaving strategy could better balance the performance and efficiency of the optimization algorithm.

Although our adaptive joint optimization method was originally implemented based on RGB-D data, it is readily applied to RGB data, by removing the depth loss from the framework. Although there is still a certain gap in the geometry details compared with our complete framework, the version without depth loss can produce reasonably clear texture and a plausible geometric model, as shown in Figure 10c. A quantitative comparison of these two pipelines performed on our Doll dataset, as shown in Table V, indicated that our pipeline produced acceptable performance with only RGB data. Our method can be extended to joint optimization with RGB or RGB-D data, while the depth information can contribute to a better geometric details.

In order to further evaluate the quality of the optimized results on both synthetic and real data, we performed a study in which we asked users to vote for the most visually realistic rendered image, as shown in Figure 12. A total of 138 participants were asked to vote for the best rendered result on both synthetic and real datasets. During the survey process, we showed the front rendering of each model, and there was no time limit for participants to vote. For some real datasets with low noise levels, it was sometimes difficult for individuals to distinguish between the rendered images produced by the different methods. Nevertheless, images produced by the method proposed in this paper was still preferred over those produced by the other methods.