Partial-View Object View Synthesis via Filtering Inversion

FINV leverages a pre-trained 3D GAN generator $G$; we use GET3D [14] as the backbone GAN. GET3D is a 3D GAN that disentangles geometry and texture. The geometry branch of the GET3D generator differentiably outputs a surface mesh of arbitrary topology, and the texture branch produces a texture field that can be queried at the surface points to produce texture maps. In our ablation studies, we also use EG3D [5] as the backbone and discuss both backbones’ advantages and disadvantages.

At a high level, GET3D samples two input vectors from a Gaussian distribution $\mathbf{z}_{\text{geo}}\in\mathbb{R}^{512}$ and $\mathbf{z}_{\text{tex}}\in\mathbb{R}^{512}$. Then, following StyleGAN [23, 21, 22], GET3D uses non-linear mapping networks $f_{\text{geo}}$ and $f_{\text{tex}}$ to map $\mathbf{z}_{\text{geo}}$ and $\mathbf{z}_{\text{tex}}$, respectively, to intermediate latent vectors $\textbf{w}_{\text{geo}}=f_{\text{geo}}(\mathbf{z}_{\text{geo}})$, $\textbf{w}_{\text{tex}}=f_{\text{tex}}(\mathbf{z}_{\text{tex}})$. These intermediate latent vectors are further used to produce a textured mesh. We use $G_{M}(\textbf{w}_{\text{geo}};\phi)$ to denote the rendered binary object mask, where $\phi$ represents the parameters of the geometry branch. Letting $\textbf{w}=(\textbf{w}_{\text{geo}},\textbf{w}_{\text{tex}})$, we use $G_{I}(\textbf{w};\phi,\theta)$ to denote the rendered RGB image, where $\theta$ represents the parameters of the texture branch. The mask only depends on the geometry branch, and the final rendering depends on both the geometry and texture branches. Note that the image generation also depends on camera intrinsics and extrinsics, which we omit from the above notation for simplicity.

Figure 4 shows a high-level workflow of this phase. Given a pre-trained generator $G$, a set of RGB observations $\mathbf{I}_{0:T}$, and their corresponding object masks $\mathbf{M}_{0:T}$ up to the current time $T$, we aim to optimize a randomly initialized latent code (in practice, we use a set of latent codes) $\textbf{w}=(\textbf{w}_{\text{geo}},\textbf{w}_{\text{tex}})$ that best encodes the observed object. In greater detail, we keep the parameters of the generator $G$ fixed and we optimize with the following objective:

$\mathcal{L}_{\text{LPIPS}}$ is the LPIPS metric [55], and $\mathcal{L}_{\text{MASK}}$ is binary cross entropy in our experiments. We use $\textbf{w}^{p}=(\textbf{w}^{p}_{\text{geo}},\textbf{w}^{p}_{\text{tex}})$ to denote the latent code obtained after the inversion process.

However, the process of inversion can be unstable in practice due to the misalignment between the training and real-world distributions [38]. For example, in our experiments, we train the generator $G$ on synthetic object models, where this problem is particularly pronounced. To combat this issue, we propose filtering inversion, a process that combines inversion with a filtering process akin to particle filtering.

The filtering process is described as follows. At time $t=0$, we randomly initialize a set of latent codes $\{\textbf{w}_{1},...,\textbf{w}_{N}\}$ and invert them with Eq. (1) in parallel, taking image $\mathbf{I}_{0}$ as the reference. At every following time step $t>0$, we observe a new image $\mathbf{I}_{t}$ and perform a filtering and optimization step. In the filtering step, we filter the latent codes, keeping only those that meet the following criterion:

where $\text{Rank}(\cdot)$ is the Percentile Ranking that gives a percentage score of the loss out of all the latent codes, and $\gamma$ is a hyper-parameter which we set to $30\%$ in our experiments. In other words, we keep the top $\gamma N$ latent codes that can better reconstruct the object at all views, including the newly observed view $\mathbf{I}_{t}$. Next, we proceed to the optimization step, where we again apply Eq. (1) to update all the latent codes on all the observations up to and including the new observation $\mathbf{I}_{t}$.

We then proceed to time $t+1$ and repeat the filtering and optimization steps. This filtering and optimization process is executed repeatedly to discard latent codes stuck in local minima and keep improving our representation of the object with every new observation.

This phase aims to refine the representations by fine-tuning the generator to reconstruct objects while maintaining the essential priors learned by the generative model. Since our goal is to reconstruct objects in the wild, we will inevitably run into instances with geometry or texture that have not been observed before in the training set. In such cases, simply having the filtered inversion phase (phase I) will be insufficient, as there may not exist a latent code in the learned latent space that enables the generator to produce the required texture and geometry. If we simply use a random or mean latent code and fine-tune the entire generator around it, the representation tends to overfit to reconstructing the observed views and produces highly-distorted novel views [35]. Instead, we fine-tune the generator while fixing $\textbf{w}^{p}$, which can be used to generate a textured mesh that is similar to the observed object in real life. In this way, we expect to be able to minimize the “distortion” of the originally well-behaved latent space.

One key insight here is that geometry and texture are of different complexities for different object categories. The disentangled architecture of GET3D allows us to fine-tune the geometry and texture of the object separately and with different amounts. For example, cars have fewer geometric variations than texture variations, while wooden tables come in many different shapes but have more uniform textures. Thus, we can impose more regularization on the training of geometry to avoid overfitting to the imperfect and partial observations and distorting the learned latent space.

Let $\textbf{w}^{p}=(\textbf{w}^{p}_{\text{geo}},\textbf{w}^{p}_{\text{tex}})$ be a latent code obtained after phase I inversion at time $t$. We fine-tune the generator using the following objectives:

where $\mathcal{L}_{\text{MSE}}$ is the MSE calculated on pixels and $\lambda_{\text{MSE}}$ is a coefficient. Recall that $\phi$ and $\theta$ represent the parameters for the generator’s geometry and texture branch, respectively. By having the geometry and texture branch’s objectives disentangled, we can optimize $\phi$ and $\theta$ to different degrees. In our experiments, we observe that reconstructing geometry tends to be easier than texture and we simply use early-stopping as a regularization on Eq. (3) during the refinement phase.

All methods are evaluated on real-world images. We assume a sequential observation of RGB images and instance segmentation masks from a camera moving in the scene relative to the object. We evaluate the model with either the first 1, 3, or 5 frames as input and report reconstruction quality metrics on frames 6 and onward. We report reconstruction quality metrics given the first 1, 3, and 5 frames of the object. This allows us to assess the ability of each method to incorporate information from multiple frames in an online setting. Note that we do not use depth information but assume known camera poses and intrinsic. In this paper, methods compared within the same table are provided with the same estimated poses and segmentations.

Table 2 shows the results on ScanNet [10] chairs and tables, showing the PSNR, SSIM [46], and LPIPS [55] metrics for 1, 3 and 5 input views. Our method outperforms all prior work across all metrics for all numbers of input views. For example, on ScanNet Chairs with five input views, we outperform the most competitive prior work in each case by 7.1%, 7.0%, and 16.3% relative improvement in PSNR, SSIM, and LPIPS, respectively. In the most challenging single-view setting, this improvement increases to 7.2%, 17.4%, and 48.7%, respectively, demonstrating the superior ability of our method to effectively use the learned prior to reconstruct objects from partial views. The results on ScanNet Table data repeat this finding.

Table 3 shows the results on NuScenes cars. As before, our method consistently outperforms all other methods that were pre-trained on synthetic ShapeNet data across all numbers of input views. Specifically, we show 2.9%, 6.4%, and 8.3% relative improvement in PSNR, SSIM, and LPIPS, respectively, compared to the most competitive baseline in each case in the single-view setting. With 5 views, we still outperform all baselines in SSIM and LPIPS; however, Instant-NGP reports higher PSNR. Since LPIPS is the metric that more accurately reflects human perception [55, 52], this points to the fact that despite not reconstructing cars the best on a pixel level, our method still gives semantically better reconstructions on target views. We explore this finding deeper in Section 4.3, showing that our model actually achieves a better overall reconstruction. Surprisingly, despite using only synthetic training data, we achieve comparable results to $\text{AutoRF}^{*}$ (bottom row), which was trained on in-domain real-world images taken from the same distribution as the testing images, in all metrics except multi-view LPIPS. This demonstrates the effectiveness of our method in bridging the substantial sim-to-real gap.

Figures 6 and 7 present qualitative results, highlighting the challenge of our setting: objects are often partially observed and not in full view. Although the baselines are able to generate reasonable reconstructions when the target viewpoint is close to the input view, they fail to render unseen parts of the object from viewpoints far from the input. We hypothesize that this is because pixelNeRF and IBRNet learn local priors since they are optimized to render from local pixel features. Although IBRNet uses a ray transformer so that density predictions on individual rays are coherent, it still learns local priors since the rays are organized in patches. In contrast, we limit our method to update within the latent space of a learned generative model. The learned structure of the latent space serves as regularizer and imposes a “global prior”. Intuitively, during the filtered inversion phase, we are asking the model to answer the question: “given these observations (constraints), what would the entire object look like?”. Subsequently, in the refinement phase, we ask the model to better adapt its answer to the above question to real-world observations. This two-stage process results in more globally coherent reconstructions. AutoRF also imposes a global prior, but it uses an encoder network instead of filtered inversion to compute the latent codes. The encoder overfits to its training distribution in the source domain, yielding simulated-looking renderings. Successful results with AutoRF are only achieved when trained on target-domain data ($\text{AutoRF}^{*}$). Figure 5 shows additional qualitative results.

In addition to the main experiments shown above, we perform additional experiments to answer the following questions: Q1: Do FINV’s filtering process in the filtered inversion phase and the use of the refinement phase boost performance? Q2: How does the choice of a mesh-based backbone GAN (e.g., GET3D) compare with radiance field-based backbone GANs (e.g., EG3D)? Q3: How does FINV handle rotational deltas between input and target views compared to baselines?