Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video

Score Distillation Sampling (SDS) is first proposed in DreamFusion (Poole et al., 2023) for text-to-3D tasks. It enables the use of a 2D text-to-image diffusion model as a prior for optimization of a NeRF. We denote the NeRF parameters as $\theta$, text-to-image diffusion model as $\phi$, text prompt as $\rho$, the rendering image and the reimage as $\mathbf{x}$ and $\mathbf{z}$, the SDS loss is defined as:

$$\nabla_{\theta}\mathcal{L}_{SDS}\left(\phi,\mathbf{x}\right)=\mathbb{E}_{\tau,\epsilon}\left[\omega(t)\left(\hat{\epsilon}_{\theta}\left(\mathbf{z}_{t};\rho,\tau\right)-\epsilon\right)\frac{\partial{\mathbf{x}}}{\partial{\theta}}\right],$$

(1)

where $\tau$ is timestamps in diffusion process, $\epsilon$ denotes noise, and $\omega$ is a weighted function. Intuitively, this loss perturbs $\mathbf{x}$ with a random amount of noise corresponding to the timestep $\tau$, and estimates an update direction that follows the score function of the diffusion model to move to a higher density region.

Besides text-to-3D, SDS is also widely used in image-to-3D tasks. Zero123 (Liu et al., 2023) is one prominent representative. It proposes a viewpoint-conditioned image-to-image translation diffusion model fine-tuned from Stable Diffusion (Rombach et al., 2022), and exploits this 3D-aware image diffusion model to optimize a NeRF using SDS loss. This image diffusion model takes one image, denoted by $\mathbf{I}_{in}$, and relative camera extrinsic between target view and input view, denoted by $\left(\mathbf{R},\mathbf{T}\right)$, as the input, and outputs the target view image. Compared with the original text-to-image diffusion model, text prompt in Equation 1 is not required in this model cause the authors translate the input image and the relative camera extrinsic to text embeddings. Then Equation 1 could be re-written as:

$$\nabla_{\theta}\mathcal{L}_{SDS}\left(\phi,\mathbf{x}\right)=\mathbb{E}_{\tau,\epsilon}\left[\omega(t)\left(\hat{\epsilon}_{\theta}\left(\mathbf{z}_{t};\mathbf{I}_{in},\mathbf{R},\mathbf{T},\tau\right)-\epsilon\right)\frac{\partial{\mathbf{x}}}{\partial{\theta}}\right],$$

(2)

K-planes (Fridovich-Keil et al., 2023) is a simple and effective dynamic NeRF method which factorizes a dynamic 3D volume into six feature planes (i.e., hex-plane), denoted as $P=\{P_{o}\}$, where $o\in\{xy,yz,xz,xt,yt,zt\}$. The first three planes correspond to spatial dimensions, while the last three planes capture spatiotemporal variations. Each of the planes is structured as a $M\times M\times F$ tensor in the memory, where $M$ represents the size of the plane and $F$ is the feature size that encodes scene density and color information. Let $t$ denote the timestamp of a video clip, given a point $p=(x,y,z,t)$ in the 4D space, we normalize the coordinate to the range $[0,M)$ and subsequently project it onto the six planes using the equation $f(p)_{o}=P_{o}(\iota_{o}(p))$, where $\iota_{o}$ is the projection from a space point $p$ to a pixel on the $o$’th plane. The plane feature $f(p)_{o}$ is extracted via bilinear interpolation. The six plane features are combined using the Hadamard product (i.e., element-wise multiplication), to produce a final feature vector as follows:

$$f(p)=\prod_{o\in\{xy,yz,xz,xt,yt,zt\}}f(p)_{o},$$

(3)

Then, the color and density of $p$ is calculated as $c(p)=c(f(p))$ and $d(p)=d(f(p))$, where $c$ and $d$ denotes mlps for color and density, respectively.

Existing DyNeRF methods mainly assume the supervision signals are temporally coherent, however, this assumption does not hold in our task since our main supervision signal is from an image diffusion model. In order to minimize the impact of temporal discontinuity in the supervision signals, we are prone to 4D representations which naturally guarantee a certain level of temporal continuity. Therefore, we build our DyNeRF based on K-planes (Fridovich-Keil et al., 2023) which explots temporal interpolation, an operator naturally inclined to temporal smoothing. Empirically, reducing the time resolution of spatiotemporal planes helps enhance temporal consistency, however, this results in over-smoothed renderings where finer details are lost. In contrast, increasing the time resolution leads to crisp renderings, but the continuity of images within the same time series diminishes. To achieve both temporal continuity and high image quality, we adjust the multi-scale technique in K-planes and introduce Cascade DyNeRF.

Let us denote the scale index by $s$. In K-planes, multi-scale features are exploited by concatenation along feature dimension, then the color and density could be calculated as:

$$c(p)=c(\mathrm{concat}(\{f(p)^{s})\}_{s=1}^{S}),\;d(p)=d(\mathrm{concat}(\{f(p)^{s})\}_{s=1}^{S}),$$

(4)

where $S$ is the number of scales. In our setting, the experiment results show simple concatenation is hard to balance between image quality and temporal consistency. So we propose to leverage the cascade architecture, which is commonly used in object detection algorithms (Cai & Vasconcelos, 2018; Carion et al., 2020), to first output a coarse yet temporally coherent dynamic object from low-resolution planes and then let the high-resolution planes learn the residual between the coarse and fine results. That is, the color and density of the object after scale $s$ is:

$$c(p)^{s}=\sum_{k=1}^{s}c(f(p)^{k}),\;d(p)^{s}=\sum_{k=1}^{s}d(f(p)^{k}),$$

(5)

where k indicates the scale index. Note that losses are applied to the rendering results of each scale to guarantee that planes with higher resolution learn the residual between results from previous scales and the target object. In this way, we can improve temporal consistency without sacrificing much object quality. But Cascade DyNeRF alone is not enough for spatiotemporal consistency, so we resort to extra regularization, please see in the next section.

Video generation methods usually train an inter-frame interpolation module to enhance the temporal consistency between keyframes (Ho et al., 2022; Zhou et al., 2022; Blattmann et al., 2023). Inspired by this, we exploit a pre-trained light-weighted video interpolation model and propose Interpolation-driven Consistency Loss to enhance the spatiotemporal consistency of the 4D generation.

The interpolation model adopted in this work is RIFE (Huang et al., 2022), which takes a pair of consecutive images as well as the interpolation ratio $\gamma~{}(0<\gamma<1)$ as the input, and outputs the interpolated image. In our case, we first render a batch of images that are either spatially continuous or temporally continuous, denoted by $\{\mathbf{x}\}_{j=1}^{J}$, where $J$ is the number of images in a batch. Let us denote the video interpolation model as $\psi$, the interpolated image as $\hat{\mathbf{x}}$, then we calculate the Interpolation-driven Consistency Loss as:

	$\displaystyle\hat{\mathbf{x}}_{j}$	$\displaystyle=\psi(\mathbf{x}_{1},\mathbf{x}_{J},\gamma_{j}),$		(6)
	$\displaystyle\mathcal{L}_{ICL}$	$\displaystyle=\sum_{j=2}^{J-1}\lVert\mathbf{x}_{j}-\hat{\mathbf{x}}_{j}\rVert_{2},$

where $\gamma_{j}=\frac{j-1}{J-1}$, and $2\leq j\leq J-1$.

This simple yet effective loss enhances the continuity between frames thus improving the spatiotemporal consistency in dynamic object generation by a large margin. Moreover, we find the spatial version of this loss alleviates the multi-face problem in 3D generation tasks as well. Please refer to the experiment sections to see quantitative and qualitative results. The Interpolation-driven Consistency Loss and some other regularization losses are added with SDS loss in Equation  2, details of which can be found in the experiment section.

Sometimes image sequence rendered from the optimized DyNeRF suffers from artifacts, such as blurry edges, small floaters, and insufficient smoothness, especially when the object motion is abrupt or complex. To further improve the quality of rendered videos, we design a lightweight video enhancer and optimize it via GAN, following pix2pix (Isola et al., 2017). The real images are obtained with image-to-image technique (Meng et al., 2021) using a super-resolution diffusion model, and the fake images are the rendered ones.

To better exploit video information, We add cross-frame attention to the UNet architecture in pix2pix, i.e., each frame will query information from two adjacent frames. We believe this could enable better consistency and image quality. Denote the feature map before and after cross-frame-attention as $\mathbf{F}$ and $\mathbf{F}_{j}^{\prime}$, we have:

	$\displaystyle F_{j}^{\prime}$	$\displaystyle=\mathrm{Attention}(\mathcal{Q}_{j},\mathcal{K}_{j},\mathcal{V}_{j}),$		(7)
	$\displaystyle\mathcal{Q}_{j}=\mathrm{flatten}(F_{j}),$	$\displaystyle\mathcal{K}_{j}=\mathcal{V}_{j}=\mathrm{flatten}(\mathrm{concat}(F_{j-1},F_{j+1}),$

where $\mathcal{Q}$, $\mathcal{K}$ and $\mathcal{V}$ denotes query, key, and value in attention mechanism, and $\mathrm{concat}$ denotes the concatenation along the width dimension.

Loss for the generator and discriminator are the same as pix2pix.

Data Preparation For each input video, we initially segment the foreground object utilizing SAM (Kirillov et al., 2023) and subsequently sample 32 frames uniformly. The majority of the input videos span approximately 2 seconds, with some variations extending to around 1 second or exceeding 5 seconds. For the ablation study of video sampling, please refer to the appendix A.3.

Training During SDS and interpolation consistency optimization, we utilize zero123-xl trained by  Deitke et al. (2023) as the diffusion model for SDS loss. For Cascade DyNeRF, we set $s=2$, i.e., we have coarse-level and fine-level DyNeRFs.The spatial and temporal resolution of Cascade DyNeRF are configured to 50 and 8 for coarse-level, and 100 and 16 for fine-level, respectively. We first train DyNeRF with batch size 4 and resolution 64 for 5000 iterations. Then we decrease the batch size to 1 and increase the resolution to 256 for the next 5000 iteration training. ICL is employed in the initial 5000 iterations with a probability of 25%, and we sample consecutive temporal frames at intervals of one frame and sample consecutive spatial frames at angular intervals of 5^∘-15^∘ in azimuth. SDS loss weight is set as 0.01 and reconstruction loss weight is set as 500. In addition to SDS and ICL, we also apply foreground mask loss, normal orientation loss, and 3D smoothness loss. The learning rate is set as 0.1 and the optimizer is Adam. In the post-video enhancing stage, we train the video enhancer with a modified Unet architecture. The learning rate is set as 0.002, the batch size is 16, and the training epoch is 200. The main optimization stage and the video enhancing stage cost about 2.5 hours and 15 minutes on a single V100 GPU. For details, please refer to the appendix A.4.

To date, few methods have been developed for 4D generation utilizing video obtained from a static camera, so we only manage to compare our method with D-NeRF (Pumarola et al., 2021) and K-planes (Fridovich-Keil et al., 2023).

Quantitative Results To quantitatively evaluate the proposed video-4D generation method, we select and download seven animated models, namely Pistol, Guppie, Crocodie, Monster, Skull, Trump, Aurorus, from Sketchfab (ske, 2023) and render the multi-view videos by ourselves, as shown in Figure 3 and appendix A.2. We render one input view for scene generation and 4 testing views for our evaluation. The per-frame LPIPS (Zhang et al., 2018) score and the CLIP (Radford et al., 2021) similarity are computed between testing and rendered videos. We report the scores averaged over the four testing views in Table. 1. Note that the commonly used PSNR and SSIM scores were not applied in our scenario as the pixel- and patch-wise similarities are too sensitive to the reconstruction difference, which does not align with the generation quality usually perceived by humans. As shown in Table  1, our dynamic 3D generation produces the best quantitative results over the other two methods on both the LPIPS and CLIP scores, which well aligns with the qualitative comparisons shown in Figure  3.

Qualitative Results The outcomes of our method and those of dyNeRFs are illustrated in Figure 3. It is observable that both D-NeRF and HyperNeRF methods struggle to achieve satisfactory results in novel views, owing to the absence of multi-view information in the training data. In contrast, leveraging the strengths of the generation model, our method proficiently generates a 360^∘ representation of the dynamic object. For additional results, please refer to the appendix A.1.

Cascade DyNeRF In Figure 4, we conduct an ablation study for Cascade DyNeRF. Specifically, we substitute Cascade DyNeRF with the original K-planes architecture, maintaining all other settings unchanged. In the absence of the cascade architecture, the training proves to be unstable, occasionally yielding incomplete or blurry objects, as demonstrated by the first and second objects in Figure 4. In some cases, while the model manages to generate a complete object, the moving parts of the object lack clarity, exemplified by the leg and beak of the bird. Conversely, the proposed Cascade DyNeRF exhibits stable training, leading to relatively satisfactory generation results.

Interpolation-driven Consistency Loss The introduction of Interpolation-driven Consistency Loss (ICL) stands as a significant contribution of our work. Therefore, we conduct extensive experiments to investigate both its advantages and potential limitations. Figure 5(a) illustrates the ablation of both spatial and temporal Interpolation-driven Consistency Loss (ICL) in the video-to-4D task. Without ICL, the objects generated exhibit spatial and temporal inconsistency, as evidenced by the multi-face/foot issue in the blue jay and T-rex, and the unnatural pose of the seagull. Additionally, color discrepancies, such as the black backside of the corgi, are also noticeable. Employing either spatial or temporal ICL mitigates the multi-face issue, and notably, the use of spatial ICL also alleviates the color defect problem. Utilizing both spatial and temporal ICL concurrently yields superior results. We further perform a user study, depicted in Figure 2(a), which includes results w/ and w/o ICL for 20 objects. For efficiency in evaluation, cases in which both methods fail are filtered out in this study. 20 users participate in this evaluation, and the results unveiled a preference for results w/ ICL in 75% of the cases.

We further explore whether ICL could alleviate multi-face problems for text-to-3D tasks. We compared the success rate of DreamFusion implemented w/ and w/o the proposed ICL loss. For the sake of fairness and rigor, we collect all prompts related to animals from the official DreamFusion project page, totaling 230. 20 users are asked to participate in this non-cherry-pick user study, where we establish three criteria for a successful generation: alignment with the text prompt, absence of multi-face issues, and clarity in geometry and texture. We visualize the statistics in Table 2(b) and Table 2. The results show that although users have different understandings of successful generation, results w/ ICL always outperform results w/o it. For a comprehensive understanding, qualitative comparisons are presented in Figure 5(b), which indicates the proposed technique effectively alleviates the multi-face Janus problem and thus promotes the success rate. Implementation details about text-to-3D can be found in the appendix A.4, and we also analyze the failure cases and limitations of ICL in A.5.

Cross-frame Video Enhancer In Figure 11 (see in appendix), we show the proposed cross-frame video enhancer could improve uneven color distribution and smooth out the rough edges, as shown in almost all figures, and remove some floaters, as indicated by the cat in the red and green box.