Beyond Skeletons: Integrative Latent Mapping for Coherent 4D Sequence Generation

To learn the high-quality 4D sequence distribution with color via diffusion, we propose an integrative latent mapping approach that models both 3D shape and color information within a unified latent space. The 3D geometry is first represented by a continuous SDF with a shape latent vector $z_{i}\in\mathbb{R}^{d_{1}}$. To establish an initial shape representation, we use an auto-decoder $D_{\theta}$ to decode a coarse ${SDF}_{c}^{i}$.

where latent vector $z_{i}\in\mathbb{R}^{d}$ encapsulates the implicit shape information for the $i$-th object, $x_{i}\in\mathbb{R}^{3}$ denotes the spatial query coordinate. $D_{\theta}$ is used to reconstruct the initial shape, yielding a coarse ${SDF}_{c}^{i}$. Building upon this coarse representation ${SDF}_{c}^{i}$, the 3D U-Net $U_{\phi}$ further refines the geometry details, resulting in the final ${SDF}^{i}$, providing an accurate shape $S_{re}^{i}$ representation.

For geometrically consistent color learning, we unify the shape and color representation rather than using an additional isolated latent vector to encode color information (see Fig.2). We employ a remeshing method [20] to subdivide the reconstructed shape $S_{re}^{i}$ surface, and increase the number of vertices to enhance the color details. This process is followed by aligning the vertices and using a KD-tree to calculate the color values $P_{c}^{i}$ from ground truth object $S_{gt}^{i}$. We sample random coordinates $p_{i}$ and random color $(R^{p}_{i},G^{p}_{i},B^{p}_{i})$ from $P_{c}^{i}$.

where $(R^{p}_{i},G^{p}_{i},B^{p}_{i})_{gt}$ is the color of random coordinate $p_{i}$ from vertices $v_{i}$ of the $i$-th object and $\Phi_{\theta}$ is the interpolation function. Concurrently, we construct a continuous color field in the neighborhood of shape surface through interpolation to ensure that the decoder can fully learn the color distribution of the object and is generalizable to a broader range of geometries.

We employ a decoder $C_{\theta}$ that is capable of decoding the color of shape, which is inherently mapped to the latent shape representation $z_{i}$.

where $(R_{i}^{p},G_{i}^{p},B_{i}^{p})$ is the color of random coordinates $p_{i}$ of the $i$-th object, $c_{i}\in\mathbb{R}^{d_{2}}$ is the randomly initialized color latent vector, optimized during training.

At training time, we simultaneously optimize the shape decoder $D_{\theta}$, 3D U-Net $U_{\Phi}$, and color decoder $C_{\theta}$.

where $\mathcal{L}_{s}$ is the shape representation loss, $\mathcal{L}_{c}$ is the color mapping loss, and $\lambda_{1}$ and $\lambda_{2}$ are balancing weights. $\mathcal{L}_{s}$ maximizes the joint log posterior over all the training shapes to optimize the shape decoder and is defined as

where $x_{i}$ is the query coordinates, $SDF^{i}_{gt}$ is the ground truth SDF and $L_{1}$ is the L1-norm. The color decoder $C_{\theta}$ is optimized by a combination of regularized L1-norm loss and structural similarity loss since 3D shapes share structural similarities at local scales. We find that the color decoder optimized with structural similarity loss achieves significantly better results compared to using only L1-norm loss.

where $\lambda_{3}$ and $\lambda_{4}$ are balancing weights for the two terms.

At inference time, after training and fixing the model parameters, a shape latent code $\hat{z_{i}}$ and color latent code $\hat{c_{i}}$ for target object $S_{i}$ can be estimated via Maximum-a-Posterior (MAP) estimation as:

This demonstrates that despite employing an auto-decoder architecture, our model is capable of accepting unknown inputs during inference and providing an implicit representation for the object of new inputs. Accordingly, the integrative latent mapping for unified 3D shape and color modeling is capable of simultaneously decoding the $SDF^{i}$ values and color $(R_{i},G_{i},B_{i})$ of shape $S_{i}$.

Our 4D sequence representation marks a significant innovation by eliminating reliance on traditional skeletal structures. We propose a skeleton-free latent joint representation, and then extend the implicit expression in the temporal dimension. Equipped with the trained models $D_{\theta}$, $U_{\phi}$, and $C_{\theta}$, we decode the shape and color of an animated 3D object $S_{i}$ at arbitrary frame from its shape latent vector $z_{i}\in\mathbb{R}^{d_{1}}$ and color latent vector $c_{i}\in\mathbb{R}^{d_{2}}$. This allows us to represent a 4D sequence as a matrix $\mathcal{M}\in\mathbb{R}^{N\times(d_{1}+d_{2})}$, which jointly concatenates the latent vectors $\{z_{i}\in\mathbb{R}^{d_{1}}\}$ and $\{c_{i}\in\mathbb{R}^{d_{2}}\}$ across all frames, effectively capturing the 4D distribution of the sequence.

where $N$ denotes the number of frames, with each column of $\mathcal{M}$ corresponding to a specific frame $n$, and $z_{n}$ and $c_{n}$ representing the shape and color latent vector at a specific frame, respectively. The full 4D sequence is now represented by a single 2D matrix $\mathcal{M}$ in a low-dimensional space. $\mathcal{M}$ provides a compact and efficient representation for the entire 4D sequence. We can then train a diffusion model (see Sec 3.3) with the previously created matrix $\mathcal{M}$ as data points, enabling manipulation and generation of animated objects $S_{1}\cdots S_{N}$.

After the diffusion denoising process, the generated matrix $\mathcal{M^{\prime}}$ is used to reconstruct the full 4D sequence (see Fig. 3). We split the matrix $\mathcal{M^{\prime}}$ along the temporal dimension to get the shape and color latent vectors $z_{n}$ and $c_{n}$ for each frame. We then use the trained models $U_{\Phi},D_{\theta}$ and $C_{\theta}$ to decode each frame $S_{n}$ of the 4D sequence. We first decode the shape $\{v_{n},f_{n}\}$ by:

where $v_{n}$ and $f_{n}$ are the vertices and triangle faces of shape $S_{n}$ at time step $n$, MCubes denoted as marching cube algorithm to reconstruct vertices and triangle faces from SDF values, $x_{n}$ is the query coordinate same to $x_{i}$ in Sec.3.1.

We significantly reduce the dimensionality and computational complexity and export high quality 4D sequence by using $C_{\theta},c_{n}$ and $v_{n}$ to decode the color $(R_{n},G_{n},B_{n})$ and get the object frame $S_{n}$:

where Eq. 9 and 10 represent the process of decoding one frame $S_{n}$ of 4D sequence $S_{1}\cdots S_{N}$ using $\mathcal{M}$. We denote the above operations for brevity as $\mathcal{F}$. We repeat this process to reconstruct the full 4D sequence as

The 4D representation $\mathcal{M}$ allows the diffusion model to efficiently operate across the temporal dimension while imposing temporal consistency (details in Sec. 3.3), resulting in the generation of intricate 4D sequences $S_{1}\cdots S_{N}$ with high fidelity and temporal coherence.

We can now apply diffusion to the 4D sequence representation $\mathcal{M}$ to generate sequences, rather than to the individual 3D animated frames. During the forward diffusion process, data is transformed into Gaussian noise following a Markov chain, while the reverse process performs iterative denoising to recover data from a single Gaussian distribution. For the forward process of the latent diffusion, the original 4D representation $\mathcal{M}_{0}\sim q(\mathcal{M}_{0})$ is gradually corrupted in pre-defined $T$-steps noised distribution following the Markov’s chain assumption until the Gaussian distribution is reached. Based on the Markov property, the joint distribution $\mathcal{M}_{1:T}$ is derived from the original 4D representation $\mathcal{M}_{0}$:

Our diffusion model operates in the latent space to generate a 4D latent of the animated object sequence conditioned by the image-text input (see Fig. 1). We apply a unified description template $p_{n}$ embedded with frame index $n$ to condition coherent sequences. Additionally, to maintain visual consistency, we replicate the image features, using the same image condition $I$ for generation across each frame:

where $\tau_{\beta}$ and $\tau_{\theta}$ are the text encoder and image encoder of CLIP [31], $t_{n}$ is the text condition of $n$-th frame, $\circ$ means concatenation, $F_{n}^{T},F_{n}^{I}$ are the text and image features for the $n$-th frames.

We employ attention mechanisms $\emph{Att.}(Q,K,V)=\text{Softmax}(\frac{QK^{T}}{\sqrt{d}})V$ to guide the generative process, striking a balance between conditional guidance and the temporal coherence of the latent representation $\mathcal{M}_{t}$ (see Fig.4). A frame coherent self-attention module is first used to enhance the consistency among the latent vectors $z_{t}$ and $c_{t}$ within the 4D sequence representation $\mathcal{M}_{t}$:

where $\phi_{i}(\mathcal{M}_{t})$ denotes a intermediate status of $\mathcal{M}_{t}$ in the noise prediction network $\epsilon_{\theta}$, and $W_{Q}^{(i)}\in\mathbb{R}^{d\times d_{\epsilon}^{i}}$, $W_{K}^{(i)}\in\mathbb{R}^{d\times d_{\epsilon}^{i}}$, & $W_{V}^{(i)}\in\mathbb{R}^{d\times d_{\epsilon}^{i}}$ are learnable projection matrices. The encoded features $F_{n}^{T}$ and $F_{n}^{I}$ are then used to control the denosing process of $\mathcal{M}_{t}$ with condition injection cross-attention module:

where $\tau_{i}$ is the task-specific encoder for the $i$-th modality.

We adopt classifier-free guidance for image-text conditional generation of 4D representation $\mathcal{M}_{t}$. The simplified objective proposed by  [33] reads as follows:

where $D$ is a dropout operation enabling classifier-free guidance, and $F$ refers to a simple concatenation.

We conduct extensive ablation studies first.

We conduct a comparative analysis of the quality of 3D shape reconstructions among DeepSDF  [28], SDFusion [9], and our method. SDFusion generates a 3D shape by using a 3D latent diffusion model with a VQVAE decoder, then the texture rendering based strategy is used to color the shape with an off-the-shelf 2D diffusion model. DeepSDF uses a MLP decoder to decode SDF values without color of query points under given latent vector. We compare the accuracy of our model for only geometry on ShapeNet (the car, chair, and airplane) and 3DBiCar datasets (the object’s T-pose) with them. Tab.5 shows that our method outperforms SDFusion and DeepSDF across all metrics on geometry generation, and outperforms the texture rendering based color generation method SDFusion on color generation, which confirms that our proposed latent mapping for unified modeling can learn more shape details and retain the shape-consistent color.

Fig. 7 illustrates the detailed performance of our model and HyperDiffusion [12] in generating 4D sequences, highlighting our model’s ability to maintain temporal consistency while generating sequences with significant motion amplitudes and preserving fine details.

Fig. 8 demonstrates the capability of our 4D representation $\mathcal{M}$ to capture different actions of the same object. It showcases our model’s ability to generate diverse poses for the same object, indicating that our model not only produces 4D sequences rich in detail but also supports a variety of poses.

We then compare our method with [32] as shown in Fig. 9. To maintain consistency with DreamGaussian4D, we train our latent diffusion model only with image conditions, though our approach supports image-text conditions. We set the number of supervision views for DreamGaussian4D to 1 due to the GPU memory limitation. Through the exported 4D sequence $S_{1}\cdots S_{N}$ from DreamGaussian4D, We render images from multiple viewpoints. If viewpoints are outside the range of supervised views used for training, the rendering quality is poor, e.g. incomplete shape and blurred color. This is because single-view optimization will reduce the performance of the texture and geometric generation. Consequently, our approach achieves superior results under the same GPU resource, striking a balance between memory consumption and high-quality sequence generation.

Finally, we showcase our colored 4D sequence generation results in Fig. 10. The results in Fig. 10 indicate that our method can balance the three-way tradeoff between the 3D shape quality, color and sequence coherence since we can effectively generate 4D representation $\mathcal{M}$ and decode it to 4D sequence $S_{1}\cdots S_{N}$ under given the image-text condition.