4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives

The field of novel view synthesis has received widespread attention initiated by Neural Radiance Fields (NeRF) [1]. Querying a MLP for hundreds of points along each ray, NeRF is significantly limited in training and rendering. Subsequent works attempted to improve the speed [15, 16, 17, 18, 19, 20] and enhance rendering quality [21, 3, 2, 22, 23]. Later, 3D Gaussian Splatting (3DGS) [24] achieves real-time rendering at high resolution thanks to its fast differentiable rasterization pipeline while retaining the merits of volumetric representations. Subsequent works [25, 26] have improved rendering [27, 28, 29, 30, 31] and geometric accuracy [32, 33, 34]. Here, we explore the potential of Gaussian primitives in representing dynamic scenes.

Synthesizing novel views of a dynamic scene in a desired time is a more challenging task. The intricacy lies in capturing the intrinsic correlation across different timesteps. Inspired by NeRF, one research line attempts to learn a 6D plenoptic function represented by a well-tailored structure [4, 6, 7, 5, 8]. However, these methods struggle with the coupling between parameters. An alternative approach explicitly models continuous motion or deformation [9, 10, 11, 12]. Among them, point-based approaches have consistently been deemed promising.

Despite impressive novel view synthesis performance, 3DGS imposes substantial storage requirements due to the associated attributes of a large number of Gaussians. To reduce the memory footprint, [35] uses the hash grid to encode the color and quantifies other parameters through residual vector quantization (RVQ) [36], while utilizing a learned volume mask to eliminate Gaussians with small size. [37] introduces a scale- and resolution-aware redundancy score as a criterion for removing redundant primitives. [38] compresses parameters of part of Gaussians in a sensitivity-aware manner. [39] stores the related Gaussians in the same anchor and dynamically spawns neural Gaussians during rendering. [40] further organizes sparse anchor points using a structured hash grid. We systematically explore these compression approaches with our dynamic scene representations.

There have been recent efforts [12, 41, 42, 43, 44, 45, 46, 47] in extending 3DGS for dynamic scenes. A straight approach is to incrementally reconstruct the dynamic scene via frame-by-frame training [12]. [41, 42, 43] model the geometry and dynamics of scenes by joint optimizing Gaussians in canonical space and a deformation field. [44] encourages locality and rigidity between points by factorizing the motion of the scene into a few neural trajectories. These representations hold the topological invariance and low-frequency motion prior, thus well-suited for reconstructing dynamic scenes from monocular videos. However, they assume that dynamic scenes are induced by a fixed set of 3D Gaussians and that all scene elements remain visible. Circumventing the need to maintain ambiguous global tracking, our approach can more flexibly deal with complex real-world scenes.

Efficient simulation of realistic sensor data in diverse driving scenes is crucial for scaling autonomous driving systems. Early works [48, 49] employ traditional graphics techniques but suffer from domain gap of synthesized data and laborious manual modeling. [50] creates realistic novel driving videos by inserting 3D assets according to the depth map.  [51, 52, 53, 54, 55] utilize neural implicit fields to reconstruct the scene from captured driving data, enabling novel view synthesis. Recent works [56, 57, 58, 59] have applied 3DGS to driving scenes, significantly improving fidelity while achieving real-time simulation. Other notable efforts focus on reconstructing scenes without relying on 2D masks or 3D bounding boxes of dynamic objects. Among them, SUDS [60] and EmerNeRF [61] decompose dynamic and static regions by optimizing optical flow, while $S^{3}$Gaussian [62] and PVG [63] achieve this through self-supervision.

Generative capabilities of 3D/4D representations have been explored. In 3D generation, DreamFields [64] and DreamFusion [65] utilize NeRF as a base 3D representation. Magic3D [66] incorporates a textured mesh based on the deformable tetrahedral grid in the refinement stage. Also, 3DGS has been applied to 3D generation [67, 68, 69, 70]. For 4D generation, 4D radiance fields [7, 71, 6] have been adopted [72, 71, 14], followed by Gaussian-based representations [41, 42, 13, 46] explored in [73, 74, 75, 76, 77, 78]. We show the advantages of 4DGS in terms of both generation quality and speed in video-to-4D generation task.

Following the success of Segment Anything Model (SAM) [79] in 2D image segmentation, recent works [80, 81, 82, 83, 84, 85, 86, 87] extend NeRF and 3DGS for scene understanding and segmentation. SAGA [81] applies multi-granularity segmentation to 3DGS using a scale gate mechanism. SA4D [80] conducts 4D segmentation based on deformed 3DGS with single-granularity. We validate the general efficacy of 4DGS for 4D scene understanding.

In 3DGS, a scene is represented as a cloud of 3D Gaussians. Each Gaussian has a theoretically infinite scope and its influence on a given spatial position $x\in\mathbb{R}^{3}$ defined by an unnormalized Gaussian function:

where $\mu\in\mathbb{R}^{3}$ is mean vector, and $\Sigma\in\mathbb{R}^{3\times 3}$ is an anisotropic covariance matrix. Although the Gaussian function defined as equation (1) is unnormalized, it also holds desired properties of normalized Gaussian probability density function critical for our methodology, i.e., the unnormalized Gaussian function of a multivariate Gaussian can be factorized as the production of the unnormalized Gaussian functions of its condition and margin distributions. Hence, for brevity and without causing misconceptions, we do not specifically distinguish between equation (1) and its normalized version in subsequent sections.

In [24], the mean vector $\mu$ of a 3D Gaussian is parameterized as $\mu=(\mu_{x},\mu_{y},\mu_{z})$, and the covariance matrix $\Sigma$ is factorized into a scaling matrix $S$ and a rotation matrix $R$ as $\Sigma=RSS^{\top}R^{\top}$. Here $S$ is summarized by its diagonal elements $S=\mathrm{diag}(s_{x},s_{y},s_{z})$, whilst $R$ is constructed from a unit quaternion $q$. Moreover, a 3D Gaussian also includes a set of coefficients of spherical harmonics (SH) for representing view-dependent color, along with an opacity $\alpha$.

All of the above parameters can be optimized under the supervision of the rendering loss. During the optimization process, 3DGS also periodically performs densification and pruning on the collection of Gaussians to further improve the geometry and the rendering quality.

In rendering, given a pixel $(u,v)$ in view $\mathcal{I}$ with extrinsic matrix $E$ and intrinsic matrix $K$, its color $\mathcal{I}(u,v)$ can be computed by blending visible 3D Gaussians that have been sorted according to their depth, as described below:

where $c_{i}$ denotes the color of the $i$-th visible Gaussian from the viewing direction $d_{i}$, $\alpha_{i}$ represents its opacity, and $p_{i}(u,v)$ is the probability density of the $i$-th Gaussian at pixel $(u,v)$.

To compute $p_{i}(u,v)$ in the image space, we linearize the perspective transformation as in [88, 24]. Then, the projected 3D Gaussian can be approximated by a 2D Gaussian. The mean of the derived 2D Gaussian is obtained as:

where $\mathrm{Proj}\left(\cdot|E,K\right)$ denotes the transformation from the world space to the image space given the intrinsic $K$ and extrinsic $E$. The covariance matrix is given by

where $J$ is the Jacobian matrix of the perspective projection.

To enable Gaussian splatting for modeling dynamic scenes, reformulation is necessary. In dynamic scenes, a pixel under view $\mathcal{I}$ can no longer be indexed solely by a pair of spatial coordinates $(u,v)$ in the image plane; But an additional timestamp $t$ comes into play and intervenes. Formally this is formulated by extending equation (2) as:

Note that $p_{i}(u,v,t)$ can be further factorized as a product of a conditional probability $p_{i}(u,v|t)$ and a marginal probability $p_{i}(t)$ at time $t$, yielding:

We consider the underlying $p_{i}(x,y,z,t)$ follows a 4D Gaussian distribution. As the conditional distribution $p(x,y,z|t)$ is also Gaussian, we can derive $p(u,v|t)$ similarly as a planar Gaussian whose mean and covariance matrix are parameterized by equations (3) and (4), respectively.

Subsequently, it comes to the question of how to represent a 4D Gaussian. An intuitive solution is that we adopt a distinct perspective for space and time. Consider that $(x,y,z)$ and $t$ are independent of each other, i.e., $p_{i}(x,y,z|t)=p_{i}(x,y,z)$, then equation (6) can be implemented by adding an extra 1D Gaussian $p_{i}(t)$ into the original 3D Gaussian. This design can be viewed as imbuing a 3D Gaussian with temporal extension, or weighting down its opacity when the rendering timestep is away from the expectation of $p_{i}(t)$. However, our experiment shows that while this approach can achieve a reasonable fitting of 4D manifold, it is difficult to capture the underlying motion of the scene (see Section VIII-C for details).

To address the mentioned challenge, we suggest treating the time and space dimensions equally by formulating a coherent integrated 4D Gaussian model. Similar to [24], we parameterize its covariance matrix $\Sigma$ as the configuration of a 4D ellipsoid for easing model optimization:

where $S=\mathrm{diag}(s_{x},s_{y},s_{z},s_{t})$ is a diagonal scaling matrix and $R$ is a 4D rotation matrix. On the other hand, a rotation in 4D Euclidean space can be decomposed into a pair of isotropic rotations, each of which can be represented by a quaternion. Specifically, given $q_{l}=(a,b,c,d)$ and $q_{r}=(p,q,r,s)$ denoting the left and right isotropic rotations respectively, $R$ can be constructed by:

The mean of a 4D Gaussian is represented by four scalars as $\mu=(\mu_{x},\mu_{y},\mu_{z},\mu_{t})$. Thus far we arrive at a complete representation of the general 4D Gaussian.

Subsequently, the conditional 3D Gaussian can be derived from the properties of the multivariate Gaussian with

where the numerical subscripts denote the index across each dimension of the matrix. Since $p_{i}(x,y,z|t)$ is a 3D Gaussian, $p_{i}(u,v|t)$ in equation (6) can be derived in the same way as in equations (3) and (4). Moreover, the marginal $p_{i}(t)$ is also a Gaussian in one-dimension:

So far we have a comprehensive implementation of equation (6). Next, we can adapt the highly efficient tile-based rasterizer proposed in [24] to approximate this process, through considering the marginal distribution $p_{i}(t)$ when accumulating colors and opacities.

The previous formulation treats the unnormalized Gaussian defined in equation (1) as a specialized probability distribution and assumes it can be factorized into the product of the unnormalized Gaussian functions of its conditional and marginal distributions. In this paragraph, we provide a concise proof for this property.

First we demonstrate that Gaussian conditional probability formula also holds in equations (5) and (6), i.e.

where

To prove equation (11), since Gaussian conditional probability formula holds for normalized version: $\mathcal{N}(u,v,t|\mu,\Sigma)=\mathcal{N}(t|\mu_{t},\Sigma_{t})\mathcal{N}(u,v|\mu_{uv|t},\Sigma_{uv|t})$, we only need to prove

From Gaussian property, we know that $\Sigma_{uv|t}=\Sigma_{uv}-\Sigma_{uv,t}\Sigma_{t}^{-1}\Sigma_{t,uv}$. Then from the decomposition of $\Sigma$ by

equation (15) holds immediately. The proof can be easily extended onto $p(x,y,z,t)=p(t)p(x,y,z|t)$.

The view-dependent color $c_{i}(d)$ in equation (6) is represented by a series of SH coefficients in the original 3DGS. To more faithfully model the dynamic scenes of the real world, we need to enable appearance variation with both varying viewpoints and evolving time.

Leveraging the flexibility of our framework, a straightforward solution is to directly use different Gaussians to represent the same point at different times. However, this approach leads to duplicated and redundant Gaussians that represent an identical object, making it challenging to optimize. Instead, we choose to exploit 4D extension of the spherical harmonics (SH) that represents the time evolution of appearance. The color in equation (6) could then be manipulated with $c_{i}(d,\Delta t)$, where $d=(\theta,\phi)$ is the normalized view direction under spherical coordinates and $\Delta t$ is time difference between the expectation of the given Gaussian and the viewpoint.

Inspired by studies on head-related transfer function, we propose to represent $c_{i}(d,\Delta t)$ as the combination of a series of 4D spherindrical harmonics (4DSH) which are constructed by merging SH with different 1D-basis functions. For computational convenience, we use the Fourier series as the adopted 1D-basis functions. Consequently, 4DSH can be expressed as:

where $Y^{m}_{l}$ is the 3D spherical harmonics. The index $l\geq 0$ denotes its degree, and $m$ is the order satisfying $-l\leq m\leq l$. The index $n$ is the order of the Fourier series. The 4D spherindrical harmonics form an orthonormal basis in the spherindrical coordinate system.

In optimization, we only use the rendering loss as supervision. In most cases, combining the representation introduced above with the default training schedule as in [24] is sufficient to yield satisfactory results. However, for scenes with more dramatic changes, we observe issues such as temporal flickering and jitter. We consider that this may arise from suboptimal sampling techniques. Rather than adopting the prior regularization, we discover that batch sampling in time turns out to be effective, resulting in more seamless and visually pleasing appearance of dynamic contents.

In terms of density control, simply considering the magnitude of spatial position gradients is insufficient to assess under- and over-reconstruction over time. To address this, we incorporate the gradients of $\mu_{t}$ as an additional density control indicator. Furthermore, in regions prone to over-reconstruction, we employ joint spatial and temporal position sampling during Gaussian splitting.

Table I presents a quantitative evaluation on the Plenoptic Video dataset. Our approach not only significantly surpasses previous methods in terms of rendering quality but also achieves substantial speed improvements. Notably, it stands out as the sole method capable of real-time rendering while delivering high-quality dynamic novel view synthesis within this benchmark. To complement this quantitative assessment, we also offer qualitative comparisons on the “flame salmon” scene, as illustrated in Figure 3. The quality of synthesis in dynamic regions notably excels when compared to other methods. Several intricate details, including the black bars on the flame gun, the fine features of the right-hand fingers, and the texture of the salmon, are faithfully reconstructed, demonstrating the strength of our approach.

We also evaluate our approach on monocular dynamic scenes, a task known for its inherent complexities. Previous successful methods often rely on architectural priors to handle the underlying topology, but we refrain from introducing such assumptions when applying our 4D Gaussian model to monocular videos. Remarkably, our method surpasses all competing methods, as illustrated in Table II. This outcome underscores the ability of our 4D Gaussian model to efficiently exchange information across various time steps.

Our novel approach involves treating 4D Gaussian distributions without strict separation of temporal and spatial elements. In Section III-B, we discussed an intuitive method to extend 3D Gaussians to 4D Gaussians, as expressed in equation (6). This method assumes independence between the spatial $(x,y,z)$ and temporal variable $t$, resulting in a block diagonal covariance matrix. The first three rows and columns of the covariance matrix can be processed similarly to 3D Gaussian splatting. We further additionally incorporate 1D Gaussian to account for the time dimension.

To compare our unconstrained 4D Gaussian with this baseline, we conduct experiments on two representative scenes, as shown in Table III. We can observe the clear superiority of our unconstrained 4D Gaussian over the constrained baseline. This underscores the significance of our unbiased and coherent treatment of both space and time aspects in dynamic scenes. We also qualitatively compare the sliced 3D Gaussians of these two variants in Figure 4. It can be found that under the No-4DRot setting the rim of the wheel is not well reconstructed, and fewer Gaussians are engaged in rendering the displayed frames after filtering, despite a larger total number of fitted Gaussians under this configuration. This indicates that the 4D Gaussian in the No-4DRot setting has less temporal variance, which impairs the capacity of motion fitting and exchanging information between successive frames, and brings more flicker and blur in the rendered videos.

Incorporation of 4D rotation to our 4D Gaussian equips it with the ability to model the motion. Note that 4D rotation can result in a 3D displacement. To assess this scene motion capture ability, we conduct a thorough evaluation. For each Gaussian, we test the trajectory in space formed by the expectation of its conditional distribution $\mu_{xyz|t}$. Then, we project its 3D displacement between consecutive two frames to the image plane and render it using equation (6) as the estimated optical flow. In Figure 5, we select one frame from each scene in the Plenoptic Video dataset to exhibit the rendered optical flow. The result reveals that without explicit motion supervision or regularization, optimizing the rendering loss alone can lead to the emergence of coarse scene dynamics.

If the 4D Gaussian has only local support in time, as the 3D Gaussian does in space, the number of 4D Gaussians may become very intractable as the video length increases. Fortunately, the anisotropic characteristic of Gaussian offers a prospect of avoiding this predicament. To further unleash the potential of this characteristic, we set the initial time scaling to half of the scene’s duration as mentioned in Section VIII-A2.

In order to more intuitively comprehend the temporal distribution of the fitted 4D Gaussian, Figure 6 presents a visualization of mean and variance in the time dimension, by which the marginal distribution on $t$ of 4D Gaussians can be completely described. It can be observed that these statistics naturally form a mask to delineate dynamic and static regions, where the background Gaussians have a large variance in the time dimension, which means they are able to cover a long time period. Actually, as shown in Figure 7, the background Gaussians are able to be active throughout the entire time span of the scene, which allows the total number of Gaussians to grow very restrained with the video length extending.

Moreover, considering that we filter the Gaussians according to the marginal probability $p_{i}(t)$ at a negligible time cost before the frustum culling, the number of Gaussians actually participated in the rendering of each frame is nearly constant, and thus the rendering speed tends to remain stable with the increase of the video length. This locality instead makes our approach friendly to long videos in terms of rendering speed.

To examine whether modeling the spatiotemporal evolution of Gaussian’s appearance is helpful, we ablate 4DSH in the second row of Table III. Compared to the result of our default setting, we can find there is indeed a decline in the rendering quality. Moreover, when turning our attention to 4D spacetime, we realize that over-reconstruction may occur in more than just space. Thus, we allow the Gaussian to split in time by sampling new positions using complete 4D Gaussian as PDF. The last two rows in Table III verified the effectiveness of the densification in time.

Consistent4D test dataset [14]: comprises seven videos with 32 frames, each containing $32\times 4$ ground truth images from four viewpoints at every timestamp. In this dataset, we initialize 50,000 random points inside a sphere of radius 0.5 with uniformly distributed timestamps.

By default, we conduct training with a total of 500 iterations and halted densification at the 100$th$ iteration. The weights in equation (32) are set as $\lambda_{\text{img}}=10,000$ and $\lambda_{\text{sds}}=1$. The other settings for hyperparameters are the same as reconstruction in Section VIII-A2.

We include cascade dynamic NeRF (C-DyNeRF) introduced in [14] and Deform-GS [42] as alternative representations for comparison. Due to different convergence speeds, we train C-DyNeRF for 1000 iterations and Deform-GS for 500 iterations. The other training settings follow those specified in the respective papers.

For each 4D representation, we report the total generation time needed for complete convergence and the time required for a single training iteration. We report the CLIP similarity and LPIPS between rendered images and groundtruth images.

Table IX reports the results on the Consistent4D test dataset. Our 4DGS significantly surpasses all alternatives in terms of both generation time and quality. Figure 11 shows that 4DGS is fastest to converge. We note C-DyNeRF fails to converge in 500 iterations, whereas Gaussian-based representations succeed.

We set feature dimensions $D=32$ and $k=16$ in k-nearest neighbors. We train the features and scale mapping based on compact 4DGS (see Section IV), pre-trained on the Sear Steak sequence from the Plenoptic Video dataset. The training is conducted with 10,000 iterations at a batch size of 1. In inference, as [81] we cluster Gaussians in 4D space based on the learned features using HDBSCAN, achieving automatic scene decomposition. For 4D segments, we first extract the 2D feature from the rendered feature map at a user-specific 2D point and then filter the 4D Gaussians whose smoothed scale-gated feature similarity with the query feature is below a certain threshold (typically 0.8). The filtered 4D Gaussians represent a certain component most similar to the 2D point and can be rendered from any views at any timestamp. For both scene decomposition and 4D segments, we adjust the input scale from 0 to 1 to achieve multi-granularity segmentation.

Figure 10 shows the multi-granularity segmentation results. The learned features enable the 4D Gaussians to cluster automatically and adaptively as the scale varies. At scale 0, individual elements like the man’s hair, clothing, and arms are distinctly segmented. As the scale increases to 1, the entire figure of the man is segmented as a whole. This multi-granularity segmentation capability also extends to 4D segments. For example, the Gaussians representing the man’s whole body, head, or arm can be identified at different scales with minimal blurry floaters.