Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction

In recent years, we have witnessed significant progress in the field of novel view synthesis empowered by Neural Radiance Fields. While vanilla NeRF (Mildenhall et al., 2020) manages to synthesize photorealistic images for any viewpoint using MLPs, numerous subsequent works focus on acceleration (Fridovich-Keil et al., 2022; Hu et al., 2022; Hedman et al., 2021; Müller et al., 2022), real-time rendering (Chen et al., 2022b; Yu et al., 2021a), camera parameter optimization (Bian et al., 2023; Lin et al., 2021; Wang et al., 2023), few-shot learning (Zhang et al., 2022; Yang et al., 2023; Yu et al., 2021b), unbounded scenes (Barron et al., 2022; Gu et al., 2022), improving visual quality (Barron et al., 2021, 2023), and so on.

More recently, a novel framework 3D Gaussian Splatting (Kerbl et al., 2023) has received widespread attention for its ability to synthesize high-fidelity images for complex scenes in real-time along with a fast training speed. The key insight lies in that it exploits a point-like representation, referred to as 3D Gaussians. However, these works are mainly restricted to the domain of static scenes.

To extend neural rendering to dynamic scenes, current efforts primarily focus on deformation-based (Pumarola et al., 2021; Park et al., 2021b; Tretschk et al., 2021) and flow-based methods  (Li et al., 2022, 2021b; Du et al., 2021; Xian et al., 2021). However, these approaches share similar issues as NeRF, including slow training and rendering speed. To mitigate the efficiency problem, various acceleration techniques like voxel (Fang et al., 2022; Liu et al., 2022) or hash-encoding representation (Park et al., 2023), and spatial decomposition (Fridovich-Keil et al., 2023; Shao et al., 2022; Cao & Johnson, 2023; Wu et al., 2022) have emerged. Given the increased complexity brought by dynamic scene modeling, there still exists a gap in rendering quality, training time, and rendering speed compared to static scenes.

Concurrent with our work, methods like Deformable 3D Gaussians (D-3D-GS) (Yang et al., 2024), 4D-GS (Wu et al., 2024), and Dynamic 3D Gaussians (Luiten et al., 2024) leverage 3D-GS as the scene representation, expecting this novel point-like representation can facilitate dynamic scene modeling. D-3D-GS directly integrates a heavy deformation network into 3D-GS, while 4D-GS combines HexPlane (Cao & Johnson, 2023) with 3D-GS to achieve real-time rendering and superior visual quality. Dynamic 3D Gaussians proposes a method that simultaneously addresses the tasks of dynamic scene novel-view synthesis and 6-DOF tracking of all dense scene elements. While our method also takes 3D-GS as the scene representation, unlike any of the aforementioned methods, our main motivation is to aggregate 3D Gaussians with similar deformations into a superpoint to significantly decrease the computational expense required.

There exists a long line of research works on superpixel/superpoint segmentation and we refer readers to the recent paper (J & Kumar, 2023) for a thorough survey. Here we focus on neural network-based methods.

On one hand, methods including SFCN (Yang et al., 2020), AINet (Wang et al., 2021), and LNS-Net (Zhu et al., 2021) adopt a neural network for generating superpixels. SFCN utilizes a fully convolutional network associated with an SLIC loss, while AINet introduces an implantation module and a boundary-perceiving SLIC loss for generating superpixels with more accurate boundaries. LNS-Net proposes an online learning framework, alleviating the demand for large-scale manual labels. On the other hand, existing methods for point cloud over-segmentation can be divided into two categories: optimization-based methods (Papon et al., 2013; Lin et al., 2018; Guinard & Landrieu, 2017; Landrieu & Obozinski, 2016) and deep learning-based methods  (Landrieu & Boussaha, 2019; Hui et al., 2023, 2021).

Our approach can be treated as an over-segmentation of dynamic point clouds, which is an unexplored realm. Existing superpixel/superpoint methods cannot be directly applied to our task since it is challenging to maintain superpoint-segmentation consistency across the temporal domain. Moreover, prevalent methods either employ computationally intensive backbones or do not support parallelization, making the segmentation a heavy module, which will hinder the fast training and rendering speed of our approach.

3D Gaussian Splatting(3D-GS) (Kerbl et al., 2023) propose a novel point-like scene representation, referred to as 3D Gaussians $\mathcal{G}=\{G_{i}:\bm{\mu}_{i},\bm{s}_{i},\bm{q}_{i},\bm{\sigma}_{i},\bm{h}_{i}\}$. Each 3D Gaussian $G_{i}$ is defined by a 3D covariance matrix $\mathbf{\Sigma}_{i}$ in world space (Zwicker et al., 2001a) and a center location $\bm{\mu}_{i}$, following the expression:

For differentiable optimization, the covariance matrix $\mathbf{\Sigma}_{i}$ can be break down into a scaling matrix $\mathbf{S}_{i}$ and a rotation matrix $\mathbf{R}_{i}$, i.e., $\mathbf{\Sigma}_{i}=\mathbf{R}_{i}\mathbf{S}_{i}\mathbf{S}_{i}^{\top}\mathbf{R}_{i}^{\top}$, where $\mathbf{S}_{i}$ is represented by a 3D vector $\bm{s}_{i}$ and $\mathbf{R}_{i}$ is represented by a quaternion $\bm{q}_{i}\in\mathbf{SO}(3)$.

In the process of rendering a 2D image, 3D-GS projects 3D Gaussians onto a 2D image plane using the EWA Splatting algorithm (Zwicker et al., 2001b). The corresponding 2D Gaussian, defined by a covariance matrix $\mathbf{\Sigma}^{\prime}$ in camera coordinates centered at $\bm{\mu}^{\prime}$, is calculated as follows:

where $\mathbf{W}$ is the world-to-camera transformation matrix, and $\mathbf{J}$ is the Jacobian matrix of the affine approximation of the projective transformation. After sorting 3D Gaussians by depth, 3D-GS renders the image using volumetric rendering (Drebin et al., 1988) (i.e. $\alpha$-blending). The color $C(\bm{p})$ of pixel $\bm{p}$ is computed through blending $P$ ordered 2D Gaussians, as expressed by:

where $\sigma_{i}$ represents the opacity of each 3D Gaussian, $\bm{c}_{i}$ is the RGB color computed using the spherical harmonics coefficients $\bm{h}_{i}$ of the 3D Gaussian and the view direction.

To optimize a static scene and facilitate real-time rendering, 3D-GS introduced a fast differentiable rasterizer and a training strategy that adaptively controls 3D Gaussians. Further details can be found in 3D-GS (Kerbl et al., 2023), and the loss function utilized by 3D-GS is $\mathcal{L}_{1}$ combined with a D-SSIM term:

where $\lambda$ is set to $0.2$.

It is evident that 3D-GS is suitable solely for representing static scenes. Therefore, when confronted with a monocular/multi-view video capturing a dynamic scene, we opt to learn 3D Gaussians in a canonical space and the deformation of each 3D Gaussian across the temporal domain under the guidance of aggregated superpoints. Since we assume there are only rigid transformations for every single 3D Gaussian, only the center location $\bm{\mu}_{i}$ and rotation matrix $\mathbf{R}_{i}$ of a 3D Gaussian will vary with time, while other attributes (e.g., opacity $\bm{\sigma}_{i}$, scaling vector $\bm{s}_{i}$, and spherical harmonics coefficients $\bm{h}_{i}$) remain invariant.

To model dynamic scene, we divide 3D Gaussians into $M$ superpoints $\{\mathbb{S}_{j}\}_{j=1}^{M}$ (i.e. disjoint sets). Each superpoint $\mathbb{S}_{j}$ contains several 3D Gaussians, while each 3D Gaussian has only one correspondent superpoint $\mathbb{S}_{j}$. Following the principle of As-Rigid-As-Possible, 3D Gaussians in the same superpoint $\mathbb{S}_{j}$ should have similar deformation, which can represented by relative translation $\Delta\bm{t}_{j}$ and rotation $\Delta\mathbf{R}_{j}$ based on their center locations and rotation matrices in the canonical space. Therefore, the center location $\bm{\mu}_{i}^{t}$ and rotation matrix $\mathbf{R}_{i}^{t}$ of the $i$-th 3D Gaussian at time $t$ will be:

So as to predict the relative translation $\Delta\bm{t}_{j}^{t}$ and rotation $\Delta\mathbf{R}_{j}^{t}$ of the $j$-th superpoint at time $t$, we directly employ a deformation neural network $\mathcal{F}$ that takes the timestep $t$ and canonical position $\bm{p}^{c}_{j}$ of the $j$-th superpoint as input, and outputs the relative transformations of superpoints with respect to the canonical space:

where $\gamma$ denotes the positional encoding:

In our experiments, we set $L=10$ for $\gamma(\bm{p}^{c}_{j})$ and $L=6$ for $\gamma(t)$.

During inference, to further decrease the rendering time, we can pre-compute the relative translation $\Delta\bm{t}^{t}$ and rotation $\Delta\mathbf{R}^{t}$ of superpoints predicted by the deformation network $\mathcal{F}$ at all timesteps. When rendering novel view images at a new timestep $t$ in the training set, the deformation of $j$-th superpoint can be calculated through interpolation:

where the linear interpolation weight $w=(t-t_{1})/(t_{2}-t_{1})$, and $t_{1}$ and $t_{2}$ are the two nearest timesteps in the training dataset.

The key insight of superpixels/superpoints lies in that pixels/points with similar properties should be aggregated into one group. Following this idea, given an arbitrary timestep $t$, properties including the position $\bm{p}^{t}$, the relative translation $\Delta\bm{t}^{t}$ and relative rotation $\Delta\mathbf{R}^{t}$ should be similar within one superpoint. We utilize a learnable association matrix $\mathbf{A}\in\mathbb{R}^{P\times M}$ to establish the connection between 3D Gaussians and superpoints, where $P$ is the number of 3D Gaussians and $M$ is the number of superpoints. Notably, only $K$ nearest superpoints of each Gaussian should be considered. Therefore, the associated probability $a_{ij}$ between Gaussian $G_{i}$ and superpoint $\mathbb{S}_{j}$ can be calculated as:

where $\mathcal{N}_{i}$ is the set of $K$-nearest superpoints for the $i$-th Gaussian in the canonical space.

With the associated probability $a_{ij}$, the properties $\bm{u}_{j}\in\{\bm{p}^{t}_{j},\Delta\mathbf{R}_{j}^{t},\Delta\bm{t}_{j}^{t}\}$ of $j$-th superpoint can be reconstructed from the properties of Gaussians:

where $\bm{v}_{i}$ denotes the properties of $i$-th Gaussian, and ${i\mid j\in\mathcal{N}_{i}}$ means all 3D Gaussians $i$ with the $j$-th superpoint in $\mathcal{N}_{i}$. It is noteworthy that the relative rotation $\Delta\mathbf{R}_{i}^{t}$ is represented by Lie algebra $\mathfrak{s}\mathfrak{e}3$, which enables linear rotation interpolation. On the other hand, the properties $\bm{v}_{i}$ of the $i$-th Gaussian can also be reconstructed through adjacent superpoints:

Ultimately, the property reconstruction loss is employed to ensure the consistency between the original properties $\bm{v}_{i}$ of Gaussians and the reconstructed properties $\bm{v}^{\prime}_{i}$:

where $\bm{v}^{\prime}=R_{sp\to g}(R_{g\to sp}(\bm{v}))$, and $\|\cdot,\cdot\|$ denotes the mean square error(MSE). And the more similar the Gaussian properties within the same superpoint are, the smaller this loss will be, thereby fully exploiting the As-Rigid-As-Possible feature.

Furthermore, the corresponding superpoint $\mathbb{S}_{j}$ of Gaussian $G_{i}$ is the superpoint with the highest association probability:

It is noteworthy that $j^{*}$ is the same as the $j$ in Eq. 5.

The computation of the overall loss function is:

where $\lambda_{\bm{v}}$ represents hyper-parameters controlling the weights, with $\lambda_{\bm{\mu}^{t}}=10^{-3}$ and $\lambda_{\Delta\mathbf{R}^{t}}=\lambda_{\Delta\bm{t}^{t}}=1$.

We implement our SP-GS with PyTorch, and $\mathcal{F}$ is a 8-layer MLPs with 256 hidden neurons. The network is trained for a total of 40k iterations, with the initial 3k iterations training without the deformation network $\mathcal{F}$ as a warm-up process to achieve relatively stable positions and shapes. 3D Gaussians in the canonical space will be initialized after the warm-up training, and for the initialization of superpoints, $M$ Gaussians are sampled using the farthest point sampling algorithm, and the canonical positions $\bm{p}^{c}$ of superpoints are equal to the centers of the sampled Gaussians. Moreover, the $A_{ij}$ of the learnable association matrix $\mathbf{A}$ will be initialized as 0.9 if the $j$-th superpoint is initialized with the $i$-th 3D Gaussian. Otherwise, $A_{ij}$ will be initialized as 0.1. Before each iteration, we calculate the canonical position of superpoints with Eq. 10.

The Adam optimizer (Kingma & Ba, 2015) is employed to optimize our models. For 3D Gaussians, the training strategies are the same as those of 3D-GS unless stated otherwise. For the learnable parameters of $\mathcal{F}$, the learning rate undergoes exponential decay, ranging from 1e-3 to 1e-5. The values for Adam’s $\beta$ are set to (0.9, 0.999).

Given the potential existence of non-rigid deformation in a dynamic scene, another optional non-rigid deformation network $\mathcal{G}$ is employed to learn the non-rigid deformation of each Gaussian for time $t$:

By combining rigid motion with non-rigid deformation, the final center $\bm{\mu}^{t}_{i}$ and rotation matrix $\mathbf{R}^{t}_{i}$ of Gaussian $G_{i}$ can be computed as below:

For the version incorporating the non-rigid deformation network $\mathcal{G}$ (abbreviated as SP-GS+NG), the model is initialized with the pretrained model from the version with only $\mathcal{F}$ and trained for 20k iterations using the loss $\mathcal{L}_{img}$. Besides, $\mathcal{G}$ is a 3-layer MLPs with 64 hidden neurons.

The D-NeRF dataset consists of 8 videos, each containing 50-200 frames. The frames together with camera pose serve as the training data, while test views are taken from novel views. Quantitative and qualitative results are shown in Tab.1 and Fig.2. Though rendered at a resolution of $800\times 800$, we achieved a much higher FPS than previous non-Gaussian-Splatting based methods. As for D-3D-GS, they directly apply a deformation network to every single 3D Gaussian for a higher visual quality, leading to a much lower FPS than ours. We achieve superior or comparable results against previous state-of-the-art methods in terms of all metrics. It’s noteworthy that Lego is excluded while calculating the metrics as we observed a discrepancy in all methods. Please refer to Fig. 2 for a visualized results. Per-scene comparisons are provided in Appendix C.1.

The HyperNeRF dataset (Park et al., 2021a) and NeRF-DS (Yan et al., 2023) serve as two real-world benchmark dataset captured using either one or two cameras. For a fair comparison with previous methods, we use the same vrig scenes, a subset of the HyperNeRF dataset. Quantitative and qualitative results of the HyperNeRF dataset are shown in Tab.2 and Fig.4, while results of NeRF-DS are shown in Tab. 3 and Fig. 3. As shown in Tab. 2 and Tab. 3, our method outperforms baselines by a large margin in terms of FPS while achieving a superior or comparable visual quality. As shown in Fig.4, our results exhibit notably higher visual quality, particularly in the hand area.

In our paper, we introduce two hyperparameters: the number of superpoints and the number of nearest neighborhoods $\mathcal{N}_{i}$. We conduct experiments on testing how sensitive our method is to the variation of the two hyperparameters. Tab. 4 shows the performance of our approach when varying these hyperparameters, and our method appears to be robust under all these variations. Besides, we introduce property reconstruction loss to facilitate grouping similar Gaussians together. Tab. 5 demonstrates that property reconstruction loss can improve rendering quality. We provide more ablations on property reconstruction loss in Appendix.D.

As depicted in Fig. LABEL:fig:vis, we provide a visualized result of the 3D Gaussians and superpoints for the hook scene of D-NeRF (Pumarola et al., 2021). Notably, we observe that the superpoints are uniformly distributed in the space while nearby 3D Gaussians will be aggregated into one superpoint. For a more intuitive understanding, readers can refer to our project page for more videos.

In scenarios where a 3D-GS based model $\mathcal{H}$ exhibits superior performance, which also predicts the deformation over time, we can distill such a model into SP-GS to improve the visual quality. The concept is straightforward: we can directly replicate the state of $\mathcal{H}$ at any given time as the canonical state of SP-GS. Subsequently, we optimize the association matrix $\mathbf{A}$, the superpoint deformation network $\mathcal{F}$, and optionally the non-rigid deformation network $\mathcal{G}$ by incorporating $\mathcal{L}$ loss and mean square error(MSE) $\mathcal{L}_{err}=\sum_{\bm{u}\in\{\bm{\mu},\mathbf{R}\}}\lambda_{\bm{u}}\sum_{i}\|\bm{u}^{t}_{i},\bm{u}^{\prime t}_{i}\|$, where $\bm{u}^{t}_{i}$ and $\bm{u}^{\prime t}_{i}$ are the properties of the teacher and the student respectively. Tab. 6 shows the quantitative results of distilling D-3D-GS model into SP-GS model on the “As” scene of NeRF-DS dataset. While D-3D-GS cannot achieve real-time rendering on V100 (20.65 FPS), our distillated student model can achieve significantly higher rendering speed (164.04 FPS). Therefore, model distillation provides a trade-off between visual quality and rendering speed, leaving users with more choices to meet their requirements.

Our SP-GS support estimating the 6-DoF pose of each superpoint for new images in the same scene. To be specific, we can solely learn the translation and rotation for each superpoint with other parameters of SP-GS fixed. This can be potentially used in motion capture where novel view images are given and one wants to know the motion of each components (superpoints). Experiments are conducted on the jumpingjacks scene of D-NeRF dataset (Pumarola et al., 2021). We first train the complete model (SP-GS) using the beginning 50 images. Subsequently, we initialize the learnable translation and rotation parameters of superpoints with the 50-th frame and directly optimize them with rendering loss using the Adam optimizer for 1000 iterations. The program terminates upon completing the pose estimation for all images. Fig. LABEL:fig:repose illustrates the change in PSNR across images 51-88 for the jumpingjacks scene. Notably, it demonstrates a gradual decrease in PSNR.

As depicted in Figure LABEL:fig:edit, scene editing tasks such as relocating parts between scenes or removing parts from a scene can be accomplished with ease. This capability is facilitated by the explicit 3D Gaussian representation, enabling relocation or deletion from the scene. Our method further streamline the process since it is no longer necessary to manipulate over the 100,000 3D Gaussians. Moreover, our superpoints provide some extent of semantic meanings, enabling a reasonable editing of the scene.