EGSRAL:An Enhanced 3D Gaussian Splatting based Renderer with Automated Labeling for Large-Scale Driving Scene

Deformable 3D GS Network. To reduce data dependency, we use only image data for driving scene reconstruction. We begin by initializing a set of 3D Gaussians $G_{0}\left(\boldsymbol{x},\boldsymbol{r},\boldsymbol{s},\sigma\right)$ through SfM, where $\boldsymbol{x}$, $\boldsymbol{r}$, $\boldsymbol{s}$, and $\sigma$ represent the position of the Gaussian primitive, quaternion, scaling, and opacity, respectively. To better model dynamic 3D Gaussians, the Deformable 3D GS introduces a deformation field for the 3D Gaussians. The deformation field takes the position $\boldsymbol{x}$ and time $t$ as inputs to predict offsets $\delta\boldsymbol{x}$, $\delta\boldsymbol{r}$, and $\delta\boldsymbol{s}$ for $\boldsymbol{x}$, $\boldsymbol{r}$, and $\boldsymbol{s}$, respectively. Subsequently, these deformed 3D Gaussians $G_{1}\left(\boldsymbol{x}+\delta\boldsymbol{x},\boldsymbol{r}+\delta\boldsymbol{r},\boldsymbol{s}+\delta\boldsymbol{s},\sigma\right)$ are then passed through a differentiable tile rasterizer (Kerbl et al. 2023) to render the novel image:

where $C_{\{u,v\}}$ denotes the rendered color of the $(u,v)$ pixel, $T_{i}$ is the transmittance defined by $\Pi_{j=1}^{i-1}(1-\alpha_{j})$, $c_{i}$ is the color of each Gaussian primitive, and $\alpha_{i}$ is calculated by evaluating a 2D Gaussian with covariance matrix (Yifan et al. 2019) multiplied by a learned per-primitive opacity.

Deformation Enhancement Module (DEM). To further enhance the modeling of dynamic objects, we propose a deformation enhancement module that fine-tunes the deformation field based on the time $t$ and the state attribute $\boldsymbol{d}$ of the Gaussian primitive. Specifically, we use the state attribute $\boldsymbol{d}$ and time encoding $\gamma(t)$ as inputs to output an adjustment factor $\alpha_{p}$ for each Gaussian primitive’s deformation field:

where $\mathcal{F_{\beta}}$ represents the dynamic encoding network’s parameters, $\boldsymbol{c}_{view}$ is the current camera’s coordinate position, and $sigmoid(\cdot)$ is the activation function.

Opacity Enhancement Module (OEM). As seen in the image rendering formula (Equation 1), the rendering of the current pixel depends on the color and opacity of the Gaussian primitive. To enhance the capacity of opacity prediction, we initialize the Gaussian primitive by initializing opacity to a trainable parameter $\boldsymbol{\sigma}^{{}^{\prime}}\in R^{16\times 1}$ and introduce a lightweight network to accelerate opacity optimization:

where $\mathcal{F_{\eta}}$ is the opacity enhancement module’s parameters.

Grouping Strategy (GPS). Previous methods (Zhou et al. 2024; Yan et al. 2024) for reconstructing driving scenes using 3D GS have not accounted for large-scale scenes. To address this, we propose a grouping strategy. As illustrated in Figure 2(a), we divide the scene into $N$ groups using a fixed image interval and assign each Gaussian primitive a group identification $id$, which allows us to perform subsequent cloning, splitting, and rendering based on these group $id$s. By grouping strategy, we solve the problem of unreasonable fields of view. As shown in Figure 2(c), the original 3D GS rendering field of view includes distant Gaussian primitives (green rectangle), which is impractical. By grouping (red rectangle), we limit the field of view to a specific range, thus reducing the optimization burden by excluding Gaussian primitives outside this range.

Adaptor Requirement. Our method leverages image sequences (dataset) to construct scene point clouds and estimate camera poses and parameters using SfM methods. The generated point clouds and camera poses are defined in a new coordinate system generated by the SfM methods. However, the 3D annotations of the image sequences and corresponding camera poses are defined in the original world coordinate system, as in nuScenes (Caesar et al. 2020). Consequently, there exist two different coordinate systems: the original world coordinate system (OWCS) and the estimated world coordinate system (EWCS) from SfM methods.

Our Adaptor. We take a neural network to model the transformation relationship of camera poses from the two coordinate systems, which takes camera poses in the OWCS as input and predicts the corresponding camera poses in the EWCS. We represent the camera pose as a 3$\times$4 matrix that includes both rotation and translation information. Therefore the shapes of input and output are both 3$\times$4. We take MLPs to build our adaptor, as shown in Figure 3, the backbone of our adaptor is composed of 8 layers of MLP, and the output head is a simple linear layer, which is used to predict camera poses in the EWCS. To optimize our adaptor, we introduce three constraints during the training process.

Constrains for Adaptor. To effectively train the adaptor, we need camera poses from the dataset in both coordinate systems. It is straightforward to obtain the existing camera pose pairs from these systems, where each pair consists of camera poses of the same frame from both the OWCS and the EWCS. We adopt the $smooth_{L_{1}}$ loss (Girshick 2015) to ensure that the adaptor’s predictions closely match the corresponding SfM camera poses in the EWCS, given the original camera poses in the OWCS. The loss function for camera pose constraints is:

where $L_{p}$ is the loss of the existing camera pose constraint, $P_{A}$ is the pose predicted by the adaptor given camera pose $P$ in the OWCS, and $P_{S}$ is the corresponding pose generated by SfM in the EWCS. After applying the initial constraint, the adaptor can transform existing camera poses between the two coordinate systems. However, due to the limited number of camera pose pairs, the adaptor’s generalization ability is restricted, resulting in poor performance with novel camera poses not included in the existing pairs. To address this issue, we introduce new camera pose constraints.

The novel camera pose constraints aim to enhance the adaptor’s generalization ability by utilizing existing data to constrain new camera poses. Based on the projection consistency rule of the same objects in two camera coordinate systems, we can construct projection constraints for new camera poses. Specifically, for a camera pose $P$ in the dataset, we use a Random Position Transformation (RPT) module to sample a nearby novel pose $P_{nov}$. This pose has a corresponding camera pose $P_{nov\_S}$ in the EWCS, which is implicit and not directly available. Next, we take the following $N$ camera pose pairs after the current camera pose and project them onto the corresponding novel camera pose planes in both coordinate systems. The corresponding projected points should have the same pixel coordinates. As shown in Figure 3, the coordinate conversion module (CCM) is used to convert points (the $N$ camera poses) in the OWCS to the camera coordinate system firstly by leveraging coordinate transformation formulas shown in Equation 7. In the coordinate system of the novel camera pose, the relative pose of $P_{n+i}$ is represented as $\hat{P}_{n_{nov}}^{n+i}$.

where $P_{w}^{n+i}$ is the camera pose of the $frame_{n+i}$ in the OWCS, $P_{w}^{n_{nov}}$ is the novel camera pose, sampled nearby the current frame $frame_{n}$ in the OWCS. Then, we project these points to the novel camera pose plane. As shown in Figure 3, the point project module (PPM) is used to project points from the camera coordinate system to the pixel coordinate system. Next, we calculate the coordinate of $\hat{P}_{n_{nov}}^{n+i}$ in the pixel coordinate system of the novel camera pose plane by utilizing the camera intrinsic parameters, as illustrated in Equation 8 and 9 below:

where $K_{intr}$ is camera intrinsic parameter matrix, and $T_{\hat{P}_{n_{nov}}^{n+i}}$ is the translation of $\hat{P}_{n_{nov}}^{n+i}$. Then the projection constraint can be summarized as shown in Equation 10.

where $\tilde{P}_{{n_{nov}}\_A}^{n+i}$ is the pixel coordinate of the relative camera pose $\hat{P}_{n_{nov}\_A}^{n+i}$ in the EWCS projected to the novel camera pose $P_{n_{nov}\_A}$ plane. $P_{n_{nov}\_A}$ is predicted by our adaptor fed with novel camera pose $P_{n_{nov}}$ as shown in Figure 3. Our adaptor aims to ensure that $P_{n_{nov}\_A}$ is equal to $P_{n_{nov}\_S}$ by making $\tilde{P}_{{n_{nov}}\_A}^{n+i}$ equal to $\tilde{P}_{{n_{nov}}\_S}^{n+i}$. $\tilde{P}_{{n_{nov}}\_S}^{n+i}$ is estimated by using the projected pixel coordinates from the corresponding camera poses in the OWCS as shown in Figure 3 based on the projection consistency rule.

where $\hat{P}_{{n_{nov}}\_A}^{n+i}$ is the 3D coordinate of the relative camera pose of camera pose $P_{S}^{n+i}$ in the EWCS relative to novel camera pose $P_{n_{nov}\_A}$, as shown in Figure 3. $\hat{P}_{{n_{nov}}\_S}^{n+i}$ is the estimated 3D coordinate of the relative camera pose of the camera pose $P_{S}^{n+i}$ in the EWCS, relative to novel camera pose $P_{n_{nov}\_S}$, obtained using the 3D coordinates from the corresponding camera poses in the OWCS. $L_{3D}$ is the loss of novel camera pose constraints in 3D coordinates, $L_{proj}$ is the loss of novel camera pose constraints in project pixel coordinates, $w_{1}$ is the weight of $L_{p}$, $w_{2}$ is the weight of $L_{3D}$, $w_{3}$ is the weight of $L_{proj}$.

Annotation Generation. During the inference phase, as illustrated in Figure 1, we start with the original camera pose $P_{ori}$ in the OWCS from the dataset. This pose undergoes an affine random transformation to obtain $P_{nov}$ by the RPT module shown in Figure 3. Then we feed $P_{nov}$ into the adaptor to produce the corresponding pose in the EWCS, denoted as $P_{nov\_s}$. Feeding $P_{nov\_s}$ into the renderer generates the corresponding novel view image, called $image_{nov}$. Similarly, applying the same affine transformation to the original dataset’s annotations $anns_{ori}$ results in $anns_{nov}$, which corresponds to $image_{nov}$. Combining $image_{nov}$ with $anns_{nov}$ results in a novel view image with annotations, effectively accomplishing auto labeling.

KITTI Dataset. KITTI dataset (Geiger, Lenz, and Urtasun 2012) contains a wide range of driving scenes. Our experiments focus on a single distinct scene: City, which consists of 1424 train images and 159 test images.

NuScenes Dataset. NuScenes (Caesar et al. 2020) stands as a comprehensive autonomous driving dataset, featuring 3D bounding boxes for 1000 scenes. We follow the different scenes selected by S-NeRF (Xie et al. 2023) and DrivingGaussian (Zhou et al. 2024) for our experiments.

Implementation Details. Our implementation is primarily based on Deformable 3D GS (Yang et al. 2024), maintaining consistency with its training iterations and optimizer settings. Additionally, the DEM and the OEM are optimized using Adam with a learning rate of 0.001.

Adaptor Model Training Settings. Our training dataset is nuScenes poses $P$ and poses $P_{S}$ generated by SfM as data pairs. We need to train on 17 different scenes, each with approximately 230 pose pairs. Each scene requires training for 1000 epochs. We use the Adam optimizer with a learning rate of 2e-4 and a batch size of 16. It takes about 2 hours to train using AMD MI100. Adaptor model loss weights $w_{1}$, $w_{2}$ and $w_{3}$ in Equation 12 are 50, 0.1 and 1, respectively, and the $N$ camera pose is 15.

2D/3D Detection Settings. To verify the effectiveness of synthetic data, we use mainstream detection model to perform 2D and 3D detection tasks on four categories: car, bus, truck, and trailer. 2D detection tasks use Co-DETR (Zong, Song, and Liu 2023a) and 3D tasks use MonoLSS (Li, Jia, and Shi 2024). For the test dataset, we select the first fifteen scene data in the nuScenes validation set.

Evaluation on KITTI. We compare our method with state-of-the-art (SOTA) methods, as shown in Table  1. Our approach demonstrates superior performance compared to other 3D GS methods, with PSNR and SSIM improvements of 2.27 and 0.074, respectively, and 0.114 reduction in LPIPS over Deformable 3D GS. Additionally, compared to other methods like READ, our method enhances the PSNR metric to 23.60 while achieving comparable SSIM results. However, the 3D GS-based method has a slight deficiency in modeling high-frequency information, such as ground texture, which leads to a higher LPIPS. For detailed visualization results, please refer to the supplementary materials.

Evaluation on nuScenes. Following the scenes used by DrivingGaussian, as shown in Table 2, our method achieves the highest performance, with an increase of 0.3 and 0.018 in PSNR and SSIM, respectively, and a decrease of 0.075 in the LPIPS. It is worth noting that DrivingGaussian trains the dynamic objects and the static backgrounds separately, training an individual 3D GS model for each dynamic object then merging them. In contrast, our method requires only a single training process and does not need any annotations, resulting in better performance and reducing the dependence on data annotations.

Performance of 2D/3D Object Detection. We select 17 different scenes containing 3,898 images (674 of which are sample images with annotations) from the nuScenes dataset as the baseline dataset for the detection task. We produce the annotations for the non-sample images based on the annotations of sample images by camera coordinate system transformation. To augment the dataset, we synthesize additional images using an adaptor, increasing the total number of images to two and three times that of the baseline dataset. The 2D detection results are presented in Table 4, and the 3D detection results are presented in Table 5. In these tables, $sample$ refers to the 674 sample images, while $all$ refers to the 3,898 images, including both sample and non-sample images. The results demonstrate that, without any additional input, our data augmentation method improves the accuracy of the detection model in both the $sample$ and the $all$ dataset.

KITTI Selected by READ. For the KITTI dataset, we follow the setup of READ (Li, Li, and Zhu 2023) and use the single camera as input, with a resolution of $1232\times 368$. We select every 10th image of cameras in the sequences as the test set. See Table 7 for the detailed hyperparameter settings.

NuScenes-S Selected by S-NeRF. For the NuScenes-S dataset, we follow the setup of S-NeRF (Xie et al. 2023) and only use the front camera as input, with a resolution of $1600\times 900$. We select every 4th image of cameras in the sequences as the test set. The specific scene tokens are 164, 209, 359, and 916. We report the average results of all camera frames on the selected scenes and assess our models using the average score of PSNR, SSIM, and LPIPS. See Table 7 for the detailed hyperparameter settings.

NuScenes-D Selected by DrivingGaussian.

3D Detector. MonoLss (Li, Jia, and Shi 2024) achieves state-of-the-art performance in the primary categories of car, cyclist, and pedestrian on the KITTI monocular 3D object detection benchmark, as shown in Table 10. It also demonstrates strong competitiveness in cross-dataset evaluations on the Waymo, KITTI, and nuScenes datasets. At the time of our experiments, MonoLss is the leading open-source 3D detection model that relies solely on monocular image input.

Effect of the Group Number in GPS. In Table 13, we present the results of our grouping strategy under different numbers of groups. With a small number of groups, the large scene leads to an unreasonable field of view after grouping, preventing effective training for each group. Conversely, with too many groups, the correct field of view is restricted, increasing the optimization burden of 3D GS. Ultimately, we achieve the best performance with 8 groups.

Effect of the Number of Camera Poses. For the adaptor, we conduct an ablation study on the number of camera poses, as shown in Table 14. During the adaptor’s training process, the number of camera poses $N$ is set to 15. By comparing the 2nd, 3rd, and 4th rows, it is evident that, compared to using 10 or 20 camera poses, selecting 15 has improved the accuracy of 3D frame alignment in novel-view images, enhancing the performance of the 3D detection task.

Complexity. Our rendering model builds upon and improves Deformable 3D GS. To demonstrate the effectiveness of our approach, we compare the average rendering speed between Deformable 3D GS and our method on a reconstructed KITTI scene. Specifically, on an NVIDIA V100 with approximately 4 million Gaussian primitives, Deformable 3D GS requires 0.768s to render a single frame, whereas our method (with 8 groups) only takes 0.254s. By applying our grouping strategy to Deformable 3D GS, the rendering speed is further improved to 0.199s. These results highlight that, despite the additional computational complexity introduced by our two enhancement modules, our method still achieves superior rendering speed compared to Deformable 3D GS. This improvement is primarily attributed to our novel grouping strategy, which efficiently groups Gaussian primitives and mitigates issues related to an excessive field of view. As a result, the number of Gaussian primitives required for rendering is reduced, significantly accelerating the overall rendering process.

Limitations. Since our rendering method is based on the 3D GS framework, it inherits the limitations of this approach. The 3D GS process begins by initializing the attributes of the 3D GS primitives—such as center position, opacity, and 3D covariance matrix—using point clouds generated by the SfM method. This is followed by an optimization and training phase. Consequently, the quality of the reconstruction depends on the accuracy of the point clouds produced by the SfM method.