Sketch-guided Cage-based 3D Gaussian Splatting Deformation

Sketching is a widely used modeling paradigm in geometric modeling and computer graphics. Early methods such as Teddy [13] and Fibremesh [26] construct smooth shapes guided by user-specified 2D sketches. After generating the initial shape, deformations can be done by drawing reference strokes or deforming the generated curve.

Some works [25, 17, 42, 19] focus on the sketch-based deformation that uses individual drawn strokes as a deformation clue. (For a thorough review of the 3D shape modeling, we refer to [6].) All these methods combine the user constraints from sketches with some geometric regularizer such as the Laplacian. Whereas, these geometric energies were designed to preserve certain properties, such as smoothness, and couldn’t take into account the semantic information of the object. This can sometimes lead to unnatural deformation, e.g. bending the straight shape.

More recently, data-driven methods were applied to sketch-based 3D shape modeling [9, 23, 21, 3]. Some works [9, 3] trained neural networks to generate 3D meshes conditioned on the input sketch. These methods always need large-scale datasets to train the networks and the editing can only be done for generated shapes. For more general sketch-based editing,  [23, 21] edited the shapes (represented by Neural Radiance Field) by 2D sketch matching and employed Score Distillation Sampling(SDS) [27] to produce natural-looking editing that satisfied the semantic of the text prompt.

Score Distillation Sampling(SDS) was first introduced in  [27] as a 3D shape generation method based on 2D diffusion prior. The core idea of SDS is to make the renderings of generated 3D shapes look natural from any random viewpoint. Since it is an image-based score, it can be applied to 3D editing of any representation, e.g. triangular mesh [36, 38], NeRF [23] and 3D GS [20]. However, the original SDS used a 2D image diffusion model with only 2D knowledge, leading to the view inconsistency problem of SDS. Recently, by using 3D data in training the diffusion model, Multi-view diffusion [29, 22, 12] was introduced to generate 3D shapes with better geometric consistency. By replacing the image diffusion model in the SDS with multi-view Diffusion, the cross-view consistency can be improved in the 3D shape editing [28, 21].

Neural Radiance Field(NeRF) [24] and 3D Gaussian Splating(3D GS) [16] are new 3D representations designed for novel-view synthesis that the 3D scene can be reconstructed from a set of images. Some work has been done for NeRF editing [23, 10, 30, 41]. Since our work focuses on sketch-based deformation of 3D GS, we will talk in more detail about editing 3D GS. [34] introduce a method that edits the 3D Gaussian Splatting(GS) scene with text instruction, powered by LLM and 2D image diffusion prior, which can achieve object texture editing and environment changing. [5] also uses 2D image diffusion prior as the guidance of the editing but introduced the Hierarchical GS to improve the editing quality. It enables object removal and addition by employing inpainting techniques.

Instead of guiding the editing by 2D image diffusion prior, [4, 35] introduced methods that edit the rendered image of original 3D GS from multi-views with consistency control, and fit the changes in the edited images to 3D GS directly, which improve the efficiency and quality of editing significantly.

Some works focus on the deformation of the 3D GS. Align-Your-Gaussians(AYG) [20] and DreamGaussian4D [28] introduced methods that animate a static 3D GS object to a 4D GS sequence. AYG [20] employs Video Diffusion prior to drive the temporal deformation between frames and using 2D Diffusion prior for every frame respectively to constrain the deformation validly. Instead of using Video Diffusion prior, DreamGaussian4D [28] uses a reference video in one specific view to animate the static 3D GS and applies 3D-aware Score Distillation Sampling(SDS) to propagate the deformation in every frame. The reference video was generated from an image-to-video Diffusion model based on the rendered image of static 3D GS from that specific view. Compared to AYG, DreamGaussian4D is more efficient but limited by the generation quality and universality.

PhysGaussians [37] seamlessly integrates physically grounded Newtonian dynamics within 3D GS to achieve high-quality novel motion synthesis. It adapts the Material Point Method(MPM) to 3D GS which enriches 3D GS with meaningful kinematic deformation and mechanical stress attributes.

SuGaR [8] employs new training energies based on original 3D GS [16] that produce 3D GS with better surface alignment and more even density distribution. These new properties make it possible to extract mesh from 3D GS only using traditional methods, such as Poisson reconstruction [15]. Given the extracted mesh, The 3D GS can be bound to the mesh and deformed by mesh deformation algorithms, such as ARAP [31]. Similar to SuGaR, [7] also proposes to bind the 3D GS kernels to the mesh, which is reconstructed by the existing methods from the input multi-view images directly. However, the result significantly depends on the extracted mesh, which can fail in scenes with complex geometry and transparent components.

A $3D$ Gaussian is defined by a full $3D$ matrix $\Sigma\in\mathbb{R}^{3\times 3}$ and a centroid $\mu\in\mathbb{R}^{3}$, defining the density of the Gaussian:

$$G(x)=e^{-(1/2)(x-\mu)^{T}\Sigma^{-1}(x-\mu)}.$$

(1)

Given a diagonal scaling matrix $S\in\mathbb{R}^{3}$ and rotation matrix $R\in SO(3)$, the corresponding $\Sigma$ is constructed as:

$$\Sigma=RSS^{T}R^{T},$$

(2)

with $S$ and $R$ being the variables that are optimized during the reconstruction of the scene using $3D$ Gaussians. We consider a collection of such Gaussians, $\left(\Sigma_{i},\mu_{i}\right)_{i=1}^{n}$.

Deforming 3D Gaussians entails assigning a new centroid position $\mu$ and a new covariance matrix $\Sigma$ to each Gaussian. However, we wish to regulate the space of possible deformations, to avoid Gaussians floating apart, and exposing only meaningful deformations of the object the Gaussians represent. Towards this goal, we design a novel, tailor-made deformation scheme, incorporating two components: 1) a cage-based deformation [2], tailored to Gaussian Splats; 2) a method to control this cage deformation, inspired by neural jacobian fields [1]. We detail these two components next.

We use Score Distillation Sampling(SDS) [27] to guide our deformation to be plausible. In short, SDS renders a 3D model from different view points, and then leverages a trained 2D image diffusion model, and backpropagates the diffusion process through the differentiable renderer, to the degrees of freedom of the 3D model. we further make use of a 3D-aware image diffusion model [22] to enable a more accurate 3D consistency of the generated images. In short, this model can be conditioned on a specific viewing direction, and produces more consistent images of the same object from different view point. See Supplemental for the full details.

The overview of the editing pipeline is shown in Figure 2. The user first selects a viewpoint $vp$ (blue camera). This viewpoint is used to extract a silhouette of the object. We render the $3D$ GS from viewpoint $vp$ to get an image $I_{vp}$ (Figure 2 B). The user additionally deforms the silhouette into $S_{vp}$ (Figure 2 C). To obtain the deformed image $I^{CN}_{vp}$ (Figure 2 D) guided by the user’s sketch, the rendering $I_{vp}$ is fed into an image-to-image diffusion model conditioned on the $S_{vp}$ by using ControlNet [39]. Our loss measures the silhouette difference $\mathcal{L}_{sil}$ between $I^{CN}_{vp}$ and $I^{def}_{vp}$ (Figure 2 E), the rendering of deformed 3D GS from viewpoint $vp$:

$$\mathcal{L}_{sil}=\|I^{CN}_{vp}-I^{def}_{vp}\|_{2}^{2}.$$

(7)

We chose to only penalize the silhouette and not the full RGB deformed image, as our experiments showed the texture of the objects can be otherwise changed drastically.

To keep the deformation natural in all views, we apply 3D-aware SDS on randomly sampled views (red camera) in every iteration. Thus the final gradient used during optimization is:

$$\nabla\mathcal{L}_{total}=\alpha\nabla\mathcal{L}_{sil}+\nabla\mathcal{L}_{SDS},$$

(8)

$\alpha$ is set to $10000$ for all examples. When optimizing the 3D GS by the objective function 8, the 3D GS is deformed by the differentiable Cage-based block as shown in section 3.2.

The cage is generated automatically by first extracting a mesh from the GS model using the coarse stage of the SuGaR [8] method. This mesh is very large, sometimes it is not a closed manifold and it has many fold-overs so it would not be suited for a cage. Therefore, we compute a triangulated offset surface using a function from Libigl [14] that applies marching cubes on a grid of signed distance values from the input triangle mesh. The resulting mesh is a close manifold triangular mesh with an adjustable number of vertices depending on the model size and level of detail desired. We used StableDiffusion-XL as the diffusion model of the ControlNet. For each deformation, we optimized for $2000$ iterations, with a learning rate of $0.002$, and optimized by Adam optimizer [18]. Except for the reference view, $4$ random views were sampled for the 3D-aware SDS. The diffusion model used in 3D-aware SDS is zero-1-to-3 XL [22]. All experiments were run on a single Nvidia RTX A6000 GPU. The running time was related to the number of Gaussian splats in different scenes. For the scene with less than 100k Gaussian splats, the running time is less than 10 minutes. More efficient diffusion models is a growing area of research and our method would be directly accelerated by the many methods that are being developed [32, 11].

We tested our method on various 3D objects, from human-made objects to animals and humans, as shown in Figure 1, 5, and 9. It shows that our method can deform the objects precisely with the guidance of the sketch and produce natural-looking results. These 3D shapes were collected from the SketchFab and TurboSquid and transferred into a 3D GS scene. The 3D shapes were originally mesh and converted into 3D GS by training from 100 sampling images from random views of the shape.

Moreover, we also explored the ability of our method to deform large-scale real-world captured data. We used the UAV-captured dataset of a great Oratory and a Clock Tower to reconstruct the 3D GS scene respectively. As shown in Figure 6.

As one of the important applications of 3D deformation is animation, we also conducted experiments of animating static 3D GS by our method. As shown in Figure 1 (C), 7, and the accompanying video, the user can provide multiple input sketches as some key-frame sketches in the animation sequence. Then, by running our method for each key frame and interpolating between the deformed cage of key frames, we can get animations of the static 3D GS.

We compared our Cage-based deformation for 3D GS with the SOTA method MLP+HexPlane [28]. For comparison, we deform the source 3D GS using our pipeline and render the sketch view of the deformed 3D GS as the reference image of the MLP+Hexplane. Thus, except for the difference in the deformation method, MLP+Hexplane also has extra RGB image guidance in the sketch view when our pipeline is only guided by silhouette. Even in that case, our pipeline can deform the 3D GS with much higher fidelity than using MLP+Hexplane. The qualitative comparison was shown in Figure 5, because of the good space continuity of the cage deformation, the local detail features can be fully preserved, e.g. the pattern on the statue, and the teeth of the dinosaur. Though the Hexplane was applied as a geometric regularizer, the detailed features can still be destroyed and cause fuzzy renderings when using MLP+Hexplane. We also quantitatively compare the visual quality of the deformed shapes based on image quality. We use the CLIP-IQA [33], a metric measuring the image quality based on the CLIP score, to evaluate 8 deformed results. For every deformed 3D GS, 8 views were rendered as the images to be evaluated. Since our task is to measure how the image quality was changed after the deformation process, we report the relative CLIP-IQA score, which is calculated as the CLIP-IQA score of the undeformed renderings divided by the CLIP-IQA score of the deformed renderings. As shown in table 1, the average decrease of the CLIP-IQA score for the MLP+Hexplane method is $24.68\%$, however, our method almost keeps the same CLIP-IQA score as original renderings with less than $1\%$ decreasing.

We tested the effects of two important components in our method, decomposed NJF and 3D-aware SDS. We explored the difference of optimizing directly on the cage vertices, via NJF and our proposed decomposed NJF. As shown in Figure 8, optimizing directly on the cage vertices can lead to undesired fuzzy rendering which is caused by entanglement of the cage during the optimization. As shown in Figure 8 (B), the detailed feather of the statue and the door on the side of the clock tower were destroyed. Applying NJF can solve this problem since it works as a geometry regularizer for the cage. However, it can be too strong to fit the sketched silhouette precisely. As shown in Figure 8 (C), the wing of the statue is not fitted on the end and the clock tower is not bent enough. When using decomposed NJF as stated in section 5, we can obtain a more efficient optimization which leads to a lower $\mathcal{L}_{sil}$ (average of $49.36\%$ decreasing in these two examples), but with same rendering visual quality compared to NJF (Figure 8 A).

We also explored the effect of 3D-aware SDS in our pipeline. As shown in Figure 9, the 3D-aware SDS plays a critical role in preserving undesirable deformation in the views except for the sketch view in the pipeline. For instance, when the pistol was deformed only by $\mathcal{L}_{sil}$, the barrel was bent, which can be fixed by adding the 3D-aware SDS.