DynamicAvatars: Accurate Dynamic Facial Avatars Reconstruction and Precise Editing with Diffusion Models

Several techniques have been proposed for tasks involving 3D dynamic scenes. Neural Radiance Fields (NeRF) have introduced a groundbreaking perspective in 3D-aware generation through exceptional volumetric rendering. Efficiency improvements have been achieved by optimizing data organization structures and numerical methods. For example, works such as InstantNGP [27] and Mip-NeRF[2] have made significant advancements. However, the implicit information storage approach used by NeRF can be computationally intensive and less efficient compared to explicit methods like voxel grids, point clouds, and Gaussian points.

In our work, we adopt 3D Gaussian Splatting [17] (3DGS) to balance reconstruction quality in 3D head avatar tasks with effective editing capabilities.

A widely adopted approach for modeling dynamic scenes involves encoding color and volume density information into a Multi-Layer Perceptron (MLP) using spatial-temporal coordinates $(x,y,z,t)$ or relative coordinates in a 4D canonical space like [35, 9]. More recent methods like [43, 21], which attach mesh triangles to 3D Gaussians for explicit editing and fine-tuning, have achieved remarkable results in facial scene reconstruction. Building upon these advancements, we introduce a Gaussian tracing method integrated with mesh adjustment to enable flexible and efficient dynamic editing.

Using diffusion models in generating photorealistic images[13] has become a standard approach for image generation like [16, 37, 38]. Foundational works using prompts to guide the diffusion model, such as InstructPix2Pix [3], achieved this by constructing conditional diffusion models and refining prompts with the help of Large Language Models (LLMs) like ChatGPT. Subsequent studies, such as Prompt2Prompt [12], combined structural-aware priors with original text information to enhance control over the generation process.

Recent advancements, such as LLM-grounded Diffusion [19], have highlighted the pivotal role of LLMs in defining prompts and indirectly influencing the quality of generated results. For instance, a self-correcting, LLM-controlled model proposed in SLD [36] improved text-to-image alignment by iteratively refining prompts during the generation process.

Building on this line of research, we aim to explore and optimize the editing process further, ultimately contributing to improved precision and accuracy in fine-grained image editing tasks.

Generative Adversarial Networks [10] (GANs) are widely applied in 3D reconstruction tasks, often in combination with techniques such as Convolutional Neural Networks [28] (CNNs) or implicit functions to enhance generated results. These approaches enable the inference of realistic 3D models from limited visual information, enforce geometric coherence across multiple views, and reduce surface texture and geometry blurring.

In the context of digital avatars, GANs have been employed for generating 3D faces and bodies for some time. StyleGAN [15] is a prominent example, showcasing how GAN-based architectures facilitate expressive face synthesis and demonstrate the potential to generate novel expressions and poses from an initial avatar, while [34] combined GAN with diffusion model which showed positive results. Related works such as EG3D [6] and GenN2N [22] provide practical methods for combining GANs with NeRF to improve synthetic novel-view rendering of facial avatars.

In our research, we observed that the original outputs often suffered from inaccurate colors and texture distortions, particularly around sensitive regions such as the eyes and teeth. To address these issues, we implemented a specialized GAN-based generator module designed to specifically target and correct problematic areas, significantly improving the overall quality and realism of the results.

Gaussian Splatting is based on creating a collection of Gaussian splats, where each splat is characterized by position x and covariance matrix ${\Sigma}$. While the mean of a Gaussian splat is the center point ${\mu}$, for each position $x\in{{\mathbb{R}}^{3}}$ we have:

Worth to notice that the covariance matrix should be differentiable during optimization process. A matrix decomposition method was introduced to solve this problem as follows:

where R refers to rotation matrix and S refers to scaling matrix.

Each Gaussian Splat can also be described by a set of physical parameters like position vector ${\mu\in{{\mathbb{R}}^{3}}}$ , scaling vector $s\in{{\mathbb{R}}^{3}}$, rotation quaternion $q\in{{\mathbb{R}}^{4}}$ and color $c\in{{\mathbb{R}}^{3}}$. The rendering process involves tracing every 3D points along a ray to determine its contribution to the final color of a specific pixel. This is achieved by accumulating the influence of Gaussian splats intersected by the ray, resulting in the rendering formula:

where ${{c}_{i}}$ represents the color of each 3D point on the trace and ${{\alpha}_{i}}$ is the corresponding density. Note that in an optimized version, we can sort the splats by depth in order to cut off those invisible points.

One of the most important parts of editing stage is to ensure the diffusion model could generate an image which is fully correspond to the text-based instructions. To achieve this, many previous works designed an architecture similar to the follows:

1)Define Let ${I_{\text{original}}\ \in\mathbb{R}^{H\times W\times C}}$ represent the input image, where ${H,W,}$ and ${C}$ denote the image height, width, and number of channels, respectively. The user provides a natural language prompt ${T}$, which is a sequence of tokens:

where ${\mathcal{V}}$ is the vocabulary space, and ${n}$ is the number of tokens in the prompt.

The goal is to generate an edited image ${I_{\text{edit}}}$ such that the semantics of ${T}$ are incorporated into ${I_{\text{original}}}$ , satisfying:

where ${\mathcal{L}_{\text{edit}}}$ is a loss function capturing the alignment between ${I_{\text{edit}}}$ and ${T}$.

2)Encode The input image ${I_{\text{original}}}$ is passed through an image encoder ${D_{\text{img}}:\mathbb{R}^{H\times W\times C}\to\mathbb{R}^{d}}$, producing a latent representation:

where ${z_{\text{img}}\in\mathbb{R}^{d}}$ is a feature vector in the latent space of dimension ${d}$.

The textual prompt ${T}$ is mapped to the same latent space using a text encoder ${D_{\text{txt}}:\mathcal{V}^{n}\to\mathbb{R}^{d}}$:

3)Align To align image and text features, a joint latent space ${\mathcal{Z}}$ is defined, where:

Here, ${g_{\text{align}}}$ is a cross-attention mechanism or a shared embedding model that fuses ${z_{\text{img}}}$ and ${z_{\text{txt}}}$. The resulting latent representation ${\hat{z}_{\text{img}}\in\mathbb{R}^{d}}$ captures prompt-specific modifications.

4)Edit The latent code ${\hat{z}_{\text{img}}}$ is updated iteratively to incorporate desired edits. For diffusion-based models, this is achieved via a reverse diffusion process:

where ${\sigma_{t}}$ represents noise intensity at step ${t}$, and ${\epsilon_{t}}$ is sampled from a Gaussian distribution ${\mathcal{N}(0,I)}$.

5)Decode The output image is further refined through optional denoising or super-resolution processes ${\mathcal{S}}$:

To achieve flexible editing of expressions, textures, and additional accessories on head avatars, it is essential to use a technique that can both reconstruct an accurate head model and facilitate easy editing. Gaussian Avatars proposed using the FLAME[18] mesh to model head avatars, enabling expressive changes. However, this method lacks control over accessories, such as hair, rings, or hats, which are not modeled by FLAME.

As shown in Figure.3, we introduce a novel mesh-Gaussian binding method that differs from Gaussian Avatars [31]. In our approach, we introduce two Gaussian tracking patterns for this stages of the process. The input video is processed with a photometric head tracker to fit the FLAME parameters. Each frame includes multi-view observations, timestep parameters, and known camera parameters. Initially, we track Gaussian splats across each triangle, similar to the method in Gaussian Avatars, to ensure that areas with significant changes can be modeled with high precision. Next, we apply an independent facial composition identifier to generate a semantic mask. This allows us to assign semantic labels to each Gaussian splat when rendered into images, ensuring that the same splats are tracked and manipulated consistently throughout the dynamic scene, maintaining temporal consistency during the editing process. Meanwhile, the rendered results are compared to the real images to train the avatar. In the next stage, we decouple the relationship between the Gaussian splats and the FLAME mesh, enabling the addition and modification of accessories, such as rings and hats.

To enhance rendering quality, we apply adaptive density control operations to adjust the density of the Gaussian splats, selectively densifying and pruning them as needed. We labels the new generated splats and track them to ensure that new Gaussian splats remain connected to the FLAME mesh and semantic masks, preserving the overall structure. Additionally, we optimize the position and scaling of the splats to improve the quality at the same time.

Traditional 3D editing [20] approaches rely on static 2D or 3D masks to restrict changes to specific regions. However, this approach presents challenges, as dynamic updates during training can render static masks inaccurate, limiting their effectiveness. Additionally, static masks in NeRF editing constrain changes to fixed spatial regions, which hampers natural content expansion beyond the mask boundaries.

Recently, semantic segmentation-based masks were introduced by GaussianEditor [7], which partly address this issue by directly assigning semantic labels to individual Gaussian points. This technique increases the accuracy and speed of the editing process compared to previous approaches. However, applying these masks to dynamic scenes directly introduces new challenges in editing stage. For a labeled splat at time ${t_{0}}$, it may not contribute to the color of the corresponding semantic region at time ${t_{1}}$. Conversely, unlabeled splats could still influence the rendering process at later times. This inconsistency can lead to problems such as color mismatches, which modify the appearance of the head avatar, especially when the avatar undergoes significant pose changes over time.

To address this issue, we propose a method, illustrated in Figure.4, that aims to consider all Gaussian splats that contribute to the results across different times and poses. Our approach identifies the shape of the desired editing mask at different times and camera poses. By using the mapping net that generate selected areas across the entire timeline, we can track the Gaussian splats contributing to the target area throughout the dynamic scene. Details of the mapping net can be find in Figure.5.

Next, we edit each image in the selected set to produce an edited image set. Finally, we apply a learning process with a conditional adversarial loss, which helps regulate the Gaussian splats and maintain temporal consistency. This method allows us to edit the entire dynamic model, incorporating the desired changes arbitrarily and efficiently.

Previous 3D scene editing works based on diffusion model [29] have difficulties in retain the stability of edit and relatively low comprehension ability when facing extreme detailed prompts including described information like directions and relative positions. Works like SLD [36] proposes a LLM-controlled self-correcting model that analyzes the components of prompts and execute instructions in order in latent space. However, this method still lacks the ability of understanding direction and relative position information. Work like [1], bring up a feasible direction to utilize the LLM model to assistant the fine editing of images.

In order to improve the generate quality of results when facing such difficult conditions, we focus on solving misplaced and misunderstand problems related to edit and add accessories based on precise detailed prompts. As Figure.6 shows our method, we propose a framework similar to SLD which can provide a practical way of fine editing. We tend to readjust the structure of prompts based on the LLM and then carefully modify the images generated from the previous stage. This image correction is based on latent space manipulations, and it contains the alignment of multi-view consistency in our method.

The primary loss should focus on the rendered images. Hence we deploy a color loss as follows:

It should be noticed that this loss is used under different $\lambda$ value whenever we have the groundtruth image and need to learn the image-level information. In our experiment, we set $\lambda$ equals to 0.9 during our modeling stage and 0.7 during editing stage, it has been tested to be effective during ablation study in our work.

The second constraint we need to pay attention is the tracking loss. This one concentrates on dealing with the relative position between the mesh and Gaussian splats, and affiliation between the certain semantic area and Gaussian splats.

To supervise the positions and distributions of the Gaussian splats during the editing stage in order to conserve the basic structure of the model, we need to employ a loss that can punish the misplacement of the splats. Meanwhile, we also need to be aware of optimizing the physical parameters of each Gaussian splat.

where ${G({{x}_{j}})}$ represents Gaussian splat $j$, $i$ represents corresponding sub property of $j$ like positions, translations and color. For different parameters, we apply the coefficient ${{\lambda}_{i}}$ independently. Finally, we define the adversarial loss in editing stage. The GAN method is used to reduce the color difference between the original one and the objective images.

The overall loss function should be:

In our experiments, we use the Adam algorithm as the optimizer, setting the learning rate to ${1\times 10^{-3}}$ at the beginning of the modeling stage, which gradually decreases to ${1\times 10^{-5}}$. Each head avatar is trained independently from a scratch model, and we evaluate the fine-tuning ability and the effects of various editing methods on the distinct models constructed.

For each training session, the entire reconstruction process takes approximately 1 hour to complete. Densification and splitting frequencies are determined every 2048 iterations. However, we have not yet investigated the optimal timing for densifying and splitting the Gaussian splats. We believe that future work could involve developing a module capable of detecting Gaussian splat areas and automatically adjusting the densification and splitting processes to optimize performance.

Training: We implement our benchmark using the NeRSemble dataset, where each subject has 16 different camera views around the human head. The model is trained with the same parameter settings and methods as GaussianAvatars, using two RTX 4090 GPUs.

Evaluation: We evaluate our approach based on the following settings:

Reconstruction Evaluation: To demonstrate that our method maintains high reconstruction quality, we compare our results with those of other models. Editing Evaluation: We assess our model’s editing ability from various perspectives, including: 1) Driving the avatar with novel poses and expressions, 2) Modifying the style and identity of the head avatar based on prompts, 3) Adding accessories using prompts that contain detailed position and style information.

We conduct experiments and compare our results with baseline models for both self-reenactment and cross-identity reenactment tasks. For reconstruction and non-prompt-based editing metrics, we use PSNR, SSIM, and LPIPS to evaluate the visual quality of the rendered images. For prompt-based editing, we evaluate edit accuracy using SLM and text-image direction similarity with the InstructPix2Pix metric.

Figure.7 presents qualitative comparisons of novel-view synthesis from various methods. Our approach demonstrates exceptional performance compared to the listed works. Notably, GaussianEditor produces images with noticeable artifacts around the mouth and eye regions, and struggles to generate avatars with novel expressions.

This can be attributed to two key factors: 1) GaussianEditor does not utilize a mesh model for constraints, which limits the regularization of the Gaussian splat distribution. 2)The editing algorithm in GaussianEditor is prompt-based, and such edits lack the necessary information to effectively alter the expressions of head avatars. Additionally, the inherent randomness of the diffusion model makes it challenging to maintain stable control over fine-grained edits, such as expressions.

As shown in Table.1, we present quantitative comparisons between our method and several other prominent approaches.

As illustrated in Figure.8 and Figure.9, DynamicAvatars demonstrates outstanding scene editing and the ability to render finely edited images compared to other methods. It is evident that our model effectively interprets the information described in the prompt and generates photorealistic images, thanks to the preprocessing module. Moreover, our model excels in style modification while maintaining accurate expressions and texture colors, owing to the GAN framework that supervises facial image generation.

Table.2 presents the statistics of the quantitative comparisons regarding editing ability. Our approach outperforms other methods in handling complex editing scenarios, such as detailed prompts and mixed style-expression edits.