HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splatting

Animatable Head Prior Model. FLAME [39] is a vertex-based linear blend skinning (LBS) model, with $N=5023$ vertices and $4$ joints (neck, jaw, and eyeballs). The head animation can be formulated by a function:

where ${\bm{\beta}}\in\mathbb{R}^{|{\bm{\beta}}|}$, ${\bm{\gamma}}\in\mathbb{R}^{|{\bm{\gamma}}|}$ and ${\bm{\psi}}\in\mathbb{R}^{|{\bm{\psi}}|}$ are the shape, pose and expression parameters, respectively (we refer readers to  [44, 39] for the blendshape details).

Deformable Gaussian Texture. To mitigate the limitations of animatable head prior model, we use 3D Gaussian points to model the texture. The key point is to make sure these points can be deformed semantically by the head prior model. Following Qian et al. [54], we assume every 3D Gaussian point is connected with a FLAME mesh. The FLAME mesh moves and deforms the corresponding points. Given pose and expression, the FLAME mesh can be calculated by Eq. 4. Then, we quantify the mesh triangle by its center position ${\bm{t}}$, rotation matrix $\tilde{{\bm{R}}}$ and area $a$, which describe the triangle’s location, orientation and scaling in world space, respectively. Among them, the rotation matrix is a concatenation of one edge vector, the normal vector of the triangle, and their cross-product. Formally, we deform the corresponding 3D Gaussian point as

where ${\bm{\mu}}^{\prime}$, ${\bm{s}}^{\prime}$ and ${\bm{R}}^{\prime}$ are the position vector, scaling vector and rotation matrix of the deformed Gaussian for rendering. Intuitively, the 3D Gaussian point will be rotated, scaled, and translated by the mesh triangle. In this way, Gaussians can be seen as a residual term of FLAME to represent intricate geometry and texture. As a result, FLAME enables the 3DGS to animate semantically, while 3DGS improves the texture representation and rendering efficiency of FLAME.

Joint Learning of Shape, Texture, Animation. The intricate texture can be modeled by the deformable Gaussian texture $\theta_{\mathrm{3DGS}}$. Besides, we assume the shape of head prior model $\theta_{\mathrm{FLAME}}=\left\{{\bm{\beta}}\right\}$ is learnable. The learnable shape allows for modeling character more precisely. For example, characters like the Hulk in Marvel have larger heads, whereas characters like Elsa in Frozen have thinner cheeks. Meanwhile, we notice that excessive shape updates can negatively impact the learning process of 3DGS due to deformation changes. Thus, we stop the shape update after a certain number of training steps to ensure stable learning of 3DGS. As a result, a head avatar can be represented by an animatable head Gaussian as $\theta=\theta_{\mathrm{FLAME}}\cup\theta_{\mathrm{3DGS}}$.

Super-dense Gaussian Initialization. The supervision signal of SDS loss [53] in head avatar generation is sparse. It inspires us to initialize 3D Gaussians that thoroughly cover the head model for faster convergence and improved representation. In specific, each mesh triangle is initialized with $K$ evenly distributed points. The positions of the deformed 3D Gaussians ${\bm{\mu}}^{\prime}$ are calculated by sampling on the FLAME model (with standard pose), with all mesh triangles sharing the same sampling weight. The deformed scaling ${\bm{s}}^{\prime}$ is the square root of the mean distance of its K-nearest neighbor points. Then, we initialize the position and scaling of 3D Gaussians by the inversion of Eq. 5: ${\bm{\mu}}_{init}=\tilde{{\bm{R}}}^{-1}(({\bm{\mu}}^{\prime}-{\bm{t}})/\sqrt{a});{\bm{s}}_{init}={\bm{s}}^{\prime}/\sqrt{a}$. The other learnable parameters in $\theta_{3DGS}$ are initialized following vanilla 3DGS [35].

Animation-based Text-to-3D Distillation. The vanilla text-to-3D distillation [53] produces satisfactory performance in static but falls short in animation. We attribute it to the absence of new poses and expressions in training. Therefore, we design a new text-to-3D distillation that adapts to animation.

Training with Animations. We first incorporate the new pose and expression into training [41, 73]. Specifically, we sample pose and expression from real-world motion sequences, such as TalkSHOW [71], to ensure that the avatar satisfies the textual prompts with a diverse range of animation.

FLAME-based Control Generation. Training with animations is crucial for dynamic avatar generation. However, the direct introduction of new pose and expression results in Janus (multi-faces) problem [30], due to the data bias in the diffusion model. This issue, represented as portrait bias with front-view, straight-looking, and closed mouths, hinders its application in animation-based distillation. To address this issue, we introduce the MediaPipe [45] facial landmark map $C$, a fine-grained control signal marking the regions of upper lips, lower lips, eye boundary, eyeballs, and facial boundary [21, 42], for more precise and detailed guidance. It can be extracted from an animatable head Gaussian, which ensures that the control signal aligns well with the Gaussian points when the shape, pose, and expression change. The loss gradient is formulated as:

Denoised Score Distillation. According to our experiments, we find the generated avatars have non-detailed and over-smooth textures. To solve this issue, we consider the distilled score to be noisy [24, 65, 34]. Hertz et al. [24] indicates that the score can be seen as the noise when the rendered image matches the textual prompt. Following NFSD [34], we assume the score with a large timestep $t\geq 200$ is noisy, and the rendered image can be seen as matching the negative textural prompts, such as $y_{\mathrm{neg}}=$ “unrealistic, blurry, low quality, out of focus, ugly, low contrast, dull, dark, low-resolution, gloomy”. Besides, the score with a small timestep $t<200$ is relatively clean. As a result, we reorganize the SDS into a piece-wise function:

where $s_{neg}$ is a pre-defined CFG scalar for negative textual prompts. Intuitively, we get a cleaner score to improve the avatar’s texture.

Adaptive Geometry Regularization. To deform semantically, the 3D Gaussians should closely align with the rigged mesh triangle. Introducing a regularization term for the 3D Gaussians, such as ${\|{\bm{\mu}}\|}_{2}$, will lead to the 3D Gaussians being overly concentrated around the mesh center. Thus, the regularization should inversely scale with the triangle size. For instance, in the eye and mouth region, where the mesh triangle is small, the rigged Gaussians should have a relatively small scaling and position. Following Qian et al. [54], we introduce the position and scaling regularization. For each triangle, we initially calculate the maximum distance among its center ${\bm{t}}$ and three vertices, termed as $\tau$, to describe the triangle size. Then, we formulate the regularization term as:

where $\tau_{\mathrm{pos}}=0.5\tau$ and $\tau_{\mathrm{s}}=0.5\tau$ are the experimental position tolerance and scaling tolerance, respectively.

Animatable Head Gaussian Details. In 3DGS, Kerbl et al. [35] employs a gradient threshold to filter points that require densification. Nevertheless, the original design cannot handle textual prompts with varying gradient responses. To address this, we utilize a normalized gradient to identify the points with consistent and significant gradient responses. Furthermore, the cloned and split points will inherit the same mesh triangle correspondence of their parent [54]. The densification and pruning iterations setting are following [43]. The FLAME’s shape size is $|{\bm{\gamma}}|=300$, the expression size is $|{\bm{\psi}}|=100$ and the pose size is $|{\bm{\gamma}}|=3\times 4$ (neck, jaw, left eyeball and right eyeball).

Text to Avatar Optimization Details. We initialize animatable head Gaussian with $K=10$ per triangle. Besides, we commonly set $s=7.5$ and $s_{neg}=1$ in animation-based text-to-3D distillation [34]. In our experiment, we default to using Realistic Vision 5.1 (RV5.1) [3] and ControlNetMediaPipeFace [75, 1]. To alleviate the multi-face Janus problem, we also use the view-dependent prompts [30].

Training Details. The framework is implemented in PyTorch and threestudio [20]. We employ a random camera sampling strategy with camera distance range of $\left[1.5,2.0\right]$, a fovy range of $\left[40^{\circ},70^{\circ}\right]$, an elevation range of $\left[-30^{\circ},30^{\circ}\right]$, and an azimuth range of $\left[-180^{\circ},180^{\circ}\right]$. We train head avatars with a resolution of 1024 and a batch size of 8. The entire training consists of 10,000 iterations. The overall framework is trained using the Adam optimizer [36], with betas of $\left[0.9,0.99\right]$, and learning rates of 5e-5, 1e-3, 1e-2, 1.25e-2, 1e-2, and 1e-3 for mean position ${\bm{\mu}}$, scaling factor $vs$, rotation quaternion ${\bm{q}}$, color ${\bm{c}}$, opacity ${\bm{\alpha}}$, and FLAME shape ${\bm{\beta}}$, respectively [43]. Note that we stop the FLAME shape optimization after 8,000 iterations. The entire optimization process takes around two hours on a single NVIDIA A6000 (48GB) GPU.

Static Head Avatar Generation. We evaluate the avatar generation quality in terms of geometry and texture. In Fig. 4, we evaluate the geometry through novel-view synthesis. Comparatively, the head-specialized methods produce avatars with superior geometry compared to the text-to-3D methods [53, 49, 14, 65]. This improvement can be attributed to the integration of FLAME, a reliable head structure prior, which mitigates the multi-face Janus problem [30] and enhances the geometry.

Dynamic Head Avatar Generation. We evaluate the efficiency of animation in terms of semantic alignment and rendering speed. For the evaluation of semantic alignment, we visually represent the talking head sequences, which are controlled by speech [71]. In Fig. 5, we compare HeadStudio with TADA [41]. The yellow circles in the first row indicate a lack of semantic alignment in the mouths of Hulk and Geralt, resulting in misplaced mouth texture. Our approach achieves excellent semantic alignment and smooth expression deformation. On the other hand, our method enables real-time rendering. When compared to TADA, such as Kratos (52 fps vs. 3 fps), our method demonstrates its potential in augmented or virtual reality applications. Furthermore, the comparison in Fig. 6 indicates the semantic alignment of the method proposed by Bergman et al. [6]. But it lacks in terms of its representation of appearance and geometry.

Effect of Super-dense Gaussian Initialization. In Fig. 7, we present the effect of super-dense Gaussian initialization. Since the SDS supervision signal is sparse, super-dense Gaussian initialization enhances point coverage on the head model, leading to a favorable initialization and improved avatar fidelity.

Effect of Animation-based Text-to-3D Distillation. As illustrated in Fig. 8, we visualize the effect of each component in text to avatar optimization. Our method shows the improvements in the following three aspects: 1) Shape (a vs. c): FLAME offers precise control signals to address multi-face issues, ensuring ID consistency. 2) Texture (a vs. d): Denoised score distillation alleviates the over-smoothing problem in texture by eliminating unnecessary gradients. 3) Animation (a vs. b): Training with animations is crucial for artifact elimination (highlighted in yellow box) in deformation.

Effect of Adaptive Geometry Regularization. In Fig. 7, we also present the effect of adaptive geometry regularization. Firstly, adaptive geometry regularization could reduces the outline points. Nevertheless, overly strict regularization weaken the representation ability of animatable head Gaussian, such as the beard of Kratos (fourth column in Fig. 7). To address this, we introduce an adaptive scale factor to balance restriction and expressiveness based on the area of mesh triangle. Consequently, the restriction of Gaussian points rigged on jaw mesh has been reduced, resulting in a lengthier beard for Kratos (third column in Fig. 7).

Effect of Different Diffusion Models. In this paper, we use Realistic Vision 5.1 (RV5.1) [3] as the default diffusion model. Compared to SD2.1 [56] and SD1.5, we observe that RV5.1 is capable of producing head avatars with a more visually appealing appearance. Meanwhile, we show the results of using SD2.1 (same as TADA [41]) and SD1.5 in Fig. 9. HeadStudio can generate avatars with better semantic alignment (texture alignment in mouths) and faster rendering speed (53 fps vs. 3 fps) compared with TADA [41].

0. Initialization. We evenly sample $K=10$ points per triangle from FLAME with the standard pose, and initialize the scaling via the square root of the mean distance of K-nearest neighbor points. The 3D Gaussians rigged with a large mesh triangle are initialized with a larger radius, compared to the ones rigged with a small mesh. As a result, it initializes 3D Gaussians that can thoroughly cover the head model. The further discussion of $K$ selection can be found in Sec. B.2.

1. Random camera and animation sampling. At each iteration, the animation inputs, pose and expression are sampled from the FLAME sequences (pre-calculated based on the real-world talk show videos [71]). Meanwhile, a camera position is randomly sampled as described in Sec. 4.3.3.

2. Deform and render animatable head Gaussian. We detail the deformation process in Fig. 11. Given the pose and expression, FLAME with learnable shape is driven according to Eq. 4, deforming the mesh triangles. Then, we utilize the mesh triangle’s center position, rotation matrix and area to translate, rotate and scale the corresponding rigged 3D Gaussians (Eq. 5). Following this, we render the deformed 3D Gaussians at a resolution of $1024\times 1024$ based on the sampled camera pose.

3. Optimization with animation-based text-to-3D distillation. Based on the FLAME model, we initially draw a facial landmark map in MediaPipe format as the diffusion condition. Then, we calculate the gradients of Eq. 6 w.r.t. the animatable head Gaussian parameters, which force the rendering to satisfy the text prompt in any pose, expression, and camera view.

4. Optimization with geometry regularization. We constrain the position and radius of 3D Gaussians w.r.t. the size of their rigged mesh triangle according to the Eq. 8. Furthermore, an adaptive scaling factor is introduced in Eq. 9 for modeling elements outside the space of FLAME. The impact of the regularization is discussed in Fig. 16.

Evaluation on different $K$ in super-dense Gaussian initialization. We discuss the impact of the hyperparameter $K$ in HeadStudio. As shown in Fig. 13, the proposed initialization is essential for generation. In the default configuration ($K=1$), the animatable head Gaussian is unable to grow up through cloning and splitting [35], leading to a poor appearance. We attribute it to the sparse guidance provided by score distillation-based loss. On the other hand, the density of 3D Gaussians is similar to the resolution of the image. A denser 3D Gaussians will have a better representation ability. Therefore, with the increase of $K$, the dense initialization results in better generation results. However, a large $K$ will result in additional time and memory costs. Therefore, we opt for $K=10$ as the default experimental setup.

Evaluation on Adaptive Geometry Regularization. First, we investigate geometry regularization and explore the impact of its weight in HeadStudio. As depicted in Fig. 15, geometry regularization is crucial for semantic deformation. In the absence of geometry regularization ($\lambda_{pos}=0,\lambda_{s}=0$), the 3D Gaussians fail to align semantically with FLAME, resulting in the problem of mouths sticking together (first column in Fig. 15). On the other hand, the weight shows a trade-off between alignment and representation. For instance, the Thor in the third column, generated with a large constraint weight, shows good alignment in the mouth but lacks representation (the helmet is missing). Then, we analyze the proposed adaptive scaling factor. We choose the area of the mesh triangle as an adaptive scaling factor (shown in Fig. 16), which is small around the eyes and mouth, and large on jaw and over head. With the help of the adaptive scaling factor, the generation demonstrates semantic alignment and adequate representation simultaneously (fourth column in Fig. 15). It highlights the importance of the adaptive scaling factor in geometry regularization, which effectively balances the alignment and representation.

Evaluation on Animal Character. We evaluate the generalization of HeadStudio with various animal character prompts. As shown in Fig. 18 and Fig. 17, HeadStudio effectively generates animal characters, such as the lion, corgi, bear, raccoon and chimpanzee. However, we believe that the human head prior model, FLAME [39], could limit the animation quality. In the future, replacing FLAME with an animal prior model like SMAL [84] in HeadStudio could improve animal avatar generation.