TalkingEyes: Pluralistic Speech-Driven 3D Eye Gaze Animation

It’s a challenging task to generate 3D eye gaze animation from speech, and most existing methods were proposed a decade ago. The methods can be roughly categorized into rule-based [29, 30] and learning-based [31, 32, 33].

Rule-based methods construct the mapping between conversation states and eye gaze patterns by defining animation rules. Masuko and Hoshino [29] designed a set of empirical rules to map the conversation states (e.g. utterance, listening and waiting) to the amount of gaze and gaze duration. Marsella et al. [30] proposed a rule system to sequentially convert the audio and text features to communicative functions, behavior classes, specific behaviors and finally animation.

In contrast, learning-based methods capture the relationship between speech signal and eye movements by training on data. Mariooryad and Busso [31] employed three Dynamic Bayesian Networks to model the coupling between speech, eyebrow and head motion. Le et al. [32] utilized non-linear dynamical canonical correlation analysis to synthesize gaze from head motion and speech features. Jin et al. [33] proposed a LSTM-based machine learning model to predict the direction-of-focus of all the interlocutors in a three-party conversation. All the above learning-based methods can generate head-eye animation from speech, but they only establish a one-to-one mapping between speech feature and eye gaze motion, without the modeling of uncertainty in nonverbal communication.

To reduce uncertainty, some recent scene-driven (not solely speech-driven) methods [25, 26] forced the animated eye gaze to focus on the interlocutors or on the salient points within the scene. Our method only takes the speech as input and achieves the goal of generating plausible eye gaze motion in a speech-centric setting.

Compared with speech, using video to generate eye gaze motion is more straightforward and convenient. The researchers in computer graphics have proposed several 3D face trackers for video. These trackers can reconstruct not only 3D face model but also 3D eye gaze in each frame, by fitting the parameters of both 3D face state and eye gaze state to the image cues. Wang et al. [40] utilized a random forest classifier to extract the iris and pupil pixels in the eye region, and then used them to sequentially infer the most likely state of the 3D eye gaze at each frame in the MAP framework. Wang et al. [41] improved the work [40] by using CNN rather than random forest classifier for the iris and pupil pixels classification. Wen et al. [42] introduced a method for estimating eyeball motions for RGBD inputs by minimizing the differences between a rendered eyeball and a recorded image.

The above 3D eye gaze fitting methods require accurate segmentation of iris and pupil pixels and user-specific calibration, which is difficult to be achieved in low resolution videos.

The computer vision community starts very early to estimate gaze from image. The estimated gaze is usually in the form of a unit gaze direction vector in 3D space or a point of gaze (PoG) on 2D plane. Early gaze estimation methods are mainly template-matching-based, appearance-based and feature-based, while recent methods are mostly based on deep learning [43]. Zhang et al. [44] introduced the first CNN-based method for gaze estimation, which jointly utilizes eye images and head poses to predict gaze directions. Accounting for the potential asymmetry between human eyes, Cheng et al. [45] proposed to firstly predict the 3D gaze direction for each eye individually and then employ a reliability evaluation network to adaptively adjust the weighting strategy throughout the optimization. Park et al.[46] introduced a novel graphical representation of 3D gaze direction, which serves as intermediate supervision to improve the estimation accuracy. Owing to individual differences, some methods [47, 48] employed a few-shot approach to achieve user-specific gaze adaptation. To address the challenge of occluded eyes, Kellnhofer et al. [49] utilized visible head features to predict gaze direction, even when the eyes are completely obscured, and indicated the prediction’s limited accuracy by outputting a higher uncertainty value accordingly.

These deep-learning-based gaze estimation methods focus on eyes, without considering the relationship between gaze estimation and 3D face reconstruction.

Methods in this field are roughly categorized into linguistics-based [50, 51, 52, 53] and learning-based [12, 14, 15, 16, 17, 21, 19, 22, 23, 24]. Linguistics-based methods typically establish a set of mapping rules between phonemes and visemes, thus providing explicit control over the animation. JALI [50] is a representative recent linguistics-based method, which utilizes two anatomical actions (jaw and lip) to animate a 3D facial rig. As a lot of manual effort is required for tuning animation parameters, a wide variety of data-driven methods have been proposed as an alternative. The generative models are utilized in these methods to directly learn the mapping from speech features to 3D facial motion, which include CNN [12, 14], Transformer [15, 16, 22, 23, 19] and Diffusion Model [17, 21, 24]. VOCASET [12] and BIWI [13] are two widely used 3D audio-visual datasets to train the audio-to-mesh regression networks in the learning-based methods.

Considering that Learn2Talk [19] has achieved impressive lip-sync facial animation, we choose it as the 3D facial motion generator in our proposed method.

The way of utilizing motion capture systems or eye tracking devices to capture eye gaze and microphone to record audio in a laboratory setting is limited in its ability to collect large-scale and diverse data. Considering that 2D audio-visual datasets are more accessible and offer broader coverage, it’s possible to construct the audio-gaze dataset by reconstructing both 3D face mesh and 3D eye gaze from video. Motivated by this, we propose TKED which is obtained by performing 3D face reconstruction and 3D eye gaze fitting on the VoxCeleb2 [9] and HDTF [11] datasets.

VoxCeleb2 is a large-scale dataset collected from YouTube, comprising short interview videos cropped and resized to a $224\times 224$ resolution. The speakers in the dataset span a wide range of different ethnicities, accents, professions and ages. A distinct aspect of the VoxCeleb2 videos is the frequent movement of the speakers’ eyeballs, which contributes to a diverse range of eyeball rotation patterns. To mitigate the potential bias in TKED towards the specific eyeball movement style present in the VoxCeleb2 videos, we have added a small number of videos from HDTF as the complement. HDTF is a high-quality collection of talking face videos of 720P or 1080P, carefully gathered from YouTube. In contrast to VoxCeleb2, the videos from HDTF exhibit relatively stationary eyeballs.

TKED consists of 5,982 videos featuring 669 subjects, amounting to approximately 14 hours of footage in total. Out of the 5,982 videos, 5,912 are from VoxCeleb2, with a total duration of about 12 hours, while the remaining 70 videos are from HDTF, totaling about 2 hours. To ensure consistency, the videos from HDTF have been resampled to match the 25 fps in VoxCeleb2. We have conducted 3D face reconstruction and 3D eye gaze fitting on each frame of these videos to obtain the pseudo ground truth FLAME parameters [34], which include two eyeballs’ rotation, head rotation and facial motion parameters. Fig. 2 shows four sequences of the reconstructed 3D FLAME model. Along with the audio stored in the original videos, TKED is first dataset in the research community to simultaneously contain the three types of resources: 3D talking eyes, 3D talking head and 3D talking faces. The comprehensive comparison between TKED and existing audio-mesh datasets [12, 13, 27, 28, 21] is presented in Tab. I. The proportion of training, validation and testing data in TKED is 8:1:1.

As shown in Fig. 3, the pipeline mainly consists of four stages: video pre-processing, 3D face reconstruction, 3D eye gaze fitting and data post-processing.

The deep-learning-based gaze estimation methods [43] proposed in computer vision provide a number of potential solutions for reconstructing eye gaze from video, but their estimated 3D gaze directions are not always compatible with the reconstructed 3D face mesh. Alternatively, 3D eye gaze fitting methods [40, 41, 42] from computer graphics seem to offer a better fit, but their code has not been publicly released.

The proposed LightGazeFit is a lightweight yet effective 3D eye gaze fitting method tailored for processing videos from VoxCeleb2 and HDTF. Existing fitting methods [40, 41] heavily rely on accurate segmentation of iris and pupil pixels to formulate mask or edge likelihood. However, due to the relatively poor quality of VoxCeleb2 videos, which are only available in a $224\times 224$ resolution, achieving precise segmentation of iris and pupil is nearly impossible. LightGazeFit overcomes this issue by utilizing only the pupil center to form the likelihood, yet it still generates good results across all videos from VoxCeleb2 and HDTF.

The 3D eyeball model used in LightGazeFit comes from FLAME and it’s shown in Fig. 4. We manually annotate the iris vertices, the pupil vertices and the pupil center vertex on the 3D model. Then, we assign a brown color to the iris vertices and a black color to the pupil vertices. As the precise segmentation of iris and pupil is not available, LightGazeFit eliminates the need of eyeball calibration [40, 41, 42] which estimates the iris and pupil size for each subject. With the iris and pupil size fixed, we found that it performs well in the eyeball rotation estimation.

The objective function used in LightGazeFit is defined as:

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters and are set to $1e-4$ and $1e-2$ respectively. The pupil center loss ${\mathcal{L}_{\text{center}}}$ is defined as the difference between the predicted 2D pupil center $\mathbf{y}$ and the point $\hat{\mathbf{y}}$ projected from the pupil center vertex on 3D eyeball model to the 2D image plane:

The 2D pupil center $\mathbf{y}$ in the image is predicted by a neural network [41]. Given the 2D pupil center $\mathbf{y}$, their exists many 3D eyeball rotation solutions to generate $\hat{\mathbf{y}}$. To reduce such ambiguity, we use regularization loss to make the eyeball rotate as little as possible. The regularization loss $\mathcal{L}_{\text{reg}}$ is defined as:

where $\mathbf{p}_{eye}\in\mathbb{R}^{6}$ denotes the 3D rotation of two eyeballs. Finally, based on the assumption that left and right eyeball share the similar behaviour, the symmetry loss $\mathcal{L}_{\text{sym}}$ is defined as:

To model the weak correlation between speech signals and non-verbal cues in conversation, such as head gesture [58], hand gesture [59, 60] and body poses [61], the generative models VAE [1, 62] and VQVAE [2, 63] are commonly used. These models are capable of learning a compact representation for the non-verbal motion data. Through mapping the speech to the compact latent space, the difficulty in capturing the weak correlation between speech and motion is significantly attenuated and hence promotes the quality of motion synthesis. Following this, TalkingEyes also employ VAE and VQVAE to learn the latent space of motion.

On the other hand, there is a key difference between VAE and VQVAE in terms of motion diversity. VAE tends to output the mean of a Gaussian distribution, thus limiting the diversity in outputs. VQVAE uses discrete codebook to capture a diverse set of latent representations, allowing for a wider range of possible outputs. Since the rotation range of head usually exceeds that of eyeballs, the variation of the head motions is generally higher than that of eye gaze motions. As summarized in Tab. II, VAE is capable of producing head motions with adequate diversity (e.g. PoseVAE [58, 19]), while VQVAE may sometimes produce excessive diverse and exaggerated head motions. In contrast, VAE struggles to produce sufficiently diverse eye gaze motions, while VQVAE effectively achieve this. To support our claims, we have conducted quantitative experiments which are introduced in Sect. V-D. To conclude, we use VAE in the latent space learning for head motions, and use VQVAE in that for eye gaze motions.

The pipeline of TalkingEyes is shown in Fig. 5, featuring the translations from speech signals to 3D eye gaze motions $\mathbf{\hat{P}}^{e}_{1:T}\in\mathbb{R}^{T\times 6}$ (left and right eyes) and 3D head motions $\mathbf{\hat{P}}^{h}_{1:T}\in\mathbb{R}^{T\times 3}$ in one framework, where $T$ denotes the number of frames.

The training of TalkingEyes comprises two main stages: firstly learn a discrete latent space for eye gaze motions by pre-training VQVAE, and then jointly train the VAE-based speech-to-head translation and the VQVAE-based speech-to-gaze translation in an autoregressive manner. In the first stage, we pre-train a VQVAE to model the latent space of eye gaze motions as a discrete codebook $\mathcal{Z}$. The codebook is learned by self-reconstruction over the ground truth motions $\mathbf{P}^{e}_{1:T}$. In the second stage, a conditional VAE is employed to reconstruct the head motions $\mathbf{\hat{P}}^{h}_{1:T}$ from the ground truth motions $\mathbf{P}^{h}_{1:T}$. This motion reconstruction is conditioned on the speech embedding $\mathbf{A}_{1:T}$ that is extracted by the speech encoder (wav2vec2.0 [64]) from the input speech. Then the predicted head motions $\mathbf{\hat{P}}^{h}_{1:T}$ and the previously predicted eye gaze motions $\mathbf{\hat{P}}^{e}_{1:t}$ are used as conditions in the mapping of the speech embedding $\mathbf{A}_{1:T}$ to the target motion codes. This mapping is accomplished through multi-modal alignment and codebook $\mathcal{Z}$ lookup. The motion codes are further decoded into the eye gaze motion $\mathbf{\hat{P}}^{e}_{t+1}$ at current frame $t+1$, which will serve as one frame of past motions in the next round of autoregression.

In the inference, the ground truth head motions $\mathbf{P}^{h}_{1:T}$ and the head motion encoder in VAE are not used. The head latent code $\mathbf{c}$ in VAE is randomly sampled from a standard Gaussian.

We construct a discrete codebook $\mathcal{Z}=\{{\mathbf{z}_{k}}\in\mathbb{R}^{C}\}_{k=1}^{N}$ to form the discrete latent space, thus allowing any frame in a eye gaze motion sequence to be represented by a codebook item $\mathbf{z}_{k}$. The Transformer-based VQVAE model that consists of a motion encoder $E_{\text{VQ}}$, a motion decoder $D_{\text{VQ}}$ and a codebook $\mathcal{Z}$, is pre-trained under the self-reconstruction of ground truth motions $\mathbf{P}^{e}_{1:T}$. As shown in Fig. 5, the motions $\mathbf{P}^{e}_{1:T}$ is firstly encoded into a temporal feature vector $\hat{\mathbf{Z}}$. Then, $\hat{\mathbf{Z}}$ is quantized to the feature vector ${\mathbf{Z}_{q}}$ via a element-wise quantization function $Q(\cdot)$ that maps each item in $\hat{\mathbf{Z}}$ to its nearest entry in codebook $\mathcal{Z}$:

${\mathbf{Z}_{q}}$ is finally decoded into the reconstructed motions $\mathbf{\hat{P}}^{e}_{1:T}$ via:

Similar to [2, 16], the training losses of VQVAE include a motion-level loss and two intermediate code-level losses:

where the first term is a motion reconstruction loss, the latter two terms update the codebook by reducing the distance between codebook $\mathcal{Z}$ and embedded feature $\hat{\mathbf{Z}}$, $sg(\cdot)$ stands for a stop-gradient operation and $\beta$ is a weighting factor.

We adopt a conditional VAE model to model the translation from speech to head motions. The head motion encoder $E_{\text{VAE}}$ accepts ground truth motions $\mathbf{P}^{h}_{1:T}$ concatenated with speech embedding $\mathbf{A}_{1:T}$ as input. The encoder $E_{\text{VAE}}$ outputs the continuous latent code $\mathbf{c}$ that adheres a Gaussian distribution $\mathcal{N}(\mu_{h},\sigma_{h}^{2})$ with the learned mean $\mu_{h}$ and learned variance $\sigma_{h}^{2}$. The latent code $\mathbf{c}$ is further concatenated with $\mathbf{A}_{1:T}$, and then be inputted to the head motion decoder $D_{\text{VAE}}$ to predict head motions as $\mathbf{\hat{P}}^{h}_{1:T}$. All the $T$-frames of motions in $\mathbf{\hat{P}}^{h}_{1:T}$ are generated by once reconstruction, which can be formulated as:

Subsequent to the VAE model, a VQVAE-based autoregressive model is cascaded for the eye gaze motion generation. The predicted head motions $\mathbf{\hat{P}}^{h}_{1:T}$ is used as condition to guide the translation from speech to eye gaze motions. Specifically, we adopt a Transformer cross-modal decoder $D_{\text{cross-modal}}$ to algin three different modalities, namely speech audio, head motions and eye gaze motions. $D_{\text{cross-modal}}$ is equipped with causal self-attention to learn the dependencies between each frame in the context of $t$-frames of past eye gaze motions $\mathbf{\hat{P}}^{e}_{1:t}$. Additionally, $D_{\text{cross-modal}}$ is equipped with cross-modal attention to align the past eye gaze motions $\mathbf{\hat{P}}^{e}_{1:t}$ respectively with the head motions $\mathbf{\hat{P}}^{h}_{1:T}$ and the speech embedding $\mathbf{A}_{1:T}$. We set the cross-attention keys (K) and values (V) as the concatenation of the predicted head motions embedding and the speech embedding, and set the queries (Q) as the past eye gaze motions embedding. The cross-modal decoder $D_{\text{cross-modal}}$ outputs the features $\hat{\mathbf{Z}}^{1:t}$ as:

where $\mathcal{P}^{e}_{\text{emb}}$ and $\mathcal{P}^{h}_{\text{emb}}$ are linear projection layers to extract eye gaze motions embedding and head motions embedding respectively. $\hat{\mathbf{Z}}^{1:t}$ is further quantized into $\mathbf{Z}^{1:t}_{q}$ via Eq. 5 and decoded by the pre-trained VQVAE decoder into $\mathbf{\hat{P}}^{e}_{t+1}$ via Eq. 6, which can be formulated as:

The newly predicted motion $\mathbf{\hat{P}}^{e}_{t+1}$ is used to update the past motions, in preparation for the next prediction.

Training Losses. We train the head motion encoder $E_{\text{VAE}}$, the head motion decoder $D_{\text{VAE}}$, the cross-modal decoder $D_{\text{cross-modal}}$, the projection layers $\mathcal{P}^{e}_{\text{emb}},\mathcal{P}^{h}_{\text{emb}}$ and part of the speech encoder, while keeping the codebook $\mathcal{Z}$ and eye gaze motion decoder $D_{\text{VQ}}$ frozen. The training losses consist of four terms: reconstruction loss ${\mathcal{L}_{\text{rec}}}$, velocity loss ${\mathcal{L}_{\text{vel}}}$, KL-divergence loss ${\mathcal{L}_{\text{kl}}}$ and feature regularity loss ${\mathcal{L}_{\text{reg}}}$.

The reconstruction loss ${\mathcal{L}_{\text{rec}}}$ measures the difference between predicted motions and ground truth motions for both head and eye gaze:

The velocity loss ${\mathcal{L}_{\text{vel}}}$ measures the discrepancy between the first-order derivative of predicted motions and that of ground truth motions for both head and eye gaze:

The KL-divergence loss ${\mathcal{L}_{\text{kl}}}$ forces the Gaussian distribution of the predicted head motions $\mathbf{P}^{h}_{1:T}\sim\mathcal{N}(\mu_{h},\sigma_{h}^{2})$ close to a standard Gaussian $\mathcal{N}(0,1)$. ${\mathcal{L}_{\text{kl}}}$ has a simplified form [1]:

The feature regularity loss ${\mathcal{L}_{\text{reg}}}$ measures the deviation between the predicted eye gaze motion feature $\mathbf{\hat{Z}}^{1:T}$ and the quantized feature $\mathbf{Z}^{1:T}_{q}$ from codebook:

The final training loss is formulated as follows:

where $\lambda_{1}$ is set to 1e-4.

To generate more vivid animation, we have also implemented the synthesis of eye blinks. Blinking is primarily influenced by personal habits, environmental factors and other conditions. Liu et al. [65] counted the number of blinks in their dataset and found that the frequency of human blinking, measured in blinks per minute, typically follows a log-normal distribution. Following their work, we conducted a statistical analysis on our dataset TKED to derive a suitable blink frequency that well aligns with our eye gaze animation. We counted the number of blinks in each video in TKED and then converted these counts into blinks per minute based on the video’s duration. For blink detection, we calculated the Eye Aspect Ratio (EAR) [66], which is defined as the ratio of the eye’s height to its width. When the eyes are closed, the EAR approaches zero. After statistical analysis, we found that the blinks per minute follow a Gaussian distribution with a mean of 40.10 and a deviation of 15.42. When generating animations, we first sample the number of blinks per minute from the distribution and then convert that into blink intervals. Subsequently, we adjust the expression parameters within a 5-frame window for each blink to create the blinking action.

VQVAE is pre-trained by Adam optimizer for 100 epochs with a learning rate $1e-4$. The pre-training takes about $1$ hour on a NVIDIA RTX 4090 GPU. Then the VAE and the autoregressive model are trained by Adam optimizer for 100 epochs with a learning rate $1e-4$. The training takes about $5$ hours on the same GPU.

We compare our LightGazeFit with a 3D eye gaze fitting method, Wang et al.’s method [40], and a deep-learning-based gaze estimation method, Gaze360 [49].