Assembling a Pipeline for 3D Face Interpolation

It is common to observe many realistic 3D animated characters being used in video games and movies. Behind the scenes, large data sets of images are often used in order to achieve realistic appearances. Designers are also often responsible for editing the characters in order to improve the final results. While significant advances have been achieved in facial animation, it is still difficult to make human-like 3D faces that always look realistic. People are very good at recognizing the differences between real humans and digital humans.

3D character animation technologies are also starting to be used in a variety of new innovative products. For example, the newest iPhone (smartphone by Apple) has an application that can take pictures or videos of people and place 3D models covering their faces and moving in coordination with the real faces. Such an application illustrates the growing need for simple and efficient approaches for achieving realistic face animation. This paper describes a pipeline built with open source tools for achieving animations by interpolation of 3D facial expressions taken from pictures.

A significant amount of previous work has relied on datasets of human faces in order to build face models. For example, the approach of Blanz and Vetter [1] applied pattern classification on their dataset of human faces in order to reconstruct a 3D face model from a single 2D face image. Booth et al. [2] introduced the Large Scale Facial Model (LSFM), which is able to automatically construct 3D models of a variety of human faces from a data set of 9,633 distinct facial identities. Tran et al. [3] also proposed a framework to construct 3D models from a large set of face images, but without requiring to collect 3D face scan data. Huber et al. [4] presented the Surrey Facial Model, a multi-resolution model that can build a facial model in different resolution levels.

Facial animation can be accomplished by various approaches. Chuang and Bregler [5] describe an approach for facial animation that is based on motion capture data and interpolation of blend shapes. Noh and Neumann [6] proposed the work Expression Cloning, an approach that makes a facial movement by transferring vertex motion data from one source to another. Lee et al. [7] introduced an approach to generate facial expressions based on muscle information from real face data.

There are also available products that allow the creation and manipulation of 3D facial models. For example, Face Poser is a system presented by Lau et al. [8] and Poser [9] is a software system that facilitates modeling and editing 3D faces with a comprehensive graphical user interface.

In this work a solution based on available open source tools is presented. 3D face models are created from photos based on the Surrey Facial Model [4]. In order to achieve a simple approach for face animation, this work focuses on animating the interpolation between a pair of input facial expressions.

Our overall method consists of three main steps. First, 3D face models from pairs of input photos are created. In this work we have used 4 pairs in order to obtain 4 animations between different facial expressions. These pairs are presented in Figure 8.

Second, for each pair, the corresponding texture images are morphed in order to obtain a texture image for a given interpolation factor $t\in[0,1]$, where $t$ allows to obtain results from the initial model ($t=0$) to the goal model ($t=1$).

Finally, the coordinates of the 3D models are interpolated according to parameter $t$ and associated with the intepolated texture image computed in the previous step. These steps are detailed in the next subsections.

Figures 15 to 18 at the end of this paper illustrate the interpolated 3D face models obtained for all 4 pairs that were considered. The presented in-between models were obtained with interpolation parameter $t$ starting from 0 to 1, by 0.1 increments.

The morphed 3D models for all pairs look natural both with respect to the mesh and texture image deformations. Table 1 presents performance measurements showing how long it takes to compute one interpolated 3D model frame at different resolutions with a growing number of vertices. A 3D face model with 29587 vertices is still processed fast enough to produce smooth animations.

The used Surrey Facial Model [11] allowed us to construct 3D face models that were natural; however, it did not produce good results for all types of faces. Problems were encountered especially for people with rounded faces. The Surrey Facial Model was constructed by analyzing 169 scan data sets and nearly 60% of the scanned data were from Caucasian people. This might be the reason why it was difficult for the Surrey Facial Model to achieve a precise face scaling for all types of faces.

While the obtained results were smooth and of good quality, inorder to achieve a complete result additional facial objects have also to be considered. For example, models for eyeballs, teeth, tongue, eyebrows, eyelashes and hair are important for achieving complete animated faces.

Improved rendering techniques are also important to be considered. The presented results were rendered only using a standard Phong illumination model with a single frontal light source. Special skin illumination characteristics are important to be included with dedicated shaders in order to replicate skin properties and achieve improved illumination results.

The described approach presents a simple pipeline to achieve fast and natural 3D facial animation between given expressions. The approach can be replicated with the use of open source tools and animations can be obtained just from input face photos of different expressions.