APB2FaceV2: Real-Time Audio-Guided Multi-Face Reenactment

Generative Adversarial Networks. Since the concept of generative adversarial network was first proposed [9], many excellent works have been proposed to generate photorealistic images. Generally, these methods mainly fall into two categories: the vector-based method [10, 11, 12] that only uses a noise or embedded vector as input to generate target image, and the pixel-based method [13, 14] that uses the image as input. Theoretically, each of these methods contains a generator G with parameter $\theta_{g}$ to capture the data distribution for generating photorealistic images, as well as a discriminator D to authenticate generated images for enhancing the capability of G in an adversarial manner. To learn the distribution of G over data $x$ from a prior distribution $p{{}_{z}}(z)$ ($G(z;\theta_{g})\in p_{data}(x)$), D plays a two-player minimax game with G in the following value function $V(D,G)$:

Face Reenactment via Audio. Many works have yielded good results in the audio-to-face task, which uses audio as input for providing adequate information about orofacial movements. Works [15, 2, 1, 3] use the audio signal to predict parameters of the predefined 3D model, while Suwajanakorn et al. [16] and Prajwal et al. [17] propose to predict the lip rather the full face. Thus these methods need extra post-operations such as 3D rendering or face fusion, which is cumbersome and not suitable for practical applications. We wish design an end-to-end method to directly generate the full face like [18, 19, 7, 20, 21], and supplement some auxiliary signals simultaneously to control the facial areas that are not related to the audio information. So based on the previous work [6], we design a new framework named APB2FaceV2 that can not only generate photorealistic face end-to-end but also reenact multiple faces in real time by a unified model.

Audio-aware Fuser. The Audio-aware Fuser module inputs audio, head pose, and eye blink signals, which are further extracted by $\boldsymbol{\phi}_{A}$, $\boldsymbol{\phi}_{H}$, and $\boldsymbol{\phi}_{E}$ to obtain $\boldsymbol{f}_{A}$, $\boldsymbol{f}_{H}$, and $\boldsymbol{f}_{E}$ respectively. Specifically, $\boldsymbol{\phi}_{A}$ contains 5 convolutional layers for extracting the feature of each time node and additional 5 convolutional layers for fusing them, while both modules $\boldsymbol{\phi}_{H}$ and $\boldsymbol{\phi}_{E}$ contain three linear layers. Subsequently, the three features are concatenated and further extracted through $\boldsymbol{\phi}_{D}$ to obtain the embedded representation of the facial geometric feature $\boldsymbol{f}_{Geo}$, where the facial landmark is used as the supervisory signal in the training stage.

$\displaystyle\boldsymbol{f}_{Geo}=\boldsymbol{\phi}_{D}(\boldsymbol{f}_{Fus})=\boldsymbol{\phi}_{D}([\boldsymbol{f}_{A},\boldsymbol{f}_{H},\boldsymbol{f}_{E}]),$

(2)

Multi-face Reenactor. Given a reference face image $\boldsymbol{I}_{R}$ and the extracted facial geometric feature $\boldsymbol{f}_{Geo}$, the Multi-face Reenactor $\boldsymbol{\psi}$ reenacts the target face $\boldsymbol{\hat{I}}_{T}$ that has matched facial expression with the input signals, i.e. audio, head pose, and eye blink. Specifically, $\boldsymbol{\psi}$ consists of a chain of sub-modules: an image encoder $\boldsymbol{\psi}_{E}$, a feature transformer $\boldsymbol{\psi}_{T}=\left\{\boldsymbol{\psi}_{T}^{1},\boldsymbol{\psi}_{T}^{2},\dots,\boldsymbol{\psi}_{T}^{L}\right\}$ ($L$ represents the number of module repetitions and is set to 9 in the paper), and an image decoder $\boldsymbol{\psi}_{D}$. The process can be described as:

$\displaystyle\boldsymbol{\hat{I}}_{T}=\boldsymbol{\psi}_{D}(\boldsymbol{\psi}_{T}(\boldsymbol{\psi}_{E}(\boldsymbol{I}_{R}))),$

(3)

Note that the feature transformer $\boldsymbol{\psi}_{T}$ is designed in a decoupling idea that simultaneously learns appearance information from $\boldsymbol{I}_{R}$ as well as the geometric information from $\boldsymbol{f}_{Geo}$, and the new proposed AdaConv is used to inject geometric information on each block.

Adaptive Convolution. Different from APB2Face [6] that injects facial movement information by first plotting the landmark image and then concatenating it with deep features, we propose an elegant information injection module, i.e. Adaptive Convolution (AdaConv). As shown in the top right of Figure 1 (Highlighted in green), the AdaConv layer inputs a geometric vector $\boldsymbol{f}_{Geo}$ and a deep feature map $\boldsymbol{F}_{in}^{l}$, and outputs a modified feature map $\boldsymbol{F}_{out}^{l}$. In detail, $\boldsymbol{f}_{Geo}$ goes through two linear layers to generate a set of parameters, i.e. $\boldsymbol{f}_{Geo}\rightarrow\boldsymbol{W}^{l}$, which will be reshaped and applied to a convolutional layer. As the formula shows below:

	$\displaystyle\boldsymbol{F}_{out}^{l}$	$\displaystyle=Conv(\boldsymbol{F}_{in}^{l};\boldsymbol{W}^{l})$		(4)
		$\displaystyle=Conv(\boldsymbol{F}_{in}^{l};Linear^{l}(\boldsymbol{f}_{Geo})),$

The parameter $\boldsymbol{W}^{l}$ of the convolutional layer contains $k\times k\times C_{g}\times C$ weight parameters and $C$ bias parameters, where $k$, $C$, and $C_{g}$ are kernel size, channel number, and group number for the convolution. Thus we can control the amount of injected information by controlling the values of these parameters. Specifically, AdaConv reduces to AdaIN [22] when we set $\{k=1,C_{g}=1\}$, and we set $\{k=3,C_{g}=1\}$ in the paper for balancing computation and model effect.

Objective Function. In the training stage, we adopt geometry and content losses to supervise geometric information and generated image quality, as well as adversarial loss to further improve the quality and authenticity of the reenacted image. The overall loss function is:

$\displaystyle\mathcal{L}_{All}=\lambda_{G}\mathcal{L}_{G}+\lambda_{C}\mathcal{L}_{C}+\lambda_{Adv}\mathcal{L}_{Adv},$

(5)

where $\lambda_{G}$, $\lambda_{C}$, and $\lambda_{Adv}$ represent weight parameters to balance different terms, and are set 1, 100, and 1 respectively.

Dataset. In the paper, almost all of experiments are conducted on AnnVI dataset that contains six announcers (three men and three women) and 23790 frames totally with corresponding audio clip, head pose, eye blink, and landmark [23].

Implementation Details. We use Adam optimizer [24] with $\{\beta_{1}=0.5$, $\beta_{2}=0.999\}$ and train the model for 110 epoch. The learning rate is set to $2e^{-4}$, and the batch size is 16. PatchGAN [13] is used as the discriminator, and the training setting is in accord with the reenactor.

Qualitative Results. Some qualitative experiments are conducted on AnnVI dataset to visually demonstrate the high quality of reenacted images and the flexibility of the proposed approach. Specifically, we randomly select one face of each identity (6 faces totally) as the reference face, and one drive frame of each identity for supplying audio, head pose, and eye blink signals. As indicated by the red rectangles in Figure 2, our proposed method can reenact photorealistic faces among multiple persons with one unified model that achieves multi-face reenactment task. Thanks to the decoupling design of our method, APB2FaceV2 can use input signals of other persons to reenact the target face that is consistent with the identity of the reference face. Experimental results show that our method has strong generalization ability, where the model can use non-self audio as input to reenact photorealistic faces.

Quantitative Results. As shown in Table 1, SSIM metric is chosen to quantitatively evaluate our method with the state-of-the-art (SOTA) method, and experimental results indicate that the proposed method can generate more photorealistic faces where the SSIM score goes up from 0.799 to 0.805, even though using only one unified model (The work [6] need train 6 models totally for 6 persons). Nevertheless, our method still obtains a higher Detection Rate (DR), i.e. 99.1%, which also demonstrates the superiority of our method than SOTA.

Comparison with SOTAs. We further conduct a comparison experiment with most related SOTA methods on the Youtubers dataset [5]. As show in Figure 3, our approach obtains a better visual effect than others, as well as the best SSIM and FID scores. Moreover, our method significantly reduces the number of parameters by nearly 6 times and 4 times compared to Wav2Pix and APB2Face respectively, and can run in real time, i.e. 22.5 FPS in CPU (i7-8700K @ 3.70GHz) and 158.9 FPS in GPU (2080 Ti), as shown in Table 2.

Decoupling Experiment. A decoupling experiment is further conducted to demonstrate that our proposed method is capable of disentangling input signals, i.e. audio, head pose, and eye blink. As shown in Figure 4, the first three rows are generated results that only change one component of the head pose signal, i.e. yaw, pitch, or roll, while the last row shows the results that only change the eye blink signal. Experimental results indicate that our method can control the properties of the generated face that is flexible for practical applications.