Semantic-aware One-shot Face Re-enactment with Dense Correspondence Estimation

We adopt an advanced 3D parametric model, FLAME (Li et al. 2017), to explicitly decompose face semantics into various disentangled components. Specifically, FLAME depicts the geometry of a portrait image with coefficients of camera $\alpha$, pose $\beta$, expression $\theta$, shape $\phi$, lighting condition $\lambda$ and facial texture $\mu$. As shown in Fig. 2, an encoder network, denoted as $E_{3D}:I\rightarrow C$ ($C=\{\alpha,\beta,\theta,\phi,\lambda,\mu\}$), is proposed to regress FLAME coefficients $C$ from a given face image $I$. Concretely, coefficients for disentangled semantics of $I_{s}$ and $I_{d}$ could be computed as $C_{s}=E_{3D}(I_{s})$ and $C_{d}=E_{3D}(I_{d})$, respectively. Thus, the set of coefficients for the re-enacted face $I_{r}$ could be obtain by simply mixing the element in both $C_{s}$ and $C_{d}$, which could be denoted as $C_{r}=\{\alpha_{d},\beta_{d},\theta_{d},\phi_{s},\lambda_{s},\mu_{s}\}$ (geometry-related coefficients obtained from $I_{d}$, and the rest from $I_{s}$).

Unlike most existing methods merely using low-dimensional 3DMM coefficients for re-enactment control, we also make use of the generative ability of FLAME to convey more semantic information in the spatial dimension. To this end, given the set of parameters $C_{r}$, a differentiable rendering network (Feng et al. 2021) (denoted as $R:C\rightarrow I$) is adopted to generate the corresponding proxy image $P_{r}=R(C_{r})$. Notably, $P_{r}$ helps mitigate the identity gap as it shares the same identity with $P_{s}=R(C_{s})$, i.e. the textured mesh of $I_{s}$, and contains abundant yet compact spatial information (e.g., no background but adequate facial texture) to help estimate the desired face motion field.

Intuitively, the mismatch between $I_{s}$ and $I_{d}$ in both identity and image content (e.g., background and high-frequency texture) makes it difficult to accurately estimate the desired face motion field for animating $I_{s}$. However, according to the discussion in Section 2.1, the advantages of textured face proxies rendered by FLAME (i.e., $P_{s}$ and $P_{r}$) help mitigate those gaps and provide an aligned space for motion field estimation. In this work, we model the face motion between $I_{s}$ and $I_{d}$ with a dense correspondence field $\mathcal{F}$, where each element $f_{i,j}\in\mathcal{F}$ denotes the correlation between two image patches $p_{i}$ and $p_{j}$ within $P_{s}$ and $P_{r}$, respectively.

Specifically, we use an encoder network $E_{corr}$ followed by two sequences of ResBlocks, denoted as $E_{res}^{(s)}$ and $E_{res}^{(r)}$, to extract features from $P_{s}$ and $P_{r}$, respectively. Feature maps computed by a pre-trained HR-Net (Sun et al. 2019) (denoted as $L_{s}$ and $L_{r}$) are also involved to represent the distribution of facial key-points, which help the network better explore the inner relationship between facial components. So far, both face proxy ${P}$ and landmark embedding ${L}$ only describe the content within facial area, and do not contain the features of the rest of images. Therefore, to make the network learn about the spatial layout beyond facial regions, we also feed the parsing map of $I_{s}$ (denoted as $M_{s}$) computed by a pre-trained BiSeNet (Yu et al. 2018) to the feature extraction network (see Fig. 3).

Mathematically, the feature vector for the source face image could be written as

$$f_{s}=E_{res}^{(s)}(E_{corr}(P_{s}),L_{s},M_{s})$$

(1)

and similarly, we have

$$f_{r}=E_{res}^{(r)}(E_{corr}(P_{r}),L_{r})$$

(2)

describing the feature of $P_{r}$. Therefore, the dense correspondence field $\mathcal{F}$ could be represented by the correlation matrix between $f_{s}$ and $f_{r}$ (Zhang et al. 2020b; Zhou et al. 2021), which could be formulated as

$$\mathcal{F}=softmax(\hat{f}_{r}\cdot\hat{f}_{s}^{T})\in\mathbb{R}^{hw\times hw}$$

(3)

where $\hat{f}_{s}\in\mathbb{R}^{hw\times 1}$ and $\hat{f}_{r}\in\mathbb{R}^{hw\times 1}$ denote the flattened and normalized version of $f_{s}\in\mathbb{R}^{h\times w}$ and $f_{r}\in\mathbb{R}^{h\times w}$, respectively.

Notably, with the column-wise softmax operation in Eq. 3, warping of $I_{s}$ could simply be achieved by matrix multiplication, i.e. $I_{s}^{warp}=\mathcal{F}\cdot\tilde{I}_{s}$, where $\tilde{I}_{s}\in\mathbb{R}^{hw\times 1}$ denotes $I_{s}$ after down-sampling and flattening. The advantage is two-fold: 1) it could be easily extended to other spatial representations of $I_{s}$, e.g. textured face proxy $P_{s}$ ($P_{s}^{warp}=\mathcal{F}\cdot\tilde{P}_{s}$) or parsing map $M_{s}$ ($M_{s}^{warp}=\mathcal{F}\cdot\tilde{M}_{s}$), to provide multi-modal prior knowledge for the subsequent generation process (see Section 2.3), and 2) the reverse mapping could also be implemented with a simple matrix multiplication operation, which facilitates the computation of cycle consistency constraints (see Section 2.4).

Although $I_{s}^{warp}$ could theoretically approximate the ideal re-enactment result and serve as a generative prior, it is often blurred due to the loss of high-frequency details in $I_{s}$, and distorted due to the error of estimated warping field $\mathcal{F}$ (see Fig. 3). To solve this problem, we propose a Facial Prior-guided Generator to synthesize the final re-enactment result with high visual quality, which incorporates various warped spatial representations of the source subject (i.e., $I_{s}^{warp}$, $M_{s}^{warp}$, and $P_{s}^{warp}$) as the guiding prior knowledge.

As shown in Fig. 3, the generator takes the down-sampled concatenation of $I_{s}^{warp}$, $M_{s}^{warp}$, and $P_{s}^{warp}$ as input, and generates the final output with a sequence of ResBlocks. For ResBlocks on coarse levels, $I_{s}^{warp}$, $M_{s}^{warp}$, and $P_{s}^{warp}$ are involved to introduce prior knowledge on the spatial layout of re-enacted faces. Since they contain different semantic information of $I_{r}$, we design an Attention-based Fusion Module (AFM, as shown in Fig. 4) to fuse them into a single-channeled feature map, which is injected into the main body of the generator for semantic guidance.

Concretely, since the information of image background is only embedded in $I_{s}^{warp}$, while that of the foreground region (i.e., facial skin area) is contained in both $I_{s}^{warp}$ and $P_{s}^{warp}$, the feature of background ($f_{bg}$) and foreground ($f_{fg}$) could be computed by

$$f_{bg}=conv(STN(I_{s}^{warp}))$$

(4)

$$f_{fg}=conv(concat(STN(I_{s}^{warp}),STN(P_{s}^{warp})))$$

(5)

where $STN$ denotes the Spatial Transformer Network (Jaderberg et al. 2015), $conv$ stands for the convolutional block, and $concat$ refers to the channel-wise concatenation operation. Then, $f_{bg}$ and $f_{fg}$ are merged by an attention map $A$ computed based on all three input, as shown in Fig. 4. The final output $f_{out}$ could be obtained by merging $f_{bg}$ and $f_{fg}$ via

$$f_{out}=A\cdot f_{fg}+(1-A)\cdot f_{bg}$$

(6)

Afterwards, $f_{out}$ is integrated into the corresponding Resblock via spatially-adaptive normalization (Park et al. 2019). In contrast, due to the inevitable error of warping field estimation, we do not incorporate $I_{s}^{warp}$, $M_{s}^{warp}$, and $P_{s}^{warp}$ into ResBlocks on finer levels, since they may increase the chance of introducing noise on high-frequency textural details.

The dense correspondence estimator and facial prior-guided generator are trained simultaneously with the following objective functions.

Similar to (Siarohin et al. 2019), on FF++ and Celeb-DF, we randomly select videos of $80\%$ subjects for training, and use the rest data for testing with no identity overlap with the training split. VoxCeleb1 is used as a driving dataset to test the performance of cross-domain face re-enactment where source images are selected from the testing split of FF++ and Celeb-DF.

According to comparison results shown in Fig. 6, both One-shot and Latent-pose fail to completely disentangle the identity information in driving faces, and thus they interfere with the generation results and cause large face shape distortion. Although FOMM shows better performance in identity preservation, it fails to accurately capture pose changes and subtle movement of facial components (especially for eyes and mouth). Also note that FOMM requires image sequences as input for driving at inference time, while our method could work in cases where only one single driving image is available. Moreover, our method outperforms Bi-layer in synthesizing more realist hair region and background texture preservation.

Quantitative results for both single- and cross-domain face re-enactment are reported in Table 1. Since One-shot enforces the shape of re-enacted faces close to that of driving images, it produces lower error in re-enactment fulfillment but completely fails to preserve the identity of source faces (CosSim values lower than 0.35 in all cases). Latent-pose also suffers from poor identity preservation performance, where CosSim values are around 0.36 to 0.37. In contrast, our method achieves much better performance in preserving the source identity, indicating the success of using 3DMM to bridge the identity gap in one-shot face re-enactment. Although CosSim scores obtained by FOMM are slightly higher than ours, it produces larger errors in pose and expression transformation compared to our methods in most cases. Moreover, FOMM requires a sequence of faces with no abrupt change for driving, while our method only needs one driving image to achieve face re-enactment, which has a much broader range of application.

In this subsection, we validate the effectiveness of the network design for each component in our method. Specifically, we propose the following variants of DCG-GAN for investigation: 1) In DCG-GAN-sD (simple Dense correspondence estimation), $\mathcal{F}$ is computed merely based on $M_{s}$ and $M_{r}$ with no $L_{s}$, $L_{r}$ or $P_{s}$ involved; 2) DCG-GAN-sG (simple Generator) refers to the setup where no AFM network is used to guide the generation process, and $I_{s}^{warp}$, $M_{s}^{warp}$, $P_{s}^{warp}$ are fed to the generator only at the bottom layer; and 3) DCG-GAN-sDG combines the configuration in both previous settings for comparison.

As shown in Fig. 8, removing the facial landmark ($L_{s}$ and $L_{r}$) and parsing map ($P_{s}$) impairs the accuracy of motion field estimation, especially for hair and mouth regions. The reason for this might be that the model fails to extract sufficient information of both the image layout (contained in $P_{s}$) and shape of facial components (embedded in $L_{s}$ and $L_{r}$). Results obtained by DCG-GAN-sG have severe problem in identity preservation, as no detailed textural information is provided for ResBlocks at higher resolutions. DCG-GAN-sDG is very unstable to train and we only present results obtained by the latest checkpoint before collapse, which suffer from severe artifacts. Quantitative results of reported in Table 1 are in line with our previous observations. Our model consistently outperforms its variants under all measurements, indicating the effectiveness of the network design for each module.