Video2StyleGAN: Encoding Video in Latent Space for Manipulation

The StyleGAN [31, 32, 29, 30] generator consists of a 9-level pyramid structure operating on spatial resolution from $4\times 4$ to $1024\times 1024$. Using adaptive instance normalization (AdaIN) [26], the 18 512-dimension vectors, often called styles, are injected into different resolution levels to control the appearance of the output image. The lower-resolution layers control the high-level layout/shape of the face, while the higher-resolution layers control the face details. The space of these vectors is denoted as $\mathcal{W}+\in\mathbb{R}^{18\times 512}$. In our work, a pretrained StyleGANv2 [32] generator is treated as a black-box renderer. Our goal is to project each frame of the video into a latent vector $\mathbf{w}$ in $\mathcal{W}+$, so that the pretrained StyleGAN reconstructs the exact input frame.

As discussed in Sec. 1, video GAN inversion has three challenges: I. Keeping the temporal consistency between consecutive output frames; II. Running at real-time; III. Retaining subtle motion and expression. We will try to solve these challenges in this work.

Inspired by the structure of StyleGAN, many single image GAN inversion works [39, 44, 50] use a hierarchical structure, which encodes the input image into the latent vectors. For the video GAN inversion, however, in order to follow the same structure, we need to adapt some recurrent structures like convolutional LSTM to incorporate temporal information. But, such designs not only make the network difficult to train due to the high GPU memory demand, but also is inefficient. Unlike existing GAN inversion works, we adapt a vision transformer (ViT) [19] in our framework. It unwraps the input image into patches and embeds them into a low dimensional space. Its benefit is twofold: 1) It embeds neighbor frames into the same space, which enables involving temporal information at minimum cost; 2) It is efficient in speed. So, it can solve Challenge I and II simultaneously.

For simplicity, we use “encoding” to represent GAN inversion in the rest of our discussion. Fig. 2 depicts the overview structure of our framework, which consists of the Single Frame Encoding Network (FrEN) and Video Encoding Network (ViEN). The FrEN is trained to encode each frame separately. It directly regresses the $\mathcal{W}+$ vector $w\in\mathbb{R}^{18\times 512}$ from the class token of ViT as shown in Fig. 2(a). As aforementioned, encoding a video frame-by-frame will cause visual jitters in videos. To remove jitters, the ViEN also takes the latent vector of the previous frame as input, other than the current frame. The first frame of a video is encoded by the FrEN.

To illustrate the importance of temporal consistency, we visualize the jitters in Fig. 3. We align 6 reconstructed neighbor frames to their center frame and compute the accumulated root mean square error (RMSE) between the aligned neighbor frame and the center one. The total RMSE is shown at the bottom of each error map. A larger value (red color) indicates the reconstructed video has more visual jitters. In the 2^nd column, although the individual frames are reasonably reconstructed from the latent vectors, single frame encoding results in large visual jitters (e.g. lighting and facial appearance). Such jitters are obvious in human eyes, please watch our supplementary video #2 for better effects. In our experiment, we observe that these appearance jitters originate from the higher resolution layers in StyleGAN. So we try to use a Gaussian filter to smooth the 9-18 layers of $\mathbf{w}$ along the time dimension. The results in the 3^rd column of Fig. 3 indicate the Gaussian smoothing can alleviate the jitters to some extend. However, the naive smoothing in the latent space can also average out the motion of the face as shown in the supplementary video #2. This may due to that the face pose is not completely disentangled in the $\mathcal{W}+$ space. To remove jitters, we need to fix the high resolution layers in $\mathbf{w}$, and also appropriately compensate the face motion caused by directly using high resolution layers from the previous frame.

Let us denote the high and low resolution layers of the latent vector $\mathbf{w}$ as $\overline{\mathbf{w}}\in\mathbb{R}^{10\times 512}$ and $\underline{\mathbf{w}}\in\mathbb{R}^{8\times 512}$, respectively. As shown in Fig. 2(b), the input of the ViEN is the current frame and the latent vector $\mathbf{w}_{t-1}\in\mathbb{R}^{18\times 512}$ from the previous frame. Using an MLP projector, we encode the latent vector from the previous frame as an extra embedding input for the “video transformer”. The output of the ViEN is the lower resolution layers $\underline{\mathbf{w}}_{t}\in\mathbb{R}^{8\times 512}$. We obtain the encoded latent vector as $\mathbf{w}_{t}=\underline{\mathbf{w}}_{t}\oplus\overline{\mathbf{w}}_{t-1}$, where $\oplus$ represents concatenation. Please note that at inference time, $\mathbf{w}_{t-1}$ is the output of ViEN of the previous frame. The first frame of a video is actually encoded by the FrEN. The final reconstructed frame $\widehat{I}_{t}$ can be rendered by a pretrained StyleGAN $G$ as: $\widehat{I}_{t}=G(E(I_{t},\mathbf{w}_{t-1}))$, where $E$ denotes the video encoding network.

Having guaranteed temporal consistency by reusing the high resolution portion of the latent vector (Sec. 3.2), we design losses solely focusing on high fidelity reconstruction of the face for an individual frame. Image GAN inversion works have proved the effectiveness of $L_{2}$ and perceptual loss, which directly evaluate the similarity between the input and reconstructed image. Let the current frame be $I_{t}$ and the latent vector of last frame as $\mathbf{w}_{t-1}$. The $L_{2}$ loss in our training is then defined as:

$$L_{2}(I_{t},\mathbf{w}_{t-1})=\left\|I_{t}-\widehat{I}_{t}\right\|_{2}.$$

(1)

LPIPS [54] loss compares extracted intermediate features via a pretrained backbone network to evaluate the perceptual similarity between two images. We apply LPIPS loss to further enhance the image quality:

$$L_{\{vgg,alex\}}(I_{t},\mathbf{w}_{t-1})=P_{\{vgg,alex\}}(I_{t},\widehat{I}_{t}),$$

(2)

where both VGG and AlexNet are used as the backbone.

Most importantly, as discussed in Challenge III, being able to accurately encode the facial expression and motion is critical for video encoding. This did not gain attention in previous works, since human vision is not sensitive to the subtle discrepancy in still comparisons. However, in videos, people tend to easily observe inaccurate face motions (e.g. eye blink). To tackle this challenge, we seek to leverage high-level face information, i.e. sparse facial landmarks and dense 3D face mesh, to guide the network to implicitly capture the face motion. As a result, we propose a sparse facial landmark loss, which enforces the reconstructed face to have the same detected landmarks as the input face. This loss is defined based on the 2D FAN [13] landmark detector:

$$L_{f}(I_{t},\mathbf{w}_{t-1})=\left\|\mathbf{F}(I_{t})-\mathbf{F}(\widehat{I}_{t})\right\|_{2}.$$

(3)

FAN estimates the position of 68 facial landmarks using $\mathop{\mathrm{argmax}}$ on the 68 channel heatmaps inferred by the network, which is not differentiable. Therefore, in Eqn. 3, we define the landmark loss over the intermediate heatmaps $\mathbf{F}\in\mathbb{R}^{68\times H\times W}$ ($H$ and $W$ are the frame height and width) to evaluate the landmark similarity between the input frame and the reconstructed one. It is worth noting that, our way to define the loss differs from most previous face landmark losses [36, 57] which utilize L2 loss on explicit 2D facial landmarks. The continuous feature maps can preserve the original response from a face landmark detector, which provides more implicit guidance for the training of the video encoding network.

In addition, some face motions (e.g., lifting eyebrows) cannot be accurately described by sparse facial landmarks. Hence, we further use 3DDFAv2 [24] to model these face geometry details. Based on the dense 3DMM [11] mean face mesh, 3DDFAv2 [24] estimates the shape and expression parameters so that the geometry of the face mesh matches the input image. Let $\mathbf{M}$ and $\mathbf{V}$ denote the mean and result 3D face vertices. The relation between the input frame and the result 3D mesh can be written as:

$$\mathbf{V}(I)=\mathbf{R}(I)(\mathbf{M}+\mathbf{A}_{id}\mathbf{\alpha}(I)+\mathbf{B}_{exp}\mathbf{\beta}(I))+\mathbf{T}(I),$$

(4)

where $\mathbf{A}_{id}$ and $\mathbf{B}_{exp}$ are the principal components of identity and expression provided by 3DMM; $\mathbf{\alpha}$, $\mathbf{\beta}$, $\mathbf{R}$ and $\mathbf{T}$ are the identity, expression and rigid transformation parameters inferred by 3DDFAv2. We then define the dense 3D loss as:

$$L_{v}(I_{t},\mathbf{w}_{t-1})=\left\|\mathbf{V}(I_{t})-\mathbf{V}(\widehat{I}_{t})\right\|_{2}.$$

(5)

As discussed in Sec. 4.2, our sparse facial landmark loss and dense 3D loss enable to handle arbitrary face pose and expression without explicitly face alignment as a pre-processing. The final loss function of our network can be written as:

$$L=L_{2}+\lambda_{p}(L_{vgg}+L_{alex})+\lambda_{f}L_{f}+\lambda_{v}L_{v},$$

(6)

where $\lambda_{p}$, $\lambda_{f}$ and $\lambda_{v}$ are regularization values for the corresponding losses.

To train the video encoding network, we collected a face video dataset ( see Sec. 4.1). The training procedure consists of two stages: First, we set $\lambda_{p}=0.8$, $\lambda_{f}=100$ and $\lambda_{v}=0$ and train the network for 90,000 iterations. In our experiment, we observed that the result is initially of poor quality, and 3DDFAv2 [24] mostly returns abnormal face geometries. Therefore, we only apply the soft constraint provided by 2D FAN [13] in this stage.

Second, from the 90,000^th iteration, we set $\lambda_{p}=0.8$, $\lambda_{f}=0$ and $\lambda_{v}=0.0001$ and train until converge. In this stage, the quality of the encoded frames becomes more suitable for 3DDFAv2. We apply a relatively small weight on the dense 3D loss, so that the training is robust to failure cases in 3DDFAv2.

Since the training of our video encoding network requires knowing the previous frame latent vector, we train the FrEN first using the above procedure. After the FrEN converges, we train the ViEN using two consecutive video frames drawn from our face video dataset. The latent vector of the first frame is inferred by the pretrained single frame encoding network, and the ViEN is trained using the second frame and the latent vector of the first frame. For all training steps, we use the Ranger optimizer following pSp [39]. The learning rate is set to 0.0001.

To train the video GAN inversion network, we collected a dataset containing 16300 face videos from the Internet. The number of frames in each video is between 50 and 100. For each frame, we crop the face region with 256x256 resolution. The total number of individual frames is 1,648,849. The sources are TV News and interviews, which cover a wide range of gender, ethnicity, appearance and expression. These variations play an important role in training the pose-aware video GAN inversion and the pose editing network.

For the GAN inversion task, we seek to obtain a sequence of latent vectors so that the pretrained StyleGAN can reconstruct the input video using the latent vectors. We compare our method with learning-based image GAN inversion methods pSp [39], e4e [44], Re-Style [5] and optimization based method Image2StyleGAN [1] and IDInvert [56]. Fig. 4 shows the qualitative comparison.

Example (1) is a relatively neutral expression similar to the images in the StyleGAN training set FFHQ [31]. Example (2) and (3) are typical frames in a talking face video with different expressions compared to FFHQ. Example (4) contains significant expressions, i.e., eye blinking, which is very challenging for existing GAN inversion methods. In general, existing GAN inversion methods fail to properly reconstruct the face in the video due to the variety of facial expressions and head poses. Even if they are able to generate reasonable-looking faces, the result typically contains wrong identity (pSp, e4e and IDInvert in example (1), (3) and (4)) and inaccurate expression (example (4)). Optimization based approach Image2StyleGAN is able to reconstruct identity and expression to some extent, but it may take a significantly longer time to converge to a better result (See Table. 1), and also could not guarantee the visual quality and temporal consistency. Since we introduce losses (Eqn. 3 and Eqn. 5) to implicitly guide the video encoding network, our model automatically perceives the location of the face and is able to handle arbitrary pose and expression. We recommend readers to watch our supplementary video #3 for better comparison, since some attributes of the videos (for example, face motion and temporal consistency) can only be observed in dynamics.

To quantitatively evaluate the performance, we collect a 38 face video test set. In the upper part of Table 1, we evaluate the accuracy of video GAN inversion using a variety of metrics. The PSNR and SSIM measure the image domain similarity. We also include Fréchet inception distance (FID) for the perceptual similarity, since PSNR and SSIM do not evaluate the general fidelity of the reconstructed image. The facial landmark error (LM) is the mean squared error (MSE) between the facial landmarks of the input video and the result, measuring the accuracy of the face pose and expression. To evaluate the temporal consistency of the resulting video, we use the accumulated RMSE of 6 aligned neighbor frames defined in Sec. 3.2, denoted as TC in Table  1.

In general, existing learning-based single image GAN inversion methods pSp, e4e, ReStyle and IDInvert fail to reconstruct a satisfactory appearance (low PSNR and SSIM, high FID) and expression (high LM) of face videos. Optimization based method Image2StyleGAN [1] is able to reconstruct facial expression details (high SSIM and low LM), but the overall appearance performance (PSNR and FID) remains similar to the learning-based methods. Note that the temporal consistency (TC) of the comparison single image methods are all large, especially Image2StyleGAN [1]. This indicates that existing methods cannot be generalized to videos directly. Our method is significantly better in all the metrics, since our GAN inversion network structure guarantees temporal consistency (Sec. 3.2) and is trained with face-aware losses like sparse face landmark loss (Eqn. 3) and dense 3D face mesh loss (Eqn. 5). Also note the per-frame runtime on the right column, our method is the only real-time method that is realistic for video processing, compared to the prohibitive processing time of IDInvert and Image2StyleGAN.

To demonstrate the effectiveness of the loss design discussed in Sec. 3, we show the comparison of our models trained with different loss setups in Fig. 5. In column (b), the results produced by the network trained only with $L_{2}$ loss (Eqn. 1) are blurry. Adding the perceptual loss (Eqn. 2) promotes the overall quality of the image in column (c).

However, due to the lack of high-level understanding of face semantics, the accuracy of the face pose(3^rd row) and expression (1^st and 2^nd rows) are not satisfactory. By applying the sparse facial landmark loss (Eqn. 3, our network is able to perceive the facial expression in the video, as shown in column (d). The dense 3D loss (Eqn. 5) further guides our network to capture subtle expression, making the final GAN inversion result accurately reconstructs the input (columns (e) and (f)). Similar to Fig. 3, we include the accumulated RMSE for the result of single frame encoding and video encoding networks. By reusing the high resolution part of the latent vector, our video encoding network is able to reduce the appearance jitters compared to single frame encoding. In the bottom part of Table 1, we also show the quantitative comparisons of these loss setups. Note that although setup (b) and (c) is temporally consistent (low TC) and achieve better PSNR and SSIM, they produce undesired blurry and inaccurate facial expression (high FID and LM) since they are trained solely on image domain losses.

Due to the complexity of the human face, physically modeling and editing the face appearance [42, 43, 49, 55] is difficult in general. One benefit of encoding a video into the latent space is to control the face pose and expression editing, which is important for video generation. Face pose editing is a challenging task, especially for videos. Due to occlusion, part of the face is often invisible in certain frames, e.g. ears and teeth. Adapting the edited face with other regions in the video, e.g. neck and hair, is also difficult. Traditional methods build sophisticated algorithms for face editing, i.e. learning a warp field to transform high dimensional features [49]. However, these methods could be subject to robustness issues due to the complexity of real-world cases.

On the other hand, since StyleGAN latent space encodes the general face semantics, the rendered face is guaranteed to lie in this space, making the pose/expression editing much easier. We are able to focus on changing the face pose/expression, but do not need to put extra effort into maintaining the fidelity of the rendering. Moreover, since temporal consistency is guaranteed by our video encoding network and the same edits are performed on the latent codes, the editing does not need to consider temporal consistency. Our approach of the network is shown in Fig. 6. Thanks to our face video dataset, we are able to train the pose editing network using the complementary face pose variations in the dataset. For each iteration, we select 2 frames ($I_{1}$ and $I_{2}$) from the same video clip. To incorporate enough pose and expression difference, $I_{1}$ and $I_{2}$ are 10 frames apart. We use 3DDFAv2 [24] to estimate shape $\mathbf{\alpha}_{1},\mathbf{\alpha}_{2}$, expression $\mathbf{\beta}_{1},\mathbf{\beta}_{2}$, rotation $\mathbf{R}_{1},\mathbf{R}_{2}$ and translation $\mathbf{T}_{1},\mathbf{T}_{2}$ on $I_{1}$ and $I_{2}$. We concatenate the shape difference $\Delta\mathbf{\alpha}=\mathbf{\alpha}_{2}-\mathbf{\alpha}_{1}$, expression difference $\Delta\mathbf{\beta}=\mathbf{\beta}_{2}-\mathbf{\beta}_{1}$, rotation difference $\Delta\mathbf{R}=\mathbf{R}_{2}-\mathbf{R}_{1}$ and translation difference $\Delta\mathbf{T}=\mathbf{T}_{2}-\mathbf{T}_{1}$ as the input of a simple 3-layer MLP, which is responsible for inferring the latent vector displacement $\Delta\mathbf{w}$. Using our single frame encoding network, we obtain the latent vector for the first frame $\mathbf{w}_{1}$. Finally, we use the same training losses (Eqn. 6) to enforce the frame reconstructed from $\mathbf{w}_{1}+\Delta\mathbf{w}$ to be similar as $I_{2}$.

Fig. 7 demonstrates examples on pose and expression editing. Since we define the input based on the 3DMM [11] face model parameters, we are able to edit both pose and expression in a flexible way. Following the coordinate system shown in the upper left image, the face can be rotated in the 3D coordinates. The other parts of the frame, e.g. connection with neck, hair and clothes, are automatically handled. The occlusion relation between face and ears is also handled without any artifacts. In the expression editing (last two columns), although the teeth are not visible in the input frame, our method is able to generate them without any human intervention. Since we have shown that existing single image GAN inversion methods cannot handle videos (Sec. 4.2), they could not produce satisfactory video editing results. Therefore, due to limited space, we only compare with IDInvert [56] and pSp [39] if a pre-trained single image editing model is provided for a task discussed in our paper. Note that they only support binary editing with a specific attribute, and generate artifacts (first two examples) and change identity (third example). Please watch our supplementary video #5 for a better view of the temporal consistency of our results.

Another benefit of our work is the convenient face appearance manipulation. Similar to pose editing, temporal consistency does not need to be considered thanks to our video encoding network. In the first row of Fig. 9, we add a random vector $\delta\mathbf{w}$ to the latent vectors of a face video, resulting in random faces performing the same face motion. A typical use case of this is protecting the identity in the video by perturbing the look of the person. Following the previous discussion in StyleSpace [51], we can also manipulate individual attributes of the video by adjusting specific dimensions in the StyleSpace. In the second row of Fig. 9, we demonstrate the results of editing individual attributes in the video including lip color, smile, hair color, hairstyle and eye shape.

By learning a fixed direction in the latent space, we can also manipulate the face video with specific semantic meaning. One example is to edit the age of the face in the video. For the implementation details of our age editing network, please refer to the supplementary material. In Fig. 8, we demonstrate the results of our latent space age editing for face videos. Note that wrinkles are learned without explicit efforts to model. On the right part of Fig. 8, we provide age editing results generated by single image method IDInvert [56]. Since the latent codes for video frames have a significantly different distribution from StyleGAN-like images, the pre-trained latent direction of IDInvert produces wrong attributes like glasses (first three examples) and color bias (last example).

Apart from face editing in the video, our face video encoder network also enables common image-domain processing as an image-to-image translation task. Colorization is an operation to recover colored video from the grayscale input video. Super-resolution is to increase the resolution of the input video while reconstruct the high frequency visual details. For colorization task, we simply train the video encoding network with grayscale frames. For super-resolution task, we bicubic downsample the input video then upsampling 4x to its original frame size. The training procedure of these tasks are the same as described in Sec. 3.4. With these simple modifications, our model can be adapted to applications like reconstructing color videos (first row) and improve the visual quality of low-resolution videos (second row), as shown in Fig. 10. The pre-trained super-resolution model of pSp [39] yield wrong head pose and identity in videos.