Towards a Pipeline for Real-Time Visualization of Faces for VR-based Telepresence and Live Broadcasting Utilizing
Neural Rendering

Natural face-to-face communication is three-dimensional. A conversation includes not only the verbal communication channel but also the non-verbal channel. In particular, eye contact, facial expressions as well as gestures performed with arms and hands (kinesics), and even the physical distance between each other (proxemics) are essential information carriers during a conversation [LG18]. Currently, common mainstream technologies for computer-mediated communication are video conferencing applications such as Skype or FaceTime. While these allow reading the facial expressions of the counterpart, no “real” eye contact is possible, wide gestures may be cut off in the camera image, deictic gestures are difficult to interpret spatially, and perceptual physical body distance between the participants does not exist.

Current head-mounted displays for VR are capable of delivering believable and immersive 3D experiences including telepresence. However, this does not fully apply to real-time social interactions in VR. When the face of a person is covered by an HMD, it is impossible to read its non-verbal facial communication cues, which is, in fact, a crucial communication channel between individuals. This is not only relevant for face-to-face meetings in VR (for example in VRChat [vrc22], Altspace [alt22], or Meta’s Horizon Worlds [met22]) but also in VR application scenarios in which only one VR user wears an HMD and tries to engage with their audience. For example, such a VR user could be an architect who presents ideas for a new building in VR to their clients, a Virtual YouTuber (VTuber), a Twitch streamer in front of a green screen who broadcasts themselves from inside a VR environment, or friends playing a VR game together in a living room.

The classic way for creating and rendering photo-realistic humans in real time is costly and requires a lot of manual effort such as scanning, modeling, and manual texturing from a skilled 3D artist. Furthermore, today’s HMDs usually lack adequate sensors for face tracking. Only a few research groups have so far addressed this problem and presented methods that can generate authentic face avatars for VR without extensive manual modeling [TZS⁺18, LSSS18, WSS⁺19, RZS⁺21, GPL⁺21]. These approaches are not available to the public, and many of them require expensive hardware [LSSS18, WSS⁺19].

Human avatars and their perception have been studied in multiple domains. A systematic review of social presence concludes: ”…multiple studies show that the vivid perceptions of another person often lead to greater enjoyment and social influence…” [OBW18]. Although, the Media Richness Theory [DL84] is almost 40 years old, recent studies still confirms it [ILC19]. It suggests that the most comprehensive exchange of information between people happens during face-to-face conversations compared to all other digital or analog communication possibilities. A higher quantity and quality of data shared typically leads to more effective communication. A key issue in this context of face-to-face telepresence in VR is the occurrence of the Uncanny Valley effect [MMK12]. Humans are markedly sensitive to minimal and unnatural discrepancies in faces. As soon as a virtual human does not perfectly resemble a real human, it is often subconsciously classified as unlikeable, unpleasant, or even creepy.

One technique that has successfully bridged the Uncanny Valley in recent years is Generative Adversarial Networks (GANs). Today, GANs serve as the core technology behind Deepfakes. They enable such authentic results that their methods and algorithms are the subject of current research to distinguish fake images from real ones, as the human eye is no longer able to reliably do so [RCV⁺19]. Therefore, we use algorithms in the context of this work that are also employed to create Deepfakes. We extend this approach with an additional dimension (textured 2.5D point cloud instead of only an RGB image) to generate realistic representations of 2.5D face avatars. We do not create a full 3D head because we only capture the face of a person from a static frontal position with an RGBD sensor. This implies that we do not generate realistic textures from side views. However, we maintain a stereoscopic perception of the reconstructed face during face-to-face conversations in a virtual environment.

We present an end-to-end learning system that has low hardware cost compared to others, requires moderate computational resources, and generates results with frame rates suitable for VR applications. Our research contributes to GANs playing a key role in authentic 3D telepresence applications in the near future. In addition to sharing our insights in this paper, we make the code of our prototype publicly available under: https://github.com/Mirevi/face-synthesizer-JVRB.

This work is an extension and improvement of our previous neural rendering pipeline [LPG20] and complements our recent work on how to build an HMD with face tracking capabilities [LPDG20]. The contributions of this work are the creation of a face capture pipeline as well as the introduction of novel Generative Adversarial Networks (GAN) [GPAM⁺14] that are tailored for the authentic reconstruction of faces in three-dimensional telepresence and live broadcasting applications. The motivation is to use a commodity graphics card to capture and reconstruct the individual characteristics of a person’s face with a high level of (personal) details and VR-enabled frame rates in order to create an authentic avatar that goes beyond the capabilities of today’s avatar creation tools such as VRChat [vrc22], Altspace [alt22] or Meta’s Horizon Worlds [met22].

Face reconstruction for telepresence and (live) broadcasting with an HMD occluding a person’s face is a young research field. Olszewski et al. [OLSL16] presented a system that uses an RGB camera to transfer facial expressions from the lower face area to an avatar. Li et al. [LTO⁺15] extended this approach with pressure sensors placed in the foam of an HMD capturing a person’s facial expressions. The idea of using sensors within the HMD is similar to our concept, but we use personalized avatars that are trained in advance and synthesized in a final step.

Casa et al. [CFA⁺16], Früh et al. [FSK17], and Thies et al. [TZS⁺18] used stationary RGBD cameras to create personalized avatars of users. In the first step, the user was captured by the camera without an HMD in order to create a virtual avatar. When the user wore the HMD, the stationary RGBD camera recognized facial expressions. Due to the fixed position of the camera, the range of head motion was limited. Eye movements were registered by eye-tracking cameras and transferred to the user’s face avatar. The approaches of Casa et al. [CFA⁺16] and Früh et al. [FSK17] evoked the Uncanny Valley effect to varying degrees. To mitigate this, Früh et al. [FSK17] did not completely remove the HMD, but rendered it as a semi-transparent object. These systems are similar to our approach in the way that they create a personalized avatar using an RGBD camera and produce almost photo-realistic avatars. The approach of Thies et al. [TZS⁺18] demonstrated better results by using a 3D morphable model (3DMM) as underlying head mesh template and as an inductive bias to their system that provides fundamental data about the composition of a human face. They optimize the visual quality by an analysis-by-synthesis approach [BV99] and achieve photo realistic results with only a few image artifacts. While the visual quality is convincing, this approach only provides stereoscopic renderings without the ability to freely choose the perspective around the reconstructed face because the final results are based on a given 2D video. Furthermore, it inherently does not allow for manipulation of the head’s rotation, scale, and position in the final result.

The systems of Lombardi et al. [LSSS18], Wei et al. [WSS⁺19], and Raj et al. [RZS⁺21] create photo-realistic avatars with authentic facial expressions. While previous works completed the generation of personalized avatars within a few minutes, the system of Lombardi et al. requires computational time of more than a day. The three-dimensional avatar is generated with the aid of a large number of high-resolution images from different angles and facial expressions with an expensive hardware setup that generates a large amount of data for further processing. The created face avatar can be controlled by three RGB cameras attached to an HMD. A key component of this system is the use of Variational Autoencoders (VAEs). Both VAEs and GANs have been proven several times to be suitable for authentic face reconstruction. However, since literature shows that VAEs and only a L1 loss tend to produce blurry results more often, we use GANs [JH19]. The latter concept was first presented by Goodfellow et al. [GPAM⁺14], and Radford et al. [RMC15] improved it in a sustainable way. Furthermore, Karras et al. [KALL17] achieved photo-realistic portrait images that are indistinguishable from real photographs by using the principle of Progressive Growing GAN. However, according to Karras et al., the GAN has little to no external control over the appearance of the generated object or face because the input to the network is a latent vector without any direct relation to a face property such as hair color, facial expression, or gender. In further works Karras et al. [KLA18] enhanced the architecture of the GAN and were able to automatically separate higher-level attributes (e.g. pose, identity) from stochastic variations (e.g. freckles, hair). Nevertheless, this approach does not allow to explicitly control the facial expression.

Conditional GANs (cGANs) have been shown to be able to learn and reproduce specific relationships between inputs and outputs that are understandable for humans. For example, Mirza and Osindero [MO14] have extended the input to the generator and discriminator with a label ${y}$, which makes it possible to generate images from a particular category ${y}$. This method for conditioning GANs was developed further by Radfort et al. [RMC15] with the DCGAN and by Isola et al. [IZZE17] with the Pix2Pix GAN. They replaced the noise input vector $z$ with a user-defined input vector. Without a noise vector, there is no latent space $Z$ (since ${z\in Z}$). If the stochastic aspect contained in the noise vector is not compensated, the GAN will only memorize the training examples. Any inputs that deviate from the training data would lead to inadequate results, as described by Isola et al. [IZZE17]. By using a U-net architecture [RFB15] with dropouts in the Pix2Pix GAN, the stochastic aspect as well as the missing latent space can be otherwise integrated into the generator. The discriminator of the Pix2Pix GAN receives the same input image $x$ as the generator as well as its output image $y_{fake}=G(x)$ or the image $y_{real}$ matching $x$ from the dataset. This is basically equivalent to the idea of cGANs [MO14] where not only the output of the generator is evaluated but also its difference from the input. Unlike the cGAN, the output of the discriminator of the Pix2Pix GAN is not a scalar that decides between “real” or “false” but a matrix. By using convolutional layers (cf. Radford et al. [RMC15]), each entry in the output matrix represents an $n*m$-sized region (so-called patch) of the input image. This allows abstract representations to be admitted as matrices (e.g. images) for conditioning the network to have a controlled influence on the output of the generator. This approach was further developed by [WLZ⁺17] with the Pix2PixHD GAN to generate images with a higher resolution and more details. In this paper, we adapt the idea of cGANs, especially of the Pix2Pix and Pix2PixHD frameworks, and tailor them to our application domain.

In the following, we explain the process and structure of the proposed system, as shown in Fig. 1, and then discuss the steps in more detail in the subsequent sections.

Our process starts with the acquisition of a personal RGBD dataset. The acquired data is preprocessed by an automated procedure. A Facial Landmark Map (FLM) per RGB image is extracted and saved beside the corresponding RGB image. It decodes the facial expression of the respective RGB image in a binary image as so-called landmarks as shown in the second image from left in Fig.1. Our proposed GAN is trained with the captured RGBD images as well as with the corresponding FLM. For each person our GAN must be trained from scratch. We do not use any inductive biases like a 3DMM [BV99] and the system does not learn correspondences between persons. After the training, the system can be used for real-time telepresence or live broadcasting. In the VR application scenario, the user would wear a face-tracking head-mounted display that could create an FLM in real time, which we then feed into the trained generator module of our GAN. This paper does not focus on building and implementing a face-tracking HMD. Further hardware-related implementation details are described in our previous work [LPDG20]. The GAN could create an RGB and a D image of the “learned” person based on the FLM, and finally, we fuse the generated RGB and D images into a textured point cloud.

GANs are difficult to train because of their adversarial training procedure to find the balance between the learning rate and losses of the generator and discriminator networks. As we use many different losses, a long line of hyper parameter tuning was necessary to find the optimal settings. Fig. 5 shows the reconstruction results for four different persons with the best training parameters described in section 3.4.

We did not use the face-tracking HMD for the evaluation, similar to [LPDG20], because it can cause slight tracking errors and could decrease the quality of the results for comparison. Our intention is to directly compare the improvements of our pipeline to our previous system described in [LPG20]. For a comparison in motion, we refer the reader to the corresponding video that can be found via the GitHub link.

As a quantitative metric for the assessment of the reconstruction quality, we use Structural Similarity (SSIM) [ZBSS04] and Learned Perceptual Image Patch Similarity (LPIPS) [ZIE⁺18]. Our previous system [LPG20] performed on average with an SSIM of 0.851 and a value of 0.114 on LPIPS. Our proposed system reaches on average 0.910 for SSIM (higher is better) and 0.082 for LPIPS (lower is better). The numerical results of the SSIM metric are comparable to a JPEG compression of about half the original file size of the images in column 3. In contrast, the previous system [LPG20] only achieved reconstruction quality of less than a quarter of the original file size by means of a comparison with the JPEG compression.

All results and metrics shown in Fig. 5 and 6 are generated from LFMs from the test set. This means, no backpropagation was conducted on the test set. The train/test split was 85% to 15%. The dataset for the first participant in Fig. 5 and 6 was split 1238/207, for the second 1500/250, for the third 1620/271 and for last 2413/403. Please note that the dataset for the last participant was around 1000 items larger compared to the previous participants and lead to slightly better quantitative results for the best and worst result according to SSIM and LPIPS (the last two images per participant). This allows the conclusion that a larger data set tends to lead to better image quality in our scenario.

The main issue of our previous approach was the sharpness of the generated images [LPG20, LPDG20], as shown in Fig. 4. Due to the new architecture and loss functions, the system produces images with more details. Even skin pores are reconstructed well, e.g. on the user’s forehead, as can be seen in Fig. 5, rows E to H. Furthermore, we noticed a better reconstruction quality in areas with high-frequency details such as facial hair, as shown in Fig. 4. Also, the temporal consistency is improved. Please see the linked video on our GitHub page for details.

The error between the generated and ground truth depth values is mostly below 4 mm, as depicted in column 6. Exceptions are the reconstruction results with the worst SSIM and LPIPS metrics per dataset of a person, such as shown in Fig. 5, row D and H, as well as in Fig. 6, row E and J. Furthermore, outliers can be seen in column 8. Note that we use the raw depth image of the Kinect and the raw output of our GAN. We do not filter or smooth the images, therefore, we assume that the network has also learned the depth noise of the sensor, which causes additional depth errors.

To compare the faces between the images of columns 2 and 3 without measuring background changes, we determined the facial area based on depth values and rejected all other pixels. At the border area between the faces and the background, large differences in the SSIM and the depth difference visualization can be seen. These differences are caused by the fact that the cropped areas of the real and generated images do not always align perfectly due to minimal differences in the generated faces. In addition, we also apply erosion and clipping to the face to reject parts of the background, which can cause the minimal differences.

Although our quantitative metrics indicate better results, our system still shows limitations in the reconstruction quality of the eyes, lips and oral cavity. Especially teeth and the tongue are often reconstructed with noisy artifacts, as can be seen in Fig. 5, row D, column 2. The error increases with a graceful degradation when the expression moves towards exaggerated expressions that are far from the neutral face expression. The eyes are reconstructed with less artifacts than the oral cavity, but we observed that even a little amount of image artifacts can evoke the Uncanny Valley Effect, as can be seen especially in Fig. 5, row B, column 2. Please zoom in for details.

In our experiments we used PyTorch 1.8 and Python 3.7 on Windows 10. We trained the same data set with approximately 1600 elements (an element is a set comprised of an RGBD image and an FLM) on two different machines. This took 8 hours on a system equipped with an Nvidia RTX2080Ti, an AMD Ryzen Threadripper 2990WX and 128GB RAM. With a newer system, comprised of an Nvidia RTX3090, an Intel i7-9900K and 32GB RAM, training took only 4 hours and 13 minutes. Our previous system took 17-20 hours on the machine with the RTX2080Ti for only 600 elements.

The time required for a forward pass of the generator module with an image size of $512\times 512$ pixels is between 3 and 4 ms (333–250 fps) on an Nvidia RTX3090 and between 6 and 7 ms (167–143 fps) on an Nvidia RTX 2080. Our previous system was faster (between 1 and 3 ms) but generates only images with a size of $256\times 256$ pixels. The timings of the present system are still suitable for VR-based application, where 75 to 120 fps are common frame rates. However, note that many face tracking systems only work with 60 or even 30 fps, which can limit the frame rate of the pipeline.

As mentioned before, one of the major issues is the low reconstruction quality of exaggerated expressions of the eyes, lips and oral cavity. The artifacts can induce the Uncanny Valley Effect and must be avoided for telepresence or broadcasting applications. As a further step, we plan to use a 3DMM [BV99] such as the FLAME model [LBB⁺17] as a better inductive bias to regularize depth and color information more efficiently. Furthermore, we observed that landmark tracking is not sufficient for faithful lip movement during speech. Therefore, an additional input signal besides the landmarks is necessary. A conditioning of speech as audio signals could provide a solution. Another issue to improve is the uncomfortable helmet mount. A solution with a stationary RGBD camera placed on a tripod or table is a favorable approach for future research. Although, we achieve real-time frame rates on a gaming GPU, the generator network could still be too slow for using on a current stand alone headset like a Meta Quest 2 or a Vive XR Elite. Further experiments, hardware and performance improvements could solve this problem in the future.

We presented an improved end-to-end pipeline compared to previous approaches [LPG20, LPDG20] and a new GAN architecture that can learn facial identity and individual expressions of a user and reproduce them as a textured point cloud with frame rates that are suitable for Virtual Reality, telepresence and broadcasting environments. We have incorporated and extended the architecture, losses, and processing pipelines of several approaches from the field of neural rendering. Compared to previous works, our proposed system generates higher quality image results with slightly longer run time at inference. We achieved this goal by mainly changing the architecture of the discriminator while keeping the architecture of the generator lean. The reconstruction results partially lie in the Uncanny Valley, but they still convince with an authentic visualization of the respective person’s identity and individual facial expressions. We believe that neural rendering will be a crucial part of photo-realistic rendering of humans in real-time applications in the future. Our work is a further step into this direction and hopefully helps to understand, improve and apply this technology.

We thank the MIREVI group at the University of Applied Sciences Düsseldorf and the Promotionszentrum Angewandte Informatik (PZAI) in Hessen, Germany. This work is sponsored by the German Federal Ministry of Education and Research (BMBF) under the project numbers 16SV8182 (HIVE-Lab), 13FH022IX6 (iKPT 4.0) and 16SV8756 (AniBot).