TIFace: Improving Facial Reconstruction through Tensorial Radiance
Fields and Implicit Surfaces

To address the View Synthesis Challenge for Human Heads (VSCHH), we use TensoRF [1] as our baseline. TensoRF utilizes 4D tensors to model the radiance field of a scene. The key idea mainly focuses on the low-rank factorization of 4D tensors to achieve better rendering quality and smaller model size with fast speed. Following the baseline model, we implement a vector-matrix decomposition version of TensoRF with some modifications to fit the ILSH dataset. As shown in Figure 3, although the baseline model achieves relatively accurate facial reconstruction, the reconstructed results often have floating artifacts, which severely affects the reconstruction quality of the edges of the face. We speculate that this is because there are light sources in the background area of the rendered image, which causes a sudden change in the background color. To avoid this, we use an efficient way to get the masks corresponding to the input images. Furthermore, we improve both explicit and implicit rendering methods and build additional constraints based on masks to improve rendering quality.

Recently, SAM-based image segmentation methods have attracted a lot of attention [7]. Following ViTMatte [15], we obtain the corresponding masks of the input images through a small number of label points as prompts. As shown in Figure 2, we get masks fine enough to distinguish the foregrounds and the backgrounds of the images, even in the hair strand region.

A natural way to utilize masks is to combine masks with RGB images into RGBA images, as implemented on the Blender dataset. This actually uses masks to set the values of background area in the image to a fixed color (e.g. black). Although it appears to remove the effect of cluttered backgrounds on rendering faces, this method still exhibits some artifacts in our experiments, as shown in Figure 4 and Figure 5. We speculate that the reason is that the generated mask is not completely accurate and that the model lacks constraints on the background, resulting in a not so fine sampling on the surface.

To address this issue, we propose a constraint on the mask and demonstrate its effectiveness through experiments. For each pixel, we march along a ray, sampling $Q$ shading points along the ray and computing the accumulated density weights:

$$T=\sum_{q=1}^{Q}\tau_{q}(1-\exp(-\sigma_{q}\Delta_{q})),\tau_{q}=\exp(-\sum_{p=1}^{q-1}\sigma_{q}\Delta_{q}).$$

(1)

Here, $\sigma_{q}$ is the density at the sample location $x_{q}$, $\Delta_{q}$ is the ray step size, and $\tau_{q}$ denotes the ray transmittance. The mask loss is:

$$\mathcal{L}_{mask}=\frac{\lambda}{|B|}\sum_{x}T_{x}^{2}\mathbb{I}(x\in B),$$

(2)

where $x$ is the pixel corresponding to the ray, $B$ is the background areas according to the masks, $\mathbb{I}(\cdot)$ is the indicator function. Here, we set $\lambda=0.01$ in our experiments. Note that we also tried to use cross entropy in mask loss function, and found that it is not as effective as the proposed loss.

Finally, the total loss is defined as:

$$\mathcal{L}_{T}=\mathcal{L}_{color}+\alpha\mathcal{L}_{reg}+\beta\mathcal{L}_{mask}+\gamma\mathcal{L}_{sparsity},$$

(3)

where $\mathcal{L}_{color}$ is a MSE loss on RGB color, $\mathcal{L}_{reg}$ is the regularization term and $\mathcal{L}_{sparsity}$ is the total variation (TV) loss that measures the difference between neighboring values in the matrix or vector factors.

To further improve the face rendering quality, we explore another way to render photo-realistic faces. We observe that although T-Face shows promising results, it does not perform well in facial reconstruction details. Inspired by InstantNGP [10] and NeuS [14], we use implicit surface rendering for face reconstruction based on Instant-NeuS implementation [4]. Before this, we also tried Instant-NeRF [4] (a combination of InstantNGP [10] and NeRF [9]) to complete face reconstruction, but gave up due to the difficulty of imposing constraints on neural radiance fields to remove artifacts. In the experiment, we use the binary cross entropy loss as the mask loss, as mentioned in NeuS [14]. To avoid floating artifacts, we also add sparsity loss:

$$\mathcal{L}_{sparsity}=\frac{1}{|S|}\sum_{y\in S}\exp(-\gamma d(y)),$$

(4)

where $y$ is the sampled point, $d(y)$ is the corresponding SDF values, $S$ represents the collection of sampling points. Here we set $\gamma=0.5$. Finally, the total loss is defined as:

$$\mathcal{L}_{I}=\mathcal{L}_{color}+\alpha\mathcal{L}_{reg}+\beta\mathcal{L}_{mask}+\gamma\mathcal{L}_{sparsity},$$

(5)

where $\mathcal{L}_{color}$ is a MSE loss on RGB color and $\mathcal{L}_{reg}$ is the Eikonal regularization [2].

To further improve the results, we use a simple and effective linear weighted ensemble method to obtain the final results. In practice, we use a set of linear weights $(0.1,0.6,0.3)$ corresponding to the baseline, T-Face, and I-Face to group the rendered images.

We implement our T-Face and I-Face in PyTorch. In T-Face, we follow the baseline configurations, using Adam optimizer [6] with $(\beta_{1},\beta_{2})=(0.9,0.99)$ and initial learning rate of $0.02$ for tensor factors and $0.001$ for the MLP decoder. We train our model T-Face for 50000 steps with a batch size of 4096 rays on a single NVIDIA RTX 3090 (40-60 minutes per scene). Following TensoRF [1], we start from an initial coarse grid with $128^{3}$ voxels and then upsample them at steps 2000, 3000, 4000, 5500, and 7000 to a fine grid with $300^{3}$ voxels. In I-Face, we use AdamW optimizer [8] with $(\beta_{1},\beta_{2})=(0.9,0.99)$ and the learning rate $0.01$. We train our model I-Face for 20000 steps with a dynamic batch size (256-8192) on a single NVIDIA RTX 3090 (10-20 minutes per scene).

To demonstrate the effectiveness of the proposed improvement, we ablate our methods in the first three scenes of the ILSH dataset.