Inverse Rendering of Translucent Objects using Physical and Neural Renderers

There has been a lot of work on estimating depth [48, 18, 31], BRDF [1, 2, 35, 33, 15, 42], and illumination [19, 20, 25] separately.

With the recent rise of deep learning, more and more research is focusing on simultaneous parameter estimation. Li et al. [37] propose the first single-view method to estimate the shape, SVBRDF, and illumination of a single object. They use a cascaded network to tackle the ambiguity problem and an in-network rendering layer to create global illumination. A few research groups demonstrate that a similar idea can be applied in more complex situations, such as indoor scene inverse rendering by using a Residual Appearance Renderer [50] or spatially-varying lighting [34, 54, 74]. Li et al. [36] extend the inverse rendering of indoor scenes by considering more complex materials like metal or mirror. Boss et al. [8] tackle the ambiguity problem under a saturated highlight by using a two-shot setup. Sang et al. [49] implement shape, SVBRDF estimation, and relighting by using a physically-based renderer and a neural renderer. Deschaintre et al. [16] introduce polarization into shape and SVBRDF estimation. Wu et al. [58] train the model in an unsupervised way with the help of the rotational symmetry of specific objects. Lichy et al. [39] enables a high resolution shape and material reconstruction by a recursive neural architecture. Our model can be interpreted as an extended version of these methods that also take SSS into account.

Subsurface scattering is essential for rendering translucent objects such as human skin, minerals, smoke, etc. The reconstruction and editing of these objects heavily rely on scattering parameter estimation. However, inverse scattering is a challenging task due to the multiple bounces and multiple paths of light inside the object. Some works [30, 21, 28, 71, 22] combine analysis by synthesis and Monte Carlo volume rendering techniques to tackle this problem. However, they suffer from some problems like local minimal and long optimization time.

Recently, Che et al. [11] used a deep neural network to predict the homogeneous scattering parameters as a warm start for analysis by synthesis. However, they do not estimate parameters such as geometry, illumination, etc. This means that their model can not handle applications like scene reconstruction, material editing. In addition, they only consider pure SSS. However, most translucent objects in our real-world consist of both surface reflectance and SSS. Our model estimates geometry, illumination, surface reflectance, and SSS simultaneously.

Differentiable renderers are broadly used in inverse rendering field for reconstructing human faces [51, 55], indoor scenes [50, 34], buildings [43, 65], and single objects [37, 8, 49]. However, most of them use the physically-based renderer that only considers direct illumination. Such a renderer cannot produce high-quality images as they do not allow for effects like soft shadows, interreflection, and SSS. Recently, there have been some general-purpose differentiable renderers [46, 32, 67, 66] that consider indirect illumination. However, such a Monte Carlo path tracing graph takes a lot of memory and computing resources, especially when a large sample per pixel (spp) is used. Instead, some works [37, 44, 50, 49, 64, 68] splice a neural network behind direct illumination only renderer to enable global illumination or scene editing. Inspired by them, we also designed a renderer-neural network architecture for the SSS task.

Scene editing is a broad research topic, and recent research advances have been dominated by deep learning. A bunch of works has been reported for relighting [47, 40, 65, 51, 60, 72], material editing [41, 52] and object manipulation [62, 57, 29, 70]. We take the first step on the scattering parameter editing of translucent objects by observing only two images.

Scene representation For the geometry part, we use a depth map $D$ to roughly represent the shape of the object and a normal map $N$ to provide local details. For the surface part, we use a microfacet BSDF model proposed by [53]. A roughness map $R$ is used to represent BSDF parameters. The homogeneous SSS is modeled by three terms: an extinction coefficient $\sigma_{t}$ which controls the optical density, a volumetric albedo $\alpha$ which determines the probability of whether photons are scattered or absorbed during a volume event, and a Henyey-Greenstein phase function [24] parameter $g$ which defines whether the scattering is forward ($g>0$), backward ($g<0$) or isotropic ($g=0$). We estimate the spherical harmonics $sh$ as a side-prediction to assist the other part of the model. A flashlight intensity $i$ is also predicted, taking into account that the intensity of the flashlight varies from device to device.

Model design Inspired by Aittala et al. [2] and Boss et al. [8], we also use a flash and no-flash image setup. Our motivation for this design is that the visibility of translucent objects will vary at different light intensities. For example, if a bright light is placed on the back of a finger, we can clearly see the color of the blood. Such a property facilitates scattering parameter estimation and enables better disentanglement of surface reflectance and SSS.

So, given a translucent object of unknown shape, material, and under unknown illumination, our target is to estimate these parameters simultaneously. In addition, we also enable material editing by manipulating the estimated parameters. Figure 2 shows an overview of our model. The inputs of our model are three images: a flash image $I_{f}\in\mathbb{R}^{3\times 256\times 256}$, a no-flash image $I\in\mathbb{R}^{3\times 256\times 256}$, and a binary mask $M\in\mathbb{R}^{256\times 256}$. For each scene, the estimated parameters are:

• •

  A depth map $D\in\mathbb{R}^{256\times 256}$ and a normal map $N\in\mathbb{R}^{3\times 256\times 256}$ to represent the shape.

• •

  A roughness map $R\in\mathbb{R}^{256\times 256}$ used in the microfacet BSDF model.

• •

  The spherical harmonics $sh\in\mathbb{R}^{3\times 9}$ and a flashlight intensity $i\in\mathbb{R}^{1}$ to represent illumination.

• •

  The extinction coefficient $\sigma_{t}\in\mathbb{R}^{3}$, volumetric albedo $\alpha\in\mathbb{R}^{3}$, and Henyey-Greenstein phase function parameter $g\in\mathbb{R}^{1}$.

Taking advantage of recent developments in deep learning, we use a deep convolutional neural network as our estimator. We use the one encoder and multiple heads structure. Our motivation for this design is that estimating shape, material, and illumination can be thought of as multi-task learning. Ideally, each task can assist the other to learn a robust encoder. The encoder extract features from the input images and each head estimate parameters accordingly. So, given a flash image $I_{f}$, no-flash image $I$, and a binary mask $M$, the estimated physical parameters are:

$$\tilde{D},\tilde{N},\tilde{R},\tilde{sh},\tilde{i},\tilde{\sigma_{t}},\tilde{\alpha},\tilde{g}=f_{e}(I,I_{f},M),$$

(1)

where $f_{e}$ denotes the estimator. $\tilde{D},\tilde{N},\tilde{R},\tilde{sh},\tilde{i},\tilde{\sigma_{t}},\tilde{\alpha},\tilde{g}$ are the estimated depth, normal, roughness, spherical harmonics coefficient, flashlight intensity, extinction coefficient, volumetric albedo, and Henyey-Greenstein phase function parameter.

The use of reconstruction loss to supervise network training is a widespread technique in deep learning. Specifically, reconstruction in inverse rendering means re-rendering the scene with the estimated parameters. In addition, the rendering module must be differentiable to pass the gradient of reconstruction loss to the estimator.

One potential option is to use a general-purpose differentiable renderer such as Mitsuba[46]. However, we have two problems. First, current general-purpose differentiable renderers have memory and training speed problems for consumer-grade GPUs, especially when using a high sample rate. Because the computational cost of path tracing is much larger than that of standard neural works. For example, differentially rendering a single translucent object image with $256\times 256$ resolution and 64 samples per pixel (spp) requires more than 5 seconds and 20 GB memory on an RTX3090 GPU. Moreover, the rendering time and memory will increase linearly when spp increase. Second, re-rendering the object with SSS requires the entire 3D shape (e.g., a 3D mesh). However, it is not easy to estimate the complete 3D information at a single viewpoint. That is the reason we only estimate a normal and a depth map as our geometry representation. This representation is sufficient for rendering surface reflectance but not for SSS. Another option [58, 8] is to use a differentiable renderer that only considers direct illumination. This sacrifices some reconstruction quality but dramatically improves efficiency. However, such a method cannot be applied to a translucent object because SSS depends on multiple bounces of light inside the object.

With the assist of recent advances in image-to-image translation [27, 73], many works have demonstrated the successful use of neural networks for adding indirect illumination [37, 44], photorealistic effect [43, 50], or relighting[49, 64]. Inspired by them, we propose a two-step rendering pipeline to mimic the rendering performance of SSS of a general-purpose differentiable renderer. The first step is a physically-based rendering module that only considers direct illumination:

$$\tilde{I_{d}}=f_{d}(\tilde{D},\tilde{N},\tilde{R},\tilde{i},\tilde{sh}),$$

(2)

where $f_{d}$ is a physically-based renderer that follow the implementation of [49], and it has no trainable parameters. $\tilde{I_{d}}$ is the re-rendered image that only consider the surface reflectance. The second step is a neural renderer $f_{n}$ to create SSS effect:

$$\tilde{I_{f}}=f_{n}(\tilde{I_{d}},\tilde{sh},\tilde{i},\tilde{\sigma_{t}},\tilde{\alpha},\tilde{g},B),$$

(3)

where $\tilde{I_{f}}$ is the re-rendered flash image. $f_{n}$ is the proposed neural renderer, and it consists of 3 parts (See Figure 2 for reference), a Surface encoder, a Scattering encoder, and a decoder. The Surface encoder maps the estimated surface reflectance image $\tilde{I_{d}}$ into a feature map. In addition to the surface reflectance image, we also input the background image $B$ (masked-out version of $I_{f}$) to the Surface encoder to provide high frequency illumination information. The Scattering encoder consists of a few upsampling layers. It maps $\tilde{sh},\tilde{i},\tilde{\sigma_{t}},\tilde{\alpha},\tilde{g}$ to a feature map. The decoder consists of some Resnet blocks [23] and upsampling layers. The task of our neural renderer can be considered as a conditional image-to-image translation, where the condition is the SSS parameters.

The advantage of the two-renderer design is that it is naturally differentiable. At the same time, the training cost is acceptable. In addition, the physically-based renderer can provide physical hints to the neural renderer. Because we separate the reconstruction of surface reflectance and SSS explicitly, the problem of ambiguity can be alleviated.

In this subsection, we present an augmented loss to address the hidden information problem by using multiple altered images to supervise the proposed neural renderer. In Section 3.3, we propose a two-renderer structure to compute the reconstruction images to improve the parameter estimation. However, a well-known problem of reconstruction loss in deep learning is that neural networks learn to “hide” information within them [13]. This may cause our neural renderer to ignore the estimated SSS parameters and only reconstruct the input image by the hidden information. If so, the reconstruction loss cannot give correct gradients and thus fail to guide the training of the estimator.

In order to solve this problem, we enhance the supervision of the proposed neural renderer by an augmented loss. Specifically, after the parameters are estimated, we edit the estimated SSS parameters and let the Neural renderer reconstruct image based on the edited SSS parameters. We also render $K$ altered images $I_{alter}^{k}$ as their ground truth labels to train the neural renderer. The altered images have the same parameters as the original flash image, except the SSS parameters. Specifically, $I_{f}$ and $I_{alter}^{k}$ share the same shape, surface reflectance, and illumination, but different extinction coefficient, phase function, and volumetric albedo. The edited SSS parameters are randomly sampled from the same distribution as the original ones:

$$\tilde{I}_{alter}^{k}=f_{n}(\tilde{I_{d}},\tilde{sh},\tilde{i},\tilde{\sigma_{t}}+\delta_{\sigma_{t}},\tilde{\alpha}+\delta_{\alpha},\tilde{g}+\delta_{g},B),$$

(4)

where $\delta_{\sigma_{t}},\delta_{\alpha},\delta_{g}$ are the differences between target SSS parameters (subsurface scattering parameters of $I_{alter}^{k}$) and the estimated ones, $\tilde{I}_{alter}^{k}$ is the reconstructed altered images. The benefits of this design are as follows. First, the input image is not the same as the image to be reconstructed, which makes it meaningless for the neural network to “hide” the information of the original input image. Second, the variety of input parameters and output images makes the neural renderer more sensitive to the changes in SSS parameters. These two points make the neural renderer a good guide for the estimator, which allows for a more accurate estimation of the SSS parameters. Considering the time and computing resources it takes to render the training images, in practice, we set $K=3$.

The proposed model is fully supervised and trained end to end, and we compute the loss between the estimated parameters their ground truths:

	$\displaystyle L$	$\displaystyle=$	$\displaystyle L_{D}+L_{N}+L_{R}+L_{sh}+L_{i}$
		$\displaystyle+$	$\displaystyle L_{\sigma_{t}}+L_{\alpha}+L_{g}+L_{I_{f}}+\sum_{K}{L_{alter}^{k}},$

where $L_{D},L_{N},L_{R},L_{I_{f}},L_{alter}^{k}$ stand for the $L1$ loss between the estimated depth, normal, roughness, flash image, altered images and their ground truth ones. $L_{sh},L_{i},L_{\sigma_{t}},L_{\alpha},L_{g}$ stands for the $L2$ loss between the estimated spherical harmonics, flashlight intensity, extinction coefficient, volumetric albedo, Henyey-Greenstein phase function parameter and their ground truth ones.

In this work, we propose the first large-scale synthetic translucent dataset that contains both surface reflectance and SSS. We collected the 3D objects from ShapeNet [10], human-created roughness map and auxiliary normal map from several public resources, and environment map from the Laval Indoor HDR dataset [19].

For each scene, we used Mitsuba [46] to render five images: flash image, no-flash image, and three altered images. Each scene contains a randomly selected object, auxiliary normal map, roughness map, environment map. We also randomly sample the SSS parameters and flash light intensity. Similar to some SfP (Shape from Polarization) works [3], we assume a constant IoR (Index of Refraction) to reduce the ambiguity problem. A full introduction to our dataset can be found in the supplementary documentation.

To the best of our knowledge, we are the first to tackle the inverse rendering problem of translucent objects containing both surface reflectance and SSS. It is not easy to compare our method with pure surface reflectance methods [37, 8]. Although similar to our method, these methods also predict parameters like surface normal, depth, and roughness. The key problem is that the coloration of pure surface reflectance models is affected by “diffuse albedo”, which is a parameter of BRDF. However, in our proposed scene representation, the surface reflection and refraction are modeled by BSDF, and the SSS is modeled by the Radiative Transport Equation (RTE), which means that the coloration is affected by the volumetric albedo, extinction coefficient, and phase function. In conclusion, training pure surface reflectance methods on our dataset is impossible. Thus, we choose a pure SSS method proposed by Che et al. [11] to compare with our model. Their method requires an edge map as the additional input of the neural network. We follow their paper to generate the edge maps for our dataset. We train their model on our dataset with the same hyperparameters as our method. We report the Mean Absolute Error (MAE) in Table 1, and visual comparison in Figure 4. It is difficult to compare the parameter estimation accuracy without GT parameter references, so we only show the synthetic data results. It is observed that their method fails to estimate reasonable SSS parameters due to the highly ambiguous scene representation.

We conduct ablation studies to evaluate the effects of each component of the proposed model. Specifically, we start with a Baseline model, which uses the same estimator as our Full model, but only reconstructs the input scene with a neural network. So, the baseline model is essentially an autoencoder. Then, we divide the reconstruction into two steps: a physically-based renderer that reconstructs the surface reflectance and a neural renderer that create the multi-bounce illumination as well as the SSS effect. We call this model “2R”. After that, we introduce the augmented loss to the “2R” model and denote it as “2R-AUG”. Finally, we implement the Full model by adding the two-shot setting to “2R-AUG”. We report the MAE results on the synthetic data of all experiments in Table 1. It is observed that for most of the metrics, “2R” model outperforms the Baseline model. Especially for the illumination part, the accuracy improved a lot. This confirms our previous point that is explicitly separating the surface reflectance, and SSS reduces ambiguity. Although the performance of “2R” and “2R-AUG” are similar for geometry, illumination, and surface reflectance parts, the result of SSS parameters which are enhanced by the augmented loss, is improved. Comparison between the “2R-AUG” and Full model proves that the proposed two two-shot problem settings can further reduce the ambiguity problem. Figure 5 demonstrate some visual comparisons. It is observed that the SSS images of the Full model are most similar to the ground truth ones, and the other models are unstable. In addition, because of the two-shot setting, the full model’s estimation of environment illumination is also more accurate. To show the flexibility of the proposed model, we also test on several common real-world translucent objects, and illustrate the results in Figure 1. The results show that our model can estimate reasonable SSS parameters.

Given a translucent object with an unknown shape, illumination, and material, we show that the proposed inverse rendering framework can edit translucent objects based on the given SSS parameters. Figure 6 shows the editing results.