\ConferencePaper\biberVersion\BibtexOrBiblatex\electronicVersion\PrintedOrElectronic

\teaser

Our model takes RGB albedo captures as input, and reconstructs a set of biophysical properties of human skin. On the left, an example of a captured albedo vs the reconstruction of our model (top) using the estimated skin properties, of which melanin and hemoglobin are shown in the bottom. On the right, renderings using albedos modified through edits of some of the estimated parameters, under two different lighting setups (top and bottom rows).

Estimation of Spectral Biophysical Skin Properties
from Captured RGB Albedo

C. Aliaga¹ C. Hery¹ M.Xia^1,2 Meta Reality Labs Research¹ Cornell University²

Abstract

We present a new method to reconstruct and manipulate the spectral properties of human skin from simple RGB albedo captures. To this end, we leverage Monte Carlo light simulation over an accurate biophysical human skin layering model parameterized by its most important components, thereby covering a plausible range of colors. The practical complexity of the model allows us to learn the inverse mapping from any albedo to its most probable associated skin properties. Our technique can faithfully reproduce any skin type, being expressive enough to automatically handle more challenging areas like the lips or imperfections in the face. Thanks to the smoothness of the skin parameters maps recovered, the albedo can be robustly edited through meaningful biophysical properties.

{CCSXML}

<ccs2012> <concept> <concept_id>10010147.10010371.10010372.10010376</concept_id> <concept_desc>Computing methodologies Reflectance modeling</concept_desc> <concept_significance>500</concept_significance> </concept> <concept> <concept_id>10010147.10010178.10010224.10010245.10010254</concept_id> <concept_desc>Computing methodologies Reconstruction</concept_desc> <concept_significance>500</concept_significance> </concept> </ccs2012>

\ccsdesc

[500]Computing methodologies Reflectance modeling \ccsdesc[500]Computing methodologies Reconstruction

\printccsdesc

^†^†volume: 41^†^†issue: 2

1 Introduction

Creating convincing photo realistic replicas of human faces has been a long-standing goal of computer graphics, usually relying on skilled artists to fine tune a large number of spatially varying shading parameters. As a crucial part of virtual humans, significant effort has been put on accurately modeling skin. Skin is not only challenging due to its complex interaction with light, but also because the human visual system has specifically evolved for strong facial perception. A number of models have targeted the effect of the biophysical composition of the skin on its final appearance, often centered in its diffuse reflectance resulting from the complex interaction of light inside the dermal tissues [DJ06, KB04, BK10, CBKM15, IAJG15]. More recently, the interest in estimating skin properties from captures has grown, mostly focusing on inverting the biophysical properties of skin from its diffuse reflectance, leveraging biophysical constraints [AS17, GAS*19, ZDP*19, GGD*20, GGR*21].

We present a new model that, given a biophysically-inspired space of spectral skin albedos, makes use of a neural network to accurately recover, out of simple RGB albedos, the main components and structure of the skin. For this, our model builds upon a biophysical description of the main properties in human skin, which we translate into albedo by means of Monte Carlo simulations.

Thus, our high-level contributions can be summarized as follows:

•

An expressive space of spectral skin albedo constrained by physical properties of real skin, in agreement with measurements reported in tissue optics and medical research. Such space is created through a biophysically-inspired model of human skin of practical complexity.
•

A learned inverse mapping from skin albedo to its associated spectral biophysical skin properties, that enables the recovery of smooth, high resolution spatially varying maps of skin properties from RGB captured albedos.
•

Altogether, a framework to faithfully reproduce captured albedos of a wide range of skin types with minimal error, also enabling editing capabilities in a robust way through the estimated biophysical properties.

The process is composed of two steps. First, we precompute an albedo space in the form of a high dimensional tensor of skin tones resulting from all possible combinations of skin parameters. The details of our biophysical skin model are described in Section 3. Then, we learn the inverse mapping from an albedo to its associated skin properties in Section 4. After discussion of implementation details in Section 5, we show how to identically reconstruct the albedo and the results of manipulating the skin parameters in Section 6, with comparisons to related work in Section 7.

2 Related Work

Accurately capturing and modeling the appearance of human skin has been a very active area of research for decades. Here we focus on appearance modeling and reconstruction, and refer to a number of extensive surveys for further details in skin and face appearance modeling [INN07, NMM*19], face appearance capture [KRP*15, WLL*09], or a broader collection of methods acquiring the optical properties of translucent materials [FJM*20].

Modeling. There has been a bunch or works focusing on biophysical skin components. Some of these isolated the distributions of melanin and hemoglobin in human skin [THM99, TOS*03]. After an early model first introduced multi-spectral skin color [AMD01], spectral techniques that explicitly drive the appearance through biophysical properties have been developed, spanning models of different complexity, ranging from two layers and the two most relevant components (melanin and haemoglobin) [DJ06], up to seven layers and accounting for a large number of parameters related to the chemical composition and structure of human skin [CBKM15, IAJG15, KB04, BK10]. Or even a model for dynamic facial color [JSB*10], built from in vivo measurements of melanin and hemoglobin concentrations.

Capturing. Since early efforts in computer graphics to acquire the skin BRDF [MWL*99], a series of approaches focused on human face acquisition, progressively improving from the original Light Stage device [DHT*00]. Relying on such Light Stage-type scans, a number of models reconstructed the reflectance functions directly from captures [MHP*07, GHP*08, GFT*11, GTB*13], or synthesized high-resolution facial surface microgeometries [GTB*13]. Some other subsequent publications expanded on the facial appearance modeling and capture, including dynamic diffuse albedo encoding blood flow, specular intensity or high resolution normals [GRB*18, GRB*20], even single-shot [RGB*20], with editing capabilities [WMP*06]. Another method [DWd*08] used multi spectral photographs to recover spatially varying biophysical components over a layered skin model. A number of works opted to rely on Kubelka-Munk (KM) theory [KM31] to recover the skin parameter maps. An example [AS17] of this kind of models is tested and discussed in Supplementary Section 2.2. In skin research, 3D maps of human skin properties on the full face with shadows can be reconstructed from 3D hyper spectral imaging [GAS*19] via a two-flux KM model. Based on this, neural networks were trained to recover maps of oxygen rate, blood volume fraction, melanin concentration, bilirubin concentration, and epidermal thickness [GGR*21]. Also relying on hyperspectral imaging, a neural network and a more complex 7 layer skin model were presented [ZDP*19] to extract 2D maps of blood volume fraction, blood oxygen saturation, melanin content and epidermal thickness. It demonstrated the importance of the thickness of the superficial bloodless layer. However, both approaches [GGR*21, ZDP*19] lack albedos reconstructed by the model and the resulting estimated parameters, making it difficult to assess its application in a vfx or virtual reality context. Closest to our solution, a recent paper [GGD*20] reconstructs spectral skin properties through a novel measurement system, benefiting from narrow-band LEDs, based on a simplified skin model [JSB*10, DJ06].

Differently to previous works, our model is capable of faithfully reproducing with minimal error any skin type by a set of biophysical properties proven to be relevant. It relies on Monte Carlo simulations to accurately model light-skin interactions, avoiding assumptions (e.g. epidermal blood [GGD*20]) and corrections needed for KM approaches [GGR*21, Sau42]. Also, our model does not require hyperspectral captures, since it operates directly on simple RGB albedos (although it can be extended to spectral captures), and recovers smooth parameter maps that allow for intuitive edits in such parameter space.

3 A Space of Biophysically-Based Skin Albedos

In this section we describe how we create our albedo space, the details of the skin model and the data employed, together with balance of practical complexity and expressiveness.

3.1 Skin Model: Structure and Optical Properties

3.1.1 Skin Structure

We decided to restrict our model to two layers, epidermis and dermis, since a similar assumption has proven to be adequate in the past [MM02] and fits our purpose. The epidermis is composed of two parts: the living epidermis and the stratum corneum. The latter is the outermost part, and exhibits properties (surface roughness and sebum production) that affect the specular reflectance of the skin. However, it has a minimal effect on the albedo of the skin, given its low absorption and relative small thickness (5 and 20 $\mu m$ depending on the body location [CSLM19]).

On the other hand, we model the dermis as a single layer of averaged properties of scattering and absorption of its two sub-layers, the reticular dermis and the papillary dermis. It is modeled as a semi infinite medium, omitting the sub-dermal fat layer. This was decided to keep the generality of the model, with a focus on faces: next to the dermis can be not only fat, but other internal tissues like cartilage or muscles, heavily depending on the anatomical location and composition of the subject. Also, we empirically found that including the dermal thickness has a minimal effect over the resulting diffuse reflectance.

3.1.2 Absorption

Following well known multi-layered tissue optics models [Jac13], we describe the optical properties of each layer by its spectral absorption ( $\mu_{a}$ ) and scattering ( $\mu_{s}$ ) coefficients. The absorption of each layer $\mu_{a_{i}}$ is the result of the additive contribution of each chromophore absorption $\mu_{a_{c}}$ present in each layer:

\mu_{a_{i}}=\sum_{c\in C_{i}}\mu_{a_{c}}=\sum_{c\in C_{i}}\frac{V_{c}p_{c}\varepsilon_{c}}{w_{c}}

(1)

where $C_{i}$ is the set of chromophores contained in the layer $i$ ; $V_{c}$ is the volume fraction of the substance containing the chromophore $c$ ; $p_{c}$ is the concentration in g/L; $\varepsilon_{c}$ is the molar extinction of the chromophore; and $w_{c}$ is its molar weight (Table 2 in Appendix A).

As a result, the total spectral absorption of epidermis is defined as $\mu_{a_{e}}=V_{m}\left(\varphi_{m}\mu_{a_{eu}}+\left(1-\varphi_{m}\right)\mu_{a_{ph}}\right)+(1-V_{m})\left(\mu_{a_{\beta-c}}+\mu_{a_{base}}\right)$ , and the spectral absorption of the dermis is defined as
$\mu_{a_{d}}=V_{b}\left(\varphi_{h}\mu_{a_{hb}}+(1-\varphi_{h})\mu_{a_{hbO2}}+\mu_{a_{bil}}+\mu_{a_{\beta-c}}\right)+(1-V_{b})\mu_{a_{base}}$ .

In the same spirit of biophysical approaches [KB04, IAJG15, CBKM15, DJ06], we include the effect of melanin, determined by the melanosomes volume fraction $V_{m}$ , containing the two types of melanin (eumelanin $\mu_{a_{eu}}$ and pheomelanin $\mu_{a_{ph}}$ ) in the epidermis, governed by the melanin type ratio $\varphi_{m}$ , which greatly varies through skin type. We also incorporate the haemoglobin present in blood $V_{b}$ in the dermis: oxygenated haemoglobin $\mu_{a_{hbO2}}$ , responsible for the saturated reddish tint, and deoxygenated haemoglobin $\mu_{a_{hb}}$ , responsible for a desaturated purple color; $\varphi_{h}$ being the haemoglobin type ratio. Other chromophores included by the model are the beta-carotene $\mu_{a_{\beta-c}}$ in both epidermis and dermis and bilirubin $\mu_{a_{bil}}$ in the dermis, contained in blood. Additionally, we also include a baseline of skin absorption $\mu_{a_{base}}=7.84x10^{8}~{}\lambda^{-3.255}$ [Sai92] already employed in previous models.

3.1.3 Scattering

We treat epidermis and dermis as homogeneous mediums, the later being semi infinite. Both layers have an index of refraction of 1.4 resulting from the weighted (by thickness) sum of the corresponding sub layers: stratum corneum (1.53), living epidermis (1.34), papillary dermis (1.39) and reticular dermis (1.395). The interfaces of the two layers, as well as other details of multiple scattering in the tissue, are discussed in Section 3.2.

The reduced scattering coefficient $\mu_{s}^{\prime}$ is modeled through a function of wavelength fitted by Jacques [Jac13], which is generic for a wide range of human tissues :

\mu_{s}^{\prime}(\lambda)=a\left(f_{Ray}\left(\frac{\lambda}{\lambda_{r}}\right)^{-4}+(1-f_{Ray})\left(\frac{\lambda}{\lambda_{r}}\right)^{-b_{Mie}}\right)

(2)

where the wavelength $\lambda$ is normalized by a reference wavelength $\lambda_{r}=500$ nm, the scaling factor $a=\mu_{s}^{\prime}(\lambda_{r})$ , $f_{Ray}$ is the relative contribution of Rayleigh scattering, and $b_{Mie}$ characterizes the wavelength dependence of the Mie scattering component. We rely on the coefficients reported in the optics literature [BGT11], leading to $a$ = 36.4, $f_{Ray}$ = 0.48, and $b_{Mie}$ = 0.22 for human skin. In addition, we include the spectral dependency of the anisotropy factor $g$ , which has been measured in the past [VJSS89], ranging from 0.73 at 380 nm to 0.84 at 780 nm. It is expressed as $g_{e}\sim g_{d}\sim 0.62+\lambda~{}0.29x10^{-3}$ .

Refer to caption — Figure 1: Melanin and haemoglobin have a highly non-linear relationship with the final albedo. In this example, we vary melanin and haemoglobin to show the effect on the spectral reflectance (top) and corresponding final albedos (bottom). *Left:* Melanin is cubicly sampled with Mn = (0.2, 0.7, 2.3, 5.4, 10.6, 18.2, 29.9, 43.1)%, while haemoglobin is fixed at 2%. *Right:* haemoglobin is quarticly sampled with Bn = (0.1, 0.2, 0.7, 2, 4.5, 9.6, 17.7, 30)%, while melanin is fixed at 0.8%. These spectral reflectance plots are qualitatively in agreement with previous models of computer graphics [KB04], optics [ZDP*19], color research [VGI94], computer vision [AMD01], and dermatology [AP81].

3.2 Layered Model for Albedo Generation

For computing the diffuse reflectance of a skin patch, we first attempted to use a Kubelka-Munk - based layering model [AS17]. However, it suffers from lack of expressiveness to robustly recover the parameters of a wide enough range of skin types. Details on such exploration can be found in the Supplemental, Section 2.2. Thus, we decided to shift to a brute force Monte Carlo random walk. At this stage, we focus on resolving the transports through the stacking of the different layers of the skin structure (3.1.1). We conduct the walk in 2D, assuming symmetry in the azimuthal planes of each layer. The interface between the epidermis and dermis only considers the change in scattering and absorption parameters, since the index of refraction is measured to be mostly the same in both layers [LWC12].

For each skin tone, we run spectral simulations for the wavelengths comprised in the visible range, between 380 to 780 nm. We empirically found that steps of 10 nm is enough to produce stable, noise free albedos. For each wavelength, a million photons are launched, simulating the 2D random walk over the two-layered, semi infinite medium, to produce diffuse albedos (by opposition to the rendering step, where we make use of such uv-mapped albedos to conduct true 3D random walks, as outlined in Section 6). We treat the skin as an homogeneous medium with exponential decay, since the more recent statistical theories attempting to account for spatial correlations in some materials [WVH17, BRM*18, dEon19] do not have corresponding measured data of particle distributions for tissue. We use the multiple scattering parameters listed previously, including the wavelength dependent anisotropy factor $g$ from the Henyey-Greenstein [HG41] phase function, expressed in its 2D form [dEo21]. Note that the simulation starts once the photon has crossed the outermost interface and scatters diffusely into the tissue, namely choosing the initial direction from a Lambertian distribution around the inverse surface normal, in agreement with what we later use at rendering time (Section 6).

We validated the biophysical model by plotting the spectral reflectance of different skin tones, showing that our results are in agreement with existing measurements and simulations found in the literature (see Figure 1).

3.3 Albedo Space Parametrization and Sampling

Parameter	Description	Epidermis	Dermis
$V_{m}$	Melanin Volume Fraction	[0.001, 1]	-
$V_{b}$	Blood Volume Fraction	-	[0.001, 1]
$t$	Thickness [ $\mu$ m]	[10, 250]	2100
$\varphi_{m}$	Ratio of melanin types	[0.001, 1]	-
$\varphi_{h}$	Ratio of haemoglobin types	-	[0.001, 1]

Table 1: Ranges for the skin properties of our 5D albedo space.

Our biophysically-based albedo space is created by varying the skin properties in the ranges listed in Table 1. Note that we consider epidermal thickness, which proved to be critical for parameter estimation [ZDP*19]. Accounting for this parameter over the face, along with varying values in melanin concentrations, helps us achieve local dark zones, such as moles, and generalizes over any skin type. We also allow melanin and haemoglobin to go beyond the usual values for human adults measured in the literature [MM02], in order to automatically handle outliers found in the face, such as the lips, which exhibit very thin epidermis and higher blood concentration, or other cases like underlying veins and capillary veins, or areas with abnormal melanin concentration like freckles or spots. This range expansion is also reasonable for the oxygenation level, since it can vary a lot depending on the physical state of the person, and for the melanin type ratio, where there is little agreement in the available measured data.

Thus, some parameters like bilirubin and $\beta$ -carotene concentrations, remain fixed to common values of human skin measurements found in literature (see Table 2 in Appendix A). The remaining 5D parameter space is sampled as follows: melanin and haemoglobin are respectively selected cubicly ( $\sqrt[3]{V_{m}}$ ) and quarticly ( $\sqrt[4]{V_{b}}$ ), to better adjust to their non linear effect on the albedo (see Figure 1); epidermal thickness, melanin type ratio and haemoglobin type ratio all are treated uniformly.

4 Recovering the Skin Properties

Our biophysical model defines the forward mapping from the skin parameters to the skin albedo. For the inverse process, we need to characterize the mapping from the skin albedo back to skin parameters. The non-bijective nature of such mapping, where many combinations of skin properties can lead to the same albedo, makes this task challenging.

Metamerism. It is important to note that our model operates in spectral space to compute the reflectance corresponding to each skin parameter combination, but then such reflectance is integrated to RGB to learn the mapping. This is a design choice: it simplifies the capture, and provides good enough recovered skin parameter maps to both a) reconstruct the original albedo and b) perform plausible edits. However, operating in a higher dimensional space could alleviate metamerism, which could help to disentangle more accurately the skin parameters, generalize the model to any illuminant, or be robust to different levels of exposure, among others. Out of scope for this paper, further exploration can be done inspired by the literature in Spectral Reconstruction from RGB, where significant efforts have been done, specially in data-driven deep learning methods (a recent survey covers the wide variety of existing techniques [ZSR*21]).

4.1 Look-up Tensor (LUT).

We pre-compute a wide tensor of skin tones by following the sampling strategy outlined in Section 3.3. To estimate the skin parameters given an input albedo, we search the LUT on each texel of the albedo to find the best skin parameter set that minimizes a reconstruction $L_{2}$ error. Then, we can manipulate them and query the new corresponding albedo from the LUT. Some slices of the 5D tensor are shown in Figure 2. This approach is able to reconstruct the skin albedo faithfully with an error close to zero. However, the inverted parameter maps are noisy and have many discontinuities, since the mapping from RGB to skin parameters is not smooth. In turn, editing operations over neighboring pixels in such extracted components can lead to unexpected abrupt changes in the reconstructed albedos (see Figure 3). We leave out of scope of this paper the assessment of different representations or data structures suitable for more efficient search strategies, that could dramatically improve the performance of this approach.

4.2 Neural Skin Parameter Estimation

To overcome the limitations of the LUT approach, we opted to train an encoder-decoder network to recover smooth skin parameter maps. Both the encoder and the decoder are MLPs of four fully connected layers, with two hidden layers of 70 neurons each. The encoder maps a 3D skin albedo to a 5D skin parameter vector, while the decoder does the inverse.

Dataset. Using our biophysical skin model, we generate a data set of 600k pairs of 5D skin parameter vector and the corresponding 3D albedo, split 80% and 20% for training and validation respectively. For validation, we sample the skin parameters according to the uniform distribution. For training, we rely on Quasi-Monte Carlo for a better coverage of the skin parameters, using a low-discrepancy sequence (Halton [Hal64]). Both are subsequently non linearly remapped following the schemes detailed in Section 3.3, and the corresponding albedos are computed using the biophysical model.

Training. We used the Adam optimizer [KB14], with an learning rate of $10^{-4}$ and a batch size of 4096 for an optimal number of 400 epochs to prevent over fitting. A grid search was performed to find the optimal hyper parameters listed, including the number of hidden neurons. The loss consists of three parts:

\mathcal{L}=\mathcal{L}_{param}+\mathcal{L}_{albedo}+\mathcal{L}_{cycle}

(3)

where $\mathcal{L}_{param}$ denotes the parameter loss (encoder), which computes the $L2$ difference between the predicted skin properties from the input albedo and its corresponding ground truth properties. $\mathcal{L}_{albedo}$ (decoder) computes the $L1$ differences between the predicted albedo and the ground truth albedo corresponding to the input skin properties. $\mathcal{L}_{cycle}$ evaluates the full encoder-decoder cycle, computing the $L1$ difference between the predicted albedos from the predicted skin properties, and the ground truth albedos. We chose $L1$ loss for robustness against outliers in albedo. Note that, although in theory, a given albedo can invert to different sets of skin parameters, we use the skin parameter set that generates this albedo from the Monte Carlo simulation; this input serves as weak supervision and works well in practice.

5 Implementation Details

Spectral Downsampling. The albedo space is generated through spectral computations, but input albedos are in RGB space. To downsample the multi-band spectral values into RGB, we use an existing integration approach [MY19] (note that we could rely on other concurrent works [JH19] to this end). Most color spaces will work here (we use sRGB), considering that diffuse albedos have rather limited gamuts and dynamic ranges. We perform a change of illuminant for direct comparisons to previous work in Section 7, which we further detail in the Supplementary, Section 1.

Using Albedo Maps in Rendering. The final 3D lit geometries of our virtual faces are rendered using our own skin materials inside Blender Cycles [Ble20]. Aside from the specular component of the skin, which we represent as a double lobe GGX [WMLT07], we follow, in the spirit of a state of the art technique in production [WVH17], a 3D random walk subsurface scattering solution that relies on a numerical albedo inversion around the mean free path and accounting for the anisotropy factor $g$ (see Supplemental Section 2.1). At this stage, we simplify the model to be single layered, dermis and epidermis combined, as a semi infinite medium. Obviously multi-layered models could be employed, for instance directly consuming the chromophores estimations, but these are out of scope of this paper. See Figure Estimation of Spectral Biophysical Skin Properties from Captured RGB Albedo, Figure 8 and the Supplemental material for examples of path traced renders under 3 different lighting environments.

6 Results

We perform estimations and manipulations of skin parameters over several skin types covering the Fitzpatrick scale [Fit88]. With the LUT approach, using the largest tensor (256k skintones) resulted in varying times from 2 to 5 hours for 2k by 2k images, or more than 7 hours for 4k by 4k images, using brute force multi threaded (12) search on the tensor in an Intel Xeon W-2135 at 3.70GHz. Instead, the learned mapping has the advantages of little memory consumption and efficient computation (less than 2 seconds on average for 2k by 2k images, on the same Intel Xeon CPU mentioned above). We refer the readers to Figure 3, Figure 6, Figure 7, Figure 8 and the Supplemental Material to see the neural results. In Figure 6, we observe that we trade a tiny bit of reconstruction correctness (imperceptible in final renders), in favor of spatial regularization and performance.

Editing the Skin Parameters We show how we can manipulate directly in this space of inferred skin properties, scaling some of them up or down in an intuitive and predictable manner. We run the neural decoder on these modified quantities to reconstruct biophysical albedos, and finally render them on 3D faces. For skin types ranging from I to V, we perform large edits in haemoglobin and melanin content, with details explained in Figure 8. Note the edits are naive in order to cover similar ranges for all skin types. Figure 9 shows edits over the rest of the parameters of the model. The level of blood oxygenation translates into paler or more saturated skin colors. The thinning of the epidermis, which typically occurs with aging, translates into a more translucent look, revealing the heterogeneities of the underlying layers (e.g. capillary and veins), while a thicker epidermis results in a more opaque and rough appearance. Last, we vary several components to simulate tanning and flushing. While it is hard to fully validate the correctness of the recovered skin properties, we find the parametrization adequate to produce plausible human skin albedos.

7 Comparisons with Previous Work

We conduct a series of comparisons with the related and recent work from [GGD*20]. Not having access to the training data, we decided to reproduce the authors’ results from the examples in the paper with our approach. Figure 4 is a comparison on more true albedos from [GGD*20], which were obtained via the Antera device (under D65 illuminant). In Figure 5 we process one of the D65’ faces. This was a bit more complex, because of the conversion of the color spaces involved in some of their captures (which we outline in Supplemental). The general observation is that our reconstruction error is much lower, and we tested the model across more skin types. See the Supplemental Section 1 for additional tests.

8 Discussion and Future Work

In this paper, we presented our framework to recover and robustly manipulate the key biophysical properties of human skin from a given RGB albedo. This is accomplished through the combination of an expressive biophysically-inspired skin model, together with an encoder-decoder structure that maps albedo to skin properties and back. We demonstrate how such learned mapping overcomes the limitations of a Look-Up Tensor approach, proving how the latter becomes impractical and leads to quantization artifacts and shifts in colors. On the other hand, the expressiveness and robustness of our method is assessed through estimating and manipulating biophysical parameters of a variety of skin shades covering the Fitzpatrick scale.

As future work, although we consider all the chromophores in our current model, we aim to widen the albedo space to handle uncommon skin shades resulting from rare chromophore concentrations, such as the excess of bilirubin, beta-carotene, or weird pathologies related to methemoglobin. Incorporating the effect of hair roots on the light transport inside the skin would be interesting to better handle short beards or shaved skulls.

Another direction to further improve our results would be to train the neural network to learn the mapping between skin parameters and spectral albedos directly, instead of RGB. Finally, we want to look at adding the ability to automatically remove or compensate the baked lighting that exists in some captures, through analyzing the parts of the image that show incoherence in the estimated skin parameters, with the goal of reconstructing the skin properties under uncontrolled lighting scenarios like normal pictures.

Acknowledgments

We thank Olivier Maury and Akira Orikasa for general lookdev support, Christoph Lassner for conversations about the neural model, and Tony Tung, Ronald Mallet and Yaser Sheikh for their face captures. Scans and actors for Subjects A and F were provided by Eisko [Eis21].

References

[AMD01] E Angelopoulo, Rana Molana and Kostas Daniilidis “Multispectral skin color modeling” In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 2, 2001, pp. II–II IEEE
[AP81] R Rox Anderson and John A Parrish “The optics of human skin” In Journal of investigative dermatology 77.1 Elsevier, 1981, pp. 13–19
[AS17] Sarah Alotaibi and William AP Smith “A biophysical 3D morphable model of face appearance” In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2017, pp. 824–832
[BGT11] Alexey N Bashkatov, Elina A Genina and Valery V Tuchin “Optical properties of skin, subcutaneous, and muscle tissues: a review” In Journal of Innovative Optical Health Sciences 4.01 World Scientific, 2011, pp. 9–38
[BK10] Gladimir VG Baranoski and Aravind Krishnaswamy “Light and skin interactions: simulations for computer graphics applications” Morgan Kaufmann, 2010
[Ble20] Blender Online Community “Blender: a 3D modelling and rendering package”, 2020 Blender Foundation URL: http://www.blender.org
[BRM*18] Benedikt Bitterli, Srinath Ravichandran, Thomas Müller, Magnus Wrenninge, Jan Novák, Steve Marschner and Wojciech Jarosz “A Radiative Transfer Framework for Non-Exponential Media” In ACM Trans. Graph. 37.6 New York, NY, USA: Association for Computing Machinery, 2018
[CBKM15] Tenn F Chen, Gladimir VG Baranoski, Bradley W Kimmel and Erik Miranda “Hyperspectral modeling of skin appearance” In ACM Transactions on Graphics (TOG) 34.3 ACM New York, NY, USA, 2015, pp. 1–14
[CSLM19] Carolin Czekalla, Karl Heinz Schönborn, Jürgen Lademann and Martina C Meinke “Noninvasive determination of epidermal and stratum corneum thickness in vivo using two-photon microscopy and optical coherence tomography: Impact of body area, age, and gender” In Skin pharmacology and physiology 32.3 Karger Publishers, 2019, pp. 142–150
[dEo21] Eugene d’Eon “A Hitchhiker’s Guide to Multiple Scattering”, 2021
[dEon19] Eugene Eon “A Reciprocal Formulation of Nonexponential Radiative Transfer. 2: Monte Carlo Estimation and Diffusion Approximation” In Journal of Computational and Theoretical Transport 48.6 Informa UK Limited, 2019, pp. 201–262
[DHT*00] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin and Mark Sagar “Acquiring the reflectance field of a human face” In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000, pp. 145–156
[DJ06] Craig Donner and Henrik Wann Jensen “A Spectral BSSRDF for Shading Human Skin.” In Rendering techniques 2006 Citeseer, 2006, pp. 409–418
[DWd*08] Craig Donner, Tim Weyrich, Eugene d’Eon, Ravi Ramamoorthi and Szymon Rusinkiewicz “A layered, heterogeneous reflectance model for acquiring and rendering human skin” In ACM Transactions on Graphics (TOG) 27.5 ACM New York, NY, USA, 2008, pp. 1–12
[Eis21] Eisko “Photoreal Digital Doubles”, 2021 URL: https://www.eisko.com/
[Fit88] Thomas B Fitzpatrick “The validity and practicality of sun-reactive skin types I through VI” In Archives of dermatology 124.6 American Medical Association, 1988, pp. 869–871
[FJM*20] Jeppe Revall Frisvad, Søren Alkærsig Jensen, Jonas Skovlund Madsen, António Correia, Li Yang, Søren KS Gregersen, Youri Meuret and Poul-Erik Hansen “Survey of Models for Acquiring the Optical Properties of Translucent Materials” In STAR 39.2, 2020
[GAS*19] Lou Gevaux, Cyprien Adnet, Pierre Séroul, Raphael Clerc, Alain Trémeau, Jean Luc Perrot and Mathieu Hébert “Three-dimensional maps of human skin properties on full face with shadows using 3-D hyperspectral imaging” In Journal of biomedical optics 24.6 International Society for OpticsPhotonics, 2019, pp. 066002
[GFT*11] Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu and Paul Debevec “Multiview face capture using polarized spherical gradient illumination” In Proceedings of the 2011 SIGGRAPH Asia Conference, 2011, pp. 1–10
[GGD*20] Yuliya Gitlina, Giuseppe Claudio Guarnera, Daljit Singh Dhillon, Jan Hansen, Alexandros Lattas, Dinesh Pai and Abhijeet Ghosh “Practical Measurement and Reconstruction of Spectral Skin Reflectance” In Computer Graphics Forum 39.4, 2020, pp. 75–89 Wiley Online Library
[GGR*21] Lou Gevaux, Jordan Gierschendorf, Juliette Rengot, Marie Cherel, Pierre Séroul, Alex Nkengne, Julie Robic, Alain Trémeau and Mathieu Hébert “Real-time skin chromophore estimation from hyperspectral images using a neural network” In Skin Research and Technology 27.2 Wiley Online Library, 2021, pp. 163–177
[GHP*08] Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen and Paul Debevec “Practical modeling and acquisition of layered facial reflectance” In ACM SIGGRAPH Asia 2008 papers, 2008, pp. 1–10
[GRB*18] Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh and Thabo Beeler “Practical Dynamic Facial Appearance Modeling and Acquisition”, 2018
[GRB*20] Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh and Thabo Beeler “Practical dynamic facial appearance modeling and acquisition” Association for Computing Machinery, 2020
[GTB*13] Paul Graham, Borom Tunwattanapong, Jay Busch, Xueming Yu, Andrew Jones, Paul Debevec and Abhijeet Ghosh “Measurement-based synthesis of facial microgeometry” In Computer Graphics Forum 32.2pt3, 2013, pp. 335–344 Wiley Online Library
[Hal64] John H Halton “Algorithm 247: Radical-inverse quasi-random point sequence” In Communications of the ACM 7.12 ACM New York, NY, USA, 1964, pp. 701–702
[HG41] Louis G Henyey and Jesse L Greenstein “Diffuse radiation in the galaxy” In The Astrophysical Journal 93, 1941, pp. 70–83
[IAJG15] Jose A. Iglesias-Guitian, Carlos Aliaga, Adrian Jarabo and Diego Gutierrez “A Biophysically-Based Model of the Optical Properties of Skin Aging” In Computer Graphics Forum (EUROGRAPHICS 2015) 34.2, 2015
[INN07] Takanori Igarashi, Ko Nishino and Shree K Nayar “The appearance of human skin: A survey” Now Publishers Inc, 2007
[Jac13] Steven L Jacques “Optical properties of biological tissues: a review” In Physics in Medicine & Biology 58.11 IOP Publishing, 2013, pp. R37
[JH19] Wenzel Jakob and Johannes Hanika “A Low-Dimensional Function Space for Efficient Spectral Upsampling” In Computer Graphics Forum (Proceedings of Eurographics) 38.2, 2019
[JM91] Steven L Jacques and Daniel J McAuliffe “The melanosome: threshold temperature for explosive vaporization and internal absorption coefficient during pulsed laser irradiation” In Photochemistry and photobiology 53.6 Wiley Online Library, 1991, pp. 769–775
[JSB*10] Jorge Jimenez, Timothy Scully, Nuno Barbosa, Craig Donner, Xenxo Alvarez, Teresa Vieira, Paul Matts, Verónica Orvalho, Diego Gutierrez and Tim Weyrich “A practical appearance model for dynamic facial color” In ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 29.6, 2010
[KB04] Aravind Krishnaswamy and Gladimir VG Baranoski “A biophysically-based spectral model of light interaction with human skin” In Computer Graphics Forum 23.3, 2004, pp. 331–340 Wiley Online Library
[KB14] Diederik P Kingma and Jimmy Ba “Adam: A method for stochastic optimization” In arXiv preprint arXiv:1412.6980, 2014
[KM31] Paul Kubelka and Franz Munk “An article on optics of paint layers” In Z. Tech. Phys 12.593-601, 1931, pp. 259–274
[KRP*15] Oliver Klehm, Fabrice Rousselle, Marios Papas, Derek Bradley, Christophe Hery, Bernd Bickel, Wojciech Jarosz and Thabo Beeler “Recent Advances in Facial Appearance Capture” In Computer Graphics Forum (Proceedings of Eurographics - State of the Art Reports) 34.2, 2015, pp. 709–733
[LWC12] Tom Lister, Philip A. Wright and Paul H. Chappell “Optical properties of human skin” In Journal of Biomedical Optics 17.9 SPIE, 2012, pp. 1–15
[MHP*07] Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss and Paul Debevec “Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination”, 2007
[MM02] Igor V Meglinski and Stephen J Matcher “Quantitative assessment of skin layers absorption and skin reflectance spectra simulation in the visible and near-infrared spectral regions” In Physiological measurement 23.4 Iop Publishing, 2002, pp. 741
[MWL*99] Stephen R Marschner, Stephen H Westin, Eric PF Lafortune, Kenneth E Torrance and Donald P Greenberg “Image-based BRDF measurement including human skin” In Rendering Techniques’ 99 Springer, 1999, pp. 131–144
[MY19] Ian Mallett and Cem Yuksel “Spectral Primary Decomposition for Rendering with RGB Reflectance” In Eurographics Symposium on Rendering (EGSR 2019) Strasbourg, France: The Eurographics Association, 2019
[NMM*19] Augusto LP Nunes, Anderson Maciel, Gary W Meyer, Nigel W John, Gladimir VG Baranoski and Marcelo Walter “Appearance modelling of living human tissues” In Computer Graphics Forum 38.6, 2019, pp. 43–65 Wiley Online Library
[PJ] Scott Prahl and Steven L Jacques “Tabulated Molar Extinction Coefficient for Hemoglobin in Water”, https://omlc.org/spectra/hemoglobin/summary.html
[RGB*20] Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh and Thabo Beeler “Single-shot high-quality facial geometry and skin appearance capture” In ACM Transactions on Graphics (TOG) 39.4 ACM New York, NY, USA, 2020, pp. 81–1
[Sai92] Iyad Salam Saidi “Transcutaneous optical measurement of hyperbilirubinemia in neonates”, 1992
[Sau42] JL Saunderson “Calculation of the color of pigmented plastics” In JOSA 32.12 Optical Society of America, 1942, pp. 727–736
[SS06] Tadeusz Sarna and Harold A Swartz “The physical properties of melanins” In The pigmentary system: physiology and pathophysiology Wiley Online Library, 2006, pp. 311–341
[THM99] Norimichi Tsumura, Hideaki Haneishi and Yoichi Miyake “Independent-component analysis of skin color image” In JOSA A 16.9 Optical Society of America, 1999, pp. 2169–2176
[TOS*03] Norimichi Tsumura, Nobutoshi Ojima, Kayoko Sato, Mitsuhiro Shiraishi, Hideto Shimizu, Hirohide Nabeshima, Syuuichi Akazaki, Kimihiko Hori and Yoichi Miyake “Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin” In ACM SIGGRAPH 2003 Papers, 2003, pp. 770–779
[VGI94] Michael J. Vrhel, Ron Gershon and Lawrence S. Iwan “Measurement and Analysis of Object Reflectance Spectra” In Color Research & Application 19.1, 1994, pp. 4–9
[VJSS89] MJC Van Gemert, Steven L Jacques, HJCM Sterenborg and WM Star “Skin optics” In IEEE Transactions on biomedical engineering 36.12 IEEE, 1989, pp. 1146–1154
[WLL*09] Tim Weyrich, Jason Lawrence, Hendrik PA Lensch, Szymon Rusinkiewicz and Todd Zickler “Principles of appearance acquisition and representation” Now Publishers Inc, 2009
[WMLT07] Bruce Walter, Stephen R. Marschner, Hongsong Li and Kenneth E. Torrance “Microfacet Models for Refraction through Rough Surfaces” In Proceedings of the 18th Eurographics Conference on Rendering Techniques, EGSR’07 Grenoble, France: Eurographics Association, 2007, pp. 195–206
[WMP*06] Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen and Markus Gross “Analysis of Human Faces Using a Measurement-Based Skin Reflectance Model” In ACM Trans. Graph. 25.3 New York, NY, USA: Association for Computing Machinery, 2006
[WVH17] Magnus Wrenninge, Ryusuke Villemin and Christophe Hery “Path traced subsurface scattering using anisotropic phase functions and non-exponential free flights”, 2017
[ZDP*19] Evgeny Zherebtsov, Viktor Dremin, Alexey Popov, Alexander Doronin, Daria Kurakina, Mikhail Kirillin, Igor Meglinski and Alexander Bykov “Hyperspectral imaging of human skin aided by artificial neural networks” In Biomedical optics express 10.7 Optical Society of America, 2019, pp. 3545–3559
[ZSR*21] Jingang Zhang, Runmu Su, Wenqi Ren, Qiang Fu and Yunfeng Nie “Learnable Reconstruction Methods from RGB Images to Hyperspectral Imaging: A Survey” In arXiv:2106.15944, 2021

Appendix A Specifications of Skin Chromophores

Parameter	Description	Value
$\mu_{a_{hbO2}}$	Oxy-Haemoglobin absorption	2.303 $\frac{p_{hb}\varepsilon_{hbO2}}{w_{hb}}$
$\mu_{a_{hb}}$	Deoxy-Haemoglobin absorption	2.303 $\frac{p_{hb}\varepsilon_{hb}}{w_{hb}}$
$\varepsilon_{hbO2}$	Oxy-Haemoglobin Extinction	[Jac13]
$\varepsilon_{hb}$	Deoxy-Haemoglobin Extinction	[Jac13]
$p_{hb}$	Haemoglobin Concentration	150
$w_{hb}$	Molar weight of Haemoglobin	64500
$\mu_{a_{eu}}$	Eumelanin absorption	$6.6x10^{11}$ $\lambda^{-3.33}$
$\mu_{a_{pheo}}$	Pheomelanin absorption	$2.9x10^{15}$ $\lambda^{-4.75}$
$p_{bil}$	Bilirubin Concentration	0.05
$w_{bil}$	Molar weight of bilirubin	584.66
$p_{\beta-c_{e}}$	$\beta$ -carotene Concentration (Epidermis)	$2.1x10^{-4}$
$p_{\beta-c_{d}}$	$\beta$ -carotene Concentration (Dermis)	$7x10^{-5}$
$w_{\beta-c}$	Molar weight of $\beta$ -carotene	536.8726

Table 2: Chromophore specifications. The absorption coefficient is defined in

cm^{-1}

; the extinction coefficient

\varepsilon_{c}

\frac{cm^{-1}}{moles/liter}

; the concentration of the chromophore

p_{c}

in g/L; and the molar weight

w_{c}

in g/mol. The absorption of melanins

\mu_{a_{eu}}

and

\mu_{a_{pheo}}

is defined through a fit from [DJ06] to the measurements from [JM91] and [SS06] respectively. The 2.303 coefficient in

\mu_{a_{hb}}

comes from deriving a factor of

ln\left(0\right)

, since

\varepsilon

has been historically recorded in such base 10 nomenclature from measurements of old spectrometers in literature. Finally, oxy and deoxy haemoglobin extinction can be found tabulated in [PJ].