Acquisition of Spatially-Varying Reflectance and Surface Normals via Polarized Reflectance Fields

The Fresnel equations describe how the light behaves when it encounters the boundary between different optical media, involving two primary reflection components: specular and diffuse reflection. Specular reflection occurs unscattered, producing distinct reflections at any interface. In contrast, diffuse reflection results from both surface and subsurface scattering, causing light to scatter in various directions. The Fresnel equation specifies that specular reflection retains the polarization state of the incident light, while diffuse reflection remains unpolarized, regardless of the incident light’s polarization characteristics [35]. Therefore, diffuse and specular reflection can be separated with different states of polarization, which are determined by incident light conditions.

Theoretically, placing linear polarizers and analyzers in front of light sources and observers with different orientations can effectively separate diffuse and specular reflections. When the polarizer and analyzer are set perpendicular to each other, only the diffuse reflection becomes visible, and the intensity of polarized light is governed by Malus’s Law:

where $\theta$ is the angle between the polarizer’s axis and the analyzer’s axis. Given that the average of $\cos^{2}\theta$ is $\frac{1}{2}$, under identical lighting conditions, the radiant intensity of diffuse reflection $I_{d}$ and specular reflection $I_{s}$ can be measured by:

where $I_{\perp}$ and $I_{\parallel}$ are the observations under cross and parallel-polarized lighting respectively; proof can be found in supplementary material via Mueller calculus.

Materials are represented using spatially-varying BRDF, which explains light scattering on a material’s surface in various directions. Our goal is to precisely measure the following reflectance attributes with controllable polarized lighting: diffuse and specular albedo $\rho_{d}$, $\rho_{s}$, diffuse and specular normal $n_{d}$, $n_{s}$, specular variance $\sigma$, anisotropy $\varsigma$, and roughness $\gamma$. We also measure diffuse and specular visibility $\nu_{d}$, $\nu_{s}$, inter-reflection $\varsigma_{d}$, $\varsigma_{s}$, and occlusion $\tau_{d}$, $\tau_{s}$.

Table 1 summarizes the symbols and formats of the input and output data involved in our method. $\mathbb{R}_{+}$ encompasses all non-negative real numbers, and $\mathbb{B}=\{0,1\}$. $H$ and $W$ respectively refer to the height and width of the image. In our case, the capture is executed with 8 RED KOMODO 6K cameras at $H=6144$ and $W=3240$ at 30 FPS covering $N=346$ lighting directions.

The complex interplay of diffuse and specular reflections in light transport challenges accurate material capture, often resulting in imprecise measurements. To address this issue, we conduct a thorough capture process and employ statistical and optimization methods on the recorded sequences for precise material acquisition.

Our method consists of three primary steps: We start with polarized OLAT (One Light at A Time) captures, encompassing cross and parallel polarization conditions (Section 4.1). Next (Section 4.2), we analyze and preprocess the captured sequence to eliminate overexposure. Additionally, we define a set of constraints aimed at reducing the influence of inter-reflection, self-occlusion, and lens flare during the subsequent optimization process. Finally (Section 4.3), we delve into the specifics of our optimization approach, dedicated to enhancing the accuracy of material properties.

In Fig. 6 and Fig. 7, we highlight the importance of our proposed methods by showcasing the difference between utilizing or omitting the overexposure removal module and optimization. These steps are pivotal throughout the entire process: the removal of overexposure aids in cleanly separating diffuse and specular components, while optimized physically correct normals further enhance the quality of acquired material components afterward.

We visualize the measured anisotropy values $\varsigma$ along specific directions in Fig. 8. For diffuse objects, the measured anisotropy across different directions is nearly zero, indicating that the reflection lobe is evenly distributed, demonstrating isotropic behavior. Conversely, for the specular object, the measured value tends to deviate, indicating anisotropy. In contrast, for specular objects, the measured values deviate, indicating anisotropy. For the mug, the red scanline crosses both the clearcoat and diffuse base, clearly distinguishing the two materials. The anisotropy variation along the scanline shows distinct splits, confirming our method’s correctness. Furthermore, the renderings in Fig. 9 of metal soda also validate this. Without anisotropy, the specular reflections on the cylindrical object would result in a spherical specular highlight rather than a linear strip.

In Fig. 9, we showcase image-based rendering achieved in Blender, using our measured material properties and various HDRI as global illumination. The reference image for HDRI illumination is synthesized by weighting the captured OLAT images for each light, with these weights derived from averaging the pixel values within corresponding spherical areas of the HDRI light probe. In Blender, we use the principal BSDF material, incorporating the measured diffuse albedo as the base color, the diffuse normal as the object’s normal, and the measured specular albedo to set the specular IOR level along with the measured tangent. The measured roughness is also applied. Under HDRI lighting, our renderings closely match the reference image, and under area lighting conditions, our method accurately captures specular reflections on highly glossy materials.

We compare our results with [35], using synthesized gradient illumination from polarized OLAT capture, and static gradient illumination capture. The latter, designed for human skin, incorporates lighting pattern adjustments and lowers the lighting intensity to avoid lens flare and unexpected highlights, as shown in Fig. 10 with references in Fig. 11. Due to space limits, more results can be found in the supplementary.

Ma et al. [35] struggle to eliminate overexposure on object surfaces. Moreover, the mixture of specular reflection with diffuse reflection can compromise the quality of albedo. This may further affect the accuracy of captured normals when albedo is introduced in Equation 10. In contrast, our method achieves a distinct separation between the diffuse albedo and the specular albedo, effectively reducing overexposures on both maps and leading to a more accurate reflectance measurement. Furthermore, our approach enhances the captured diffuse normals and specular components while mitigating inaccuracies arising from albedo maps, accurately preserving intricate geometric details.

The Fresnel equation reveals that specular reflection maintains the incident light’s polarization state, while diffuse reflection remains unpolarized. When the polarizer and analyzer are set perpendicular to each other (cross polarization), the analyzer blocks the specular reflection, allowing only the diffuse reflection to be measured as $I_{\perp}$. Conversely, when the polarizer and analyzer are aligned in parallel (parallel polarization), measured as $I_{\parallel}$, the specular reflection remains observable.

The state of polarization of light can be represented by a Stokes vector $S=[S_{0},S_{1},S_{2},S_{3}]^{T}$ [9], where $S_{0}$ is the total light intensity, $S_{1}$ is the difference in intensity between horizontal and vertical linear polarization, $S_{2}$ is the difference between linear polarization at $\frac{\pi}{4}$ and $-\frac{\pi}{4}$ and $S_{3}$ is the difference between right-hand and left-hand circular polarization. The transformation of light’s polarization states via a linear polarizer at an angle $\theta$ relative to a reference axis is defined by Mueller Matrics, M:

When unpolarized light is incident on the polarizer, the input Stokes vector is $S=[1,0,0,0]^{T}$. As the light passes through a horizontally oriented polarizer, where $\theta=0$, the polarization state of light transforms from $S$ to $S^{\prime}$ defined as:

Upon passing through the analyzer with an angle $\varphi$ to the reference axis, the polarization state of the light further transforms from $S^{\prime}$ to $S^{\prime\prime}$ defined as:

Since the first number in $S^{\prime\prime}$ represents the total intensity, $\cos^{2}{\varphi}$ is the current intensity after the polarise and analyzer, which is also recognized as Malus’s Law (refere to Equation 1). Additionally, when adjusting the analyzer’s axis parallel ($\varphi=0$) or perpendicular ($\varphi=\frac{\pi}{2}$) to the polarizer, the transmitted light can be simplified as follows:

At $\varphi=\frac{\pi}{2}$, the analyzer completely blocks the light. This characteristic can be leveraged to eliminate specular reflection. Therefore, the diffuse reflection can be represented via $I_{\perp}$ while the specular reflection can be represented as the difference of two measurements represented via $I_{\parallel}-I_{\perp}$.

Moreover, on average, half of the light becomes polarized when passing through the polarizers [9], the total intensity from the diffuse reflection $I_{d}$ and specular reflection $I_{s}$ from the original unpolarized light can be derived via:

An initial estimation of diffuse and specular albedo $\rho_{d}^{\text{init}}$, $\rho_{s}^{\text{init}}$ can be roughly derived from the separated diffuse and specular reflection $I_{d}$, $I_{s}$. This estimation relies on the rendering equation for light transport at each surface point that inherently considers albedo.

In the presence of uniform white illumination, the observed radiant intensities for both diffuse reflection $I_{d}$ and specular reflection $I_{s}$ at any given surface point $p$ can be described as follows:

, where $\omega_{i}$ and $\omega_{o}$ represent the incoming and outgoing lighting directions, respectively, with $L(\omega_{i})$ denoting the incident radiance from the $\omega_{i}$, which remains uniform over the sphere. Additionally, $\rho_{d,p}$ and $\rho_{s,p}$ correspond to the diffuse and specular albedo at point $p$, while $n_{p}$ represents the surface point normal, and $f_{\sigma}$ stands for the specular reflection distribution function.

We can establish a local frame where the Y-axis aligns with the surface normal $n$. Therefore, we can rewrite the $\omega_{i}$ in spherical coordinates as $\omega_{i}=[\sin(\theta)\cos(\phi),\sin(\theta)\sin(\phi),\cos(\theta)]$, with $\theta$ as the polar angle and $\phi$ as the azimuthal angle within the local frame. Moreover, under uniform lighting conditions, the incident radiance $L(\omega_{i})$ over the spherical sphere is a constant $L_{0}$. This allows us to further derive $\rho_{d}$ and $\rho_{s}$ over the entire observation as follows:

Also, the $\int_{\omega_{i}\in\Omega}(n\cdot\omega_{i})\,d\omega_{i}$ can also be simulated in the local frame via the law of large numbers:

By conducting uniform sampling of $\omega_{i}$ from a unit sphere, we simulate the results as shown in Table 2.

Moreover, given that Ward’s model [44] subject to 2D normal distribution, denoted as $f_{\sigma}(\omega_{i},\omega_{o})\sim\mathcal{N}(\mu,\sigma^{2})$. It’s expected that the overall integral of $f_{\sigma}(\omega_{i},\omega_{o})$ over $\Omega$ equals 1. Further details can be found in [19].

During the capture, measurements become discrete through OLAT, with each light covering a specific area denoted as $A_{0}$ over the entire spherical surface. Consequently, we approximate the results via the captured sequence $\Lambda=\{I_{d}^{i}\}_{i=0}^{N}$:

where $R_{d}$ and $R_{s}$ denote the diffuse reflection function and the specular reflection function of the material, respectively. $N$ is the number of lighting directions. For further simplification, we introduce the constant $\kappa=(L_{0}\cdot A_{0})^{-1}$. As a result, the equation can be expressed as:

Occlusion describes the overall visibility $\nu$ of incident lighting $\omega_{i}$ from the upper hemisphere $\Omega$ when observed via $\omega_{o}$. This is mathematically expressed as $\frac{1}{2\pi}\int_{\Omega}\nu(\omega_{i},\omega_{o})\cdot(\omega_{i}\cdot n)d\omega_{i}$. Notably, since the average of $\omega_{i}\cdot n$ is $\frac{1}{2}$ over the hemisphere, the occlusion remains at $\frac{1}{2}$ when incident lighting $\omega_{i}$ from any solid angle is visible via $\omega_{o}$. To get the normalized occlusion over the hemisphere, we factor out the average of $\omega_{i}\cdot n$, resulting in:

where $\zeta$ is the average ambient noise map and $\lceil\cdot\rceil$ is the ceiling operator.

Our capture process takes place in a Light Stage with 8 RED KOMODO 6K cameras, synchronized at 30 frames per second. For each scan, the captured data, saved as $3240\times 6144$ resolution R3D files. Prior to using the data from the raw scan, we extract individual frames stored in OpenEXR format. In this work, we conducted complete Polarized OLAT scans for 26 objects. We present frontal views of several objects in our dataset in Figure 12 and further present the polarized OLAT captures for a specific object from multiview in Figure 20.

Our implementation leverages JAX [7] for efficient GPU access, ensuring a lightweight solution. We also employ JAXopt [4] for optimization, enabling batchable and differentiable solvers on large data blocks with a single workstation. We perform batch processing via vectorizing map on an RTX A6000, employing a batch size of $2^{20}$ for optimizing diffuse reflection and $2^{18}$ for optimizing specular reflection.

We implement the constraints $n\times\omega_{i}\geq 0$ to mitigate the impact of lens flare and inter-reflection. In practical terms, we apply these constraints to filter out values that do not meet the criteria before optimization. This approach reduces computational complexity and facilitates the optimizer’s convergence to the optimal solution. Unconstrained data typically introduces a considerable number of zeros, which usually leads the optimizer to generate blank results. The constraints effectively address this issue.

Throughout the optimization process, the presence of noise pixels (usually in the background) can significantly prolong the solver’s search for the local minimum, and unfortunately, this extended search doesn’t lead to a meaningful solution. To address this challenge, we choose to terminate the solver when the linear search encounters failure, which is often a consequence of noise pixels in the background. Notably, the solver tends to converge more readily when dealing with pixels located in the foreground objects.

The proposed method successfully decomposes highly glossy materials. However, capturing living creatures or humans presents challenges due to the current setup’s requirement to capture a dense reflectance field both with and without polarization. This process typically takes around 10 seconds given the current frame rate, and any slight movement of the subject can lead to color bleeding issues. Future implementations will need to incorporate frame tracking to address these movements effectively. Additionally, the method does not currently measure geometry, leading to difficulties in managing shadows caused by self-occlusion. This could be mitigated by integrating geometry data from multiview captures.