GMLight: Lighting Estimation via Geometric Distribution Approximation

As illumination maps are high-dynamic-range (HDR) images, the light intensity of different illumination maps may vary drastically, which is deleterious for the parameter regression. We thus propose to normalize the light intensity of illumination maps and design a novel geometric distribution method for illumination representation.

As illustrated in Fig. 2, the geometric distribution illumination representation decomposes the scene illumination into four parameters including light distribution $P$, light intensity $I$, ambient term $A$, and depth $D$. With a HDR illumination map, we first separate the light sources from the full image since the light sources in a scene play the most critical role for relighting. Consistent with [7], the light sources are selected as top 5% of pixels with the highest values in the HDR illumination map. The light intensity $I$ and the ambient term $A$ can then be determined by the sum of all light-source pixels and the average of remaining pixels, respectively. To formulate discrete distributions, we generate $N$ anchor points (using the method described in [16]) that are uniformly distributed on a unit sphere as illustrated in Fig. 3. Light-source pixels are assigned to their corresponding anchor point based on the minimum radian distance, and further determine the light-source value of the anchor point by summing all its affiliated pixels. Afterward, the value of light-source pixels is normalized by the intensity $I$ so that the $N$ anchor points on the unit sphere form a discrete distribution (i.e. the light distribution $P$). Since Laval Indoor HDR dataset [2] provides pixel-level depth annotations, the depth $D$ at each anchor point can be determined by averaging depth values of all the pixels that are assigned to the anchor point (similarly based on the minimum radian distance).

With a limited-view image cropped from the illumination map to server as the network input, the illumination parameters and the original illumination map serve as the ground truth for the regression network and generative projector, respectively.

In the regression network, four branches with shared backbone network are adopted to regress the four sets of parameters, respectively. The light intensity $I$ and ambient term $A$ are scalars which can be regressed with a naive L2 loss. However, the light distribution $P$ and depth values $D$ are vectors of $N$ dimension that are spatially localized on a sphere, and the naive L2 loss cannot effectively utilize the proximity of the geometric distribution and the property of the standard distribution (for light distribution $P$, the summation of all anchor point values is equal to one). Therefore, we propose a geometric mover’s loss based on earth mover’s distance [14] to effectively measure the discrepancy between distributions with consideration of the scene geometry as shown in Fig. 4.

Geometric Mover’s Loss: Our previous work EMLight proposes a spherical earth mover’s loss (SML) to achieve geometry-aware regression. However, the SML over-simplifies the scene geometry to be a spherical surface which ignores the complex geometry of real scenes. As an image is produced as a result of the scene illumination, scene geometry, etc., ignoring the scene geometry will deteriorate the inverse process of inferring illumination from a single image. We thus propose a geometric mover’s loss (GML) which effectively leverages the real scene geometry through depth values. To derive the proposed GML, two discrete distributions $U$ and $V$ with $N$ points are defined as shown in Fig. 4. According to earth mover’s distance, GML can be defined as the minimum total cost to convert distribution $U$ to distribution $V$, where the cost is measured by the product of the amounts of value (or ‘earth’) to be moved and the distance to be moved. Thus, a moving plan matrix $T$ and a cost matrix $C$ both of size $(N,N)$ are defined, where entries $T_{ij}$ and $C_{ij}$ denote the amounts of moved ‘earth’ between point $U_{i}$ and point $V_{j}$ and the unit cost of moving $U_{i}$ to $V_{j}$, respectively. In our previous spherical mover’s distance, the unit cost between point $U_{i}$ and point $V_{j}$ is measured by their radian distance along the unit sphere as shown in Fig. 5, which is unable to take advantage of the real scene geometry in regression. We thus propose to facilitate the scene geometry through a geometric distance as determined by the scene depth. As shown in Fig. 5, the distance of a GMLight anchor point to the center is measured by a depth value $D$ instead of the radius, thus effectively reflecting the real geometry of the scene. The geometric distance (or unit cost) between anchor points $U_{i}$ and $V_{j}$ can be computed according to their depth values $D_{i}$, $D_{j}$ and their spherical angle $\theta$, as follows:

With the defined transportation plan matrix $T$ and cost matrix $C$, GML can be formulated as the minimum total cost for the transport between $U$ and $V$:

On the other hand, different from the illumination distributions, the depth values of anchor points don’t form a standard distribution (the sum of all depth values is not constant). Thus, we introduce an unbalanced setting of GML for the regression of depth values. Specifically, it is handled by introducing a relaxed version of classical earth mover’s distance, namely unbalanced earth mover’s distance [33]. It aims to determine an optimal transport plan between measures (depth values) of different total masses. We formulate unbalanced GML by replacing the ‘Hard’ conservation of masses in (2) by a ‘Soft’ penalty with a divergence metric. An unbalanced GML as denoted by $GML^{u}$ can thus be formulated as follows:

where ${\rm KL}$ is the Kullback-Leibler divergence which is defined as:

Grounded in well-studied entropic regularization [34] for differentiable optimization, the entropic version of GML (2) can be formulated as below:

where $H(T)$ is the entropic term as defined by $H(T)=-\sum_{i=1}^{N}\sum_{j=1}^{N}T_{ij}\log T_{ij}$, $\epsilon$ is the regularization coefficients denoting the smoothness of the transportation plan matrix $T$. In our model, $\epsilon$ is empirically set to 0.0001. The unbalanced GML (3) can be regularized similarly by adding the entropic term $H(T)$. Then the entropic version of the problem (2) and (3) can be solved in a differentiable yet efficient way through Sinkhorn iteration [34] in training.

The proposed geometric mover’s loss makes the regression sensitive to the global geometry which effectively penalizes the spatial discrepancy between the predicted distribution and ground-truth distribution. Besides, the geometric mover’s loss is smoother than L2 loss which enables more stable optimization in training process.

The regressed light parameters indicate accurate light properties including the light intensity, ambient and light distributions. To project these parameters into illumination maps, we propose a novel generative projector based on generative adversarial networks (GANs) [35] as illustrated in Fig. 6. Specifically, the light distribution $P$, light intensity $I$, and ambient term $A$ are firstly projected to Gaussian map $M$ through spherical Gaussian function [7] as follows:

where $N$ is the number of anchor points, $v_{i}=P_{i}*I$ and $o_{i}$ denote the RGB value and the direction of an anchor point respectively, $u$ is a unit vector on the sphere and $s$ is the angular size which is set to 0.0025 empirically. Then the illumination generation can be formulated as an image-to-image translation task conditioned on the Gaussian map.

Fig. 6 illustrates the image translation process, where the input image is encoded into a feature vector to serve as the network input. To provide effective guidance in the generation process, the Gaussian map is injected into generation process through a spatially adaptive normalization (SPADE) [32] as shown in Fig. 6. As the illumination map is a panoramic image where pixels are stretched at different latitudes, the vanilla convolution suffers from heavy distortions around the polar regions of the illumination map. To address this, Benjamin et al. [15] propose to encode the invariance against latitude distortions into convolutional neural networks by adjusting the locations of convolution operations, as known as spherical convolution. We thus employ the spherical convolution (Spherical Conv) in the generative projector to synthesize panoramic illumination map which effectively mitigates the distortions of different latitudes on the illumination map.

In EMLight [3], the same conditional Gaussian map is injected into the generation process at different levels. Ideally, coarse Gaussian map guidance is expected in low generation layers to learn the overall illumination condition, while fine Gaussian map is expected in the high layers to indicate the accurate illumination distribution. Thus, we design an adaptive radius strategy in the Gaussian function to generate coarse-to-fine Gaussian maps for different generation layers, as shown in ‘Gaussian Map1’, ‘Gaussian Map2’, and ‘Gaussian Map3’ in Fig. 6.

Spatially-Varying Projection: Spatially-varying illumination prediction aims to recover the illumination of different positions in a scene from a single image. As there are no annotations for spatially-varying illumination in the Laval Indoor HDR dataset, we are unable to train a spatially varying model directly. Previous research [7] proposed to incorporate depth values into the projection to approximate the effect of spatially-varying illumination. We follow a similar idea with [7] to achieve the estimation of spatially varying illumination, as described below.

The Gaussian map is constructed through a spherical Gaussian function, as shown in Eq. (6). When we move the insertion position by $\nabla o$, the new direction of the anchor point $i$ can be denoted by $o_{i}+\nabla o$. The depth of the original insertion position and the new position are $l_{i}$ and $l_{i}+\nabla l$, which can be obtained from the predicted depth value of $N$ anchor points. The light intensity at the new insertion position can thus be approximated by $v_{i}*\frac{l_{i}}{l_{i}+\nabla l}$, and the Gaussian map $M_{\nabla}$ of the new insertion position can be constructed as follows:

The Gaussian map is then fed into the generative projector to synthesize the final illumination map. Fig. 8 illustrates several samples of predicted Gaussian maps, generated illumination maps, visualized intensity maps and the corresponding ground truth. Fig. 11 illustrates the generated spatially-varying illumination maps at different insertion positions.

Loss Functions: Several losses are employed in the generative projector to yield realistic yet accurate illumination maps. For clarity, the input Gaussian map, ground-truth illumination map, and generated illumination map are denoted by $x$, $y$, and $x^{\prime}$, respectively.

To synthesize realistic illumination maps, a discriminator with Patch-GAN [36] structure is included to impose an adversarial loss as denoted by $\mathcal{L}_{adv}$. Different from the vanilla GAN discriminator which discriminates real from faked at image level, Patch-GAN discriminator has a fully convolutional architecture and only conduct the discrimination at the scale of local patches. Image discrimination results across all the patches are averaged to provide the ultimate output of discriminator. The Patch-GAN achieves better synthesis quality and detail generation in image translation tasks [36] compared with vanilla GAN discriminator.

Then a feature matching loss $\mathcal{L}_{feat}$ is introduced to stabilize the training by matching the intermediate features of the discriminator between the generated illumination map and the ground truth:

where $D_{l}$ represents the activation of layer $l$ in the discriminator and $\lambda_{l}$ denotes the balanced coefficients. Targeting to yield similar illumination distribution regardless of the absolute intensity, a cosine similarity is employed to measure the distance between the generated illumination map and ground truth as follows:

Then, the generative projector is optimized under the following objective:

where $\lambda$ balances the loss terms. As the regression network and generative projector are both differentiable, the whole framework can be optimized end-to-end.

We benchmark GMLight with other SOTA lighting estimation models on the Laval Indoor HDR dataset [2]. To acquire paired data for model training, we crop eight images with limited fields of view from each panorama in the Laval Indoor HDR dataset, which finally produces 19,556 training pairs for our experiments. Specifically, the image warping operation as described in [2] is applied to each panorama to mimic the light locality for indoor scenes. In our experiments, 200 images are randomly selected as the testing set and the rest images are used for training. In addition to the Laval Indoor HDR dataset, we also qualitatively evaluate GMLight on the dataset ¹¹1https://lvsn.github.io/fastindoorlight/ introduced in [6].

Following [7] and [6], we use DenseNet-121 [37] as backbone in regression network. The default size $N$ of the discrete light distribution is 128. The sizes of the input image and reconstructed illumination map are $192$ and $128\times 256$. The detailed network structure of the generative projector is provided in the supplementary material. We implemented GMLight in PyTorch and adopted the Adam algorithm [38] as the optimizer that employs a learning rate decay mechanism (the initial learning rate is 0.001). The network is trained on two NVIDIA Tesla P100 GPUs with a batch size of 4 for 100 epochs.

For accurate and comprehensive evaluation of the model performance, we create a virtual scene consisting of three spheres made of gray diffuse, matte silver and mirror as illustrated in Fig. 7. By comparing the sphere images rendered (by Blender [40]) with the ground-truth illumination and the predicted illuminations, the illumination estimation performance can be effectively measured. Several metrics are employed to compare the rendered images including root mean square error (RMSE) which mainly evaluates the accuracy of light intensity, scale-invariant RMSE (si-RMSE) and RGB angular error [29] which mainly evaluate the accuracy of light directions. Besides, we also perform crowdsourcing user study through Amazon Mechanical Turk (AMT) to assess the realism of images rendered with illumination maps predicted by different methods. Two images rendered by the ground truth and each compared method are shown to 20 users who are asked to pick a more realistic image. The score is the percentage of rendered images that are deemed more realistic than the ground-truth rendering. The testing scene of three spheres is mainly used for quantitative evaluation. We also design 25 virtual 3D scenes on the testing set for object insertion to evaluate the qualitative performance in various scenes.

Besides, we introduce a Geometric Mover’s distance (GMD) based on the geometric mover’s loss as described in (2) to measure the discrepancy between light distributions of illumination maps as below:

where $U$ and $V$ are the normalized illumination maps, and the pixels in the maps form geometric distributions. The GMD metric is sensitive to the scene geometry with the cost matrix $C$, thus achieving a more accurate evaluation of illumination distribution (or directions) compared with si-RMSE.

We conduct quantitative comparison between GMLight and other SOTA lighting estimation methods as shown in Table I. Specifically, each compared method predicts 200 illumination maps from the testing set to render the testing scene (three spheres of different materials). The experimental results is tabulated in Table I, where D, S and M denote diffuse, matte silver and mirror material objects, respectively. As can be observed in Table I, GMLight consistently outperforms all compared methods under different evaluation metrics and materials. EMLight [3] simplifies the light distribution of scenes to be spherical, ignoring the complex scene geometry. GMLight introduces depth to model scene geometry which leads to more accurate illumination estimation. Gardner et al. [2] generate illumination maps directly, but it tends to overfit training data due to the unconstrained nature of illumination estimation from a single image. Gardner et al. [7] regress the spherical Gaussian parameters of light sources, but it often loses useful frequency information and generates inaccurate shading and shadows. Li et al. [12] adopt spherical Gaussian functions to reconstruct the illumination maps in the spatial domain but it often loses high-frequency illumination. Garon et al. [6] recover lighting by regressing spherical harmonic coefficients, but their model struggles to regress light directions and recover high-frequency information. Although Garon et al. [6] adopt a masked L2 loss to preserve high-frequency information, it does not fully solve the problem as illustrated in Fig. 9. NeedleLight [39] achieves lighting estimation in both spatial and frequency domains, while it is unable to recover illumination maps with fine details as it doesn’t employ a generative projector. In contrast, GMLight firstly regresses the accurate light parameters with a regression network, followed by a generative projector to synthesize realistic yet accurate illumination maps under guidance of the regressed parameters.

To demonstrate that the regressed parameters provide accurate guidance for the generative projector, we visualize the Gaussian maps, lighting maps, and intensity maps in GMLight and the corresponding ground truth in Fig. 8. As shown in Fig. 8, the predicted Gaussian maps indicate the light distributions in the scenes accurately, which enables the generative projector to synthesize illumination maps with accurate light directions. We also visualize the intensity maps of the predicted lighting maps, and the plausible intensity map compared with the ground truth demonstrate that the generative projector synthesizes accurate HDR lighting maps.

We conduct qualitative comparison between GMLight and four SOTA lighting estimation methods on object insertion task as shown in Fig. 9. Specifically, illumination maps are predicted from the input images and the inserted objects are rendered with the predicted illumination maps. As illustrated in Fig. 9, the illumination maps predicted by GMLight present realistic and accurate light sources with fine details, thus enabling to render object with plausible shading and shadow effects. On the other hand, Gardner et al. [2] generate the HDR illumination maps directly which make it hard to synthesize accurate light sources; Gardner et al. [7] and Li et al. [12] regress the representative Gaussian parameters to reconstruct the illumination maps which however causes unrealistic illumination maps with losing details especially frequency information. Garon et al. [6] and Zhan et al. [39] regress the representation parameters to recover the scene illumination but are often constrained by the limited order of representation basis, which may incur low-frequency illumination that produces weak shading and shadows effects.

Besides the testing set, we also conduct object insertion on wild images collected from the Internet as shown in Fig. 10. The accurate illumination maps and realistic rendering results demonstrate the exceptional generalization capability of the proposed method.

Spatially-varying illumination prediction aims to recover the illumination at different positions of a scene. Fig. 11 shows the spatially-varying illumination maps that are predicted at different insertion positions (center, left, right, up, and down) by GMLight. It can be seen that GMLight estimates illumination maps of different insertion positions nicely, largely due to the auxiliary depth branch that estimates scene depths and recovers the scene geometry accurately. We also evaluate spatially-varying illumination estimation quantitatively and Table II shows experimental results. We can see that GMLight outperforms other methods consistently in all insertion positions. The superior performance is largely attributed to the accurate geometry modeling of light distributions with scene depths.

Fig. 12 illustrates the 3D insertion results with the estimated spatially-varying illuminations. The sample images are from [6], where a silver sphere is employed to indicate spatially-varying illuminations at different scene positions which serve as references for evaluating the realism of 3D insertion. As Fig. 12 shows, the inserted objects (clock) at different positions present consistent shading and shadow effect with the silver sphere, demonstrating the high-quality estimation of spatially-varying illuminations by GMLight.

We developed several GMLight variants as listed in Table III to evaluate the effectiveness of the proposed designs. The baseline model is selected as SG+L2 which regresses spherical Gaussian parameters with L2 loss. GD+L2, GD+SML, and GD+GML denote regress the geometric distribution of illumination with L2 loss, SML in EMLight [3], and GML in GMLight, respectively. GD+GML+GP denotes the standard GMLight with all proposed designs. All variant models are employed to render 200 images of the testing scene (3 spheres) and the rendering results are evaluated by various metrics including the RMSE, si-RMSE, Angular Error, User Study, and the proposed GMD as shown in Table III. It can be observed that GD+L2 outperforms SG+L2 clearly which demonstrates the effectiveness of geometric distributions for lighting representation. The superiority of the proposed GML is also verified as GD+GML produces better estimation than GD+L2 and GD+SML. With the including of the generative projector (GP), the performance across all metrics is improved by a large margin which demonstrates that generative projector improves illumination realism and accuracy significantly.

We also ablate the effect of different number of anchor points as shown in Table IV. It should be noted that the experimental results in RMSE and si-RMSE are averaged on three materials. Compared with the standard 128 anchor points, the prediction performance with 64 anchor points drops slightly and increasing anchor points to 196 also doesn’t bring clear performance gain. We conjecture that the larger number of parameters with 196 anchor points affects the regression accuracy negatively. In addition, we also compare the geometric mover’s loss and cross-entropy loss for distribution regression, compare the spherical convolution and vanilla convolution for panorama generation, and study the effect of adaptive radius in Gaussian projection. We can see that GML outperforms cross-entropy loss (CEL) clearly as GML captures spatial information of geometric distributions effectively. In addition, spherical convolution performs better than vanilla convolution consistently in panoramic image generation and adaptive radius strategy in Gaussian projection also brings notably improvement to the performance.