Unsupervised Intrinsic Image Decomposition with LiDAR Intensity

Optimized based methods. IID represents an ill-posed problem that decomposes a single image into albedo and shade. To address this issue, Land et al. [19] proposed a prior that accounts for sharp and smooth luminance variations induced by albedo and shade changes, respectively. Grosse et al. [12] improved estimation accuracy by considering hue variance as well as luminance. In addition, several priors have been proposed to enhance the estimation accuracy, including the piecewise constant in albedo [26, 1], sparse and discrete values of albedo [35, 37, 39], and similar shade among neighboring pixels [10]. Bell et al. [3] formulated and minimized energy function based on these priors. To achieve edge-preserving smoothing, a combination of global and local smoothness based on superpixels was proposed [4]. Although these handcrafted priors are reasonable in small images, they are insufficient for more complex scenarios, such as outdoor scenes.

Supervised learning methods. With the development of deep learning, supervised learning methods began to be used for IID. Narihira et al. [32] first applied supervised learning methods to IID. In addition, IID was performed by learning the relative reflection intensity of each patch extracted from the image [45]. Nestmeyer et al. [34] directly estimated per-pixel albedo by trained convolutional neural network (CNN). Fan et al. [9] designed a loss function for a universal model, which works on both fully-labeled and weakly labeled datasets. Recently, many researchers have trained supervised learning models based on albedo and shade from synthetic data [29, 46], since observing the ground truth albedo and shade in a practical scenario is difficult. However, the estimation accuracy for observed data may be limited by the gap between the synthetic and observed data.

Unsupervised learning methods. In these days, semi-supervised and unsupervised learning methods have begun to be used in IID. Janner et al. [14] suggested semi-supervised learning methods that utilize a few labeled data for training and transfer to other unlabeled data. Most existing unsupervised learning methods such as Li et al. [25] require a series of images or multi-view images. Liu et al. [28] proposed unsupervised single image intrinsic image decomposition (USI³D), which is an unsupervised learning method from a single image. USI³D outperforms state-of-the-art unsupervised learning methods for IID. However, these unsupervised learning methods are currently underperforming supervised learning methods due to the lack of criteria for solving ill-posed problems. In addition, it is difficult for these methods to discriminate between cast shadows and image textures when only using an image and depth. Therefore, we propose the utilization of LiDAR intensity, which is independent of sunlight conditions, cast shadow, and shade.

Datasets for intrinsic image decomposition. MIT Intrinsics, a dataset of image decomposed data on 16 real objects, was published by Grosse et al. [12] as a dataset for IID. Since MIT Intrinsics is small for training deep learning models, synthetic data are widely used. Thus, Butler et al. [6] collected an MPI Sintel dataset, which consisted of synthetic data that included albedo, depth, and optical flow. In addition, synthetic datasets such as the ShapeNet [7] and the CGIntrinsics [24] were also published. The free supervision from video games (FSVG) dataset [18], which was extracted from video games and contains a large number of images in outdoor settings with albedo. On the other hand, Bell et al. [3] published the IIW dataset in real scenes with large number of sparse annotations. Conventionally, datasets with LiDAR intensity and annotations for IID did not exist. To validate the utility of LiDAR intensity for IID, we have created a publicly available dataset that includes RGB images, LiDAR intensity, and IID annotations.

A LiDAR calculates the object distance from the time lags between irradiating the laser and detecting the reflected light. In this process, LiDAR intensity, which is the return strength of a irradiated laser beam, is also obtained. LiDAR intensity is intrinsic to the object surface and is therefore independent of sunlight conditions, cast shadow, and shade. Thus, Guislain et al. [13] performed a shadow detection based on the un-correlation between color and LiDAR intensity. In addition, homogeneous regions are extracted with LiDAR intensity and elevation for cast shadow detection [27]. LiDAR intensity is also utilized for hyper-spectral data correction [36, 5] and object recognition [20, 16, 31]. As mentioned above, LiDAR intensity utilization has the potential to separate scene illumination from albedo, even in unsupervised manner.

When utilizing LiDAR intensity for IID, the sparsity or occlusion may cause precision degradation. Thus, we demand to densify the LiDAR intensity before or inside the IID model. In general, dense LiDAR intensity without occlusion corresponding to the supervised data does not exist, and the density of LiDAR intensity is sometimes quite low. Therefore, it is difficult to letting the network manage it implicitly in an end-to-end manner, and we selected a separate model; LID module based on DIP [40]. DIP is one of the fully unsupervised learning method used for denoising and inpainting. In the original DIP, for an image $x_{0}$ to be cleaned, a noise map $z$ is input to a deep neural network $f_{\theta}()$, where $\theta$ represents the deep neural network parameters. Then, energy optimization is performed to satisfy Eq. 11.

Based on the difference in learning speed between the noise component and the image component, a clean image is generated by stopping the iteration in the middle of the process. To apply the original DIP directly to LiDAR intensity densification, we substitute LiDAR intensity for $x_{0}$ and optimize Eq. 12.

where $m_{\rm{L}}$ is LiDAR mask which is 1 for pixels with the observed LiDAR intensity and 0 otherwise. However, original DIP tend to blur in sparse regions due to its inconsideration of RGB images. Thus, we input [RGB image, LiDAR intensity]^T and $[m_{\rm{R}},m_{\rm{L}}]^{T}$ for $x_{0}$ and $m_{\rm{L}}$, respectively. $m_{\rm{R}}$ has the same shape as the RGB image and all pixel values are 1. By simultaneous convolution of LiDAR intensity and RGB image, LiDAR intensity is densified while considering image edges and brightness. Although DIP is also used in depth completion in multi-view stereo [43], the data, objectives, and loss are all different from our model.

The estimation accuracy of IID is highly dependent on the ability to decompose images into domain variant and in-variant components. Utilizing LiDAR intensity alone does not adequately enable the network to learn domain dependence. Thus, we design an intensity consistency loss that computes the error between LiDAR intensity and gray-scaled albedo to provide a criterion for letting the network to learn domain dependence efficiently. The first term is the loss function when LiDAR intensity $L$ and the luminance of the estimated albedo $F(R(I))$ are correlated, where $F()$ is the function to convert from RGB image to gray scale. In the case where LiDAR intensity and albedo are correlated, the image divided by LiDAR intensity is correlated to the shade from Eq. 1. Thus, the second term represents the loss function when $F(I)/L$ and the luminance of the estimated shade $F(S(I))$ are correlated.

where $m_{\rm{L}}$ is the mask, which is 1 for pixels with densified LiDAR intensity and 0 otherwise. In addition, $s_{x}$ and $b_{x}$, where $x\in{1,2}$, are trainable parameters for adjusting the scale and bias of LiDAR intensity, respectively. In summary, IID-LI optimizes the loss function in Eq. 14.

As with the original paper [28], we set $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4},\lambda_{5},\lambda_{6}$ as 10.0, 0.1, 10.0, 0.1, 5.0, and 1.0, respectively. In addition, we set $\lambda_{7}$ as 20.0 in this paper.

The overview of our proposed method is depicted in Fig. 2. First, LiDAR intensity is densified by LID module since LiDAR points are usually sparse and have occlusion. Then, a RGB image and a densified-LiDAR intensity are given to USI³D-based model with the intensity consistency loss. Since images are generally gamma-corrected for human visibility, the observed brightness is converted non-linearly. Thus, to compare the images linearly with LiDAR intensity, an inverse gamma correction was performed before input to IID-LI, and gamma correction was performed on the output images.

In this paper, we attempt to utilize LiDAR intensity for IID. However, no public data with both LiDAR data and annotations for IID exists. Therefore, we employed our own dataset consisting of images, LiDAR data, and annotation. The data was collected using a mobile mapping system (MMS) equipped with RGB camera, LiDAR, global navigation satellite system (GNSS) and an inertial measurement unit (IMU). For LiDAR scanning, the ZF profiler which features 0.0009 degree angular resolution, 1.02 million points per second, 360-degree field of view and a maximum range of 120 meters. The data was recorded at Shinagawa, Yokohama, and Mitaka in Japan.

The annotation in this study follows the conventional method [3]. First, we extracted 100 samples from the obtained dataset for sparse annotation. We then utilized Poisson disk sampling with a minimum radius of 7% of image size to sample image points, and the points with over and under-saturation or those around the edge were removed in the same manner as Bell et al [3]. Finally, Delaunay triangulation was performed to generate edges, resulting in $91\pm 24$ pairs per image as sparse sampling. Dense sampling data was generated using Poisson disk sampling with a minimum radius 3% of image size, resulting in $566\pm 143$ pairs per image.

In this study, 10 annotators, who understand the concept of albedo, performed the annotation. For each pair, 5 out of 10 annotators answered the following three questions.

• •

  Do the two points have the same albedo intensity?

• •

  If not, does the darker point have a darker surface albedo intensity?

• •

  How confident (Definitely, Probably, Guessing) are you in your judgment of the above?

$a_{i,j}$ is the $i^{\rm{th}}$ annotator judgement for $j^{\rm{th}}$ question. The $a_{i,1}$ and $a_{i,2}$ value +1 for ”yes” and -1 for ”no” for the first and second questions, respectively. The $a_{i,3}$ values 1.0, 0.8 and 0.3 for ”Definitely”, ”Probably” and ”Guessing”, respectively. In this study, the judgement $j$ and the weight $w$ for each pair were calculated as following:

where, $A_{1}=\sum\limits_{i=1}^{5}a_{i,1}a_{i,3}$ and $A_{2}=\sum\limits_{i=1}^{5}a_{i,2}a_{i,3}$. Note that, annotations are utilized only for quantitative assessment of estimation accuracy.

For evaluating the estimation accuracy, we defined the threshold differences between the points used in the human judgements.

where, $R_{L}$ and $R_{D}$ are lighter and darker points of a annotated points pair. Following the conventional studies [3], we set the threshold $\delta=0.1$. We use four indices: weighted human disagreement rate (WHDR), precision, recall, and F-score. WHDR is calculated as Eq. 17.

The human judgement weights are also taken into account when computing precision, recall, and F-score.

Our prepared dataset consisted of 10000 samples with RGB images and LiDAR intensities for training. Moreover, we prepared an additional 110 samples for testing, which were combined with a total of 12,626 human judgments. Since IID-LI is based on VAEGAN, both albedo and shade domain data groups are required. Furthermore, to make training fair, the albedo and shade datasets were independent each other. Thus, the albedo dataset was created by extracting 10000 albedo samples from the FSVG dataset [18]. The shade dataset was created with no overlap with the albedo dataset.

In this section, we selected ten compared methods including both optimized-based and learning-based methods listed in Tab. 1. Firstly, ”Baseline R” represents an image where all pixel values are 1. Conversely, ”Baseline S” decomposes the input image with all the shade pixel values as 1. In addition, Retinex [12], Color Retinex [12], Bell et al. [3], and Bi et al. [4] were selected for optimized-based methods. Finally, supervised learning methods such as Revisiting [9], and unsupervised learning methods such as IIDWW [25], UidSequence [23], and USI³D [28] were employed. For all learning-based methods, publicly available parameters and pre-trained models were used as defaults. Since USI³D is the baseline method, we retrained and validated it with our dataset.

First, we quantitatively evaluated our proposed method and the compared methods. As shown in Tab. 2, our proposed method achieved the state-of-the-art in terms of recall and F-score. However, Revisiting [9] was the best for WHDR and precision. The estimation accuracy of each model may not have been properly evaluated due to the significant bias in the number of “E”, “D” and “L” annotations (E=9411, D=2554, L=661). Thus, to eliminate bias in the number of annotations, we randomly sampled the annotations so that each annotation size is the same (E=661, D=661, L=661). Tab. 3 shows the evaluating result for randomly sampled annotations. Our proposed method outperformed other unsupervised learning methods, as shown in Tab. 2 and Tab. 3. Furthermore, it delivered comparable performance to Revisiting [9], which is a supervised learning methods. The visual results are shown in Fig. 3. Cast shadows remained in conventional methods since these methods cannot differentiate between cast shadows and textures. On the other hand, the cast shadows were less noticeable in our proposed method by using LiDAR intensity. However, the images generated by our method are slightly blurred. The image blur is considered to be caused by the calibration errors between the image pixel and LiDAR point, thus addressing this issue is a future work.

In our proposed method, we have presented LID module and intensity consistency loss. Thus, the efficacy of these components has been quantified in Tab. 2 and Tab. 3. From the F-score in Tab. 3, the intensity consistency loss contributed the most to estimation accuracy.

This study used relatively high-density LiDAR data, though it can be sparse in certain scenarios. Therefore, we evaluated the estimation accuracy with reduced LiDAR data, and the numerical results for random sampled annotations are listed in Tab. 4. The density of LiDAR intensity was reduced to 50%, 10% and 1% from the original LiDAR intensity. As expected, the higher density of LiDAR intensities indicates higher estimation accuracy, as shown in Tab. 4. Specifically, LiDAR intensity reduced to 1% achieve comparable F-score to the base model, USI³D [28]. As a reference, Tab. 5 shows the estimated results without the LID module, and indicate that the proposed LID module improves the estimation accuracy. Fig. 4 shows an example of LiDAR intensity with reduced LiDAR density, and the effect of LID module. The LiDAR intensity with 1% points were densified successfully in visual, though the estimation accuracy may be limited due to its blurred detail. In a future perspective, the development of a more sparsity-robust model will be interesting.

The intensity consistency loss computes the error between LiDAR intensity and gray-scaled albedo. The validity of the intensity consistency loss is discussed in this section. First, an image in which the field of view is almost in shadow, and the corresponding LiDAR intensity is shown in Fig. 5. In addition, each corresponding pixel filled into a 2-dimensional histogram is also shown in Fig. 5 (right). The correlation coefficient between luminance and LiDAR intensity values for this sample was 0.45. Since these maps diverge slightly, we expect that the actual correlations coefficient would be a bit higher. This intensity consistency loss is considered to be effective at least in outdoor scenes with mostly achromatic materials such as concrete and wall surfaces.