Single Image Brightening via Multi-Scale Exposure Fusion with Hybrid Learning

Exposure time and ISO value are the most important factors which can be considered together in various lighting conditions to capture high quality images. For example, in back- or low-light condition, there are two common settings of exposure time and ISO value. The first one is a long exposure time and a small ISO value. The captured image is clean but blurred. The second one is a small exposure time and a large ISO value, and the image is sharp but noisy. Clearly, both settings have difficulty capturing a high quality image, especially when the captured scene is with high dynamic range (HDR). A recommended setting in [1, 2] is a small exposure time and a small ISO value. The captured image is clean and sharp, and most of the high-light regions are not over-exposed, but the image is dark. Therefore, a single image brightening algorithm is highly demanded to increase the brightness so as to obtain a clean, sharp and bright image. Four possible challenging problems in single image brightening are: 1) noise in under-exposed regions could be amplified; 2) the high-light regions could be washed out; 3) there could be lightness distortion in the brightened image [3]; and 4) color could be distorted for the pixels in the under-exposed regions [4].

There are two types of single image brightening algorithms. One is model-driven image processing technologies [5, 6, 7, 8, 9, 10, 11, 12] and the other is data-driven methods such as deep learning ones [13, 15]. Inputs to a model-driven image brightening algorithm are an image (images) to be processed and the related visual prior(s) [1]. The prior(s) is (are) achieved by researchers according to their R&D experience. The model-driven algorithms are built on human’s intelligence. No off-line training is required by the model-driven algorithm but the on-line computational cost could be high. Inputs to a data-driven image brightening algorithm are an image (images) to be processed and the corresponding ground truth image (images) [13, 15]. Off-line training is required by the data-driven algorithm but the on-line computational cost could be low. Each network is trained for each camera in [13]. Considering the pros and cons of these two types of methods, fusing a model-driven method and a data-driven one might be an effective way to study single image brightening. Such a framework is called hybrid learning. Two objectives of this paper are: 1) to explore the feasibility of such a framework rather than a sophisticated neural network for deep learning such that the model-driven and data-driven methods can compensate each other; and 2) to address the four challenging problems in single image brightening.

In this paper, a new hybrid learning framework is introduced for single image brightening under an assumption that the camera response functions (CRFs) are available. This assumption is not an issue if the proposed algorithm is embedded in a digital camera, as different exposure images can be acquired by the camera in advance to estimate the CRFs via the method in [16]. Same as the algorithm in [1], two virtual images are generated to brighten different parts of the captured image. Instead of using the fixed ratio strategy in [1], these two images are first generated via a model-driven method, i.e. by using the intensity mapping functions (IMFs) which can be obtained from the CRFs. As such, there is no lightness distortion in the brightened images. It is noted that noise is amplified and color is distorted in under-exposed regions if the IMFs are used directly [4]. To handle the color-distorted problem, a fixed ratio strategy is designed to brighten each pixel with at least one under-exposed color channel. The ratio is properly determined by solving a least-square optimization problem to prevent possibly visible seam from appearing at the boundaries among the under-exposed regions and their neighboring regions. Besides reducing the color distortion, it is also necessary to avoid amplifying noise in the under-exposed regions. A simple edge-preserving smoothing method is provided to address the problem. The input image is decomposed into a base layer and a detail layer by using a weighted guided image filter (WGIF) in [17], and only the base layer is multiplied by the ratio to brighten the underlying image.

Due to limited representation capability of the IMFs, there is visible difference between the virtual images and their ground truth ones. A deep learning method is adopted to enhance the virtual images such that they are closer to their ground truth images. Since there could be color distortion produced by the model-driven method, one more color loss function is introduced to reduce the possible color distortion in the enhanced virtual images. As such, the proposed loss function is composed of restoration loss and color loss. Under-exposed regions in a low-lighting image are usually dominated by the noise, the randomness of noise brings some trouble to train the network. An adaptive weight is thus incorporated in the restoration loss function to mitigate its influence on the feedback adjustment, making the network easier to converge. Both the convergence speed and the accuracy of the hybrid learning are improved compared to the existing residual deep learning.

Besides increasing the brightness of the input image, it is also important to prevent the brightest regions of the input image from being washed out in the brightened image. Two new weighting functions are proposed to achieved the objective. All the input image and two virtual images are fused via the multi-scale exposure fusion (MEF) algorithm in [25] to produce the final image with the new weighting functions and the weighting functions in [25]. Experimental results show that the proposed algorithm outperforms other single image brightening algorithms. Same as the network in [13], each hybrid learning framework is trained for each camera. In other words, both the proposed framework and the network in [13] could be embedded in a smart phone or a digital camera. On the other hand, our input is an sRGB image rather than a raw image in [13]. Besides brightening darkest regions of the sRGB image, brightest regions of the sRGB image are prevented from being washed out while they could be are washed out by the existing brightened algorithms. In summary, our contributions are highlighted as follows:

1) A hybrid learning is formulated by integrating data-driven and model-driven methods for single image brightening. Such solution combines the advantages of both types of methods to generate two virtual images of higher quality. This is a good example to show that the model-driven and data-driven methods can compensate each other.

2) A new restoration loss function is introduced, in which adaptively weights are assigned to the loss caused by suspicious noise, and the convergence speed is improved.

3) A database, which consists of 300 low, medium and high exposure image triplets, is built up. Only the exposure time is changed while other configurations of cameras are fixed. Both camera shaking and object movement are strictly controlled to ensure that only the illumination is changed.

The rest of this paper is organized as below. Relevant works on single image brightening are reviewed in Section II. The proposed hybrid learning framework is summarized in Section III. Two initial virtual images are generated by using the IMF in Section IV. They are enhanced via a deep learning method in Section V. The input image and the two virtual images are fused together in Section VI to produce the brightened image. Extensive experimental results are provided in Section VII to verify the proposed hybrid learning framework. Finally, conclusion remarks are drawn and future works are discussed in Section VIII.

In this section, existing works on single image brightening are summarized under two categories.

Conventional brightening algorithms such as histogram equalization, gamma correction et al. are simple and intuitive ways to enhance a low-lighting image. Although these methods can stretch the contrast of these images, and tackle the low visibility, other problems arise, such as noise amplification, detail loss in bright areas [18]. To address these problems, many single image brightening algorithms have been proposed in the past decade.

The single image brightening algorithms can be classified into two categories. One category is model-driven methods, such as low lighting image enhancement (LIME) [22], which extends the concept of Max-RGB [26] to the pixel level. The illumination of each pixel is first estimated as $\max(R,G,B)$ individually. A structure prior is then imposed on the estimated illumination to obtain the final illumination map. The final image is obtained using the illumination map. Li et al. [1] proposed an image brightening algorithm using the MEF, in which two virtual images with large exposure times are generated via a fixed ratio strategy. The input image and the two virtual images are fused via the MEF algorithm to produce the final image. The fixed ratio strategy can avoid color distortion in under-exposed regions. However, it assumes that the brightness relationship is linear between two co-located pixels in the two differently exposed images, which could result in lightness distortion [3] which is one challenging problem in single image brightening. An interesting CRF based strategy was proposed in [3] for the single image brightening. Experimental results show that this method can reduce lightness distortion. The CRF based strategy was ever used in [4] to study ghost removal for differently exposed images with moving objects. Selecting one of the differently exposed images as the reference image. All pixels in other images are classified into consistent and inconsistent pixels. All the consistent pixels are kept while all the inconsistent pixels are corrected using the CRFs. As indicated in [4], it is very challenging to correct an inconsistent pixel in an under-exposed region via the CRFs if the exposure time of the reference image is smaller than that of the image to be corrected. Color could be distorted and noise could be amplified by the CRFs, which are also two challenging problems for single image brightening.

The other category is data-driven methods such as deep learning ones. The deep learning has been widely applied to address image processing problems including single image haze removal [19, 36, 28], single image rain removal [35, 38], single image denoising [21, 32], as well as single image brightening. Li et al. [27] designed a LightNet to predict the mapping relations between the weakly illuminated image and the corresponding illumination map. It is shown that excellent performance has been achieved if the weakly illuminated image is with good quality but it fails to brighten a weakly illuminated image with low quality such as containing noise or JPEG compression distortion. Wei et al. [34] combined deep learning with a Retinex model to design a Retinex-Net which is composed of a Decom-Net for separating reflectance from illumination and an Enhance-Net for illumination adjustment. Wang et al. [15] introduced a new neural network to learn an image-to-illumination mapping rather than an image-to-image mapping. Chen et al. [13] proposed an interesting deep learning method to map a very dark raw image to a bright image, and each network is trained for each camera in [13]. They intended to generate perceptually good images in low-light conditions while the brightened image looks blurry and the high-light regions are washed out in the brightened image. The data-driven approach has an advantage to obtain a mapping function without hand-crafted parameter tuning. Nevertheless, such technique requires large amount of training data. All the deep learning based brightening algorithms focused on increasing the brightness of the input image. Unfortunately, the high-light regions of the input image could be washed out. It is necessary to address this challenging problem for single image brightening.

Due to the pros and cons of these two different categories of methods, it is worth fusing a model-driven method and a data-driven one for single image brightening. The objectives of this paper are to explore such a fusion of the model-driven method and the deep learning method so that they can compensate each other, and to address the four challenging problems for single image brightening.

In this section, a single image brightening algorithm is proposed by introducing a hybrid learning framework which is a combination of model-driven and data-driven methods. Same as [13], each hybrid learning framework is trained for each camera. The brightness of the input image is increased while the high-light regions are prevented being washed out.

Let $Z_{1}$ be an eight-bit image which is captured at back- or low-light condition. Two eight-bit virtual images $Z_{2}$ and $Z_{3}$ with large exposure times are produced by using a model-driven method. The ground truth images of $Z_{2}$, $Z_{3}$ are denoted as $Z_{T_{2}}$ and $Z_{T_{3}}$, respectively. They are captured together with the image $Z_{1}$ by using the method in [16]. Fig. 1 summarizes the pipeline of our network for a single image brightening via multi-scale exposure fusion with hybrid learning. Since one objectives of this paper is to explore the feasibility of fusing model-driven and data-driven methods rather than a sophisticated neural network for deep learning, the CNN used in the proposed framework is on top of the network in [14] while the ReLU is replaced by the PReLU.

The key component of the proposed algorithm is to generate two high-quality virtual images using a new concept of hybrid learning, which combines a model-driven method and a deep data-driven one. The necessity on such a fusion can be elaborated by borrowing wisdom from the field of nonlinear control systems [23, 24], namely, modelled dynamics and unmodelled dynamics are two concepts in the field of nonlinear control systems [23]. $Z_{2}$ and $Z_{3}$ produced by using the IMFs are the modeled information of $Z_{T_{2}}$ and $Z_{T_{3}}$ while $(Z_{T_{2}}-Z_{2})$ and $(Z_{T_{3}}-Z_{3})$ are the unmodelled information. The modelled information and the unmodelled information are analogous to the modelled dynamics and the unmodelled dynamics.

Different from existing data-driven methods and model-driven ones, the virtual images ${Z_{2}}$ and ${Z_{3}}$ are obtained by using a model-driven method in advance, then the unmodelled information $({Z_{T_{2}}}-{Z_{2}})$ and $({Z_{T_{3}}}-{Z_{3}})$ are learned by using a data-driven deep residual convolutional neural network. Clearly, the quality of the initial virtual images can be improved if part of the unmodelled information can be further represented. Such a framework can be regarded as hybrid learning.

The proposed hybrid learning has the following advantages: Firstly, compared with the model-driven method, the virtual images produced by the proposed method is enhanced by compensating unmodelled information. Secondly, compared with the data-driven solution, the hybrid learning converges fast and requires fewer training samples, because $({Z_{T_{2}}}-{Z_{2}})$ and $({Z_{T_{3}}}-{Z_{3}})$ are sparser than ${Z_{2}}$ and ${Z_{3}}$. Thus, it is easy to train the latter neural network using a residual network [31]. Clearly, the model-driven and data-driven methods can compensate each other.

Finally, the input image $Z_{1}$ and two virtual images $Z_{2}$ and $Z_{3}$ are fused together using the MEF algorithm in [25] to produce the final image. The details on the proposed algorithm are given in the subsequent sections.

Two initial virtual images $Z_{2}$ and $Z_{3}$ that are brighter than the input $Z_{1}$ will be produced by using the IMFs in this section.

Let the CRF be ${f_{l}}(\cdot)$, and the exposure times of ${Z_{1}}$, ${Z_{2}}$ and ${Z_{3}}$ be ${{\Delta}t_{1}}$, ${{\Delta}t_{2}}$ and ${{\Delta}t_{3}}$, respectively. Without loss of generality, it is assumed that ${{\Delta}t_{3}}>{{\Delta}t_{2}}>{{\Delta}t_{1}}$. It has been shown in [37] that there is relative brightness change in the fused image if the exposure ratios are too large. Thus, $\Delta t_{3}$ and $\Delta t_{2}$ are selected as $16\Delta t_{1}$ and $4\Delta t_{1}$ in this study. The IMF between input image and two virtual images can be expressed as follows:

where $l$ is a color channel. ${f^{-1}_{l}}(\cdot)$ is the inverse function of the CRF ${f_{l}}(\cdot)$.

Instead of using the fixed ratio strategy in [1], two initial virtual images are generated by using the IMFs. The lightness distortion is prevented from appearing in the brightened images. As mentioned in [4], if the pixel value $z$ is larger than a threshold ${\xi}_{L}$, ${\Lambda}_{1,i,l}(z)$ is a one-to-one mapping, which is reliable. Otherwise, it is not reliable due to a one-to-many mapping. Here, the threshold ${\xi}_{L}$ is determined by the quality of the camera. Both cases are considered to produce the two initial virtual images as follows:

Case 1: The $l$th color channel of pixel $p$, ${Z_{1,l}(p)}$ is bigger than the threshold ${\xi}_{L}$ for each channel $l$. The pixel values corresponding to the two virtual images can be computed by using the IMF. The virtual pixels $Z_{2}(p)$ and $Z_{3}(p)$ are computed as

Case 2: ${Z_{1,l}(p)}$ is smaller than the threshold ${\xi}_{L}$ for at least one channel, and the IMF is not reliable, which yields color distortion. Hence, the fixed ratio strategy in [1] is adopted to generate the corresponding virtual pixels. Two challenging problems to be addressed are: 1) the fixed ratio strategy amplifies noise in the under-exposed regions of $Z_{1}$; and 2) there are visible seams at the boundaries among the underexposed regions and their neighboring regions if the ratio is selected as in [1]. To address the first problem, $Z_{1}$ is decomposed as a base layer $Z_{1}^{b}$ and a detail layer $Z_{1}^{e}$ by the WGIF [17]. To address the second problem, the virtual pixels $Z_{i}(p)(i=2,3)$ are computed as

where the values of ${{\tilde{\gamma}}_{i}}$(i=2,3) are obtained by minimizing the following function:

and the function $\tilde{w}$ is defined as [39]:

The function ${h_{1}}(z)$ is given as

Similar to $\xi_{L}$, the value of $\xi_{U}$ is related to camera quality. The higher the quality of the camera, the larger the value of $\xi_{U}$. In this study, we use $\xi_{L}=5$ and $\xi_{U}=60$ as the default settings.

The resultant virtual images are shown in Fig. 2. The virtual images are close to the corresponding ground truth images but the color needs to be corrected. In addition, due to limited representation capability of IMFs, the images $Z_{2}$ and $Z_{3}$ do not contain all the information in the ground truth images. Thus, both intensity and color need to be adjusted.

The IMF can be regarded as a model to represent the correlation among different exposed images but its representation capability is limited. As mentioned in the Section III, the unmodelled information $({Z_{T_{2}}}-{Z_{2}})$ and $({Z_{T_{3}}}-{Z_{3}})$ can be further represented by a deep learning method. The unmodelled information is usually sparse, i.e., most values are likely to be zero or small as shown in Fig. 3. It is expected that the unmodelled information can be compensated by a deep neural network. Similarly, it is challenging for the deep neural network to compensate the IMF in the under-exposed regions due to existence of noise in the regions.

The loss function used in the proposed hybrid learning is defined as

where $w_{c}$ is a constant, which is selected as 2 in this study.

The restoration loss $L_{r}$ is usually defined as

The above loss function can be replaced by the weighted MSE in [30] which is equivalent to the SSIM but is differentiable. Since the under-exposed regions in $Z_{1}$ contain random noise [1], a content adaptive weight is introduced to the restoration loss so as to reduce the effect of noise on the adjustment parameters. The loss function $L_{r}$ is given as

where the weight function $W_{i,l}(p)$ is expressed as:

and $\nu$ is a small positive constant and it is empirically selected as 6.0 in this paper if not specified. When the pixel value in position $p$ is smaller than $v$, it may be noise, so a small weight is assigned to the loss.

To minimize the possible color distortion in the two brightened images, the color loss is defined as

where $\angle(Z_{T_{i}}(p),{f_{i}}(Z_{1}(p))+Z_{i}(p))$ is the angle between two 3D $(R,G,B)$ vectors $Z_{T_{i}}(p)$ and $({f_{i}}(Z_{1}(p))+Z_{i}(p))$. Since the $L_{r}$ metric only measures the color difference numerically, it cannot ensure that the color vectors have the same direction [15]. By employing the color loss in Eq. (17), the possible color distortion is reduced.

The enhanced virtual images is expressed as $({f_{i}}(Z_{1})+Z_{i})(i=2,3)$. It is shown in Fig. 3 that the virtual images enhanced by the deep learning method are much closer to the ground truth images, which implies the unmodelled information is reduced significantly. The color of the plants in the second row is also distorted, while our result looks more natural and much closer to the ground truth. The color of the cup in the third row is obviously distorted by the model-driven method. The results of our method are much closer to the ground truth. It is worth noting that the deep learning method works on TensorFlow, and is trained with a mini-batch size of 32. An Adam optimizer with a fixed learning rate of $10^{-4}$ is used to optimize the entire network. Mirroring and cropping are employed to augment training data.

The input image $Z_{1}$ and the two virtual images $Z_{2}$ and $Z_{3}$ are fused together to produce the final image. As shown in [25], the weighting maps of the differently exposed images play an important role in the MEF algorithm. This section focuses on defining the weighting maps for the three images while the MEF is the same as [25].

Besides increasing the brightness of the image $Z_{1}$, the brightest regions of the image $Z_{1}$ are prevented from being washed out. To achieve this objective, two different functions are adopted to define the weights for the three images.

One function is used to determine amplification factors of all the pixels in the image $Z_{1}$. The weight is defined as

where the function $h_{2}(z)$ is $(z-128)/32$. The function $\psi_{1}(z)$ is inspired by a weighting function in [39].

The other is to measure contrast, well exposedness level, and color saturation of each pixel in the three images and it is defined the same as in [25]

where $w_{c}(Z_{i}(p))$, $w_{s}(Z_{i}(p))$ and $w_{e}(Z_{i}(p))$ measure contrast, color saturation, and well-exposedness of pixel $Z_{i}(p)$, respectively.

Let $Y_{1}$ be the luminance component of the image $Z_{1}$ in YUV color space. The weight of the pixel $Z_{1}(p)$ is given as

and the weight of the pixel $Z_{i}(p)(i=2,3)$ is given as

With the above weighting maps, all the images $Z_{i}(i=1,2,3)$ are fused together via the existing MEF algorithm in [25]. It can be shown in the Eqs. (23) and (24) that the pixels in the brightest regions of image $Z_{1}$ dominate the fusion. As such, the brightest regions of image $Z_{1}$ are prevented from being washed out by the proposed algorithm.

Extensive experiments are carried out in this section to demonstrate the rationality and effectiveness of the proposed hybrid learning framework. The emphasis is to show how the model-driven method and the data-driven one compensate each other. Readers are invited to view to the electronic version of the full-size figures and zoom in these figures in order to better appreciate the differences among images. Code is available at [41].

A new hybrid learning framework is introduced to study single image brightening. This paper was focuses on the compensation of model-driven method and data-driven method rather than employing sophisticated neural networks. Two initial virtual images with large exposure times are first generated by using the model-driven method and they are enhanced by using the data-driven residual convolutional neural networks. All the input image and the two virtual images are fused to obtain a brightened image. The brightness of the input image is increased while the high-light regions are prevented from being washed out. Experimental results show that the proposed algorithm outperforms the existing algorithms.

It was assumed by the proposed algorithm that the camera response functions (CRFs) are available. This is not an issue if the proposed algorithm is embedded into a digital camera or a smart phone but it might not be true for an image downloaded from the Internet. A deep learning based algorithm could be used or a single image based estimation method such as the LECARM [22] is adopted to estimate the CRFs. Two virtual images with 2EV gaps are generated in the proposed algorithm. It is interesting but challenging to determine the optimal number of virtual images with the optimal EV gaps. One more interesting topic is to apply the proposed hybrid learning for other image processing problems. It is also interesting to develop more sophisticated deep learning methods to replace the one used in this paper. All of them will be studied in our future R$\&$D.

This work was supported by the National Nature Science Foundation of China under Project 61775172 and Project 61620106012.