Learning an Adaptive Model for Extreme Low-light Raw Image Processing

Histogram Equalization (HE) method is used extensively to process digital images. The basic idea is to improve image visibility by changing the image histogram into a uniform distribution, which effectively increases image entropy for low-light images. In colorHE it is argued that a color-invariant representation can be obtained by applying HE. However, HE tends to cause noise amplification, details disappearance and color distortion in low-light image enhancement.

On the account of the drawbacks of HE, a number of improvements have been put forward. Adaptive Histogram Equalization (AHE) ahe and its variation Contrast-limited Adaptive Histogram Equalization (CLAHE) CLAHE perform the histogram equalization considering not only the global histogram, but also the neighbors of each pixel. Hue-preserving color image enhancement Hue-preserving works on contrast-enhanceing and hue-preserving, wang1999image and kim1997contrast on preserving the original image luminance.

A number of low-light enhancement methods are based on the dehazing and Retinex model. It is observed that low-light images and inversed haze images share many similarities. For this reason, Dong et al. dong2011fast presented a method and achieved the low-light image enhancement in which the low-light images is inversed, dehazed and then inversed back again. This method was studied further by li2015a and malm2007adaptive .

On the other hand, the Retinex model land1964retinex revealed that a natural image is made up of the illumination map and the reflectance map. With Retinex decomposition and the assumption that the illumination map is smooth, researchers have proposed single-scale Retinex jobson1997properties and multi-scale Retinex 557356 . Based on image fusion, adjustments can be applied to the illumination map to improve the performance un24 . un23 focused on preserving natural characteristics with a Bright-Pass filter. In kimmel2003a , a variational Retinex model was proposed and the illumination estimation problem was formulated as a Quadratic Programming optimization problem. In refinedretinex illumination map is estimated considering local structure and retinexnoisemap focused on revealing the structure details from the reflectance map. A recent study shows that the Retinex model can combine with the atmospheric scattering modelretinexscatter . More refined models were presented by un28 ; un29 based on variational Retinex theory.

However, noise modeling is not well established in those algorithms. In dong2011fast it is assumed that noise is insignificant and can be removed before or after the Dehazing stage. And the Retinex based models rely on classic noise reduction algorithms (e.g. BM3D) to reduce noise found in the reflectance map. Therefore it can achieve good results in mild low-light images but tends to amplify noise when processing severe or extreme low-light images. Besides, the variational Retinex model is very time-consuming as it takes many iterations to solve the optimization objectives, thus it is hard to achieve real-time processing.

Image-denoising and low-light enhancement can be realized with the data-driven approach, separately or integrally.

Noise reduction was the subject of extensive studies. Burger et al.burger2012image discovered that convolutional neural networks could compete with the state-of-the-art classic denoising algorithm BM3D. The works on date-driven denoising have been extended by un31 ; un32 ; brooks2019unprocessing . Although these works were not targeted at low-light image processing, they provided some references for the research of other image processing studies.

The first application of the data-driven method is LLNET lore2017llnet , which utilized a deep auto-encoder to learn from synthetic low-light image datasets and the denoising procedure is integrated into the low-light enhancement process. Also, it was demonstrated that Retinex based methods can be further improved by deep learning wei2018deep ; shen2017msr , but those works still have undesirable denoising performance because synthetic datasets are used and they cannot reflect the characteristics of real low-light images.

To solve the drawbacks of synthetic datasets, a large multi-exposure image dataset was proposed in cai2018learning . It was found by Chen et al.chen2018learning that models utilizing raw-format images can provide much better results in low-light enhancement. In the SID model proposed in chen2018learning , it suggested that the state-of-the-art results can be achieved by using raw-format paired images and training in an end-to-end way. However, since the paired images have different exposure time, a ratio was introduced as an additional input to bridge the gap. Consequently, if the SID model is applied on images out of the dataset, manual adjustment is requires because there is no paired images to indicate the ratio, which results in many drawbacks in practical applications.

Most of the existing methods are unable to suppress the severe noise present in extreme low-light images. On the other hand, recent studies e.g. chen2018learning have concentrated on the extreme low-light enhancement with raw-format images and real-scene datasets in an end-to-end way, but unlike the classic methods, the brightness of the enhanced image cannot be automatically controlled in this framework, more specifically, an external ratio has to be manually adjusted when the input image has no paired image as the reference.

It is our aim to find a solution to enhance extreme low-light images, combining the advantages of adaptive brightness control and superior denoising performance. It is found that the problem of low-light enhancement can be considered as a special case of changing the time of exposure. With $t_{0}$ and $t_{1}$ representing exposure time, the low-light image $\mathcal{X}_{t_{0}}$ can be interpreted as a normal image $\mathcal{Y}_{t_{1}}$ corrupted by a hypothetical exposure-shifting operator $ES$, which only alters the exposure time but keeps the aperture and ISO unchanged:

In this process, more stochastic noise $\mathcal{N}$ is generated by operator $ES$ because $t_{0}<t_{1}$, corresponding to the low PSNR (Peak Signal to Noise Ratio) value in low-light images. Contrariwise, the corrupted low-light image can also be restored by the same operator $ES$:

The noise $\mathcal{N}$ is reduced because $t_{1}>t_{0}$. Thus the normal image $\hat{\mathcal{Y}}_{t_{1}}$ is reconstructed.

The latter process, named as Exposure Shifting (ES), can enhance low-light images as well as suppress image noise. More importantly, it can be learned in a data-driven and end-to-end way, thus it is expected to have superior denoising performance. However, a proper guideline exposure time $t_{1}$ always has to be provided in the reconstruction $\hat{\mathcal{Y}}_{t_{1}}$. To make our model adaptive, a Brightness Prediction (BP) procedure is introduced to achieve guideline time estimation.

In brief, the extreme low-light enhancement process is split into two procedures. Firstly, for a raw-format low-light image (with real exposure time $t_{0}$), an optimal guideline exposure time $t_{1}$ is estimated via a sub-model called Brightness Prediction Network (BPN). Secondly, the final RGB-format enhanced image is obtained by another sub-model, namely Exposure Shifting Network (ESN), using estimated guideline time $t_{1}$ to control the image brightness.

In addition, besides the image itself, some extra data are involved in both BPN and ESN, such as white balance index, ISO, $t_{0}$ and $t_{1}$. The Image Enhancing Vector (IEV) is put forward to encode the involved extra data.

Raw image array is the original data captured by the camera. Other key parameters, such as camera white balance, ISO, exposure time, time and date, are stored together with the raw image array as Exif metadata. In HDR imagingReinhard2005High , exposure time and ISO information have played an essential role in calculating the response curve and LDR to HDR conversion. Although it is not necessary to explicitly train the network to learn the camera response curve, feeding these additional parameters into the network as input avoids underfitting caused by insufficient features.

Image Enhancing Vector (IEV) is proposed to introduce extra features into the convolution neural networks. In this paper, IEV consists of ISO $u_{s}$, white balance indices for 4 raw image channels $(w_{r},w_{g},w_{b},w_{g2})$, exposure time of the low-light image $t_{0}$ and guideline exposure time $t_{1}$. Moreover, it is possible to append other Exif metadata into IEV to improve network performance when necessary,e.g. aperture and focal length.

The IEV is used as network input in both BPN (for $t_{1}$ estimation) and ESN (for exposure shifting), but a difference exists. As it is shown in Fig.2, in ES procedure the IEV can be written as:

Where $(w_{r},w_{g},w_{b},w_{g2})$ are the white balace indices of 4 raw image channels and $u_{s}$ is the ISO value (controlling the camera sensitivity).

In BP procedure the $t_{1}$ in IEV is removed. It is renamed as partial IEV (pIEV) to indicate the difference:

IEV and pIEV are fed into the convolutional neural network as additional channels of the raw image array. For example, the first extra channel provided by IEV is a uniform image filled up with pixel value $w_{r}$.

Exposure Shifting (ES) is the second procedure in our model but is trained first. The U-NETronneberger2015u is selected as the structure of the Exposure Shifting Network (ESN), Compared with the fully convolutional network, the U-NET architecture reduced the usage of GPU memory, thus it can easily process images with high resolution.

Recall that the basic unit of our Campus Image Dataset (CID) is an image group, which consists of 8 raw-format images with different exposure time but captured in the same scene. Let $\left(\mathcal{X}_{R},\mathcal{Y}_{R}\right)$ be the paired raw image extracted from one group, in which $\mathcal{Y}_{R}$ is the pre-selected ground-truth image and $\mathcal{X}_{R}$ is a randomly-selected low-light image. More specifically, $\mathcal{X}_{R}$ is randomly chosen within this image group, with $\mathcal{Y}_{R}$ and over-exposed images excluded (Fig.3). Moreover, the exposure time of $\mathcal{X}_{R}$ is denoted as $t_{0}$. The corresponding RGB-format image of $\mathcal{Y}_{R}$ is denoted as $\mathcal{Y}$ and its exposure time as $t_{g}$.

It is expected for the ESN to learn the exposure-shifting operator $ES$ from a large number of paired images. Specifically, the ESN is required to estimate an image $\hat{\mathcal{Y}}$ that resembles the $\mathcal{Y}$ as closely as possible. In this way ESN successfully changes the raw-format low-light image $\mathcal{X}_{R}$ into an RGB-format longer-exposed version of itself. More importantly, the serious noise of $\mathcal{X}_{R}$ can also be removed. This procedure can be written as:

Where ESN is denoted as $F_{ES}$, its parameters as $\Theta_{1}$. Note that since the BP procedure has not yet involved, the guideline exposure time $t_{1}$ is replace by $t_{g}$ in IEV $\mathcal{V}^{t_{0}\rightarrow t_{g}}$. The enhanced result image is represented by $\hat{\mathcal{Y}}^{\Theta_{1}}_{t_{g}}$.

Two metrics to guide the training process of the ESN model are Mean Square Error (MAE) and multi-scale Structural Similarity (SSIM)wang2003multiscale . The former is considered as a simple but effective indicator, evaluating the general similarity of the estimated image and the groud-truth image. And the later is a perceptual metric that is more sensitive to visible structures. Both of them are full-reference metrics. Let $L_{MAE}$ be the MAE loss and $L_{SSIM}$ be the SSIM loss. The width and length of the image are denoted as $m$ and $n$ respectively. With subscript $t_{g}$ omitted, the MAE loss $L_{MAE}$ and SSIM loss $L_{SSIM}$ are defined as:

The final MAE loss is the average of the channel MAE loss, for simplicity, the image channel is not presented in the equation. The calculation of the multi-scale structural similarity (SSIM) can be refered in wang2003multiscale . With subscript $t_{g}$ omitted, the loss function is defined as the linear combination of $L_{MAE}$ and $L_{SSIM}$:

Where $\alpha$ is a constant and $0\leqslant\alpha<1$. The influence of $\alpha$ will be discussed in the Experiments section.

The parameters of the ESN network $\Theta_{1}$ are learned by minimizing ${L}_{ES}$ over $K$ pairs of images in the training set:

$\Theta_{1}^{\ast}$ represents the optimized ESN parameters.

The obtained sub-model $F_{ES}$ is an approximation to the exposure-shifting operator $ES$. However, for a low-light image out of the training set, a ground-truth counterpart does not exist, thus the guideline time $t_{1}$ in IEV $\mathcal{V}^{t_{0}\rightarrow t_{1}}$ is absent, which leads to the Brightness Prediction (BP) sub-model and guideline time estimation.

The Brightness Prediction (BP) procedure provides the estimation of guideline exposure time $t_{1}$:

Where $F_{BP}$ and $\Theta_{2}$ is BPN and its parameters respectively. The pIEV is represented by $\mathcal{V}^{t_{0}}_{p}=\mathcal{V}_{p}(t_{0})$. It is the first enhancement procedure in our model but trained after the ESN. As it is illustrated in Fig.4, The Brightness Prediction Network (BPN) is a relatively small network, which consists of multiple convolution layers and then ends up with several fully connected layers. Because of the existence of fully connected layers, the width and length of the input image $\mathcal{X}_{R}$ should be uniformly changed into $512\times 512$.

With the estimated $t_{1}^{\Theta_{2}}$ obtained in this BP procedure, the $t_{g}$ item in IEV in the ES procedure can be replaced by $t_{1}^{\Theta_{2}}$. Different from $\hat{\mathcal{Y}}^{\Theta_{1}}_{t_{g}}$ derived in Equation (LABEL:L_esn), the new final result image can be formulated as:

Then an approach to train the BPN is required. To begin with, the purpose of BPN is to control the brightness of the final result image, which is achieved by adjusting the guiding exposure time $t_{1}$ in the ES procedure. Since there are image pairs $\left(\mathcal{X}_{R},\mathcal{Y}_{R}\right)$ and their exposure time $(t_{0},t_{g})$ available, intuitively it seems that the BPN can simply be learned from $\left(\mathcal{X}_{R},t_{g}\right)$ in an end-to-end way, with $\mathcal{X}_{R}$ as the input and $t_{g}$ as the label. However it cannot be achieved, the reason for which will be discussed later in the next subsection.

The proposed BPN training approach and the loss function $L_{BP}$ are as follows. We define the brightness of a single pixel $Br$ as the average pixel value of all color channels. Given a certain dark scene for image capturing and with ISO, aperture settings and the scene all fixed, then the brightness of a pixel is only determined by exposure time. Let $\mathcal{R}$ be the pixel irradiance in this scene, when the exposure time $t$ in camera settings increases, the pixel Exposure $\mathcal{E}(\mathcal{R},t)$ increases as well CRF_Estimation :

Thus the pixel becomes brighter. It is feasible to design loss function $L_{BP}$ based on the pixel brightness.

It should be noted that the relationship between pixel brightness $Br(\mathcal{E})$ and Exposure $\mathcal{E}=\mathcal{R}t$ is not linear, instead, it can be typically described by an S-shaped curve because the pixel brightness is limited within $\left[0,1\right]$ (or $\left[0,255\right]$ for 8-bit image storage) and the camera sensor is less sensitive to the change of $t$ when the pixel value is close to $0$ or $1$. In other words, as the exposure time $t$ changes, some pixels are not sensitive and there are little changes in the pixel brightness compared with some other pixels. For example, when photographing a street at night, pixels revealing the lamp bulbs are always saturated unless the $t$ is extremely small, and pixels representing the dark sky are always close to zero unless it is affected by noise. If those pixels are involved in BPN training, they can cause gradient vanishing in the back propagation algorithm because $\frac{\partial Br(\mathcal{E})}{\partial t}=\frac{\partial Br(\mathcal{E})}{\partial\mathcal{E}}\cdot\mathcal{R}$ and as it is revealed in the S-shaped culve, $\frac{\partial Br(\mathcal{E})}{\partial\mathcal{E}}$ is close to 0. On the other hand, when pixel brightness $Br$ is around $0.5$ the $\frac{\partial Br}{\partial t}$ reaches its maximum. The pixels meeting this condition can be collected into an Area of Interest (AoI), which identifies pixels that are sensitive to the process of Exposure Shifting.

First, in the ground-truth image $\mathcal{Y}$, an AoI weight map $\mathcal{W}$ can be obtained by filtering out pixels with brightness near $0.5$. More specifically, the ground-truth image $\mathcal{Y}$ is converted into a single channel grayscale image $\mathcal{Y}_{G}$, then a Gaussian culve with mean $\mu_{w}=0.5$ and variance $\sigma_{w}^{2}=0.01$ is applied on each pixel of $\mathcal{Y}_{G}$:

Then $\mathcal{W}_{G}$ is normalized, the result is the AoI weight map $\mathcal{W}$. The value at position $(i,j)$ is denoted as subscripts, namely $\mathcal{W}_{G,ij}$ and $\mathcal{W}_{ij}$:

Combining the Equation 12 and 13, they can be simplified by the softmax function:

Lastly, the loss function $L_{BP}$ is designed to check if the final result image $\hat{\mathcal{Y}}^{\Theta_{2}|\Theta_{1}^{\ast}}$ has the same AoI. Having the same AoI means that in this AoI area $\hat{\mathcal{Y}}^{\Theta_{2}|\Theta_{1}^{\ast}}$ and $\mathcal{Y}$ share similar brightness. Moreover, recall that pixels outside AoI are not sensitive to the change of exposure time $t$ or the ES process, thus it can be derived that the overall image brightness of $\hat{\mathcal{Y}}^{\Theta_{2}|\Theta_{1}^{\ast}}$ resembles that of $\mathcal{Y}$. In this way, BPN completed the adaptive Brightness Prediction objective.

Then the enhanced image $\hat{\mathcal{Y}}^{\Theta_{2}|\Theta_{1}^{\ast}}$ is converted into a grayscale image $\hat{\mathcal{Y}}^{\Theta_{2}|\Theta_{1}^{\ast}}_{G}$. When the AoI of $\hat{\mathcal{Y}}^{\Theta_{2}|\Theta_{1}^{\ast}}$ and $\mathcal{Y}$ overlaps exactly, the following loss function reaches its minimum:

Where $\sigma_{v}^{2}=0.04$ is another variance constant. The image value at position $(i,j)$ is represented by $i,j$ subscripts. Finally, BPN is trained by optimizing $\Theta_{2}$ over $K$ pairs of images in the training set:

In Algorithm 1, the training process of our model is summarized, together with the method to evaluate and apply this model.

As mentioned earlier in the last section, training BPN with input $\mathcal{X}_{R}$ and label $t_{g}$ is straightforward but not feasible. First let $F_{BP}^{\ast}$ be the hypothetical optimal BPN model, then $t_{1}^{\ast}(\mathcal{X}_{R})=F_{BP}^{\ast}(\mathcal{X}_{R})$ is the optimal estimation of guideline exposure time. For a pair of image and time label $\left(\mathcal{X}_{R},t_{g}\right)$, the $t_{g}$ can be considered as one sampling from a Gaussian distribution: $t_{g}\sim\mathcal{N}\left(t_{1}^{\ast},\sigma_{c}^{2}\right)$. The variance $\sigma_{c}^{2}$ in this process is too large, the reason are:

• •

  In every CID image group, the ground-truth image is chosen manually and is dominated by subjective human judgments.

• •

  The ground-truth image is selected from only 8 candidate images, but the optimal $t_{1}^{\ast}$ exists somewhere in the time continuum $\left(0,\inf\right)$.

Furthermore, the number of samples is very limited since each image group can only provide one $t_{g}$ sample. As a result, this straightforward approach cannot be applied in the training of BPN due to the high variance of the label $t_{g}$ and the limited number of samples.

Limited by the GPU memory, we did not adopt the Batch Normalization ioffe2015batch technique. To accelerate model training, before the training process started, each channel of the input image, as well as each element of IEV, was normalized along the sample dimension. Then the means and variances used in the normalization became invariant constants. Additionally, images for training were randomly cropped into $512\times 512$ patches.

Both ESN and BPN were trained with Adam optimizerkingma2014adam , following the procedures shown in Algorithm 1. The ESN was trained first with the learning rate slowly descending from $2\times 10^{-4}$ to $1\times 10^{-5}$. After about 300 epochs, when the loss on training sets became stable, the parameters in ESN were made untrainable. Then BPN was appended to the training graph and trained with the same learning rate. Considering BPN may require global information to decide the proper exposure time, we used bigger patches but avoid training with the whole image to prevent overfitting. Although the loss must propagate through the deep ESN before reaching BPN, the training process went smoothly and completed after 100 epochs.

Firstly, we highlight the adaptive brightness control feature of our method. Here we explicitly define the brightness as the mean value of an image $\mathcal{Y}$, denoted as $Br(\mathcal{Y})$. We expect a dynamic adjustment of the image brightness based on the image content. More specifically, while extreme low-light images suffer from severe noise along with color distortion and are barely visible before post-processing, other mild low-light images only have moderate noise and relatively lower brightness compared with normal images. Thus, the enhancement algorithm needs to make adjustments adaptively for different input images. Currently the classic methods, such as algorithms based on Retinex and Dehazing, have poor performance on the extreme low-light images, On the other hand, learning-based methods targeting at end-to-end denoising can process all kinds of low-light images, but they lack the flexibility and adaptability because they either need brightness amplification hand-tuning, or simply serve as a denoising procedure in other low-light methods. The proposed model has both the advantages and is able to process low-light images of different levels.

In order to evaluate the adaptability, we apply the trained model to enhance all images in the test set. Recall that the image group is the basic unit of our CID dataset, accordingly, instead of evaluating single images, the collaborative characteristics across different images are investigated within each image group. Note that the group identity of an image is NOT involved in model training and only serves as a tag to gather resulting images for further analysis.

For a sequence of multi-exposed images $(\mathcal{X}_{R,1},\mathcal{X}_{R,2},...)$ with increasing exposure time, it is expected that after enhancement the resulting images possess similar image brightness, namely $Br(\hat{\mathcal{Y}}_{1})\approx Br(\hat{\mathcal{Y}}_{2})\approx...$, where $\hat{\mathcal{Y}}$ denotes the enhanced image. Fig.5(a) illustrates a busy-road scene, the first row displays the RGB images produced by the camera, the second displays the enhanced results by utilizing our method on individual raw images. Note that the first image in this group is exposed for 1/20 seconds, causing motion blur to the truck, while the last image is exposed for only 1/100 seconds. It is shown that all four images have approximately uniform brightness after the image enhancement, despite the changing $t_{0}$ and the moving vehicle headlight. The performance of our method on extreme low-light conditions is shown in Fig.5(b). The last image in the sequence suffers from both low environment illumination and short exposure time and the color and shape of objects in the image are drowned out by noise. However, the resulting image has the noise reduced and still maintain acceptable global brightness.

As mentioned above, for quantitative analysis, we gathered images that belong to the same groups and calculated the brightness of all images in order to observe the shift of brightness. Let $(Br_{1},Br_{2},...)_{k}$ be the image brightness of each unenhanced low-light image in $k$-th image group and let $(Br^{\prime}_{1},Br^{\prime}_{2},...)_{k}$ be that of the enhanced ones. Then, the mean value and the CV (coefficient of variation) of $(Br_{1},Br_{2},...)$ is computed and denoted as $(\mu_{,}c_{v})_{k}$. Correspondingly, $(\mu^{\prime},c_{v}^{\prime})_{k}$ is calculated from $(Br^{\prime}_{1},Br^{\prime}_{2},...)_{k}$. In Fig.8, we plotted $(\mu_{,}c_{v})$ and $(\mu^{\prime},c_{v}^{\prime})$ for all groups. To illustrate the change after enhancement, for each group $(\mu_{,}c_{v})$ (marked with star) and $(\mu^{\prime},c_{v}^{\prime})$ (marked with circle) are linked with a dotted line. It can be observed from Fig.8 that for different groups that captured from diverse scenes, the enhanced images all end up with increased image brightness and significantly reduced CV. And yet the brightness growth among the groups can be very different, ranging from around 60% to over 1,000%. Considering together with Fig.5, Fig.8 indicates that the BP sub-model is able to adaptively control the global image brightness not only based on the brightness of the original image, but also according to the content of the scene. On the other hand, the dramatic reduction of CV suggests that the enhanced images within each group share almost identical brightness in spite of the varying exposure time $t_{0}$. To be precise, our model can control group brightness CV under 0.29 on 76.3% test set groups and 98.5% training set groups considering the influence of the diverse scenes, illumination conditions, ISO etc.

When designing our model, we expect the BPN to extract features of the scene content and then estimates a reasonable $t_{1}$ accordingly. In order to see through clearly how this black box works, the relationships among the real exposure time $t_{0}$, the BPN-estimated guideline exposure time $t_{1}$ and the brightness of enhanced images $Br(\hat{\mathcal{Y}})$ are shown in Fig.9. Similar to Fig.8, we use group average to simplify the figure, in which each point corresponds to an image group. As shown in Fig.9, the network tends to estimate $t_{1}$ positively associated with $t_{0}$. Furthermore, for groups with identical average $t_{0}$, the contents of the scenes have an impact on $t_{1}$ prediction. To be specific, if the scene of a group has more relatively bright areas, a smaller $t_{1}$ is more likely to be adopted to prevent overexposure and vice versa. And yet a few exceptions exist, suggesting the BPN sub-model has learned some more sophisticated rules in the neural network black box.

We compare our method with four classic and state-of-art methods, including multiscale Retinex (MSR)jobson1997multiscale , illumination map estimation based (LIME)guo2017lime , deep Retinex decomposition (Retinex-Net, R-Net)wei2018deep and learning-to-see-in-the-dark (SID)chen2018learning . For fair comparison, 3D transform-domain filtering (BM3D)Kostadin2007Image is applied on MSR, LIME, R-Net, since those methods do not model noise and require extra denoising process. Additionally, we train the SID model on the CID dataset and manually select ratio $\gamma$ for the SID model since it does not provide automatic brightness estimation.

Recall that the BPN utilizes a reduced reference evaluation index as the loss in its training. In this way, the exposure parameters of the ground-truth image, which is chosen by highly subjective human judgment, will have much less influence on BPN model training. As a result of this novel technique, however, PSNR and SSIM evaluation cannot be adopted because there is no reference image. We provide perceptual comparisons in Fig.6 and evaluate our method with no-reference quality metrics.

In Table.5.2, we compare the proposed method with MSR, LIME and R-Net(Retinex-Net), using several no-reference quality metrics including Statistical Noise Level Estimation (SNLE) Chen2015AnSNLE , Noise Level Estimation (NLE) liu2013single , Noise Variance (NV) immerkaer1996fast and image entropy. SNLE, NLE and NV metrics can estimate noise level from a single image, and clean images have relatively smaller values than the noisy ones. The SNLE estimates noise variance by observing the eigenvalues of images and the NLE by applying the principal component analysis (PCA) on special image patches. The NV metric is motivated by the fact that clean and noisy images have different sensitivity to the Laplacian operation. The image entropy reflects the average amount of information in an image, it is the most simple image evaluation metric. Images with proper brightness have more visible information, thus have larger image entropy compared to under-exposed or over-exposed ones. We run evaluations on about 400 CID test images and the SID method is excluded because it requires an external manually-selected amplification ratio for every image. As it is shown in Table.5.2, our method has superior performance on Noise Level Estimation. Moreover, the no-reference metrics use very different standards to assess image quality. MSR+BM3D, R-Net+BM3D and LIME+BM3D methods all have undesired scores on one or multiple indices, while our method has good performance on all metrics.

It is illustrated in Fig.6 that our method and SID method provide more significant improvements in noise removal than the classic method BM3D and can adaptively denoise images with diverse noise levels. Furthermore, our results benefit from the white balance index introduced in IEV and avoid abnormal color in extreme low-light images.

However, the CNN-based method tends to blur the object with sharp edges (e.g. text). By introducing SSIM into loss function, our method shows more promising results in images with text and has advantages over SID in keeping more edge information, e.g. the images in the first row of Fig.6. But on the other hand, in controlled experiments shown in Fig.7, it is revealed that the ratio of SSIM component $\alpha$ in loss function $L_{ES}$ has to be contained to avoid the noise being interpreted into texture and edge by ESN.

The advantage of our work lies in getting rid of the brightness control ratio while achieving state-of-the-art low-light noise suppression. Without any handcrafting parameters as network input, we can directly apply our method to raw low-light video processing.

A raw-format video can occupy a very large amount of storage space. Limited by devices, we only developed a relatively simple way to verify our method on raw low-light videos. In Fig.10, we experiment on 21 continuous raw images captured in a burst session, shot with short exposure time and identical ISO, to discuss the potential applications in raw video processing. In this scene, the camera starts from a bridge with lighting decoration, then turns to the dark riverbank, and finally to the dimly illuminated sidewalk. As is shown in Fig.10, though the enhancement process of each frame is completely independent of other frames, neither the estimated $t_{1}$ nor the resulting images show any sudden fluctuation. However, for practical considerations, it is also convenient to implement guideline time filters between ESN and BPN, which allows the value of $t_{1}$ to be dynamically filtered or amplified according to different demands.

The proposed algorithm is evaluated on the ModelArts cloud service, which is equipped with 8 vCPUs and an Nvidia-P100 GPU (16GB GPU memory). The execution time for a high-resolution image ($3968\times 2976$) is $0.27$ seconds on average, which only shows a minor increase compared with the SID algorithm ($0.21$ seconds). More specifically, the execution time of BPN and ESN sub-model is $0.05$ and $0.22$ respectively. For comparison, it takes $4.32$ seconds for BM3D (GPU implementation bm3d-gpu ) to complete the denoising process, therefore other low-light methods that depend on BM3D for denoising, including MSR, LIME, R-Net et al., are much less efficient on devices with GPUs.

In this paper, an adaptive raw image enhancement model is proposed for extreme low-light image processing. The two-stage framework provides adaptability to images with different scenes and exposure parameters. We also presented the CID dataset for model training and evaluation. The experimental test shows that our model can provide the state-of-the-art low-light image denoising as well as adaptive global brightness control.

The proposed method has many advantages over existing approaches. In the first place, no external parameters are needed after model training, a single raw image and its Exif metadata (e.g. white balance, ISO and exposure time) are sufficient to complete the process. Therefore, our model is able to process a large batch of raw images without parameter-handcrafting. Secondly, our model significantly outperforms other methods in which the denoising stage works as a separable step. Because instead of simply implementing noise-suppression algorithms (e.g. BM3D) before or after the main low-light enhancement procedure, we make denoising procedure an integrated part in our model, allowing the ESN module to learn exposure shifting and denoising simultaneously in an end-to-end way. In quantitative experiments, we adopted no-reference image quality metrics including SNLE, NLE, NV and image entropy to test the denoising performance. Our model has the smallest noise variance on the NLE metric compared to BM3D-based low-light methods and obtains near-optimal scores on the other three indices.

We also reveal the potential application in low-light video processing. The BPN module makes it possible for our model to process images with diverse exposure levels, from extreme low-light images to mild ones. But to be applied in video processing, it should face more challenges, e.g. the brightness of frames should change continuously and avoid fluctuation. We experiment on continuous raw image series, despite that no information of adjacent frames is used, the output of BPN changes continuously with the illumination condition and shows no fluctuation.

The model proposed is limited by dataset, which is a common problem in data-driven methods. On the one hand, the model is trained by raw-format and multi-exposed images, which significantly improved image quality. But on the other hand, all reference ground-truth images are captured in the real scenes, as a result, they sometimes have minor defects such as noise, halos around the light source, local over-exposure and white balance deviation. The denoising performance may be further improved if those drawbacks can be avoided. Then the image groups in CID have not covered enough nature scenes. For example, it is found that green grass and trees tend to have low color saturation in the resulting images, that is because most images in the training set are collected in winter when there is hardly any green plants in scenes. For future study, the raw-format CID dataset can be extended by unprocessing brooks2019unprocessing technique and generating raw-format images from other available online datasets.

Another limitation is the memory consumption problem. The batch size is constrained because it takes too much GPU memory to train the proposed model. Therefore training the model can be time-consuming on GPUs with small memory. We are planning to improve network architecture and investigate the possibility of application on mobile devices.

As a future expectation, the model can be further modified by introducing HDRI (High Dynamic Range Imaging). Since our CID dataset is composed of multi-exposed images, we can calculate an HDR image of each scene. Finally, the HDR images can be used as ground truth to participate in the ESN training.