Rethinking Noise Synthesis and Modeling in Raw Denoising

For the signal-dependent noise, we only consider the photo shot noise in the raw image formation since other signal-dependent noise sources only have very limited impact (less than 3% according to [15, 18]). Since the incident photon numbers follow the Poisson distribution strictly as described in Eq. (2), we can accurately synthesize the signal-dependent noise as

where $Y$ is the clean image obtained by removing all noise in Eq. (4), and $\hat{N_{p}}$ is the sampled signal-dependent noise (photon shot noise). To synthesize the photon shot noise, we invert the digital numbers $Y$ to the photon numbers by dividing the total gain $K_{d}K_{a}$. Then, we sample the signal-dependent noise from the Poisson distribution. The total gain $K_{d}K_{a}$ can be estimated by the linear curve between the mean and variance of flat-field frames [19].

For the signal-independent noise, we capture dark frames in the dark room to construct the dark-frame database, and the synthetic noise is sampled directly from the dark-frame database. In each shooting of the dark frame, all signal-dependent noise naturally disappears as we block all incident light. Formally, it can be formulated by removing the signal-dependent component in Eq. (4):

where $B$ is the dark frame, and $\hat{N}_{\mathrm{independent}}$ is the synthetic signal-independent noise. All actual signal-independent noise is accumulated and saved in the shooting of the dark frames. As a result, the synthetic signal-independent noise is most accurate and contains all kinds of noise types that cannot be modeled explicitly before.

Moreover, it is convenient and feasible to sample actual signal-independent noise from dark-frame databases of different devices with various exposure and ISO settings. For a popular camera sensor, each raw image contains more than 1M pixels, which indicates more than 1M noise values are sampled from the actual noise distribution at each shoot. We can easily obtain sufficient signal-independent noise samples to form the signal-dependent noise database.

By synthesizing the signal-dependent and -independent components separately, for a given clean image $Y$, we can generate the synthetic noisy image via Eq. (4).

where $\hat{D}$ is the synthetic noisy image.

The spatially-correlated noise becomes significant under low-light conditions. To preserve the spatially-correlated noise, we use patch sampling, which synthesizes the signal-independent noise by cropping patches from the constructed dark-frame database. In this way, the spatially-correlated noise is preserved in the dark-frame patches. Moreover, we observe that the raw Bayer pattern also affects the noise distribution, we further propose pattern-aligned patch sampling (PAP), which ensure that dark-frame patches have the exactly aligned Bayer pattern as the clean image to be corrupted. Table 1 shows the results of using pixel-wise sampling, patch sampling and pattern-aligned patch sampling (PAP) for synthesizing signal-independent noise. The proposed pattern-aligned patch sampling helps to improve the denoising performance under the extreme low-light condition.

In extreme low-light conditions, the signal-independent noise becomes the dominant noise, which requires accurate and high-bit synthetic noise. To show that, in Table 1, we quantize the signal-independent component of the physics-based methods to 14-bit (the same as the dark frames), which is denoted as “LB”. The physics-based methods show an obvious performance drop when using lower-bit synthetic noise, especially for low-light conditions (e.g., $\times 100,\times 200$). Therefore, high-bit signal-independent noise plays a crucial role in noise synthesis for low-light environments.

Theoretically, we can dump dark frames before the quantization operation in the sensor processing pipeline or save high-bit dark frames (e.g. 32-bit) to preserve actual signal-independent noise. However, most CMOS sensor manufacturers do not allow researchers to have access to such raw data. Therefore, we propose a high-bit reconstruction method to inverse the quantization first and synthesize high-bit values to refine the noise patches sampled by the pattern-aligned patch sampling.

Figure 1(a) shows a toy example illustrating the quantization effect for the continuous analog signal distribution of camera sensors. Figure 1(b) shows the details within each quantization step. The quantization process maps a set of continuous values in $[x-\frac{1}{q},x+\frac{1}{q}]$ to a single value $x$, which makes our collected dark frames have lower bits. To reconstruct the high-bit dark frames, we first estimate the continuous distribution of the low-bit noise sampled from the dark frames. Then, as shown in Figure 1(b), according to the estimated continuous distribution, we sample high-bit value (32-bit) within the quantization step $[x-\frac{1}{q},x+\frac{1}{q}]$ for each low-bit discrete value $x$, and use the sampled high-bit values to replace the $x$ in the dark frames. The sample probability of high-bit values is obtained by normalizing the estimated continuous distribution within the quantization step. In this way, the dark frames are reconstructed to higher bits while keeping the characteristics of the real noise.

We use statistical models to estimate the continuous distribution of quantized noise in the dark-frame database. Instead of modeling it with a single distribution (e.g., Gaussian [14], Tukey Lambda [29]), we construct a continuous distribution set which contains 5 bell-shaped distributions, including Student’s t, Weibull, Tukey lambda, Gaussian, and Gamma distribution, for better modeling noise of different sensors. We use the distribution set to fit the noise in the dark frames and identify the distribution that can best describe the noise distribution.

We follow previous works [29, 1, 7] and select two representative datasets to systematically evaluate our methods and compare with existing DNN-based and physics-based methods.

1. 1.

   SIDD [2] is a mobile dataset, which is collected by five different smartphones under three lighting conditions (low-light, normal and high exposure). We use three recent smartphones (iPhone 7, Google Pixel, Samsung Galaxy S6 Edge) in our experiments. Most recent DNN-based methods [1, 7, 30] have demonstrated their effectiveness on SIDD dataset.

2. 2.

   ELD [29] is a recent DSLR dataset, which is captured over a wide range exposure ratio ($\times 1\sim\times 200$). We evaluate our method on two popular cameras (SonyA7S2 and NikonD850) over all exposure ratios.

1. 1.

   Poisson-Gaussian (P-G) distribution [14] is the most typical noise model for raw image denoising. We use the accurate P-G noise model for comparison instead of approximating the Poisson distribution to the Gaussian distribution.

2. 2.

   ELD noise model [29] is the recently developed method for low-light environments. We have reproduced the results in ELD with the authors’ help and modified several lines of code to synthesize noise for a wider ratio range ($\times 1\sim\times 200$).

3. 3.

   Representative DNN-based methods including both recent GAN-based [7] and flow-based [1] methods are used for comparison. We use their released models to synthesize noise on SIDD dataset and train a new model for ELD dataset by adopting the authors’ code.

In all experiments, we adopt the U-shape net in SID [8] as our backbone for training denoising models with synthesized noisy images. For both SIDD and ELD dataset, we pack raw images to four channels and randomly sample non-overlapped $512\times 512$ patches with rotation and flipping augmentation. We follow the common training setting: $L_{1}$ loss function, Adam optimizer with default hyper-parameters, training from scratch, batch size 1, and initial learning rate of $10^{-4}$. The models are trained for 30k iterations and the learning rate halves twice at iterations 15k and 25k. During the training, we sample the ISO uniformly from the set $\{2^{n}100|n\in\mathbb{N},0\leq n\leq 5\}$. For each ISO, we capture 10 dark frames to construct the dark-frame database and use the proposed pattern-aligned patch sampling with high-bit reconstruction to synthesize realistic signal-independent noise. The synthesis of the signal-dependent noise follows Eq. (6).

For the SIDD dataset, three scenes (02, 03 and 05) are kept for testing. For evaluating on ELD dataset, we train models on SID Sony dataset and use the official testing code for all cameras and exposure ratios.

The in-camera noise profile for Poisson-Gaussian noise model can be obtained directly from the meta information of DNG file [17]. They have been used widely in existing works. However, we found that the in-camera noise profiles are not calibrated seriously.

We collect the three recent smartphones used in SIDD dataset and estimate parameters for the P-G noise model as described in the original paper [14]. As shown in Figure 2, except the parameters $\beta 1$ of Google Pixel, all other in-camera noise parameters show large differences compared with our calibrated parameters. In Table 2, we use the in-camera and calibrated noise parameters to synthesize noise for denoising model training and keep all other settings the same. The results show that using the calibrated parameters performs much better than using the in-camera parameters. This indicates that the in-camera noise parameters are not accurate.

We also calibrate the noise parameters for the ELD noise models. It uses the Tukey Lambda (TL) distribution family [20] to fit the read noise distribution and use the Probability Plot Correlation Coefficient (PPCC) [12] test to find the best shape parameter. We found that, for the mobile sensors, the estimated shape parameter is quite close to 0.14, which is an approximate Gaussian distribution in the TL distribution family. As a result, ELD noise model reduces to the P-G noise model with additional row noise and quantization noise.

We train a denoising model for each noise synthesis method with the same training setting. The denoising results on SIDD dataset are summarized in Table 2.

Same as reported in previous works, the in-camera P-G noise mode cannot outperform DNN-based methods, especially for GAN-based models. However, unluckily, all previous works are built on inaccurate noise profiles, since the re-calibrated P-G noise model shows large PSNR improvements on the SIDD dataset. It even works much better than all recent DNN-based methods in our experiments. As described before, the TL distribution in the ELD noise model reduces to the Gaussian distribution. Therefore, the results of the ELD noise model on all three cameras are quite similar to the P-G noise model.

Our method consistently outperforms both DNN-based and statistical methods over all three mobile cameras. Since the SIDD dataset is captured under the normal exposure ratio ($\times 1$), the naive pixel-wise sampling can produce comparable results as the well-calibrated P-G noise models. And , using the high-bit reconstruction also shows consistent improvements.

The noise profiles are not provided in both SonyA7S2 and NikonD850 cameras. Similar to the experiment on SIDD dataset, we use the same model of cameras and calibrate noise profiles for statistical methods (P-G and ELD noise models).

For DNN-based methods, we use Noise Flow [1] for comparison and modify the authors’ code for training the denoising model on the SID dataset. We found that the training process of Noise Flow on the extreme low-light dataset is quite unstable. See Section 4.1 for more implementation details.

Table 3 shows the denoising performances over four exposure ratios of the denoising model trained with data from different noise synthesis methods. The DNN-based method shows poor generalization on the extreme low-light datasets. It suffers from severe color distortion as shown in Figure 3.

For the physics-based methods, the ELD noise model outperforms the P-G noise model in extreme low-light environments ($\times 100,\times 200$). But for the normal lighting conditions ($\times 1,\times 10$), the P-G noise model shows better performances. The results for exposure ratio 1 and 10 are not available (NA) since there are no corresponding training pairs for ratio 1 and 10 in the SID dataset.

Our methods (denoted by PAP + HB) consistently outperforms all existing noise synthesis methods over different cameras and exposure ratios. Table 3 shows that the high-bit reconstruction plays an important role in our method, especially in extreme low-light environments. Some qualitative results are shown in Figure 3. The denoising model trained with our synthetic noise can recover more textures and vivid colors compared with existing methods.

Since our method can synthesize all kinds of signal-independent noise, it helps the denoising model remove the fixed pattern noise successfully even though we do not model it explicitly. The model trained with the real noise (paired data) can also remove it. But physics-based methods can only synthesize the noise that they are able to model. As a result, the denoising model trained with them cannot deal with the unmodeled noise.