A Review of an Old Dilemma:
Demosaicking First, or Denoising First?

This has led several recent works to propose joint demosaicking- denoising methods [26, 32, 9, 20]. For example [21] proposed a variational model to solve jointly demosaicking, denoising and deblurring. It uses a sparsifying prior based on wavelet packets and applied on decorrelated color channels. More detail about the technicalities of this sophisticated method can be found in [2]. Life has become far easier for joint denoising/demosaicking with the emergence of machine learning methods. It is, indeed, easy to simulate as many learning data as needed. This methodology can be used to obtain ground breaking demosaicking algorithms such as [52]. This paper proposed in 2018 a demosaicking CNN outperforming the best handcrafted algorithms including ARI [45] by nearly 2 decibels. In [32] a public ground truth dataset was introduced and used for one of the first joint demosaicking and denoising methods based on machine learning. In a rapid succession, two state of the art denoising+demosaicking methods based on deep learning were proposed: [20] and [37]. This last paper performs joint denoising and demosaicking by a customized neural network presented as a cascade of energy minimization methods tuned by learning. The outstanding results beat the claimed anterior best method [25] by 1 decibel. Then in 2018 we have two still better performing methods, [15] involving a GAN, which compares favorably to [20] and [54]. The method recently proposed in [38] performs joint denoising and demosaicking by inserting many residual denoising layers in a CNN. This complex method is claimed to beat [20] and [37] by a good margin. Lastly, in 2019, [17] introduced a “mosaic-to mosaic” training strategy analog to the noise-to-noise [41] and frame-to-frame [18] frameworks to handle noisy mosaicked raw data, and training both demosaicking and joint denoising and demosaicking networks without requiring ground truth. The method starts from pairs or bursts of raw images of the same scene. It registers them and learns to predict the missing colors.

Yet the question of how to combine denoising and demosaicking algorithms conceived as independent blocks remains very relevant, especially in the context of low power or portable devices, and given the fact that the main effort in denoising and demosaicking has addressed them independently. A big argument in favour of performing denoising before demosaicking is that most existing demosaicking algorithms have been developed under the unrealistic assumption of noise-free data [23, 33, 24, 49, 34, 35, 28, 56, 45, 36, 10, 55, 8, 60, 42, 20, 54, 53, 38]. Yet the performance of these algorithms can degrade dramatically when the noise level increases on the CFA raw image. Therefore, a previous denoising step is implicitly required by these algorithms.

In this paper we focus on the early CFA processing in the imaging pipeline (operating in linear space). We assume that the noise in the raw mosaic is additive white Gaussian (AWGN) and that its variance is known. This is realistic because, first, a variance stabilizing transform (VST) [5] applied to a raw image results in a nearly AWG noise and, second, because an accurate noise model is often known or can be estimated [50, 11]. In general, image denoising methods can be grouped into two major categories, the model based methods such as non-local means [7, 30, 29], nlBayes [39], CBM3D [12] and WNNM [22], and deep learning methods such as [27, 57]. The ensuing CNNs are sometimes flexible in handling denoising problems with various noise levels.

Our goal here is to determine which strategy is more advantageous for coupling demosaicking and denoising : Is it applying denoising and then demosaicking (which we will denote $DN\&DM$: $DN$ and $DM$ indicate denoising and demosaicking respectively), or is it better to apply first demosaicking and then denoising ($DM\&DN$)?

Many state of the art works [46, 47, 31, 58] support the opinion that $DN\&DM$ outperforms $DM\&DN$. Their first convincing argument is that after demosaicking noise becomes correlated, thus losing its independent identically distributed (i.i.d.) white Gaussian property. This increases the difficulty of applying efficient denoising and actually seems to discard all classic algorithms, that mostly rely on the AGWN assumption. A second obvious argument is that the best demosaicking algorithms have been designed with noise-free images.

For example, Park et al. [47] considered the classic Hamilton-Adams (HA) [23] and [16] for demosaicking, combined with two denoising methods, BLS-GSM [51] and CBM3D [13]. This combination raises the question of adapting CBM3D to a CFA. To do so, the authors apply a sparsifying 4D color transform to the 4-channel image formed by rearranging the Bayer pixels, apply BM3D to each channel, then apply the inverse color transform. In the very same vein, in the BM3D-CFA method [14] BM3D is applied directly on the CFA color array. To do so, “only blocks having the same CFA configuration are being compared to build the 3D blocks. This is the only modification of the original BM3D”. A little thought leads to the conclusion that this amounts to denoise four different mosaics of the same image before aggregating the four values obtained for each pixel. The authors compare two denoising algorithms with two different setups: a) filtering CFA as a single image and b) splitting the CFA into four color components, filtering them separately, and recombining back the denoised CFA image. This paper showed a systematic improvement over [58]. They use Zhang-Wu [59] as demosaicking method for their comparison of result after demosaicking. In our comparisons the method of [14] will be mentioned every time we consider the $DN\&DM$ setup with BM3D. We will nevertheless replace the demosaicking of [59] by RCNN [54] or RI [33], which clearly outperform it.

Similarly in [9] denoising is performed by an adaptation of NL-means to the Bayer pattern, where only patches with the same CFA configuration are being matched. This paper formulates the demosaicking as a super-resolution problem, assuming that the observed values are actually averages of four values in the high resolution image. It then guides this super-resolution problem by the NL-means weights. The method is compared with [44] and [58]. The authors of [58] also propose an $DN\&DM$ method, where the demosaicking method is [59] and the the denoising method is an adaptation of nlBayes [39] to a Bayer pattern. First, the method extracts blocks with similar configuration in the Bayer array and groups them by similarity, then it applies to them PCA and a Wiener denoising procedure which can be also interpreted as an LMMSE. In our experiments, this PCA method [58] will be considered every time we evaluate the $DN\&DM$ scheme (but combined with a more recent demosaicking algorithm such as RCNN [54]). The more recent paper [61] involves similar arguments. This paper uses [4], a linear filter to extract the luminance from the CFA. Then it remarks that this luminance is correlated, so it applies a variant of NL-means that attempts to decorrelate the noise. The same method is applied to each downsampled color channel and the high frequency of the grey level is transported back to the color channels. This method under-performs with respect to others considered here, so we shall not include it to our final comparison tables. Nevertheless, it remains of interest as a fast method compatible with low power cameras. The paper proves that it has a performance very close to a combination of [58] and [26].

The paper [48] is another method promoting denoising before demosaicking, involving dictionary learning methods to remove the Poisson noise from the single channel images prior to demosaicing. Experimental results on simulated noisy images as well as real camera acquisitions, show the advantage of these methods over approaches that remove noise subsequent to demosaicing. The paper nevertheless uses [43] which is a historic but outdated demosaicking method.

To summarize, in the $DN\&DM$ strategy all classic denoising algorithms such as CBM3D, nlBayes, nlMeans have been adapted to handle a noisy mosaic where only one of R, G or B is known at each pixel. Several of them [46, 47, 31, 58] address this realistic case by processing the noisy CFA images as a half-size 4-channel color image (with one red, two green and one blue channels) and then apply a multichannel denoising algorithm to it. The advantage of the denoising step of $DN\&DM$ is that the Poisson noise can be led back by the classic Anscombe transform to the case of i.i.d. white Gaussian, and the disadvantage is that the resolution of the image is reduced and, as a result, some details might be lost after denoising. Another issue of this strategy is that the spatial relative positions of the R, G, and B pixels are lost by handling the image as a four channel half size image.

In this paper, we address the above mentioned issues. We shall first delve into the advantages and disadvantages of $DN\&DM$ and $DM\&DN$ approaches. We shall then analyze noise properties after demosaicking and adjust two existing classic denoising algorithms (CBM3D and nlBayes) to accommodate them to this type of noise. Then, we shall perform a thorough experimental evaluation that will lead us to conclude that $DM\&DN$ (with an adjusted noise parameter) is superior to $DN\&DM$. This result is opposite to the conclusion of [46, 47, 31, 58]. The advantages of $DM\&DN$ seem to be linked to the fact that this scheme does not handle a half size 4-channels color image; it therefore uses the classic denoising methods directly on a full resolution color image; this results in more details being preserved and avoids checkerboard effects.

The rest of the paper is structured as follows. In Section 2 we present in detail the problem and the main ideas behind the proposed demosaicking and denoising strategy. Section 3 is a detailed evaluation of the proposed strategy. Section 4 concludes.

In order to solve the above two problems, we shall revisit the $DM\&DN$ scheme. Compared to the $DN\&DM$ scheme, the advantage of $DM\&DN$ is that it does not halve the image size. This is a way around the above mentioned problems. A serious drawback, though, is that chromatic and spatial correlations have been introduced by the demosaicking in the noise of the CFA image. The result is that the noise is no longer white. We next analyze some properties of the demosaicked noise.

Given a ground truth color image $(R,G,B)$ we define the demosaicked noise associated with a demosaicking method $DM$ in the following way: first the image is mosaicked so that only one value of either $R,G,DM$ is kept at each pixel, according to a fixed Bayer pattern. Then white noise with standard deviation $\sigma_{0}$ is added to the mosaicked image, and the resulting noisy mosaic is demosaicked by $DM$, hence giving a noisy image $(\tilde{R},\tilde{G},\tilde{B})$. We call demosaicked noise the difference $(\tilde{R}-R,\tilde{G}-G,\tilde{B}-B)$. In short, it is the difference between the demosaicked version of a noisy image and its underlying ground truth.

The model of the demosaicked noise depends on the choice of the demosaicking algorithm $DM$. For the demosaicking step we will evaluate the following state of the art methods, which have an increasing complexity: HA [23], RI [33], MLRI [34], ARI [45], LSSC [42], RCNN [54] and JCNN [20]. We are interested in algorithms with low or moderate power; only HA, RI, MLRI and RCNN have a reasonable complexity in this context. For the denoising step we shall likewise consider two classic hand-crafted algorithms, CBM3D and nlBayes.

Fig. 2 (c1) and (c2) shows an example where noisy CFA images with noise of standard deviation $\sigma_{0}$ were first demosaicked by RCNN and then restored by CBM3D assuming a noise parameter $\sigma=\sigma_{0}$. The output of CBM3D with $\sigma=\sigma_{0}$ has a strong residual noise. Similar results are also obtained with nlBayes (see the supplementary material). To understand empirically the right noise model to adopt after demosaicking, we simulated this $DM\&DN$ pipeline for different levels of noise $\sigma_{0}$, and applied CBM3D after demosaicking with a noise parameter corresponding to $\sigma_{0}$ multiplied by different factors $(1.0,1.1,\cdots,1.9)$.

The results are shown in Table 1, where the classic color peak signal-to-noise ratio (CPSNR) [4] is adopted as a logarithmic measure of the performance of the algorithms. It is defined by

where $X$ denotes the ground truth image and $\widehat{X}$ is the estimated color image. From $1.0$ to $1.9$, the CPSNR increases first and then decreases. We can see that the best values are distributed on the lines with factors from $1.4$ to $1.7$. A similar behavior was also observed using nlBayes for denoising as well as for other levels of noise (see the supplementary material).

This does not mean that the overall noise standard deviation has increased after demosaicking. Let us consider the noise standard deviation estimated as the mean RMSE of the demosaicked images from the Imax [60] dataset with different noise levels, given in Table 2. We observe that for low noise ($\sigma_{0}=1$) there is a serious demosaicking error, of about 4, not caused by the noise, but by the demosaicking itself. However, for $\sigma_{0}>10$ we see that the RMSE of the demosaicked image tends to roughly 3/4 of the initial noise standard deviation.

At first sight, this $3/4$ factor seems to contradict the observation that denoising with a parameter $1.5\sigma_{0}$ yields better results. This leads us to analyzing the structure of the residual noise. Fig. 3 shows an image contaminated with AWG noise with standard deviation $\sigma_{0}=20$ and its resulting demosaicked noise for respectively HA, MLRI, RCNN. In the last row of the figure, one can observe the color spaces (in standard (R,G,B) Cartesian coordinates) of each of these noises, each cloud being presented in its projection with maximal area. As expected, the AWG color space is isotropic and has an apparent diameter proportional to $4\sigma_{0}\simeq 80$. The color space of the demosaicked noise is instead elongated in the luminance direction $Y=\frac{R+G+B}{\sqrt{3}}$ to about $6\sigma_{0}\simeq 120$ and squeezed in the others. This amounts to an increased noise standard deviation for $Y$ after demosaicking, and much less noise in the chromatic directions.

This is confirmed by Table 3 that shows variances and covariances of $(R,G,B)$ and $(Y,U,V)$ respectively for an AWG noise with $\sigma_{0}=20$, and then for the demosaicked noise obtained from it after demosaiciking with RI, MLRI and RCNN. In Table 3 (a) these statistics are computed on a pure white noise image with $\sigma=20$. Hence the variance of $Y$ is $400$, as the $(R,G,B)\to(Y,U,V)$ transform is implemented as an isometry of $\mathbf{R}^{3}$. The variance of $Y$ is a growing sequence for the demosaicked noise obtained by increasingly sophisticated demosaicking: $715$ for RI, $772$ for MLRI, $972$ for RCNN. In contrast, the demosaicked noise is reduced in the $U$ and $V$ axes, with its variance passing from $400$ for AWGN to $168$ and $94$ for RI, and even down to $43$ and $55$ for RCNN. Hence, the noise standard deviation on $U$ or $V$ has been divided by a factor between $2$ and $3$. But Table 3 also shows that the residual noise on $U$ and $V$ is strongly spatially correlated, it is therefore a low frequency noise, that will require stronger filtering than white noise to be removed. This table also shows that the $Y$ component of the demosaicked noise remains almost white.

This leads to a simple conclusion: since image denoising algorithms are guided by the $Y$ component [13, 39], we can denoise with methods designed for white noise, but with a noise parameter adapted to the increased variance of $Y$.

To understand why the variance of $Y$ is far larger than the AWG noise it comes from, let us study in Table 4 the correlation between the three channels $(R,G,B)$ in the demosaicked noise of RI, MLRI and RCNN. We observe a strong $(R,G,B)$ correlation ranging from 0.6 for RI to 0.89 for RCNN, which is caused by the ”tendency to grey” of all demosaicking algorithms. Assuming that the demosaicked noisy pixel components (denoted $\widetilde{\epsilon}_{R},\widetilde{\epsilon}_{G},\widetilde{\epsilon}_{B}$) have a correlation coefficient close to $1$ then we have

This factor of about $1.7$ corresponds to the case with maximum correlation. Our empirical observation of an optimal factor near $1.5$ responds to a lower correlation between the colors.

We performed simulations with the schemes: $DN\&DM$ and $DM\&DN$. The considered demosaicking methods range from classic to very modern: HA[23], RI[33], MLRI[34], ARI [45], and RCNN[54]. For the denoising stage two classic hand-crafted patch-based denoising algorithms were considered: CBM3D [13] and nlBayes [39]. As commented in the introduction, both methods can be adapted to handle mosaics (in the $DN\&DM$ setting). In the case of CBM3D this amounts to applying the method by Danielyan et al. [14], while for nlBayes this is done by denoising the 4-channel image associated to the mosaic.

The denoising and demosaicking schemes with the above mentioned demosaicking algorithms and denoising methods were applied to the mosaic images of the Imax image dataset corrupted by additive white Gaussian noise with standard deviations $\sigma_{0}=1,3,5,10,15,20,25,30,35,40,45,50,55,60$.

Due to space constraints, in Table 5 we only report the results corresponding to one noise level $\sigma_{0}=20$. Results corresponding to other noise levels are in the supplementary material. From Table 5, we can see that $DM\&DN$ with parameter $\sigma=\sigma_{0}$ is not better than $DN\&DM$, but $DM\&1.5DN$ (which denotes denoising $DN$ with parameter $\sigma=1.5\sigma_{0}$) beats clearly $DN\&DM$.This might explain why many researchers think that the scheme $DN\&DM$ was superior to the scheme $DM\&DN$.

In addition to the good CPSNR results, one important advantage of the $DM\&DN$ schemes is the high visual quality of the final restored images. Fig. 2 demonstrates the differences between the various solutions (based on BM3D) obtained on the test image number 3 of the Imax dataset with $\sigma_{0}=20$. To save space, only crops of the full-color results and corresponding differences with the ground truth are shown here.

The $DN\&DM$ scheme shown in Fig. 2 (b1) and (b2) uses BM3D-CFA [14] for denoising; we can observe some minor checkerboard artifacts. From Fig. 2 (c1) and (c2), we can deduce that there is no checkerboard effect but that much noise remains in the restored image by $DM\&DN$ schemes with parameter $1.0\sigma_{0}$. The result of $DM\&1.5DN$ (Fig. 2 (d1) and (d2)) are smooth without checkerboard effects. Fig. 2 (e1) and (e2) correspond to the outputs of the CNN joint denoising and demosaicking method JCNN [20].

One can observe thin structures in the upper left corner of Fig 2 (a1), but they disappear in the restored image by $DN\&DM$. The proposed $DM\&1.5DN$ scheme restores them. The second column of Fig 2 illustrates a similar situation in which thin details are recovered by $DM\&DN$ and $DM\&1.5DN$ but not in the others.

In short, it appears that the $DM\&DN$ scheme with an appropriate parameter (namely $DM\&1.5DN$) outperforms the competition in terms of visual quality. This is due to the fact that it efficiently uses spatial and spectral image characteristics to remove noise, preserve edges and fine details. Indeed, contrary to the $DN\&DM$ schemes, $DM\&1.5DN$ does not reduce the resolution of the noisy image. Using an $DN\&DM$ scheme ends up over-smoothing the result. It comes to no surprise that JCNN performs slightly better than the other methods; however, it is much more computationally demanding and only works for $\sigma\leq 20$.

In a systematic comparison between the schemes involving CBM3D and nlBayes, schemes with CBM3D proved to perform slightly better. Furthermore, the $DM\&DN$ schemes with CBM3D are about four times faster than nlBayes. Hence, the following experiments are more focused on CBM3D.

To complete this comparison we went back to all $DN\&DM$ schemes proposed in the literature, and performed a systematic comparison for the two classic Kodak and Imax datasets. These databases are always used in demosaicking evaluations, because they illustrate different challenges of the demosaicking problem, Imax being difficult by its color contrast, and Kodak challenging for the recovery of fine structure. In Tables 6 and 7 we compare representative $DN\&DM$ methods from the literature with the best $DM\&DN$ methods identified above (all of them $DM\&1.5DN$):

• –

  The two best performing demosaicking before denoising methods ($DM\&1.5DN$) from on Table 5 are considered. Namely, RCNN for demosaicking followed by CBM3D (denoted RCNN+CBM3D) or nlBayes (RCNN+nlBayes) for denoising.

• –

  We also consider a ”low-cost” $DM\&1.5DN$ combination using MLRI [34] for demosaicking and CBM3D for denoising (MLRI+CBM3D).

The considered $DN\&DM$ methods from the literature are:

• –

  The BM3D-CFA filter was proposed in [14] to avoid the checkerboard effects resulting from independently applying BM3D to the color phases of CFA images. We evaluate BM3D-CFA [14] followed by Hamilton Adams demosaicking (denoted BM3D+HA), as well as followed by a state-of-the-art RCNN demosaicking [54] (BM3D+RCNN).

• –

  The CFA denoising framework of Park et al. [47] effectively compacts the signal energy while the noise is distributed equally in all dimensions by using a color representation from the principal components analysis of the pixel RGB values in the Kodak dataset and then removes noise in each channel by BM3D. This preprocessing is advantageous for the Kodak image set, but inadequate for the Imax image set. We evaluate this framework [47] with BM3D [12] followed by the RCNN demosaicking [54] (Park+RCNN).

• –

  The PCA-CFA filter proposed in [58] is a spatially-adaptive denoising based on principal component analysis (PCA) that exploits the spatial and spectral correlations of CFA images to preserve color edges and details. We evaluate PCA-CFA [58] followed by DLMM demosaiciking [59] (PCA+DLMM) and RCNN demosaicking [54] (PCA+RCNN).

• –

  Finally, as a reference, we include the deep learning based joint denoising and demosaicking (JCNN) of [20, 19]. But its results are only available for noise with $\sigma\leq 20$ because the network is not trained beyond that level.

From Tables 6 and 7 we see that the $DM\&DN$ method RCNN+CBM3D as well as RCNN+nlBayes yield the best results on the Kodak dataset, and the margin with respect to the best $DN\&DM$ method (BM3D+RCNN, i.e. BM3D-CFA [13] with RCNN [54]) is quite large: more than 1.5dB on average. In Fig. 4 we compare some results obtained on an image from the Kodak dataset. From the upper-left extract we can see that textures are better restored with RCNN+CBM3D and MLRI+CBM3D, while JCNN introduces some defects. From the extract we see that the $DM\&1.5DN$ methods preserve much more details than BM3D+RCNN, and the result is comparable to JCNN.

On the Imax database RCNN+CBM3D has the highest CPSNRs on high noise levels, by a small gap though. For low noise levels BM3D+RCNN is better, but the difference with RCNN+CBM3D is very small. The joint denoising-demosaicking network JCNN [20] yield the best results on the Imax dataset for $\sigma\leq 20$ (not trained above those levels) yet, the margin with respect to RCNN+CBM3D is again small. Overall, by looking at the average CPSNR we can say that the $DM\&1.5DN$ scheme RCNN+CBM3D is indeed much more robust than BM3D+RCNN.

We evaluated on a set of 14 raw images taken from the Small SIDD dataset [1]. For simplicity, the selected images correspond to phones from the same manufacturer. We adopted the simple pipeline proposed by the authors, which yields photo finished images that can be compared with the ground truth. The considered methods (RCNN+CBM3D, CBM3D+RCNN, and JCNN) were applied at the demosaicking stage (in linear space). Before any denoising step ($DN$) we applied a VST (squared root [5]), which whitens the noise, and invert it afterwards. The noise level was estimated using [11] and provided to the denoising algorithms and JCNN.

Table 8 reports the average CPSNR obtained on these images and the average of the estimated noise levels (after whitening). These values are consistent with the simulated results obtained on the Kodak database (Table 7). The result in Fig. 5, and the supplementary material, support the case in favor of the $DM\&1.5DN$ schemes (RCNN+CBM3D).