Joint Denoising and Demosaicking with
Green Channel Prior for Real-world Burst Images

Image denoising and demosaicking are two important steps in camera ISP pipeline. A few methods have been proposed for joint denoising and demosaicking on a single raw image (JDD-S) [9, 23, 10, 11, 12, 13, 14]. In [13], Qian et al. showed that the performance of JDD-S is generally better than performing denoising and demosaicking separately. In [9], a learning-based method was proposed for JDD-S. Henz et al. [10] proposed an auto-encoder architecture to model the color-image capturing process on each monochromatic sensor. Kokkinos et al. [12] proposed a plug-and-play framework for the JDD-S task. To enhance the performance on real-world images, Ehret et al. [11] proposed a mosaic-to-mosaic framework by finetuning the network using mosaic burst images. It should be noted that though Ehret et al. utilized burst images to fine-tune the network, the input of the network is still a single image. Liu et al. [14] proposed a self-guided JDD-S network by considering the advantages of the higher sampling rate of green channel and using this prior to guide the upsampling process. In this paper, we further analyze the noise level imbalance among different color channels in real-world photographs, and perform the JDD task using burst images instead of a single image.

Compared with single image restoration tasks, burst image processing encounters new challenges on estimating the offsets among different frames caused by camera movement and moving objects. According to the employed alignment frameworks, we partition burst image restoration methods into three categories, i.e., pre-aligned methods [17, 18], kernel-based methods [16, 24, 25, 26] and feature-based alignment [27, 28, 19, 20].

Pre-alignment methods mostly employ optical flow to estimate the motions and perform warping to compensate for temporal offset. The frame-to-frame method [18] utilizes the TV-$L_{1}$ algorithm [29] to estimate optical flow within frames. ToFlow [17] utilizes the SpyNet [30] as the flow estimation module, which is jointly trained with the denoising module. However, the restoration performance of those methods is largely affected by the accuracy of estimated optical flow, while accurate flow is difficult to obtain especially under large motion and severe noise. Kernel-based methods use convolutional neural networks to predict spatially varying kernels, which perform aligning and denoising simultaneously [16]. Compared with the original KPN [16], Xu et al. [24] and Marinc et al. [24] proposed to learn the deformable kernels and multiple kernels. Xia et al. [26] proposed to predict a set of global basis kernels and the corresponding mixing coefficients to effectively exploit larger denoising kernels.

Comparing with the above two categories of alignment methods, performing alignment in feature domain is a more promising strategy and has achieved SOTA performance on video super-resolution tasks. Liu et al. [27] proposed to use a localization net to estimate spatial transform parameters from deep feature and directly wrap the feature to align shift. TDAN [28] and EDVR [19] were proposed to estimate offset of deformable convolution which is utilized to align the shift in features domain. Comparing with [27] which needs the ground-truth information of spatial transform parameters, TDAN and EDVR do not need such parameters while still achieving SOTA performance. RviDeNet [20] also utilizes DConv to align multi-scale features for burst denoising. In this work, we perform alignment in feature domain and utilize deformable convolution to implicitly compensate for offsets. Moreover, we design an inter-frame module which not only utilizes multi-scale information, but also considers temporal constraint.

Our goal is to recover a clean full-color image, denoted by $x_{ref}^{rgb}$ , from a burst of real-world noisy CFA images, denoted as $\mathcal{Y}=\{y_{t}\}_{t=1}^{N}$. The subscript “ref” represents the index of reference frame. Usually, the noisy counterpart of $x_{ref}^{rgb}$ is the center frame in $\mathcal{Y}$.

The noise in real-world raw images is signal-dependent [31] due to the photon arrival statistics and the imprecision in readout circuitry. The noise introduced by photon sensing, i.e., shot noise, can be modeled as the Poisson distribution, while the noise introduced in readout circuitry, i.e., read noise, can be modeled by the Gaussian distribution. Denote by $x_{t}$ the desired clean raw image captured at time $t$. The corresponding noisy raw image $y_{t}$ can be written as [31]:

where $n(x_{t},\sigma_{s},\sigma_{r})\sim\mathcal{N}(0,\sigma_{s}x_{t}+\sigma_{r}^{2})$ and $\sigma_{s}$ and $\sigma_{r}$ are the scale parameters for shot noise and read noise, respectively. $\mathcal{N}$ represents Gaussian distribution.

As discussed in [22], the CMOS sensor has different sensitivity to light of different wavelengths or colors, and in most illumination conditions, green channels are brighter than red and blue channels in Bayer pattern CFA images. Since the real-world noise contains the Poissonian shot noise (see Eq. 1), and the signal-to-noise ratio (SNR) has a square root relationship between signal and noise, the brighter green channel often has a higher SNR than red/blue channels. To validate this, we compute the average SNR of different color channels of 50 real-world noisy raw images randomly chosen from the SIDD [15] benchmark dataset (which contains high-ISO noisy images captured by smartphone cameras), and show the SNR comparison in Fig. 1. One can see that the SNR of green channel is higher than that of red/blue channels for most of the noisy images. In addition, the green channel has twice the sampling rate of red/blue channels. Overall, the green channel preserves better image structure and details than the other two channels. In this paper, we call the prior knowledge that the green channel has higher SNR, higher sampling rate and hence better channel quality than red/blue channels the green channel prior (GCP), which is carefully exploited in this paper to design our JDD-B network.

With GCP, we propose a new network, namely GCP-Net, for JDD-B. Without loss of generality, we assume that the Bayer CFA pattern is used. For each raw image $y_{t}$, we reshape it as four R, G, G, B sub-images of the same size so that $y_{t}\in\mathbb{R}^{H\times W\times 4}$. We denote the noise map of $y_{t}$ as $m_{t}\in\mathbb{R}^{H\times W\times 4}$, whose value at each location is the standard deviation of signal-dependent noise at that position. The input of our GCP-Net is a sequence of noisy raw images $\mathcal{Y}=\{y_{t}\}_{t=1}^{N}$ and their corresponding noise maps $\mathcal{M}=\{m_{t}\}_{t=1}^{N}$. The output is the clean full-resolution linear RGB image $\hat{x}_{ref}^{rgb}\in\mathbb{R}^{2H\times 2W\times 3}$.

The overview of GCP-Net is illustrated in Fig. 3. GCP-Net consists of two branches, i.e., a GCP branch and a reconstruction branch. In the GCP branch, the green features $f_{t}^{g}$ are extracted from the concatenation of noisy green channels, denoted by $y_{t}^{g}$, and their noise level maps, denoted by $m_{t}^{g}$. This process can be written as:

where $M_{g}$ consists of several Conv+LReLU blocks. These GCP features are utilized as the guided information for the reconstruction branch. We utilize layer-wise guiding strategy and denote the GCP feature of the $l$-th layer as $f_{t,l}^{g}$.

The reconstruction branch utilizes the burst images, the corresponding noise maps and the GCP features to estimate the clean full color image. As illustrated in Fig. 3, it consists of three parts: the intra-frame (IntraF) module, the inter-frame (InterF) module and the merge module. The IntraF module is designed to model the deep features of each frame and it utilizes GCP features to guide the feature extraction. The InterF module is to compensate for the shift between frames by using the DConv in feature domain. To reduce the influence of noise in alignment, offset is estimated from the cleaner GCP features. The merge module is designed to aggregate the aligned features and use the GCP features to perform adaptive upsampling for the full-resolution image reconstruction. The details of these modules are presented in the following sections.

The architecture of the IntraF module is shown in Fig. 4. For the $t$-th frame, the input of IntraF includes the noisy raw image $y_{t}$, the corresponding noise level map $m_{t}$ and the GCP feature $f_{t}^{g}$. Firstly, one simple convolution layer $M_{0}$ is used to model the initial features $f_{t}^{0}$ as $f_{t}^{0}=M_{0}(y_{t},m_{t})$. Then, the initial features $f_{t}^{0}$ are passed to the concatenation of four green channel attention (GCA) blocks, where the GCP features are used to guide the feature extraction and a dual attention mechanism is designed to better deal with the channel-dependent and spatial-dependent noise. We adopt a layer-wise guiding strategy for GCA blocks and empirically find that such a strategy is favorable to the restoration results. Without loss of generality, we use the $l$-th GCA block to present the modeling process. The output features are denoted as:

where $M_{gca}^{l}$ represents the $l$-th GCA block.

The detailed structure of the GCA block is shown in Fig. 5. Inspired by [32, 33], the GCP information can be exploited by using pixel-wise scaling and bias, and the enhanced feature $f_{t,l}^{e}$ can be expressed as:

where $\gamma(f_{t,l}^{g})$ and $\beta(f_{t,l}^{g})$ are two learned modulation parameters of the guided layers. We denote the unit to implement Eq. 4 as the green guided (GG) unit. The green-guided features are estimated by two residual blocks, denoted by $M_{r}$,

where $f_{t,l}^{r}$ is the learned features. As normal Conv layers treat spatial and channel features equally, it is not appropriate to handle the real-world noise which is channel and spatial dependent. To further enhance the representational power of standard Conv+ReLU blocks, channel attention and spatial attention [34, 35, 36] are designed to model the cross-channel and spatial relationship of deep features.

The extracted features $\{f_{t}\}_{t=1}^{N}$ by the IntraF module are then aligned to the reference frame feature $f_{ref}$ in the InterF module. The InterF module aims at modeling the temporal dependency between frames, whose architecture is shown in Fig. 6. We use the deformable convolution to compensate for the offset within frames. To relieve the affect of severe noise and to better model the correlation between neighboring frames, we use the GCP features $f_{t}^{g}$ to estimate the offset. Similar to EDVR [19] and RViDeNet [20], pyramidal processing is utilized to handle possible large motions. Moreover, to better exploit the temporal constraint in the offset estimation, we introduce an LSTM regularization in the offset estimation.

For each pyramidal scale $s$ of the $t$-th frame, the inter-frame GCP feature, denoted by $f_{t,i}^{s}$, is obtained by using

where $[\cdot,\cdot]$ is the concatenation operator and $g$ is the Conv layer. Then, the temporal regularization is introduced by using ConvLSTM [37], which is a popular 2D sequence data modeling method. The ConvLSTM updates the hidden state $h_{t}^{s}$ and the cell state $c_{t}^{s}$ with:

The updated inter-frame feature $f_{t,i}^{s}$ can be written as:

As discussed in [38], the LSTM mechanism has limited ability to deal with complex motions. To handle large motions, multi-scale information is aggregated to estimate more accurately the offset $\Delta p_{t}^{s}$:

where $g_{1}$ and $g_{2}$ are two convolutional layers, $(\cdot)^{\uparrow 2}$ is the upsampling operator with factor 2.

The aligned features $f_{t}^{s}$ at each position $p_{0}$ can then be obtained by:

in which $K$ is the sampling location of deformable convolution kernel, $\Delta m_{t}^{s}$ is the modulation scalar. Following [39], since $\Delta p_{t}^{s}$ is fractional, bilinear interpolation is applied. The final aligned feature $f_{t}^{a,s}$ at scale $s$ is obtained by:

where $M_{g}$ refers to general Conv+LReLU layers.

The merge module is designed to merge the aligned features $f_{t}^{a}$ and output the estimated clean RGB image $\hat{x}_{ref}^{rgb}$. The aligned features are firstly concatenated and adaptively merged as follows:

where $M_{m}$ is the merge function by using one simple convolution layer. Then we upsample the features $f^{m}\in\mathcal{R}^{H\times W\times C}$ to full-resolution features $f^{m}_{f}\in\mathcal{R}^{2H\times 2W\times C}$ using GCP adaptive upsampling. Similar to the green guided operator in GCA block, the GCP adaptive upsampling can be expressed as:

where $f_{g}$ are the green guided features of size $H\times W\times C$, $(\cdot)^{\uparrow s}$ represents the upsampling operator by a factor $s$, $\gamma(f_{g})$ and $\beta(g_{g})$ are the two learned modulation parameters. Transpose convolution is used for the upsampling interpolation.

The final estimation $\hat{x}_{ref}^{rgb}$ can be written as:

where $M_{U}$ is designed to estimate the final clean RGB image. In this paper, we utilize a three scale U-Net [40] architecture for $M_{U}$ to exploit the multi-scale information as well as enlarging the receptive field. All the Conv kernels are of size $3\times 3$, followed by the nonlinear function LReLU. The upsampling and downsampling operators in $M_{U}$ are strided convolutions and transpose convolutions.

For the estimated clean $\hat{x}_{ref}^{rgb}$, we define the reconstruction loss in the linear color space as follows:

where $\sqrt{(\hat{x}-x)^{2}+\epsilon^{2}}$ is the Charbonnier penalty function [41], $\epsilon$ is set to $1\times 10^{-3}$. As discussed in KPN [16], computing loss in sRGB color space can produce a perceptually more relevant estimation. Therefore, we also introduce a loss in the sRGB color space:

where $\Gamma(\cdot)$ is the operator which transforms linear RGB color space to sRGB space. In this paper, $\Gamma(\cdot)$ contains white balance, color correction and gamma compression as in [21]. To sum up, the overall loss to optimize our model is:

where $\lambda$ is the trade-off parameter and we simply set to 1 in our experiments.

Obtaining ground-truth images for training is difficult for real-world image restoration tasks. In some single-image based restoration works [42, 15, 13], real-world degraded data are collected and the corresponding ground-truth images are physically and/or mathematically estimated for pair-wise training. However, for the burst-images based JDD-B task, the misalignment problem and the coherence between denoising and demosaicking make the ground-truth estimation much more difficult. Therefore, we synthesize training data by using an open high quality video dataset, i.e., Vimeo-90K [17].

Since camera sensor outputs are in the linear color space, we first convert the sRGB images $x$ into linear RGB space $x^{rgb}$ by using the unprocessing operation in [21], which can be written as $x^{rgb}=\Gamma^{-1}(x)$. The unprocessing operation $\Gamma^{-1}(\cdot)$ includes inverse gamma compression, inverse color correction, inverse tone mapping and inverse white balance. The converted linear RGB frame $x^{rgb}$ is taken as the ground-truth image. By using Eq. 1, the noisy raw image $y_{t}$ can be synthesized as:

where $M(\cdot)$ is the mosaic matrix which downsamples a linear RGB image to a Bayer CFA image. Without loss of generality, the RGGB mosaic pattern is used as $M(\cdot)$ to generate the data.

In our experiments, a number of $N=5$ neighboring frames are used as the input and the central frame is chosen as the reference frame. Following the setting in [16], the noise level parameters $\sigma_{s}$ and $\sigma_{r}$ in Eq. 1 are uniformly sampled from the ranges of $[10^{-4},10^{-2}]$ and $[10^{-3},10^{-1.5}]$, respectively. We adopt the method in [43] to initialize the GCP-Net and use the ADAM [44] algorithm with $\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.99 to update the network. The size of mini-batch is 2 and the size of each noisy raw patch is $64\times 64$ with 4 color channels (RGGB). The reconstructed RGB patch is of size $128\times 128$ with three color channels (RGB). The learning rate is initialized as $4\times 10^{-4}$ and it is decreased using the cosine function [45]. It takes about two days to train our model under the PyTorch framework using two Nvidia GeForce RTX 2080 Ti GPU.

In this section, we perform ablation studies to discuss the effect of major components in GCP-Net and the setting of some parameters. The Vid4 [46] and REDS4 [19] datasets are used in the experiments.

Since currently there is no JDD-B method publically available, for fair comparison, we combine the representative burst image denoising methods with a state-of-the-art demosaicking method, DemosaicNet (DMN) [9], to compare with our GCP-Net. We choose four widely used burst denoising algorithms: VBM3D [48], KPN [16], EDVR [19] and RViDeNet [20]. For VBM3D, the noise level is needed as the input, and we use the method in [49] for noise level estimation. For the KPN, EDVR and RViDeNet models, they are retrained using the same training data as GCP-Net. The retrained EDVR and RViDeNet models adopt 20 residual blocks to perform feature extraction and 40 residual blocks in the reconstruction module. The size of learned per-pixel kernel of retrained KPN is 7 x 7.

We also adjust the structures of EDVR and RViDeNet by adding one upsampling operator in the reconstruction step, and train these models for the JDD-B task. These models are denoted as EDVR* and RViDeNet*, respectively. In addition, we compare with two state-of-the-art JDD-S algorithms: FlexISP [7] and ADMM [8].

Two video datasets, i.e., Vid4 [46] and REDS4 [19], are adopted in the experiments. Vid4 is widely used as the test set in the study of video super-resolution, and the video clips (resolution: $480\times 720$) in Vid4 have small motion. The videos in REDS4 dataset have better quality and resolution (720p) but with bigger motions. Video clips are firstly converted to raw space using the unprocessing operator $\Gamma^{-1}(\cdot)$ introduced in Sec IV-A. Then the noise is added to the raw images by using Eq. 1. Tables IV and V list the PSNR/SSIM results of different algorithms under different noise levels. Following [16], both PSNR and SSIM are computed after gamma correction to better reflect perceptual quality. One can see that the proposed GCP-Net achieves the best PSNR/SSIM measures. Visual comparisons on Vid4 and REDS4 are presented in Fig. 8 and Fig. 9, respectively.

From Tables IV and V, we can see that the performance of JDD-S methods FlexISP and ADMM is generally far below the learning based multi-frame algorithms. However, the multi-frame based VBM3D+DMN fails to compete with the single frame based ADMM, especially on the REDS4 dataset. This is mainly because VBM3D+DMN assumes AWGN and it cannot handle the large-motion in the videos of REDS4. The CNN-based methods KPN+DMN, EDVR+DMN and RviDeNet+DMN achieve much better performance than JDD-S methods because they can learn to handle the misalignments between adjacent frames and exploit the temporal redundancy for denoising. Nonetheless, one can still see that these methods generate noticeable color artifacts and zippers (see Figs. 8(b)(c)(e) and Figs. 9(b)(c)(e)) near edges and complex textures. This is mainly because they perform burst denoising and color demosaicking separately without considering the correlations between the two tasks. For JDD-B algorithms, i.e., EDVR* and RviDeNet*, their restoration results contain less zippering and color artifacts compared with EDVR+DMN and RviDeNet+DMN, which proves the effectiveness of jointly handling denoising and demosaicking task. However, the results of EDVR* and RviDeNet* suffer from the over-smoothing problem (see Figs. 9(d)(f)). Benefiting from the GCP guidance, our GCP-Net performs the best in the JDD task, making good balance between noise removal and structure preservation (see Fig. 9(g)).

We also compare different JDD algorithms using real-world burst raw images captured by several smartphone cameras, including iPhone 7, iPhone X and Pixel 2. Since there is no ground-truth for the collected images, we can only provide qualitative comparison. The restored images are converted to sRGB domain for visualization by using the ISP operations, including white balance, color transfer and gamma compression. The white balance parameters, color matrix and the noise level are collected from the camera metadata.

In Fig. 10, we show the JDD results of several noisy images captured by the three smartphone cameras under normal lighting conditions as well as nighttime environments with a wide range of ISO values. Similar conclusions to the synthetic experiments can be made. KPN+DMN, EDVR+DMN and RviDeNet+DMN can remove noise but will produce zippering artifacts and smooth the image details. By jointly performing denoising and demosaicking, EDVR* and RviDeNet* can reduce the zippering effect but still produce over-smoothed reconstruction. In the zoom-in patch of the second image in Fig. 10, we can also see that RviDeNet* generates artifacts in the high noisy area. Our proposed GCP-Net can effectively remove noise while retaining fine textures. It is also free of the moire pattern. In the last row of Fig. 10, we show the JDD results on night-shot burst images with large motion. We can see that the restoration result of KPN+DMN contains much ghosting artifact, which is mainly caused by the large motion of object (i.e., car) in the burst images. RviDeNet* also generates some motion induced artifacts around the moving objects. The proposed GCP-Net can work stably under such large-motion scenes. This is because we utilize the pyramid offset estimation and the offset is estimated on GCP features which dilute the impact of noise.

Table VI lists the number of model parameters and the number of floating point operations (FLOPs) of our GCP-Net and the comparison methods, i.e., DMN, KPN, EDVR and RviDeNet. The FLOPs are calculated on 5 frames of size $128\times 128$. Because of the GCP branch and some attention modules, GCP-Net has more parameters than DMN, KPN and EDVR, but its model is much smaller than RviDeNet. Though the number of parameter and FLOPs of GCP-Net are 2 times and 1.5 times that of EDVR, it achieves significantly better JDD results than EDVR ($\sim$3.5dB on Vid4 and $\sim$2dB on REDS4). Since RviDeNet utilizes a pre-denoising network and non-local modules, it has very high computational cost but achieves lower JDD performance than GCP-Net. Overall, GCP-Net achieves a good balance between JDD effectiveness and efficiency.