MIPI 2023 Challenge on RGBW Fusion: Methods and Results

The RGBW fusion task aims to fuse the DbinB and DbinC of RGBW (Fig. 1 (b)) to improve the image quality of the Bayer output. By incorporating the white pixels in DbinC of higher spatial resolution and higher SNR, the output Bayer would potentially have better image quality. In addition, the binning mode of RGBW is mainly used for the preview and video modes in smartphones, thus requiring the fusion algorithms to be lightweight and power-efficient. While we do not rank solutions based on the running time or memory footprint, the computational cost is one of the most important criteria in real applications.

The training data contains 70 scenes of aligned RGBW (DbinB and DbinC input) and Bayer (ground-truth) pairs. DbinB at 0dB is used as the ground truth for each scene. Noise is synthesized on the 0dB DbinB and DbinC data to provide the noisy input at 24dB and 42dB, respectively. The synthesized noise consists of read noise and shot noise, and the noise models are calibrated on an RGBW sensor. The data generation steps are shown in Fig. 3. The testing data includes DbinB and DbinC inputs of 15 scenes at 24dB and 42dB, and the ground truth Bayer results are hidden from participants during the testing phase.

The evaluation consists of (1) the comparison of the fusion output Bayer and the reference ground truth Bayer, and (2) the comparison of RGB from the predicted and ground truth Bayer using a simple ISP (the code of the simple ISP is provided). We use

1. 1.

   Peak Signal-to-Noise Ratio (PSNR)

2. 2.

   Structural Similarity Index Measure (SSIM) [15]

3. 3.

   Learned Perceptual Image Patch Similarity (LPIPS) [22]

4. 4.

   Kullback–Leibler Divergence (KLD)

to evaluate the fusion performance. The PSNR, SSIM, and LPIPS will be applied to the RGB from the Bayer using the provided simple ISP code, while KLD is evaluated on the predicted Bayer directly.

A metric weighting PSNR, SSIM, KLD, and LPIPS is used to give the final ranking of each method, and we will report each metric separately as well. The code to calculate the metrics is provided. The weighted metric is shown below. The M4 score is between 0 and 100, and the higher score indicates the better overall image quality.

For each dataset, we report the average score over all the processed images belonging to it.

The challenge consisted of the following phases:

1. 1.

   Development: The registered participants get access to the data and baseline code, and are able to train the models and evaluate their running time locally.

2. 2.

   Validation: The participants can upload their models to the remote server to check the fidelity scores on the validation dataset, and to compare their results on the validation leaderboard.

3. 3.

   Testing: The participants submit their final results, code, models, and factsheets.

This team presents an end-to-end joint remosaic and denoise model, referred to as DEDD. As illustrated in Fig. 6, the DEDD model is composed of three components: a denoising model, a main network, and a differentiable ISP model. Specifically, in the first part, they employed a basic UNet [10], which incorporated two downsampling operations. In the second part, they utilized the state-of-the-art model in the low-level domain, NAFNet [4]. The NAFNet contains the 4-level encoder-decoder and bottleneck. For the encoder, the numbers of NAFNet’s blocks for each level are 2, 4, 8, and 24. For the decoder, the numbers of NAFNet’s blocks for the 4 levels are all 2. In addition, the number of NAFNet’s blocks for the bottleneck is 12. In the third part, they reformulated the BLC, WBC, GAMMA, and CCM modules in the conventional ISP pipeline into differentiable models, and adopted the officially provided demosaic model as the demosaic module. In the training phase, the clean model is used as a guidance for boosting the noisy restoration performance. The images were randomly cut into 256$\times$256 patches, with a batch size of 64. The optimizer used was AdamW [9], and the initial learning rate was set to 0.001, which was reduced by half every 5000 iterations. The training process is divided into two stages; initially, the denoising network is trained for 40k iterations, after which the parameters of the denoiser are fixed and the demosaic network is trained. The loss function utilized is the L1 loss. When training the demosaic network, two supervision signals are incorporated: the constraint of the Bayer domain and the constraint of the RGB domain. The model training is completed after retraining 80K iterations in an end-to-end training mode.

This team proposes a novel two-stream pipeline based NAF-blocks [4] for RGBW joint fusion and denoise task, as shown in Fig. 7. Inspired by high signal-to-noise ratio white pixels, they introduce a new module called W-guided dynamic convolution(WGDC), which aggregates the spatial and channel attributes of the white-pixel feature, guides the dynamic change of network capability according to the white-pixel characteristics. Moreover, the authors design a method for synthesizing RGBW data, which effectively reduces the gap between synthetic data and real data. They use the official simple ISP code to transfer the standard Bayer to the RGB image for loss calculation, which consists of PSNR, SSIM, LPIPS [22]. For training, the authors randomly crop the training images into 128x128-sized patches with the 8-sized batches. Bayer Preserving Augmentation [8], Cutmix [19] are used for data augmentation. They use cosine decay strategy to decrease the learning rate to $1\times 10^{-7}$ with the initial learning rate $1\times 10^{-3}$ and the entire training costs about 5 days and converges after $4\times 10^{-6}$ iters. In the final inference stage, Test-time Augmentation [11] is used to get final result.

This team proposes an RGBW fusion and denoising method based on the existing image restoration model Restormer [20], as shown in Fig. 8. During training, the Pixel-Unshuffle [13] is firstly applied to RGBW images to split them from 2 channels into 8 channels. Then, the 8-channel RGBW images are fed into the Restormer, obtaining the output of 8 channels. Finally, the Pixel-Shuffle [12] restores the output of 8 channels to the standard Bayer format. The training loss function consists of L1 loss, KLD loss, and LPIPS loss [22], calculated on the output of 8 channels and GT Bayer. Moreover, they also utilize three data augmentation strategies for training, i.e., Bayer Preserving Augmentation [8], Cutmix [19], and Mixup [21]. The whole network is trained for $3\times 10^{5}$ iterations. The learning rate is decayed from $2\times 10^{-4}$ to $1\times 10^{-6}$ with a CosineAnnealing schedule. The training batch size and patch size are set to 8 and 224. The Self-ensemble strategy, Test-time Augmentation [11], and Test-time Local Converter [6] are applied during the testing phase. The testing batch size and patch size are set to 1 and 224.

This team proposes a multi-scale hybrid attention network for RGBW Fusion and denoising task as shown in Fig. 9. Inspired by Restormer [20] and HAT [5], the proposed method employs the Spatial Attention Module as the decoder (SAE), which is decoded by the Multi-Scale Decoder (MSD) via skip-connections. The hybrid attention transformer (HAT) is also used in this strategy to consider both global and local information. The combination of these techniques enables efficient processing of high-resolution images as well as the extraction of significant information. The training process includes two stages. In the stage $1$, the network is trained with the patches of size $256\times 256$. The batch size is set to $12$ and the optimizer is ADAM by setting of $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The learning rate is initialized as $10^{-4}$ and it is decayed by a factor of $0.5$ at $50,000$, $80,000$, and $100,000$ iterations. Thanks to the network in stage $1$ to greatly improve Bayer by using the white-channel information. The stage $2$ finetunes the network from stage $1$. In stage $2$, the network is trained with the $L_{1}$ loss and LPIPS loss on RGB domain. The learning rate is initialized as $5\times 10^{-5}$ and decayed by a factor of $0.5$ at $10,000$, $20,000$, and $40,000$ iterations. In the testing phase, the results are from the two stages to achieve the best performance. To be specific, the self-ensemble is used in the period of testing single model to get a better result. The input frame is flipped and regard it as another input. Then an inverse transform is applied to the corresponding output. An average of the transformed output and original output will be the self-ensemble result. The final result is a mixed results of the outputs from different iterations.