MIPI 2022 Challenge on Under-Display Camera Image Restoration: Methods and Results

Inspired by [6, 10, 34], this team designs an enhanced multi-inputs multi-outputs network, which mainly consists of dense residual blocks and the cross-scale gating fusion modules. The overall architecture of the network is depicted in Figure 1. The training phase could be divided into three stages: i) Adopt the Adam optimizer with a batch size of $3$ and the patch size of $256\times 256$. The initial learning rate is $2\times 10^{-4}$ and is adjusted with the Cosine Annealing scheme, including $1000$ epochs in total. ii) Adopt the Adam optimizer with a batch size of $1$ and the patch size of $512\times 512$. The initial learning rate is $2\times 10^{-5}$ and is adjusted with the Cosine Annealing scheme, including $300$ epochs in total. iii) Adopt the Adam optimizer with a batch size of $2$ and the patch size of $800\times 800$. The initial learning rate is $8\times 10^{-6}$ and is adjusted with the Cosine Annealing scheme, including $150$ epochs in total. During inference, the team adopts model ensemble strategy averaging the parameters of multiple models trained with different hyperparameters, which brings around 0.09 dB increase on PSNR.

This team designs a UNet-like structure (see Figure 2), making full use of the hierarchical multi-scale information from low-level features to high-level features. To ease the training procedure and facilitate the information flow, several residual blocks are utilized in the base network, and the dynamic kernels in the skip connection bring better flexibility. Since UDC images have different holistic brightness and contrast information, the team applies the condition network with spatial feature transform (SFT) to deal with input images with location-specific and image-specific operations (mentioned in HDRUNet [4]), which could provide spatially variant manipulations. In addition, they incorporate the information of the point spread function (PSF) provided in DISCNet [9], which has demonstrated its effectiveness through extensive experiments on both synthetic and real UDC data. For training, the authors randomly cropped $256\times 256$ patches from the training images as inputs. The mini-batch size is set to $32$ and the whole network is trained for $6\times 10^{5}$ iterations. The learning rate is initialized as $2\times 10^{-4}$, decayed with a CosineAnnealing schedule, and restarted at $[50K,150K,300K,450K]$ iterations.

This team proposes Multi-Refinement Network (MRNet) for Image Restoration on Under-Display Camera, as shown in Figure 3. In this challenge, the authors modify and improve the MRNet [1], which is mainly composed of 3 modules: shallow Feature Extraction, reconstruction module, and output module. The shallow Feature Extraction and output module only use one convolution layer. Multi-refinement is the main idea of the reconstruction module, which includes $N$ Multi-scale Residual Group Module (MSRGM) and progressively refines the features. MSRGM is also used to fuse information from three scales to improve the representation ability and robustness of the model, where each scale is composed of several residual group modules (RGM). RGM contains $G$ residual block module (RBM). In addition, the authors propose to remove Channel Attention (CA) module in RBM [38], since it brings limited improvement but increases the inference time.

As shown in Figure 4, this team addresses the UDC image restoration problem with an Residual Dense Network (RDN) [39] as the backbone. The authors also adopt the strategy of multi-resolution feature fusion in HRNet [25] to improve the performance. In order to facilitate the recovery of high dynamic range images, the authors generate 6 non-HDR images with different exposure ranges from the given HDR images, where the value ranges of each split are: [0, 0.5], [0, 2], [0, 8], [0, 32], [0, 128], [0, 500], and normalize them and then stack into 18 channels of data for training. According to the dynamic range of the data, 2000 images in the training datasets (the remaining 16 are used as validation sets) are divided into two datasets: i) Easy sample dataset: 1318 images whose intensities lie in [0, 32]. ii) Hard sample dataset: easy samples plus doubled remaining hard samples. They use two datasets to train two models, each of which is first trained with 500k iterations. Then they fine-tune the easy sample model and the hard sample model by 200K iterations. In the validation/test phase, the 18-channel data is merged into 3-channel HDR images and fed to the network.

Considering the specificity of the UDC image restoration task (mobile deployment), this team designs a lightweight and real-time model for UDC image restoration using a simple UNet architecture with magic modifications as shown in Figure 5. Specifically, they combine the full-resolution network FRC-Net [40] and the classical UNet [24] to construct the model, which presents two following advantages: 1) directly stacking the residual blocks at the original resolution to ensure that the model learns sufficient spatial structure information, and 2) stacking the residual blocks at the downsampled resolution to ensure that the model learns sufficient semantic-level information.

During training, they first perform tone mapping, and then perform a series of data augmentation sequentially, including: 1) random crop to $384\times 384$; 2) vertical flip with probability $0.5$; 3): horizontal flip with probability $0.5$. They train the model for $3000$ epochs on provided training dataset with an initial learning rate $1e-4$ and batch size 4.

This team reformulates UDC image restoration as a Maximum Posteriori (MAP) estimation problem with the learned Gaussian Scale Mixture (GSM) models. Specifically, the $x$ can be solved by

where $\textbf{A}^{T}$ denotes the transposed version of A, $\delta$ denotes the step size, $t$ denotes the $t^{th}$ iteration for iteratively optimizing $\boldsymbol{x}$, $\boldsymbol{w}^{(t)}$ denotes the regularization parameters, and $\boldsymbol{u}^{(t)}$ denotes the mean of the GSM model. In [11], a UNet was used to estimated $\boldsymbol{w}^{(t)}$ and $\boldsymbol{u}^{(t)}$ and two sub-networks with 4 Resblocks and 2 Conv layers were used to learn A and $\textbf{A}^{T}$, respectively. For UDC image restoration, they develop a network based on Swin Transformer to learn the GSM prior (i.e., $\boldsymbol{w}^{(t)}$ and $\boldsymbol{u}^{(t)}$) and use 4 Conv layers, respectively. As shown in Figure 6, the team constructs an end-to-end network for UDC image restoration. The transformer-based GSM prior network contains an embedding layer, $4$ Residual Swin Transformer Blocks (RSTB) [16], two downsampling layers, two upsampling layers, a $w$-generator, and an $u$-generator. The embedding layer, the $w$-generator, and the $u$-generator are a $3\times 3$ Conv layer. The RSTB [16] contains 6 Swin Transformer Layers, a Conv layer, and a skip connection. The features of the first two RSTBs are reused by two skip connections, respectively. For reducing the computational complexity, they implement the united framework Eq. 1 with only one iteration and use the input $\boldsymbol{y}$ as the initial value $\boldsymbol{x}^{(0)}$.

The loss function $L$ is defined as

where $\boldsymbol{x}$ and $\boldsymbol{\hat{x}}$ denote the label and the output of the network, $\Delta$ represents the Laplacian operator, and $\mathcal{FFT}$ is the FFT operation.

This team takes model [6, 35] as a backbone network and introduces the multi-scale design. Besides, they reconstruct each color channel by a complete branch instead of processing them together. The overall architecture of the Multi-Scale and Separable Channels reconstruction Network (MSSCN) for UDC image restoration is shown in Figure 7. The network takes the multi-scale degraded images (1x, 0.5x, 0.25x) as inputs, and the initial feature maps of each color channel in each scale are extracted respectively. Before extracting feature maps, they use the discrete wavelet transform (DWT) [17] to reduce the resolution in each scale and improve the reconstruction efficiency. In the fusion module, this team fuses the feature maps in different scales in the same spatial resolution, concatenates the feature maps from each color channel and each scale, and processes them using a depth-wise convolution. In the feature extraction module, any efficient feature extraction block can be used to extract feature maps. Residual group (RG) with multiple residual channel attention blocks (RCAB) [38] is used as the feature extraction module. To improve the learning capacity of recovering images, they repeat the feature fusion module and feature extraction module $k$ times. Then the reconstructed images are generated by the inverse discrete wavelet transform (IWT).

As shown in Figure 8, this team designs a dense U-Net for under-display camera image restoration. The architecture aggregates and fuses feature information with Dense RCAB [38]. During training, batch size is set to 4 with the patch of $384\times 384$ and the optimizer is ADAM [14] by setting $\beta_{1}=0.9$, $\beta_{2}=0.999$. The initial learning rate is $1\times 10^{-4}$. They use Charbonnier loss to train models and stop training when no notable decay of training loss is observed.

This team presents a Wiener Guided Coupled Dynamic Filter Network, which takes advantage of both DWDN [8] and BPN [31]. As shown in Figure 9, on top of the U-net-based backbone, they first used feature level wiener for global restoration with the estimated degradation kernel, then used the coupled dynamic filter network for local restoration. The network obtains the global and local information from the feature level wiener deconvolution and the coupled dynamic filter network, and thus achieves a better restoration result since the UDC degradation was modeled with higher accuracy. Wiener Guided Coupled Dynamic Filter Network was jointly trained under the supervision of L1 loss and perceptual loss [13].

This team proposes the U-RRDBNet for the task of UDC image restoration. UDC images suffer from the strong glare caused by the optical structure of OLED display, which usually occurs and degrades a large area around the light source (e.g.street lamp, sun) in the view. Therefore a larger receptive field is required to deal with this task. The proposed architecture and processing pipeline are illustrated in Figure 10. Though the original U-net and its variants have achieved promising results on segmentation tasks, the capability of representation learning of the U-net is still limited for dense prediction in low-level tasks. So the authors apply the Residual-in-Residual Dense Block (RRDB) [28] to substitute simple Conv layers in U-net after each down/up-sampling operation to increase the network capability. They first perform the modified Reinhard tone mapping, and then feed the tone-mapped image into the end-to-end U-RRDBNet for restoration, and produce the image in the tone-mapped domain with artifacts removed. The output of the network is then transformed back to the HDR domain using the inverse of Reinhard tone mapping. The model is first trained using $L_{1}$ loss and perceptual loss [13] in the tone-mapped domain, and then fine-tuned with MSE loss for higher PSNR performance.

This team designs a deep learning model for the UDC image restoration based on the U-shaped network. In this network, they incorporate the convolution operation and attention mechanism into the transformer block architecture, which has been proved effective in image restoration tasks in [2]. Since there is no need to calculate the self-attention coefficient matrices, the memory cost and computation complexity can be reduced significantly. The overall network structure is shown in Figure 11. Specifically, they substitute the simplified channel attention module with the spatial and channel attention module [3] to additionally consider the correlation of the spatial pixels. They train the model with $L$1 and perceptual loss on the tone-mapped domain.

This team follows [15] and presents a two-branch network to restore UDC images. As shown in Figure 12, the input images are processed and fed to the restoration network. The original resolution image is processed in one branch while the blurred image with noise is processed in another one. Finally, the two branches are connected in series by the affine transformation connection. Then, they follow [20] and modify the high-frequency reconstruction branch with an encoder-decoder structure to reduce parameters. And they converted the smoothed diluted residual block into IMD block [12] so that the signals can propagate directly from skip connections to the bottom layers.

This team develops a Self-Guided Coarse-to-Fine Network (SG-CFN) to progressively reconstruct the degraded under-display camera images. As shown in Figure 13, the proposed SG-CFN consists of two branches, the restoration branch and the condition branch. The restoration branch is constructed by Feature Extraction Module (FEM) based on improved RSTBs [16], which incorporates the paralleled Central Difference Convolution (CDC) with Swin Transformer Layer (STL) to extract rich features at multiple scales. The condition branch is constructed by Fast Fourier Convolution (FFC) blocks [5] that endowed with global receptive field to model the distribution of the large-scale point-spread function (PSF), which is indeed the culprit of the degradation. Furthermore, the multi-scale representations extracted from the restoration branch are processed via the Degradation Correction Module (DCM) to restore clean features, guided by the corresponding condition information. To fully exploit the restored multi-scale clean features, they enable the asymmetric feature fusion inspired by [6] to facilitate the flexible information flow. Images captured through under-display camera typically suffer from diffraction degradation caused by large-scale PSF, resulting in blurred and detail attenuated images. Thereby, (1) incorporating CDC into the restoration branch can greatly help to avoid over-smooth feature extraction and enrich the representation capability, (2) constructing the condition branch with FFC blocks endow the global receptive field to capture the degradation information, and (3) correcting the degraded features from both local adaptation and global modulation makes the process of restoration more effective. During the testing phase, they adopt self-ensemble strategy and it brings a 0.63 dB performance gain on PSNR.

This team reproduced the MIRNetV2 [36] on the UDC dataset. The network utilizes both spatially-precise high-resolution representations and contextual information from the low-resolution representations. The multi-scale residual block contains: (i) parallel multi-resolution convolution streams to extract multi-scale features, (ii) information exchange across streams, (iii) non-local attention mechanism to capture contextual information, and (iv) attention-based multi-scale feature aggregation. The approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.

This team proposes a dual-branch lightweight neural network. Inspired by recent camera modeling approaches [7], they design a dual-branch model, depending on the tradeoff between resources and performance. As shown in Figure 14, the method combines ideas from deblurring [21] and HDR [19] networks, and attention methods [18, 29]. This does not rely on metadata or extra information like the PSF. The main image restoration branch with a Dense Residual UNet architecture [39, 24]. They use an initial CoordConv [18] layer to encode positional information. Three Encoder blocks, each formed by 2 dense residual layers (DRL) [39] followed by the corresponding downsampling pooling. The decoder blocks D₁ and D₂ consist of a bilinear upsampling layer and 2 DRL The decoder block D₃ does not have an upsampling layer, consists of 2 DRL and a series of convolutions with 7, 5, and 1 kernel sizes, the final convolution (cyan color) produces the residual using a tanh activation. Similar to [19], the additional attention branch aims to generate an attention map (per channel) to control the hallucination in overexposed areas. The attention map is generated after applying a CBAM block [29] on the features of the original image, and the final convolution is activated using a Sigmoid function.

This team implements the network architecture by Restormer [33]. Instead of exploring new architecture for under-display camera restoration, they attempt to mine the potential of the existing method to advance this field. Following the scheme of [33], they train an Restormer in a progressive fashion with $L_{1}$ loss for $300k$ iterations, with $92k$, $64k$, $48k$, $36k$, $36k$, and $24k$ iterations for patch size 128, 160, 192, 256, 320, and 384, respectively. Then they fix the patch size to $384\times 384$ and use a mask loss strategy [30] to train the model for another $36k$ iterations, where saturated pixels are masked out. After that, inspired by self-training [32], they add Gaussian noise to the well-trained model and repeat the training process.

Dual Branch Wavelet Net (DBWN) [17] that is proposed to restore the images degraded in an under-display camera imaging system. The overall network structure is similar to [22], where the network branches restore the high and low spatial frequency components of the image separately.

This team adopted TransWeather [27] model to deal with the image restoration problem for UDC images. Specifically, they combine different loss functions including $L_{1}$ loss and $L2$ loss. They train the model from scratch and it took approximately 17 hours with two 2080Ti GPUs.

The team proposes an Hourglass-Structured Fusion Network (HSF-Net) as shown in Figure 15. They start from a coarse-scale stream, and then gradually repeat top-down and bottom-up processing to form more coarse-to-fine scale streams one by one. In each stream, the residual dense attention modules (RDAMs) are introduced as the basic component to deeply learn the features of under-display camera images. Each RDAM contains $3$ dense residual blocks (RDBs) in series with channel-wise attention (CA). In each RDB, the first 4 convolutional layers are adopted to elevate the number of feature maps, while the last convolutional layer is employed to aggregate feature maps. The growth rate of RDB is set to 16.