Unleashing the Power of Self-Supervised Image Denoising: A Comprehensive Review

Soltanayev et. al [132] proposed a self-supervised image denoising method based on Therorem of Monte-Carlo Stein’s unbiased risk estimator (MC-SURE [134]) as Eq. (4, 5):

where $\tilde{n}\sim\aleph_{0,1}\in\mathbb{R}^{K}$ be independent of $n$ and $y$, and provided that $h(y)$ admits a well-defined sencond-order Taylor expansion. If not, this is still valid in the weak sense provided that $h(y)$ is tempered. So,

where $\epsilon$ is a fixed small positive value, $t$ is the transpose operator. The SURE-based method proposes to incorporate MC-SURE [134] into a stochastic gradient-based optimization algorithm for image denoising. An unbiased estimator is proposed to replace the unknown Bayesian risk for training, and with a carefully defined cost function, a learning-based image denoiser can be trained using this SURE-based method without requiring a noiseless ground truth. In addition, other deep learning-based denoisers can also utilize the MC-SURE-based training by modifying their cost function to a stochastic gradient-based optimizing format, as long as it satisfies the MC-SURE. This method assumes Gaussian noise with known variance in all simulations, but it can incorporate a variety of noise distributions like Poisson distribution and exponential families, potentially making it applicable to different applications in the measurement domain. However, the method is sensitive to hyper-parameters, and the neural network needs to be retrained if the underlying noise model changes.

C2N [137] framework trains a noise generator network $G$ to create a realistic noise map $\hat{n}$ for a given clean image $x$, which produces pseudo-noise images $\hat{y}$ as follows:

The 32-dim random vector $r$ is sampled from $\mathcal{N}(\bf{0},\bf{I^{2}})$ to reflect the stochastic behavior of noise according to the conditions of each scene, and authors use a new generator architecture to extract the diverse and complex noise distributions combining the Signal-Independent Pixel-Wise Transform, Signal-Dependent Pixel-Wise Transform and Spatially Correlated Transforms. A discriminator network $D$ distinguishes whether a given noisy image is synthesized from the generator $G$ or sampled from the real-world dataset. The two networks $G$ and $D$ are optimized adversarially with the Wasserstein distance [135] to learn the noise distribution as follows:

$P_{N}$ and $P_{C}$ denote the distribution of the real-world noisy and clean images, respectively. The real noisy image $y^{\prime}$ is sampled from $P_{N}$. Prime notation on $y$ denotes that we use noisy image unpaired to clean image $x$. $P_{r}$ is the distribution of random vector $r$. The authors use a stabilizing loss term $L_{stb}$ to prevent the color-shifting problem, which is defined as follows:

where $N$ denotes the number of pixel $i$ in mini-batch $B$ and $c$ is index of each color channels. They minimize this loss to make the channel-wise average of the generated noise approach zero. For denoising, the C2N framework uses pseudo-noise images $\hat{y}$ to train a denoising model in a supervised manner with $L_{1}$ loss. However, this denoising method may not perfectly reflect real-world noise, leading to potential deviations in practical applications.

Noisy-As-Clean (NAC) [93] assumes the noise in the image is weak enough, so the noise image can be treated as a clean image for denoising model training. NAC uses paired noise images $(y,z)$ generated by the clean image $x$ and synthetic noise $n_{s}$ that has the same type of noise as the natural weak noise $n_{0}$. The input $y$ and the target $z$ can be described as:

$E(\cdot)$ denotes the expectation and $Var(\cdot)$ denotes the variance. Hence, the noisy image $y$ should have similar expectation with the clean image $x$,

then,

so,

As can be seen from the above, in theory, when the noise in the image is weak enough, it can be treated as a clean image for denoising model training as the target. NAC can be combined with any CNN model, and it can achieve better performance than methods that use noisy-clean image pairs, especially when the noise is weak. However, NAC requires prior knowledge of the noise type and noise level estimation, which can be difficult to obtain in the real world. Although the article mentions using the Poisson-Gaussian mixture distribution when the noise type is unknown, many noises do not follow this distribution in practice.

Recorrupted-to-Recorrupted (R2R) [114] was introduced in 2021 that used a data augmentation method to train a denoising model with only single noisy images. Like N2N [107] that uses noisy-noisy image pairs to train the model, R2R generates these images from a single noisy image using different data augmentation methods for independent noise and noise correlated with the image signal, respectively. For independent noise, paired noise images $(\hat{y},\tilde{y})$ can be generated by the following:

where $y$ is the noisy image, $D$ can be any invertible matrix and $z$ is sampled from standard normal distribution $\mathcal{N}$. For noise correlated with the image signal $x$, paired noise images can be generated using a similar formula with the $x$-dependent covariance matrix $\sqrt{\sum_{x}}$ by the Eq. (21):

R2R achieved competitive results for both AWGN removal and real-world image denoising compared to the state-of-the-art self-supervised learning methods. However, it requires a significant amount of computational resources due to the data augmentation method used for training, and R2R requires prior knowledge of the noise level, which can be difficult to obtain in the real world.

Zhang et al. [97] proposed the Iterative Data Refinement (IDR) method trained with noiser-noise image pairs. First, IDR proves that the lower the noise level on the target noisy image the better the denoising effect. Therefore, it is feasible to reduce the noise level on the original noisy image to improve the denoising performance of the model. Then, IDR is proposed to be divided into two steps: (1) train a model $M_{0}$ with noise reduction ability through the noisier-noise pairs $\{y_{0}+n$, $y_{0}\}$. (2) iteratively replace the noisy image $y_{0}$ with the preliminary denoising image from the previous round $M_{0}$ until the loss function converges. Through iteration in this way, the denoising performance of the model is continuously improved. However, this iterative method is easy to lose the detail information in the image signal, resulting in the final denoised image being too smooth.

Noise2Score [115] is a denoising method that estimates the score function of an image using Bayesian statistics, specifically Tweedie’s formula [138], which can be applied to any exponential family of noise. The method involves adding certain noise at different levels to noisy images to estimate the score function, rather than the noise itself. A post-processing step with Tweedie’s formula is then used, which is determined by the specific noise model, such as Poisson or Gamma, etc. Noise2Score is more flexible than some other denoising methods that assume a specific type of noise. Additionally, it estimates the score function directly rather than the noise itself, and it can achieve better denoising performance than methods that rely on estimating the noise level.

Kim et al. [98] developed a denoising method that uses a quadratic equation for noise model estimation, which can be applied to Gaussian, Poisson, and Gamma noise models. The method requires only one additional inference step for noise level estimation, which is more efficient than Noise2Score [115], which requires multiple inferences. The method involves adding unknown noise to noisy images and estimating the noise model using the quadratic equation. The estimated noise model is then used to estimate the noise level and denoise the image. However, it assumes that the unknown noise is limited to these three common noise models, so it may not perform well with other types of noise.

CVF-SID[100] was introduced in 2022 by Reyhaneh Neshatavar et al. that proposes a Cyclic multi-Variate Function (CVF) module. The method only requires single noisy images. CVF-SID first decomposes the noisy imag $I_{n}$ into a clean image $I_{c}$, signal-dependent noise $N_{d}$ and signal-independent noise $N_{i}$:

where $F(\cdot)$ denotes a CNN model. And then it combines the three output parts arbitrarily as a new input to the previous model for decomposition training,

where $g$ is a combination function, $s_{1}$, $s_{2}$, $s_{3}$ are constants which can be one value from [-1, 0, 1]. Finally, repeat this cycle until the loss converges. To allow the CNN model to decompose noisy images, the method uses various self-supervised loss terms based on statistical behaviors of general noise. CVF-SID directly decomposes the sRGB noise image, maintaining the texture structure of the image better than BSN-based methods. However, the method relies on the assumption that the noise in the image can be decomposed into signal-dependent and signal-independent components, which may not always be true in practice. Therefore, the method may not perform well on images with noise that is not well-modeled by this assumption. Additionally, the cyclic training process used by CVF-SID is computationally expensive and may take longer to converge than other denoising methods. This may make it difficult to use in real-time applications or on large datasets. Finally, while CVF-SID is able to maintain the texture structure of the image better than some other methods [108, 109, 114], it may still introduce some degree of smoothing or loss of detail due to the decomposition and recombination process.

Vaksman et al. [104] presented the Patch-Craft Self-Supervised Training for Correlated Image Denoising paper at CVPR 2023, to address the challenge of denoising images with an unknown noise model. The method uses bursts of noisy images for self-supervised denoiser training, where one image from the burst is used as input and the rest are used for constructing targets. The targets are constructed using the patch-craft frames concept introduced in PaCNet [138], where the input shot is split into fully overlapping patches and the nearest neighbor within the rest of the burst images is found for each patch. It is important to note that the input shot is strictly excluded from the neighbor search. The paper also proposes a novel method for performing the statistical analysis of the target noise, which involves cutting the proper left tail shown on the histogram of an empirical covariance for the input and the difference between target and input. This method allows trainers to exclude faulty image pairs from the training set, which can boost the performance of the denoising method. One potential limitation of the method is that it requires bursts of noisy images for training, which may not always be available in real-world scenarios. Additionally, the method may not perform well on images with highly complex noise patterns that cannot be effectively modelled by the patch-craft frames concept. Furthermore, the method may be computationally expensive due to the neighbor search process used for constructing targets.

Noise2Self (N2S) [109] is proposed based on the assumption of noise with statistical independence. The authors first demonstrate that the optimal denoising function for a given dataset can be found by minimizing the self-supervised loss using a class of $\mathcal{J}$-invariant functions, which are functions that preserve the structure of the input image under arbitrary translations. They then show through examples that as the degree of correlation between features in the data increases, the optimal $\mathcal{J}$-invariant denoising function approaches the optimal general denoising function. Based on this observation, the authors propose a deep neural network with millions of parameters that is modified to become $\mathcal{J}$-invariant through the masking of pixels with random values. This involves adding a $\mathcal{J}$-invariant mask $\mathcal{J}$ to the input, as shown in Fig. 2b, where the original values on $\mathcal{J}$ are masked and replaced by random values. The authors conduct experiments using two neural networks, UNet [133] and DnCNN [82] on three datasets including ImageNet [139] and CellNet [140] with different noise mixture strategies. The experiments show that N2S achieves competitive performance with the same neural network architectures trained with clean targets (called Noise2Truth) and independently noisy targets (N2N [107]). However, many open questions remain regarding the optimal choice of $\mathcal{J}$ in practice. The construction of $\mathcal{J}$ must reflect the signal-dependent patterns and noise independence relations, which involves making a bias-variance trade-off in the loss of the relative sizes of each subset $J$ and increasing information for prediction.

Several denoising methods [107, 108, 110], including N2V [108], assume that the noise present in an image is uniform, which is not always the case in reality. Probabilistic Noise2Void (PN2V) [113] extends the original N2V method and addresses this limitation by incorporating a probabilistic model of the noise. PN2V models the noise distribution using a probabilistic approach, which allows it to handle non-uniform noise and capture the variability in the noise distribution across different regions of the image. PN2V builds on N2V by introducing a probabilistic noise model that incorporates the noise distribution as a prior into the loss function. This allows the model to learn the noise distribution from the data and produce more accurate denoising results. In the PN2V model training followed by N2V, noisy images are first randomly masked as inputs and original noisy images as targets, shown in Fig. 2a. The network is then trained to predict probability distribution of the pixel’s signal in the masked regions, considering only its surroundings, while also learning the noise model parameters from the training data. The loss function of PN2V can be expressed as:

$p(\cdot)$ is described by an arbitrary noise model, which in the case of PN2V [113] is a histogram-based noise model. For each pixel $i$, the network directly predicts $K$ output values ($K=800$, $s_{i}^{k}$ denotes each value), which are interpreted as independent samples, and $x_{i}$ denotes the observed noisy input value. Empirical results have shown that PN2V [113] outperforms N2V [108] in a variety of imaging applications, including fluorescence microscopy and electron microscopy. Furthermore, the probabilistic noise model allows PN2V to estimate the uncertainty of the denoised image, which is useful in downstream applications that require uncertainty quantification. However, in the real world, most noise sources do not strictly follow the Poisson-Gaussian distribution or histogram-based noise models used in PN2V [113]. Therefore, the PN2V model trained with a specific noise distribution will have limited performance when denoising real-world noise.

Noise2Same[94] proved through experiments that the denoising functions trained through mask-based blind-spot methods are not strictly $\mathcal{J}$-invariant, meaning that they do not preserve the structure of the input image under arbitrary translations. This can lead to reduced denoising performance, particularly in cases where the noise is correlated or exhibits complex statistical properties. Furthermore, Noise2Same demonstrated that minimizing the Euclidean distance between the denoised image and the original image, expressed as $E_{x}\|f(x)-x\|^{2}$, is not optimal for self-supervised image denoising. Noise2same masked the noise image same as N2S [109] to a $\mathcal{J}$-invariant distribution, but replaced the masked pixel with local average values instead of random values. It proposed to take both the original noisy image and the $\mathcal{J}$-invariant masked image as inputs and output two residual noise. The network would be trained from learning the invariance loss between the two outputs.

Self2Self With Dropout (S2S) [95] uses Bernoulli dropout to improve the performance of self-supervised denoising methods. S2S randomly masks instances of the noisy image using a Bernoulli process, which generates a set of image pairs for training the neural network. The Bernoulli dropout is applied using element-wise multiplication, and it is used in both the training and test phases to reduce prediction variance. To further improve performance, S2S uses partial convolution instead of standard convolution for re-normalization on sample pixels. Partial convolution is a convolutional operation that only convolves over the unmasked pixels, which helps to preserve the image structure and prevent artifacts. The self-prediction loss in S2S is defined on the pairs of Bernoulli sampled instances of the input image. The loss function encourages the network to predict the unmasked pixels of the noisy image from the masked pixels, while also taking into account the noise distribution and the image structure. The Bernoulli dropout and partial convolution help to regularize the denoising process and improve the performance of the network.

Blind2Unblind (B2UB) [99] is based on blind spots proposed by Wang et al. in 2022. B2UB uses a global-aware mask mapper to ensure that all the information in the image is fully utilized and uses re-visible loss to help restore more details in the image. B2UB’s mask strategy is called the global-aware mask mapper. Firstly, the noise image is divided into multiple $k\times k$ cells, and the pixels are marked with a number $i$ ($i\in[1,k^{2}]$, $k=4$, in the experiments) from left to right and top to bottom in each cell. By masking the points with the same number in the original image, $k\times k$ masked images with the same size as the original image are obtained as inputs, as shown in Fig. 2c. Secondly, the denoising model produces $k\times k$ outputs. The pixels at all blind spots in the output images are collected and put back in the previously recorded positions, resulting in an image of the same size as the original noisy image with each pixel denoised. This image is combined with the original noisy image for supervised training. In addition to the supervised loss, B2UB uses a re-visible loss as a regularization loss item to make full use of the information in the noisy image for detail restoration. The whole loss can be described as:

where $y$ is the original noise image, $F(\cdot)$ represents the denoising model, $\tilde{F}(\cdot)$ denotes the denoising model without gradient update during training, $M(\cdot)$ represents the first step of the global-aware mask mapper which creates blind spots, $M^{-1}(\cdot)$ represents the second step of the global-aware mask mapper which collects the denoised blind spots, $\beta$ is a fixed hyper-parameter (set as 1) that determines how much the blind term affects model denoising training, and $\alpha$ is a variable hyper-parameter (increase from 2 to 20 in training) that controls the strength of the visible part. B2UB uses the mask strategy of global-aware mask mapper to blind all points in the noise image once, so that all pixels participate in denoising training and make full use of the information of all pixels in the image. The re-visible loss is used to allow the model to see the complete noise image and repair the destroyed texture information in the process of creating blind spots. Extensive experiments on sRGB images with synthetic noise and rawRGB images with real noise demonstrate the superior performance of B2UB. However, B2UB cannot handle real-world sRGB images with spatially correlated noise that does not satisfy the prerequisites of BSN-based denoising methods.

In 2020, Wu et al. [111] proposed a two-stage image denoising method using unpaired clean $x$ and noisy $y$ images. They trained a denoising model with Dilated Blind-Spot Network (DBSN), as shown in Fig. 3b and knowledge distillation. In the first stage, noisy images are input into the proposed DBSN and the noise estimation model $CNN_{est}$ respectively. Through maximizing the constrained log-likelihood as the loss function to conduct self-supervised image denoising training, the denoised image and the noise level estimation function can be obtained. The loss function for the first stage when learning DBSN and $CNN_{est}$ is the constrained negative log-likelihood is:

where $tr(\cdot)$ denotes the trace of a matrix. $y_{i}$ is the real noisy image, $\hat{x}_{i}$ is the output of DBSN, $\widehat{\sum}_{i}^{n}$ and $\widehat{\sum}_{i}^{x}$ is a $C\times C$ covariance matrix output by $CNNest$ and D-BSN respectively for the position $i$. In the second stage, the DBSN denoising model trained in the first stage is used to obtain the denoised image $\hat{x}_{i}$ of the original noise image, and the noise level function is applied to the original clean image to obtain a synthetic noise image $\hat{y}_{i}$. Two pairs of image sets ($y_{i}$, $\hat{x}_{i}$) and ($\hat{y}_{i}$, $x_{i}$) are created, and they are input into any CNN model for joint training to obtain the final CNN denoiser. Its loss function is:

where $\beta=0.1$ is the trade-off parameter, convolutional denoising network ($CDN$) can be any existing CNN denoisers. Wu et al. [111] utilizes unpaired noisy-clean image pairs for training the denoising model. The use of real clean images in the training allows the model to focus on more detail restoration, and the real noisy images make the model more adaptable to the removal of noise in the real world. The two branches are jointly trained, and the final denoising model has better denoising performance. Additionally, this model is based on the fact that the noise is spatially independent. Although, the pixel-shuffle downsampling (PD, stride=2 in DBSN) first introduced in Zhou et al. [141] are adopted to break the spatial correlation of the noise, the size of the spatial connection that PD (stride=2) can break is finite. For large-scale complex noise in real-world, PD as well as the noise level estimation may not perform well.

AP-BSN [116] is a denoising method for sRGB images with real-world noise, proposed by Wooseok Lee et al. in 2022. AP-BSN uses Asymmetric PD (AP) to break the spatial connection of noise, obtains blind spots by masking the central pixel of the convolution kernels, as shown in Fig. 3b, and adopts random-replacing refinement ($R^{3}$) to restore image texture details and improve the denoising model’s performance. The stride of PD determines the down-sampling multiple, and different PD strides affect denoising performance during training and testing. The larger the stride, the larger the down-sampling multiple, the more thorough the spatial correlation of the noise is broken, but at the same time the texture information of the image signals is damaged more. AP means adopting PD with different strides during training and testing. AP-BSN uses PD with stride=5 during training and PD with stride=2 during testing for the best performance. The AP-BSN loss is the $L_{1}$-norm distance between the output of the BSN model and the noisy image and can be described as:

where $\rm{PD^{-1}}$ means the inverse of PD, $F(\cdot)$ is the BSN model, and $I_{N}$ is the noise image. Taking the noisy image as both input and target. AP-BSN is the first denoising algorithm using self-supervised BSN on real-world sRGB images. However, the stride of PD synchronization affects the spatial correlation of noise and the degree of damage to the texture structure of the image signal. Therefore, the stride of PD cannot be too large. AP-BSN’s denoising performance may be greatly reduced when the noise in the image is spatially correlated in a large area.

MM-BSN [102] is designed for real-world sRGB images with large areas of spatially correlated noise. MM-BSN is also masked in the network. Unlike AP-BSN [116], which only shields the central pixel of the convolution kernel, MM-BSN uses Multi-mask to shield pixels at multiple positions of the convolution kernel to break the large-area correlation of noise, as shown in Fig. 3c. When the spatial connection area of the noise is large enough, only the central pixel of the convolution kernel is masked, and the surrounding pixels are likely still to be noisy-related. Then, when the feature extraction of the target point is performed through the surrounding pixels, the extracted features will still contain noise. Noise in the real world has different shapes due to different reasons of formation. Therefore, in order to break the spatial correlation of noise, MM-BSN uses masks of various shapes (such as ’/’, ’\’, ’—’, ’-’, ’$\square$’ etc.) to mask the convolution kernel, the ’\’ shape sample is shown in Fig. 3c. A noisy image is likely to contain noise of multiple shapes, so it is inappropriate to use only one shape of mask in a model. A novel network structure is proposed to combine the features extracted by Multi-mask. In order for a model to handle noise of various shapes, multiple masks must be used at the same time. MM-BSN uses the same loss as AP-BSN [116]. The noisy image is input to the BSN model in which the pixels at multiple positions of the receptive field are masked for model training. MM-BSN is the first to propose the feature extraction of images by combining masks of various shapes. The Multi-mask strategy can break the structure of large-area spatial correlation noise and improve the denoising performance of the model. However, the shape of the mask is fixed and cannot be self-adjusted during training according to the shape of the noise.

Li et al. [105] proposed the Spatially Adaptive Self-Supervised Learning for Real-World Image Denoising in CVPR2023. It researches how to do the supervision for flat areas and texture areas respectively. For flat areas, it proposed the blind-neighborhood network (BNN), which is deformed from BSN but have different receptive field. At the end of each branch in BNN, the blind spot is created by applying a single pixel shift to the features and a $(2k-1)\times(2k-1)$ blind-neighborhood can be created by increasing the pixel shift size from 1 to $k$. The loss function for flat areas is:

$x_{1}$ is the ouput of BNN, and $y$ is the noisy image. In addition, to distinguish the texture areas from the flat areas, the paper first calculated the statand deviation map $\sigma$:

where $std(\cdot)$ represents the standard deviation function, which is calculated on 1-channel patches by averaging the values of RGB channels. $n$ is the local window size (set as 7 in this paper). Then, it proposed the following expression to convert the binarization $\sigma$ to the normalized $\alpha$ with scope of [0, 1].

where $S(\cdot)$ represents the Sigmoid function, $l=1$, and $\mu=5$. The higher $\alpha(i,j)$ the more local area is textured. When the texture areas are defined, then a locally aware network (LAN) is presented by stacking $k$ $3\times 3$ convolution layers and several $1\times 1$ convolution blocks with channel attention mechanism [58]. The loss function for LAN training phase is as follows:

where $\tilde{x}_{1}$ and $\tilde{x}_{2}$ denotes the output of the BNN and LAN, respectively. $sg(\cdot)$ represents stop gradient operation [142]. BNN creates blind spots in the receptive field by shifting the features, which can help to remove noise from flat areas. Although, LAN is specifically designed for texture areas and takes into account the local texture information, this approach may not work well on texture areas where the features are more complex and shifting them may cause loss of important information. Another potential limitation of BNN is that it requires careful tuning of the pixel shift size to create the blind spots, which may be sensitive to the noise characteristics of different images.