Self-Verification in Image Denoising

While DIP is an effective unsupervised approach, a drawback for real-world applications is its time-consuming training that requires human intervention for stopping. Inspired by DIP, in this paper we explore a more flexible use of the neural network image prior. Similar to DIP, we only focus on self-supervised cases; the difference is that our goal is to formulate an explicit regularization for denoising using the prior determined by the network.

We use a deep CNN to perform the denoising task in Eq. (2), so Eq. (2) can be rewritten as

Since DIP has confirmed the affinity of the deep network for learning low-level image statistics, we expect that this network will first capture low-level image statistics as well in early iterations before attempting to memorize the noise when solving Eq. (5). Since the output of the network (i.e. ${F_{\theta}}(y)$) contains the low-level statistics required to restore the image, we consider using ${F_{\theta}}(y)$ as the prior for image denoising. Put another way, can the output of the denoising network be used to construct a regularization for itself? In this paper, we give a feasible solution for performing this task.

Suppose there is a second noise-degraded image $y_{2}$ in addition to $y$ for which both share the same latent clean image $x$, and which have independent and identically distributed (i.i.d.) noise. Based on $y_{2}$ and $y$, we devise a new regularization term for image denoising. Then, we could consider for Eq. (5) the function

Since $y_{2}$ and $y$ have different noise, ${F_{\theta^{*}}}(y)=x$ is obviously a good candidate solution for minimizing Eq. (6). However, in most real-world applications, $y$ is available, but we do not have a second noisy image $y_{2}$. Instead, we use the output of the network to synthesize a new $y_{2}$,

where $D(\cdot)$ is a predefined adaptive noise degradation function and $n_{2}$ is newly generated noise. We discuss $D(\cdot)$ later. Combining Eqs. (6) and (7), we have

The first term of Eq. (8) implicitly constrains the removed noise to be zero mean. In the second term of Eq. (8), if the noise of $D({F_{\theta}}(y))$ and $y$ share similar statistical characteristics, but ${n_{2}}\neq n$, then ${F_{\theta^{*}}}(y)=x$ is still a good solution for Eq. (8). We refer to the second term of Eq. (8) as a self-verification regularization. In $\left\|{{F_{\theta}}(D({F_{\theta}}(y)))-{F_{\theta}}(y)}\right\|_{2}^{2}$, ${F_{\theta}}(y)$ can be regarded as a deep image prior learned by the network. The similarity comparison between ${F_{\theta}}(D({F_{\theta}}(y)))$ and ${F_{\theta}}(y)$ can be seen as using a prior generated by the network to verify the denoising behavior of the network itself. We update the network to minimize $\left\|{{F_{\theta}}(D({F_{\theta}}(y)))-{F_{\theta}}(y)}\right\|_{2}^{2}$. When $\theta$ converges and the network becomes a good denoiser, the discrepancy between ${F_{\theta}}(D({F_{\theta}}(y)))$ and ${F_{\theta}}(y)$ should be small.

With self-verification regularization, we further show how a deep CNN learns to denoise with self-supervision. We call this training method Self-Verification Image Denoising (SVID). SVID does not rely on other noise statistics except the assumption of zero-mean noise. To perform self-verification, SVID uses a Siamese network as the backbone. Although SVID is based only on noisy training data, its denoising results are impressive. We demonstrate the superior performance of SVID in various denoising experiments. SVID significantly outperforms other state-of-the-art self-supervised methods.

Self-verification consists of two steps: adaptive noise degradation and similarity comparison. We use a Siamese network, where the goal is first to input $y$ to a CNN to obtain a denoised image ${F_{\theta}}(y)$ and its removed noise $\hat{n}=y-{F_{\theta}}(y)$. Since low-level image statistics are easier to be captured by deep CNNs than noise, we assume that ${F_{\theta}}(y)$ is a clean image. We then synthesize $y_{2}$ by the following adaptive noise degradation function,

where ${n_{2}}=m\odot\hat{n}$ is a new noise which is adaptively synthesized with respect to $\hat{n}$. $\odot$ represents element-wise multiplication, $m$ is a random binary ($1$ or $-1$) mask that is independent of $\hat{n}$ but has the same dimension as $\hat{n}$. In each iteration of learning, a new $m$ is sampled. Each element in $m$ is $1$ with a probability of $p=0.5$, and $-1$ otherwise.

Next, the network learns to denoise by maximizing the similarity between ${F_{\theta}}(y)$ and ${F_{\theta}}(y_{2})$. This is achieved by solving the following optimization problem,

“Stopgrad” means that the gradient does not backpropagate to ${F_{\theta}}(y)$, which helps prevent trivial solutions (i.e. ${F_{\theta}}(\cdot)=c$, $c$ is a constant.). The stop-gradient procedure is a common strategy for learning Siamese networks [3].

We motivate the SVID objective of Eq. (10) as a modification of Eq. (6). From Eq. (6), we observe that we can expand the self-regularization $R(y,y_{2})\equiv\left\|{{F_{\theta}}({y_{2}})-{F_{\theta}}(y)}\right\|_{2}^{2}$ as

We observe that the squared error $\|{F_{\theta}}(y)-y\|_{2}^{2}$ is implicitly included in $R(y,y_{2})$, and therefore, modulo a rescaling of the regularization parameter $\alpha$, the squared error is not necessary in $Eq.(\ref{eq.new_reg})$. We also observe that the squared error term $\|y-{F_{\theta}}(y_{2})\|_{2}^{2}$ is between $y$ and the output using $y_{2}$. Since the noise in $y$ and $y_{2}$ are assumed zero mean and independent, we can view this as an adversarial term, since it is not possible to model noise in $y$ based on the noise in $y_{2}$, and therefore the network defaults to their shared structure.

In Eq. (3.2), we need the second noisy image $y_{2}$. We observe that if $y_{2}$ is synthesized by Eq. (9), then $y_{2}$ naturally has some desirable properties. Specifically,

1. Some image details may remain in $\hat{n}$. These details can be destroyed by multiplying with a random mask $m$, which makes $n_{2}$ a more natural-looking noise (see Figure LABEL:fig_cat).

2. $\mathbb{E}(m)=0$ leads to $\mathbb{E}(n_{2})=\mathbb{E}(m)\mathbb{E}(\hat{n})=0$. Therefore, the noise of $y_{2}$ meets the zero-mean assumption.

3. Since $n_{2}$ is composed of $\hat{n}$, the statistical characteristics of $n_{2}$ and $\hat{n}$ should be similar, as shown in Figure 4. In addition, $m$ is random, so ${n_{2}}\neq\hat{n}$. This further leads to ${y_{2}}\neq y$. If ${y_{2}}=y$, self-verification Eq. (10) is meaningless.

Our method uses a stop-gradient operation to avoid trivial solutions. We next discuss one hypothesis on why SVID is practically effective, as our experiments will show. In our method, we use the network’s output ${F_{\theta}}(y)$ and the adaptively synthesized $y_{2}$ to learn denoising, hoping that the learned knowledge can adapt to the real noisy data $y$. SVID is an EM-like algorithm. It implicitly involves two sets of variables (i.e. $\theta$ and ${F_{\theta}}(y)$), and solves two underlying sub-problems. The presence of the stop-gradient is the consequence of introducing the extra set of variables.

We re-express the objective function of SVID in the following form,

Note that Eq. (12) is consistent with the objective function in Eq. (10). For clarity, we use $\eta_{y}$ to denote ${F_{\theta}}(y)$. With this formulation, we consider solving:

The problem in Eq. (13) can be solved by an alternating algorithm, fixing one set of variables and solving for the other set. At each training step $t$, we randomly sample a noisy image $y$ from the given noisy dataset $\Omega^{noisy}$. Then, alternately solve the two sub-problems:

where “$\leftarrow$” means assigning.

Solving for $\eta_{y}$. There is no optimization in this part, but simply a redefinition of the term $\eta_{y}$ to be the output of the neural network on $y$. Moreover, previous work on DIP has revealed that the CNN naturally tends to capture low-level image statistics. Therefore, the similarity between $\eta_{y}^{(t)}$ and $x$ will gradually increase.

Solving for $\theta$. $y_{2}$ is constructed by Eq. (9) using $\eta_{y}^{(t)}$. The more similar $\eta_{y}^{(t)}$ is to $x$, the closer the noise statistics of $y_{2}$ and $y$ are (see Figure 4 for illustration). The denoising network is improved by solving Eq. (15). $\eta_{y}^{(t)}$ is fixed in this subproblem, so the gradient does not backpropagate to $\eta_{y}^{(t)}$.

In short, one output of the network is obtained at training step $t$, which serves as a “prior” target for denoising. $\eta_{y}^{(t)}$ contains the low-level image statistics required to restore a clean image. Therefore, we use $\eta_{y}^{(t)}$ as a label to improve the denoising ability of the network. This leads to the network to produce a better prior at time $t+1$. By solving the two sub-problems alternately at each training step, ${F_{\theta}}(y)$ approaches the true underlying $x$.

The denoising CNN in SVID is a simple U-Net [19]. We use PyTorch and Adam [10] with a batch size of 1 to train the network. The training images are randomly cropped into $128\times 128$ patches before being input to the network. The learning rate is fixed to $0.0002$ for the first $1,000,000$ iterations and linearly decays to 0 for the next $1,000,000$ iterations.¹¹1We will include a reference here to our code and datasets. After training, the CNN with fixed parameters $\theta^{*}$ is directly applied to other noisy images.

We collected 4744 images from the Waterloo Exploration Database [15] to synthesize noisy images for training. Several state-of-the-art denoising methods are adopted for performance comparison, including model-based methods BM3D [4] and NLH [8], self-learning methods DIP [23], Noise2Void(N2V) [11], Self2Self(S2S) [18] and Neighbor2Neighbor(Nb2Nb) [9], other deep learning methods include Noise2Noise(N2N) [13] and a common fully-supervised U-Net. U-Net is trained with noisy/clean image pairs, while N2N uses paired noisy/noisy images. For the sake of fairness, N2N, U-Net and our SVID adopt the same network architecture to perform denoising. BSD300 [16] is the test set of the following experiments.

Gaussian noise. We first study the effectiveness of SVID on additive Gaussian noise. Each training example is corrupted by zero-mean Gaussian noise with a random standard deviation $\sigma\in\left(0,50\right]$. For testing, we synthesize two sets of noisy images, using a fixed noise level $\sigma=25$ and a variable $\sigma\in\left(0,50\right]$. We present the quantitative results in Table 1. Visual comparisons can be found in Figure 5. In addition, Figure 4 shows the statistical histograms of the noise removed by SVID during training.

Speckle noise. Multiplicative speckle noise is signal dependent and is often observed in medical ultrasonic images and radar images. The speckle noise in the image is modeled as the random value multiplied by the pixel value of the latent signal $x$, which can be expressed as $y=x+x\cdot n$. In this model, $n$ is an uniform noise with a mean of $0$ and a variance of $v$. We vary the noise variance $v\in\left(0,0.2\right]$ during training. The results and the comparisons of denoising on Speckle noise can be seen in Table 1 and Figure 6.

Poisson noise. In our third experiment we consider Poisson noise, which can be used to model photon noise in imaging sensors. Poisson noise is also signal-dependent because its expected magnitude depends on the pixel brightness. We randomize the noise magnitude $\lambda\in\left[5,50\right]$ separately for each training example. Quantitative and subjective comparisons are reported in Table 1 and Figure 7, respectively.

Discussion. In the above comparisons, SVID shows excellent denoising results both subjectively and quantitatively. These results demonstrate that the output of the denoising network can be a prior for denoising, as this paper conjectures. The denoising performance of SVID outperforms other self-supervised methods (i.e. DIP, N2V, S2S, Nb2Nb) by a large margin, which confirms the effectiveness of self-verification. In SVID, the denoising ability of the network can be improved by self-verification. In addition, SVID is better than model-based methods BM3D and NLH. This is because the deep image prior captured by the network can better model the properties of natural images than the traditional non-local self-similarity prior. We also noticed that SVID is inferior to supervised methods U-Net and N2N, which is reasonable considering that our SVID has no access to paired data for supervision. In short, our SVID consistently exhibits encouraging denoising performance for various types of noise. The denoised images are clean and sharp. More importantly, SVID does not rely on paired data or pre-known noise statistics, which shows its potential value for many practical applications.

Computed tomography (CT) is a popular imaging modality in clinical diagnosis. Despite the healthcare benefits, the extensive use of CT has caused public concern about the potential risks of X-ray radiation. Reducing radiation dose is an effective way to deal with this problem. However, a decrease of radiation dose is associated with an increase of noise and artifacts in the reconstructed image, which may adversely affect subsequent diagnosis. Therefore, denoising is a critical post-processing step for low-dose CT images [21]. Since the noise statistics in CT images are very complicated and paired training data is difficult to obtain, the self-supervised denoising methods can better cater to low-dose CT images. We further evaluate SVID in the task of low-dose CT denoising.

We use a real clinical dataset authorized for the 2016 NIH-AAPM-Mayo Clinic LDCT Grand Challenge by Mayo Clinic.²²2https://www.aapm.org/GrandChallenge/LowDoseCT/ This dataset contains 5946 pairs of normal-/low- dose images with a slice thickness of 1mm. We randomly select 5000 pairs of images as the training set, and the rest as the test set. We compare SVID with a fully supervised U-Net. Note that SVID only uses low-dose CT images to learn denoising without the normal-dose images. The comparison results are reported in Figure 8 and Table 2. As can be seen, our SVID is also effective for noise in CT. SVID is successful in removing the majority of noise and restoring high-quality image details. Its performance is close to the fully supervised U-Net.

Our SVID employs a Siamese network to perform self-verification. An undesired trivial solution to Siamese networks is all outputs collapse to a constant. There are several strategies to prevent Siamese networks from collapsing, such as contrastive learning, clustering and stop-gradients [3]. In SVID, the stop-gradient operation is a natural consequence of introducing an extra variable. Here, our aim is to show that the stop-gradient is critical to preventing SVID from collapsing.

Figure 9 presents a comparison with and without using the stop-gradient. Without stop-gradients, all outputs of the denoising network collapse to 0 and the objective function Eq. (12) reaches its minimum of 0. With stop-gradients, the objective function Eq. (12) implicitly constrains the removed noise to be zero-mean. Therefore, the denoised image is prevented from collapsing to 0. Moreover, the deep CNN tends to capture low-level image statistics, which promotes the denoised image to look like a clean natural image.