Boosting of Implicit Neural Representation-based Image Denoiser

Image denoising is a fundamental inverse problem in low-level computer vision. In this setting, we are given a vectorized noisy observation $\bm{y}\in\mathbb{R}^{K}$ combined by the underlying clean signal $\bm{x}\in\mathbb{R}^{K}$ and noise, which is generally formulated as follows:

where $\bm{n}\in\mathbb{R}^{K}$ is an additive noise and typically assumed to be i.i.d. zero-mean Gaussian noise, i.e., $\bm{n}\sim\mathcal{N}(0,\sigma^{2})$, where $\sigma$ denotes the standard deviation. Extensive algorithms and models have been developed for image denoising. Before the era of deep learning, filter-based methods [1, 2, 3], thresholding methods [4, 5, 6] and block-matching-based methods [7, 8] are representative works. Nowadays, deep neural networks, such as DnCNN [9], SwinIR [10] and Restormer [11], etc., achieve remarkable results. In addition to these supervised learning models, self-supervised methods, such as Noise2Self [12], Noise2Void [13] and Self2Self [14], etc., provide another alternative to improve the signal-to-noise ratio (SNR), which only utilizes a single noisy image during training.

Recent advancements in image denoising have underscored the potential of Implicit Neural Representations (INR). Noteworthy methods include DIP [15], SIREN [16] and WIRE [17], among others. INR offers a novel approach to representing implicitly defined, continuous, and differentiable signals parameterized by neural networks [16]. Through INR, networks are trained to map specific coordinates or even random noise directly to the target denoised representation, efficiently adapting to fit the noisy observation.

However, INR models are typically overparameterized [18]. Consequently, these models are prone to overfitting during training, resulting in suboptimal results, even fitting noisy ones. Existing solutions to this problem primarily pursue the following directions. The first attempts to stop the training once the model begins to fit noise. Early Stopping [18] exemplifies this approach, which utilizes the running variance of past denoised results as a metric to determine the optimal training stopping point. Nevertheless, its performance can be inconsistent, and the hyper-parameter optimization becomes particularly intense. The second one integrates regularization to relieve the overfitting problem, such as Total-Variation (TV) [19], Regularization by Denoising (RED) [20], etc. For example, DIP can be effectively regularized by TV [21, 22], as well as RED [23], namely DeepRED. Although effective, these methods require additional computation and involve further hyperparameter tuning. Furthermore, the boosting algorithm [24, 25, 26, 27] presents a holistic strategy for enhancing the denoised outputs. Yet, despite its generality and efficacy, such methods are typically employed as post-processing measures. This divides the process into two distinct phases: primary denoising followed by the boosting stage. This bifurcation naturally demands additional computations and introduces more hyper-parameters to the process.

In this work, we propose a simple yet effective method to relieve the overfitting tendencies of INR models, thereby boosting their performance. Specifically, our method revolves around iteratively updating the supervision signal with the mean value derived from both the prediction and the original supervision signal throughout the training phase. This iterative substitution approach termed ITS, is shown to enhance the supervision signal’s SNR incrementally. As a result, the learning trajectory of INR models becomes more robust. Unlike conventional techniques, our approach introduces nearly zero extra computational overhead and seamlessly integrates into the learning process of INR models. Our experimental results confirm that the proposed method is an effective regularization tool for INR models, addressing overfitting challenges and boosting overall performance. Our contributions are summarized as follows:

• •

  We propose a general paradigm for regularizing INR models in image denoising to boost their performance.

• •

  We theoretically prove the proposed ITS approach can progressively improve the supervision signal’s SNR.

• •

  We evaluate the proposed method when integrated with typical INR models for image denoising, and experimental results demonstrate the effectiveness of our method.

The INR is trained to map from a given coordinate or random noise to the desired denoised representation during the learning process. This can be mathematically formulated as follows:

where $f_{\theta}(\cdot)$ denotes the INR model with $\theta$ representing its parameters. The term $\bm{z}$ is the given coordinate [16, 17] or random noise [15].

The overall objective function for optimizing $f_{\theta}(\cdot)$ is formulated as follows:

where $\|\cdot\|$ denotes the $\ell_{2}$ norm, and $\mathcal{R}(\cdot)$ is the regularization term for introducing explicit priors, such as TV [19, 21], RED [20, 23], and others.

For simplicity, let’s consider when $\lambda=0$, i.e., without any explicit priors, the objective function simplifies to $\mathcal{L}=\|\bm{\hat{x}}-\bm{y}\|^{2}$. Within this context, the INR models align with the dominant semantic context (represented by low-frequency signals) before they start overfitting to the noise, symbolized by high-frequency signals [28].

Additionally, when the noise levels increase, the learning becomes more challenging, and INR models are more vulnerable to noise disturbances [18, 29]. In contrast, if the noise level is small, the learning is much easier; more aggressively, in an extreme case that $\bm{y}=\bm{x}$ if $\sigma=0$, the objective function leads to fitting the clean ground truth.

These observations raise a simple question: Can we clean up the noisy supervision signal during the INR training process? If so, the INR model can learn much better because the noise level is reduced to provide a better supervision signal; even more extreme, if we clean up the original noisy supervision signal to the clean ground truth, then the issue of overfitting to noise will be eliminated.

Considering that the INR model is generally optimized using stochastic gradient descent (SGD) within $T$ iterations, then for a given $i$-th iteration, the denoised result $\bm{\hat{x}}^{i}$ can be expressed as follows [24, 25, 26, 30]:

where $\bm{e}^{i}$ denotes reconstruction error. This error can be further decomposed into two parts: $\bm{e_{n}}^{i}$ denotes the residual noise from $\bm{n}$ and $\bm{e_{x}}^{i}$ stands for the lost components of $\bm{x}$.

Assuming that at $t$ iteration, the model begins to overfit the noise, then there exists a relationship between $\text{SNR}(\bm{\hat{x}}^{i})$ at different iterations, that satisfies:

where $\text{SNR}(\bm{\hat{x}}^{i})=\frac{\|\bm{x}^{i}\|}{\|\bm{e}^{i}\|}$, and the above relationship’s shape mimics a bell curve. In a practical setting, it is infeasible to determine $t$ since only the noisy observation $\bm{y}$ is available without the information of ground truth $\bm{x}$.

However, given that the noisy observation $\bm{y}$ is described by Eq.(1), we suggest merging the internal denoised result $\bm{\hat{x}}^{i}$ with noisy observation $\bm{y}$ to generate a renewed supervision signal for $(i+1)$-th iteration:

where $\frac{\bm{n}+\bm{e}^{i}}{2}$ is a new reconstruction error. If $\bm{e}^{i}<\bm{n}$ holds true, the above substitution can yield a supervision signal with an enhanced SNR. An illustrative proof is provided in Theorem 1.

Based on this observation, we propose an iterative substitution (ITS) strategy that performs the updating from Eq.(6) with given increasing iterations. For simplicity, we introduce a hyperparameter $N$ to generate a set of iterations $S=\{N,2N,...,kN\}$, then for each iteration in $S$, we perform the substitution from Eq.(6), that is:

The ITS approach capitalizes on the iterative refinement principle during the training process. By progressively substituting the denoised results into the supervision signal, the model constantly refines its understanding of the image and prevents overfitting, thereby enhancing the denoising performance. The increasing iterations ensure that the model doesn’t hastily converge but instead gradually refines its output, potentially achieving a balance between noise suppression and preservation of crucial image details.

This work proposes a general iterative substitution (ITS) paradigm for regularizing INR models in image denoising. Our theoretical justification proves that ITS can improve the SNR of the supervision signal, thereby benefiting the learning process of INR models. Our experimental results demonstrate the effectiveness and efficiency of the proposed ITS. The major limitation of our method is that we assume the denoised result contains additive reconstruction error as illustrated in Eq.(4). Although such an assumption is widely adopted in prior works  [24, 25, 26, 30], the general assumption for the inverse problem, i.e., $\bm{\hat{x}}=\bm{A}\bm{x}+\bm{e}$, would be more appropriate, which additionally considers transformation formalized by $\bm{A}$, such as blurring. One future direction is to address the above limitation. In addition, extending our method to video denoising is one future direction. Besides, generalizing our problem and method to the general image inverse problem is another future direction.