Real-time Controllable Denoising for Image and Video

Traditional image and video denoising methods are often based on prior assumptions such as sparse image prior [3, 16, 20, 15], non-local similarity [7, 18, 14, 13], and other similar techniques [22, 42, 54]. However, with the recent development of deep learning networks, many learning-based methods have been proposed and achieved state-of-the-art performance. Early works [8] utilized multi-layer perceptron (MLP) to achieve comparable results with BM3D. In recent years, there has been rapid progress on CNN-based denoising methods [57, 59, 48, 21, 4, 10] and Transformer-based methods [56, 32, 61, 43], which have started to dominate the image/video denoising task. However, the above-mentioned works mainly focus on designing novel network architectures to improve the denoising performance and usually generate a single output. Their lack of ability to adjust the output denoising level based on user’s feedback has greatly restricted their practical use in many real-world applications. Moreover, although techniques like pruning [38, 62, 33] and quantization [46, 63] can accelerate such neural network-based methods, they are typically heavy, which restricts their application to real-time denoising control.

Most conventional deep-learning methods for image/video denoising can only generate a fixed result with a specific restoration level. Recently, some controllable image/video denoising methods allow users to adjust the restoration effect without retraining the network. DNI [52] and AdaFM [24] used the observation that the learned filters of the models trained with different restoration levels are similar in visual patterns. DNI interpolated all the corresponding parameters between two related networks to derive smooth and continuous restoration effects, while AdaFM adopted feature modulation filters after each convolution layer. CFSNet [51] proposed an adaptive learning strategy of using interpolation coefficients to couple the intermediate features between the main branch and the tuning branch. Different from these interpolation-base methods, some other methods [25, 9, 39] regarded modulation as a conditional image restoration problem and adopted a joint training strategy. CUGAN [9] proposed a GAN-based image restoration framework to avoid the over-smooth problem, a common issue in PSNR-oriented methods. However, all of the above controllable methods can only be trained with synthetic degradations because they require explicit degradation levels during training. When applied on real-world data, as shown in [23], methods that trained for blind Additive White Gaussian Noise (AWGN) [57, 35] may be overfitted and often suffer from dramatic performance drop. Besides the real-world image issue, all these controllable methods utilize an auxiliary conditional network and require one network inference for each different target restoration level at test time, which makes them almost impossible for real-time application.

Deep denoising methods trump traditional filter-based techniques by leveraging the neural networks’ robust representation learning capability. Most current denoising methods [11, 32, 47] reason about the relationship between clean and noisy images by regressing noise maps with a neural generator. Specifically, given a noisy image $\mathbf{I_{n}}$ and model $\mathcal{M}:\mathbb{R}^{H\times W\times C}\to\mathbb{R}^{H\times W\times C}$, we can derive the predicted clean image $\mathbf{I_{c}}$ by: $\mathbf{I_{c}}=\mathbf{I_{n}}+\mathcal{M}(\mathbf{I_{n}})$, where model $\mathcal{M}$ is updated by minimizing the distance between the denoising result $\mathbf{I_{c}}$ and the ground truth $\mathbf{I_{gt}}$. As we can see, this kind of approach generates a single fixed output result in a black-box manner, making it almost impossible to adjust the denoise operation explicitly.

In this section, we present Real-time Controllable Denoising (RCD), a novel deep learning-based pipeline for real-time controllable denoising. As illustrated in Fig. 3, RCD essentially consists of three parts: (1) A backbone network, i.e., $\mathcal{M}_{b}:\mathbb{R}^{H\times W\times C}\to\mathbb{R}^{H\times W\times LC}$, generates multiple fixed-level noise maps, where $L$ is the number of pre-defined noise levels (see (A) in Fig. 3). (2) A Noise Decorrelation (ND) block that enforces the editability of the generated noise maps (see (B) in Fig. 3). (3) An AutoTune module that gives a default set of control parameters to generate the best denoising result.

Specifically, the backbone network will generate multiple fixed-level noise maps, i.e., $\{{\mathcal{N}}_{i}\}_{i=1}^{L}$, for each noisy image input. The noise maps are then fed into the proposed Noise Decorrelation (ND) block, which makes noise maps orthogonal to each other. In this way, the decorrelated noise maps $\{\tilde{\mathcal{N}}_{i}\}_{i=1}^{L}$ will be zero-correlated and thus become linearly interpolable. At last, the AutoTune module will give a set of suggested control parameters $\{\bar{c}_{i}\}_{i=1}^{L}$ to generate the final denoising result as follows:

$$\mathbf{I_{c}}=\mathbf{I_{n}}+\sum_{i=1}^{L}\bar{c}_{i}\tilde{\mathcal{N}}_{i},$$

(1)

where $\sum_{i=1}^{L}\bar{c}_{i}=1$. Moreover, given the zero-correlated noise maps, users can also generate arbitrary strength denoising results by replacing $\{\bar{c}_{i}\}_{i=1}^{L}$ with their own customized control parameters $\{c_{i}\}_{i=1}^{L}$.

Given a noisy input image $\mathbf{I_{n}}$, the backbone network aims to generate multiple noise maps $\{\mathcal{N}_{i}\}_{i=1}^{L}$, corresponding to a set of pre-defined noise levels $\{l_{i}\}_{i=1}^{L}$, e.g. noise levels $\{5,10,15,...,60\}$. Hence, we have

$$\sigma(\mathcal{N}_{i})=l_{i},\forall i=1,...,L,$$

(2)

where $\sigma$ is the noise level operation that calculates the standard deviation of pixels in each noise map. To obtain multi-level noise maps, we replace the conventional last output layer of the denoising network with a convolutional layer with an output channel size of $L\cdot C$. Moreover, the level of the noise map is explicitly generated with the normalization operation, as given by

$$\mathcal{N}_{i}=l_{i}\frac{\mathcal{M}_{b}(\mathbf{I_{n}})^{(i)}}{\sigma(\mathcal{M}_{b}(\mathbf{I_{n}})^{(i)})},\forall i=1,...,L.$$

(3)

Here $\mathcal{M}_{b}(\mathbf{I_{n}})\in\mathbb{R}^{H\times W\times LC}$ is network output, and $\mathcal{M}_{b}(\mathbf{I_{n}})^{(i)}\in\mathbb{R}^{H\times W\times C}$ is the i-th component separated from the channel dimension. The derived $\mathcal{N}_{i}$ can be considered as the noise map estimated at the given noise level $l_{i}$.

Different from prior controllable denoising methods with implicit interpolation in the network, we propose to explicitly interpolate the noise maps in Eqn. 3. Thanks to the separation of noise interpolation and network inference, our RCD can achieve real-time user interaction.

However, the multi-level noise maps $\mathcal{N}_{i}$ directly obtained by convolutional layers are usually highly correlated, which leads to the problem of noise level collapse. In other words, the noise map representations in different levels are redundant, implying that the number of noise maps at different noise strengths that participate in the linear interpolation in Eqn. 1 is implicitly reduced. Without any constraint, our experiments show that the single noise map at a certain noise level would dominate in the linear interpolation for a variety of input noisy images. To address this issue, we further introduce the Noise Decorrelation block to make representations of these noise maps much more informative in the following section.

The Noise Decorrelation (ND) block is designed to regularize the backbone network to generate editable noise maps at varying levels. In particular, this block is a parameter-free computational unit that enforces $\mathcal{N}_{i}$ to be approximately zero-correlated with each other: $\mathbf{cov}(\mathcal{N}_{i},\mathcal{N}_{j})\approx 0,\forall i,j\in\{1,2,...,L\}$, where $\mathbf{cov}(\cdot,\cdot)$ is covariance operator. Inspired by the success of using the decorrelation technique in network optimization and normalization, we adopt the whitening-based methods [29, 30] for noise decorrelation here. The noise maps are decorrelated using the inverse square root of their covariance matrix.

Specifically, for each predicted fixed-level noise map $\mathcal{N}_{i}$, it will firstly be reshaped to $\mathcal{N}_{i}\in\mathbb{R}^{1\times M}$, where $M=HWC$. By stacking the reshaped $\mathcal{N}_{i}$ over the first dimension, we have $\mathbf{N}\in\mathbb{R}^{L\times M}$. We then calculate the noise covariance matrix $\sigma$ by: $\Sigma=\frac{1}{M-1}(\mathbf{N}-\bar{\mathbf{N}})(\mathbf{N}-\bar{\mathbf{N}})^{T}$ where $\bar{\mathbf{N}}$ is mean of $\mathbf{N}$ over channel $M$.

The Noise Decorrelation block needs to compute inverse square root $\Sigma^{-\frac{1}{2}}\in\mathbb{R}^{L\times L}$, which can be done by eigen decomposition or SVD. Since this kind of operation involves heavy computation [29], we instead adopt the more efficient Newton’s Iteration to estimate $\Sigma^{-\frac{1}{2}}$ as in [5, 27]. Giving a covariance matrix $\Sigma$, Newton’s Iteration calculates $\Sigma^{-\frac{1}{2}}$ by following the iterations below:

$$\begin{array}[]{l}\Sigma_{0}=I,\\ \Sigma_{k}=\frac{1}{2}(3\Sigma_{k-1}-(\Sigma_{k-1})^{3}\Sigma),k=1,2,..,T,\end{array}$$

(4)

where $k$ is the iteration index and $T$ is the iteration number (in our experiments $T$ = 3 or 4). $\Sigma_{k}$ is guaranteed to converge to $\Sigma^{-\frac{1}{2}}$, if $\left\|I-\Sigma\right\|_{2}<1$ [5]. This condition can be achieved by normalizing $\Sigma$ to $\frac{\Sigma}{tr(\Sigma)}$, where $tr(.)$ is trace operator.

The derived $\Sigma^{-\frac{1}{2}}$ can be regarded as a whitening matrix [44], which decorrelates the noise maps $\mathbf{N}$ in a differentiable manner. The decorrelated noise maps $\tilde{\mathbf{N}\in\mathbb{R}^{H\times W\times LC}}$ can be obtained by calculating: $\tilde{\mathbf{N}}=\Sigma^{-\frac{1}{2}}\mathbf{N}$. We can then have our editable fixed-level noises $\tilde{\mathcal{N}}_{i}\in\mathbb{R}^{H\times W\times C}$ by reshaping $\tilde{\mathbf{N}}$ and splitting it into $L$ noise maps. After Noise Decorrelation, we apply the same normalization as Eqn. 3 to guarantee the noise strength of the decorrelated noises.

The zero-correlated noise maps $\tilde{\mathcal{N}}_{i}$ present several excellent properties for controllable denoising. Firstly, the linearity of the noise level’s square towards $\tilde{\mathcal{N}}_{i}$ is guaranteed. In other words, given an arbitrary set of control parameters $\{c_{i}\}_{i=1}^{L}$, we have

$$\mathbf{Var}(\sum_{i=1}^{L}c_{i}\tilde{\mathcal{N}}_{i})=\sum_{i=1}^{L}c_{i}^{2}\mathbf{Var}(\tilde{\mathcal{N}}_{i}),$$

(5)

where $\mathbf{Var}(\cdot)$ denotes variance operator. Apparently, Eqn. 5 holds when elements of $\{\tilde{\mathcal{N}}_{i}\}_{i=1,2,..,m}$ are mutually zero-correlated. Eqn. 5 reveals the explicit relationship between the control parameters and the target noise level, which allows us to directly edit noises by interpolating $\mathcal{N}_{i}$ using $c_{i}$. Secondly, the Noise Decorrelation block can be regarded as a regularization tool that forces models to learn different noise formats for each level, which will increase the representation capacity of the denoising network [30].

Fig. 4 demonstrates how the Noise Decorrelation block works. With ND block, the covariance of learned noises is reduced to almost zero (without it, $|\Sigma|_{F}$ can be 751 and unignorable), allowing us to derive determined interpolated results with target noise intensity. In contrast, without the Noise Decorrelation blocks, the output noise level can not be guaranteed.

Given the decorrelated noise maps from the Noise Decorrelation block, the AutoTune module will predict a set of model-suggested control parameters, i.e., $\{\bar{c}_{i}\}_{i=1}^{L}$, to generate the default denoising result. Users can then use this set of parameters as a starting point to fine-tune their final desired denoising strength. Our AutoTune module is extremely lightweight, and is formulated as a single-layer module with temperature softmax activation. Specifically, $\{\bar{c}_{i}\}_{i=1}^{L}$ can be obtained by : $\bar{c}_{i}=\frac{e^{\frac{\mathcal{A}(f)_{i}}{\tau}}}{\sum_{j=1}^{n}e^{\frac{\mathcal{A}(f)_{j}}{\tau}}},$ where $\mathcal{A}$ is the NN layer, $f$ is the input feature maps, and $\tau$ is temperature. In our experiments, $\tau$ is set to be $0.05$ for best performance. Following the design ethos of efficiency and least coupling to the backbone architecture, we directly choose the unnormalized model outputs $\mathcal{M}(\mathbf{I_{n}})$ as $f$ (see (C) in Fig. 3).

Unlike existing methods that only modulate noise intensity, our RCD control scheme allows users to further optimize the denoising result to a given noise intensity by tuning $\{c_{i}\}$, as long as the weighted mean of $l_{i}$ towards $c_{i}$ remains the same. Eqn. 5 shows that when $\sum_{i=1}^{L}c_{i}^{2}\mathbf{Var}(\tilde{\mathcal{N}}_{i})=\sum_{i=1}^{L}c_{i}^{2}l_{i}^{2}$ is fixed, the variance of the output noise $\mathbf{Var}(\sum_{i=1}^{L}c_{i}\tilde{\mathcal{N}}_{i})$ would be also fixed. By exploring $c_{i}$ under the condition of fixed $\sum_{i=1}^{L}c_{i}^{2}l_{i}^{2}$, we can further optimize the denoising results at a specific noise level by involving different components ($\mathcal{N}_{i}$) in the noise interpolation. As shown in Fig. 5, our AutoTune module can generate high-quality results using just the reference control parameters $\{\bar{c_{i}}\}$. Users can further improve the result by artificially tuning $\{c_{i}\}$ around $\{\bar{c_{i}}\}$, even at the same noise level.

To guarantee the visual quality of arbitrarily edited noise $\sum_{i=1}^{L}c_{i}\tilde{\mathcal{N}}_{i}$ for any given set of control parameters $\{c_{i}\}_{i=1}^{L}$, we adopt a multi-level concurrent training strategy by minimizing the difference between each level’s noise output and the ground truth noise, i.e., $\mathcal{L}_{level}$, which is derived by:

$$\mathcal{L}_{level}=\frac{1}{L}\sum^{L}_{i=1}\mathcal{L}(\mathbf{I}_{gt},\mathbf{I_{n}}+\tilde{\mathcal{N}}_{i})$$

(6)

where $\mathbf{I}_{gt}$, $\mathbf{I_{n}}$ are ground truth clean image and the input noisy image. $\mathcal{L}(.)$ can be any loss functions (e.g., L2 loss or PSNR loss).

Spectacularly, $\mathbf{I_{n}}+\tilde{\mathcal{N}}_{i}$ can be regarded as corner cases of RCD when we use one-hot control parameters as input. Joint optimization of all the noise levels ensures that each element of $\mathcal{N}_{i}$ can be trained as optimal noise estimation under the condition of fixed noise level $l_{i}$.

Together with the AutoTune module optimization, our final cost function can be written as:

$$\mathcal{L}_{total}=\lambda\mathcal{L}_{level}+\mathcal{L}(\mathbf{I}_{gt},\mathbf{I_{n}}+\sum_{i=1}^{L}\bar{c}_{i}\tilde{\mathcal{N}}_{i}),$$

(7)

where $\lambda$ is the loss weight ($\lambda=0.1$ in our experiments), and $\mathcal{L}(\mathbf{I}_{gt},\mathbf{I_{n}}+\sum_{i=1}^{n}\bar{c}_{i}\tilde{\mathcal{N}}_{i})$ optimizes the denoising result derived by the model-suggested control parameters.

Experimental Setup. To fully demonstrate the effectiveness of the proposed RCD, we choose the most recent SOTA method NAFNet [11] as our backbone. Following [56], we first conduct denoising experiments on several widely-used synthetic color image benchmarks (DIV2K [2], BSD400 [36], Flickr2K [55] and WaterlooED [34]) with additive white Gaussian noise ($\sigma\in[0,60]$). The training patch size is 128 × 128 and the batch size is 64. We train our model with Adam [31] optimizer and learning rate $1e-3$ for total 60K iterations. Consistent to [11], PSNR loss is adapted as the loss function. Both the baseline model (NAFNet) and its RCD variants (NAFNet-RCD) are trained from scratch. For settings of RCD, we initialize $L=12$ and $\{l_{i}\}=[5,10,...,60]$ for synthetic denoising training.

Complexity analysis. Extensive adjustments of controllable parameters are often required to obtain one satisfying result for users. Therefore, editing time is vital for controllable methods. This section compares the inference and editing latency of our RCD and conventional controllable pipelines on GTX 1080Ti. As shown in Tab. 2, the proposed RCD not only outperforms other conventional controllable pipelines on inference time, but more importantly, can overwhelm those traditional controllable designs on editing time, which can be more than 2000 times faster (as editing process of RCD is network-free, without reliance on sub-networks). This comparison confirms that our RCD is more than enough for real-time image editing.

Results Analysis. We evaluate our proposed method on widely used synthetic noise datasets CBSD68 [37], Kodak24 [19], McMaster [60] and Urban100 [28] with noise levels $\sigma(15)$, $\sigma(15)$ and $\sigma(50)$. RCD is evaluated with denoising results using AutoTune outputs$\{\bar{c}_{i}\}$. As shown in Tab. 1, NFANet-RCD achieves comparable performance to the baseline NFANet consistently on multiple datasets, indicating that our plug-in RCD module enables real-time controllable denoising for NAFNet without sacrificing its original denoising performance. Please note that NAFNet-RCD can yield comparable results to the backbone just by using the AutoTune outputs, and the performance can be further improved by manually tuning the control parameters ( See Sec. 3.6.) We further show the qualitative performance of NAFNet-RCD in Fig. 6. NAFNet-RCD can recover more details of some degraded images, which may be benefited from RCD’s richer representation capacity by integrating multiple noise maps.

Slimmer Model Variants. Towards the goal of evaluating the compatibility and robustness of RCD, we conduct ablations by applying RCD to different-sized backbones. Specifically, we shrink the width and block numbers of NAFNet, denoting derived models as NAFNet-small ($\frac{1}{4}\times$) and NAFNet-tiny ($\frac{1}{16}\times$). Tab. 3 reports the results of RCD with those scaled backbones. It can be observed that the RCD-variants can achieve comparable and even slightly better denoising results compared to their baselines, which further demonstrates RCD’s robustness and effectiveness for different-sized backbones.

Experimental Setup (Real Image) Unlike existing controllable denoising methods [24, 51] which focus on synthetic benchmarks, we are the first solution that attempts to extend controllable denoising to real-world SIDD datasets. SIDD consists of real noisy images captured by smartphones with $\sigma\in[0,50]$. Instead of using full SIDD data, we choose subsets of SIDD with $\sigma\in[0,12]$ (around 70% of the entire dataset) to train our RCD model, which is initialized with $L=4$ and $\{l_{i}\}=[3,6,9,12]$. The main reason is the lacking of high $\sigma$ data at given levels in SIDD because of SIDD’s highly long-tailed noise level distribution. Specifically, most noisy images in SIDD gather in $\sigma<12$ and the samples distribute sparsely when $\sigma$ is large. Consistent to Sec. 4.1, we adopt NAFNet (SOTA methods for SIDD challenge [11]) as our backbone at two scales ($1\times$, $\frac{1}{16}\times$). Both NAFNet-RCD and the corresponding baselines are trained on this subset with the same training settings as in [11].

Results and Analysis We conduct blind denoising experiments on SIDD with different RCD model scales to evaluate its adjustability to the real-world dataset. As shown in Tab. 4 (left), our RCD (AutoTune results) can achieve high-quality controllable real-world denoising in both model scales. However, we note that enabling controllable denoising with RCD may still result in a slight decrease in quantitative results (about 0.3dB), which may be a result of unbalanced data for each level and short noise level interval ($|l_{i+1}-l_{i}|$, see more discussion in Sec. 4.4).

SIDD with synthetic noise. We extensively conduct synthetic denoising experiments on SIDD to further show the compatibility of RCD on SIDD datasets. Following Sec. 4.1, we add random Gaussian noise $\sigma\in[0,60]$ to SIDD training data, and both methods are evaluated on $\sigma=50$ SIDD test samples. As shown in Tab. 4 (right), RCD models slightly outperform their baselines, demonstrating RCD’s compatibility for SIDD. Moreover, this result can also indicate that RCD’s performance drop on SIDD real image may arise from the noise distribution and RCD configurations, rather than RCD’s adaptive capacity to SIDD data. See Appendix for more results and visualizations.

Experiment Setup Following common practice [47, 45, 32], we train our models on DAVIS training set and use DAVIS-test and Set-8 for benchmarking. Like in [47], we add Gaussian noise with random standard deviation between 5-50 on the DAVIS clean videos for training. The DAVIS set contains 30 color sequences of resolution $854\times 480$, which will be randomly cropped into $128\times 128$ patches during training. Other training settings and hyperparameters are kept the same as [47] for a fair comparison.

Choice of Basic model. We choose FastDVD [47] as our backbone model. Although recent methods [49, 32] outperform FastDVD by 1-2 PSNR at most, they actually introduce huge models and extra heavy operations like patch clustering [49] and layer-wise frame-to-frame wrapping using optical flow [32] ( $>100\times$ slower than FastDVD).

Results and Analysis. Like [45], we evaluate our video denoising models with the input length of one frame and five frames. We denote RCD models for video denoising as “FastDVD-RC” and compare their quantitative AutoTune denoising results to baseline FastDVD in Tab. 5. Consistent with preceding sections, AutoTune results of FastDVD-RCD can demonstrate comparable performance to the default FastDVD, which means our RCD can also achieve lossless real-time noise editing in video scenarios. Unlike previous heavy controllable denoising methods, our real-time RCD can even allow users to do online video denoising editing without any latency.

Selection of Denoising Levels. Differing from conventional denoising methods, RCD requires a group of predefined noise levels $\{l_{i}\}_{i=1}^{L}$. To evaluate how the selection of $\{l_{i}\}_{i=1}^{L}$ affects RCD’s performance, we conduct ablation studies on FastDVD-RCD by changing the number of noise maps ($L$) (See Tab. 6.) All of the models are trained on noisy images with $\sigma\in(0,60]$ and uniformly sampled noise levels that $\{l_{i}=\frac{60}{L}*i\}_{i=1}^{L}$. We observe that larger $L$ means more fine-grained control on denoising, but it may incur a performance drop. In fact, when $n$ is large we find that $\mathcal{L}_{level}$ will also keep large and be hard to optimize. Trading-off performance and control precision, we empirically choose $L=12$ and noise level interval $|l_{i+1}-l_{i}|=5$ as defaults.

Control Capacity. This section discusses the representation capacity of $c_{i}$ as control parameters. Generally, $c_{i}$ controls the denoising process on two aspects: intensity and components. Firstly, the noise levels of RCD outputs are identical and can be derived by $\sigma=\sqrt{\sum_{i=1}^{L}c_{i}^{2}l_{i}^{2}}$ (See Sec. 3.4), which allows us to control the denoising intensity by changing $\{c_{i}\}$. Fig. 7 depicts visualizations of RCD-controlled denoising under different intensity settings. Besides, as discussed in Sec. 3.6, RCD supports further optimization of the denoising results at specific noise intensity by tuning $c_{i}$ by involving different components of $\tilde{\mathcal{N}_{i}}$. (Please be reminded that $\tilde{\mathcal{N}_{i}}$ is trained by $\mathcal{L}_{level}$, denoting learned optimal denoising results at every fixed level $l_{i}$.)