Unsupervised Coordinate-Based Video Denoising

Our Feature Generator processes a batch of uniformly sampled coordinate grids $\{G_{t}|t=1,...,B\}$, where each $G_{t}\in\mathbb{R}^{H\times W\times 3}$, to generate a corresponding batch of feature maps $\{F_{t}|t=1,...,B\}$, each being $F_{t}\in\mathbb{R}^{H\times W\times C}$. Here, $B$ is the batch size, while $C$ is the number of feature channels.

Before the coordinate grid is fed into $\mathcal{F}_{\theta}$, each coordinate $\textbf{p}_{t}=(x,y,t)$ is subject to positional encoding nerf ; mipnerf ; rawnerf . This encoding step transforms low-dimensional input coordinates into a higher-dimensional space, enabling the model to better learn and represent high-frequency details inherent in the image data. The positional encoding function we adopt is given as:

In this equation, $L$ serves as a hyperparameter controlling the level of detail or high-frequency information in the output. By selecting a smaller value for $L$, we can effectively reduce the level of high-frequency noise in the image data, as noise often manifests as high-frequency information. For our experiments, we set $L=30$. The input coordinates, normalized to the range $[-1,1]$ using a mesh grid, are passed through the encoding function $\gamma(.)$. The resulting high-dimensional output $\gamma(G_{t})\in\mathbb{R}^{H\times W\times 6L}$ is subsequently fed into the feature generator $\mathcal{F}_{\theta}$:

where $F_{t}\in\mathbb{R}^{H\times W\times C}$ is the feature maps corresponding to each noisy frame, and $C$ denotes the channel size of the output features.

Our feature generator is composed of 6 convolution layers, each featuring a kernel size of 3 and identical padding to maintain the size throughout the model layers. Each layer has 256 feature channels with batch normalization (BN) applied solely to the first two layers. ReLU activation is employed for all layers, except for the last one.

The denoiser network takes the concatenated feature maps output from the feature generator as input, and generates a denoised central frame $\hat{I}_{B}$:

where the concatenation is applied along the feature channel dimension. This allows $\mathcal{D}_{\phi}$ to learn spatial-temporal patterns along the neighboring time frames eliminate the noise. Note that the output of $\mathcal{D}_{\phi}$ may be somewhat blurred, because we’ve set a low $L$ in the feature generator.

The architecture of $\mathcal{D}_{\phi}$ includes 6 convolutional layers. ReLU activation is used in the first five layers, and a sigmoid activation function is used in the last layer. Unlike in the feature generator, batch normalization is not applied in this stage. Each layer uses 256 filters with a kernel size of 3, except for the last two layers which have 96 and the number of color channels filters, respectively, with kernel sizes of 1.

The refine-net is built upon the backbone of the Sinusoidal Representation Networks (SIREN) siren , which is a type of coordinate-based network that uses periodic activation functions, particularly sine functions, in place of traditional activation functions like ReLU. SIREN’s unique characteristic lies in its ability to naturally model the high-frequency details of complex patterns by leveraging its intrinsic periodic activation functions. In our context, $\mathcal{R}_{\eta}$ uses the SIREN network to take the coordinates grid of the central frame as input and generate the refined image $\hat{I}_{R}$.

where $G_{t}^{c}$ is the coordinates of the central frames in the input batch.

Note that the SIREN-based refine-net is particularly beneficial in our case, as it helps to further refine the denoised output from $\mathcal{D}_{\phi}$ by enhancing the finer details and correcting any blurring introduced in the denoising process. The output from $\mathcal{R}_{\eta}$ represents the final denoised and refined video frames.

In the first stage, we jointly train the feature generator $\mathcal{F}_{\theta}$ and the denoiser $\mathcal{D}_{\phi}$ in an end-to-end fashion. The objective function for this stage is composed of two parts. The first part is the $l_{1}$ loss, which measures the difference between the denoised central frame $\hat{I}_{B}$ and the central frame $I^{c}$ of the input batch of noisy frames $\{I_{t}|t=1,2,\dots,B\}$:

The second part of the objective function ensures that the feature maps generated by $\mathcal{F}_{\theta}$ capture the image information. This is achieved by enforcing similarity between the central channels of each feature map and the corresponding noisy frame:

where $F_{t}^{c}$ is the central channels of each feature map $\mathcal{F}_{\theta}(\gamma(G_{t}))$ and $I_{t}$ is the corresponding noisy frame, as shown in Fig. 2 (a).

The final loss function for the first stage is the sum of these two losses:

where $\lambda_{1}$ is a weight parameter that controls the trade-off between the two terms. To ensure our model doesn’t simply learn to reproduce the noise present in the input, we train the network for approximately 2000 epochs.

In the second stage, we focus on training the Refine-Net $\mathcal{R}_{\eta}$. Unlike in the first stage, where $\mathcal{F}_{\theta}$ and $\mathcal{D}_{\phi}$ were trained together, here we fix $\mathcal{F}_{\theta}$ and $\mathcal{D}_{\phi}$ and solely train $\mathcal{R}_{\eta}$.

We use the coordinates of the central frame $G_{t}^{c}$ as input to $\mathcal{R}_{\eta}$, which outputs a refined denoised image $\hat{I}_{R}$. As this network is designed to further improve the quality of the denoised frames, the loss function for this stage is defined to measure the difference between the output of $\mathcal{R}_{\eta}$ and both the noisy central frame $I_{t}^{c}$ and the denoised central frame $\hat{I}_{B}$ from the first stage. Specifically, the loss function is defined as:

where $\lambda_{2}$ and $\lambda_{3}$ are weight parameters controlling the contribution of each term. These parameters help balance the network’s objectives of reducing noise (by making the output similar to $I_{B}^{c}$) and preserving details (by making the output similar to $I_{t}^{c}$).

This strategy allows the Refine-Net to leverage the advantages of both the noisy and denoised frames, by enhancing details and suppressing noise. The training of $\mathcal{R}\eta$ is performed until satisfactory results are obtained, typically for about 2000 epochs. It’s important to note that the second stage training does not affect the training of $\mathcal{F}_{\theta}$ and $\mathcal{D}_{\phi}$, which is crucial for preserving the generalization capability of the whole framework.

Our two-stage optimization can effectively remove the noise in the video frames. The first image on the left represents the initial state, which is typically a noisy video frame. The joint training of the feature generator $\mathcal{F}_{\theta}$ and the denoiser $\mathcal{D}_{\phi}$ shows a noticeable reduction in noise (middle frame), but some blurriness might still be present due to the low $L$ we used in our feature generator. The right image showcases the result of the refining stage $\mathcal{R}_{\eta}$. At this stage, the high-frequency details that might have been lost in the denoising process are recovered, resulting in a crisp, clean frame that retains the original structure and details of the scene, demonstrating the step-by-step improvement of our method.

Our approach is tested on a variety of datasets, encompassing both synthetic and real-world scenarios, to provide a comprehensive evaluation of its performance. Synthetic data are derived from established benchmarks and intentionally corrupted with diverse types of noise, while real-world data are sourced from calcium imaging experiments. Moreover, we outline the specifics of our computational setup and training parameters, detailing the choices that were made to optimize our model’s performance. This thorough experimental setup is aimed at providing a robust assessment of our proposed video denoising method and its potential applicability to different types of video data.

To facilitate a quantitative evaluation of our approach, we employ two widely-accepted metrics in the image and video processing community: the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). These measures allow us to objectively compare our results with those of the current state-of-the-art unsupervised video denoising algorithm, UDVDudvd .

The outcomes, as displayed in Table 1, highlight that our method consistently outperforms UDVD across all tested datasets and noise types. This superior performance is primarily attributed to our approach’s effective utilization of spatial-temporal information from the video sequence. Our algorithm leverages this information to efficiently eliminate noise while simultaneously preserving high-frequency details within the frames.

When examining the visual comparison results, as shown in Fig. 3, it becomes clear that UDVD often struggles to effectively remove noise when dealing with short input sequences. This challenge is particularly evident when confronted with Poisson and Impulse noise types, where UDVD tends to produce noticeable artifacts. Conversely, our method shows remarkable resilience even in these demanding situations. We tested our approach on ten short video sequences, each consisting of ten frames, and the results consistently demonstrated our method’s superior noise removal capability and robustness.

Overall, the quantitative and qualitative evaluations underscore the potential of our proposed video denoising method as a robust and effective solution, particularly for applications dealing with short, noisy video sequences.

A visual comparison between our method and UDVD on the highly noisy calcium imaging sequences further underscores our superior performance, as shown in Fig. 4. In the noisy frames, it can be challenging to distinguish individual cells due to high levels of noise. UDVD, while effective in reducing some noise, often blurs the intricate cellular structures, making it difficult to identify individual cells.

In contrast, our approach not only removes the noise effectively but also preserves the intricate cellular structures, allowing for better visualization and identification of individual cells. This difference is particularly notable in regions with a high density of cells, where our method is able to maintain the distinct boundaries between cells, whereas UDVD tends to blur them together. This visual comparison highlights our method’s ability to handle real-world data with significant noise, offering promising potential for applications in biological and medical imaging.

We conduct an ablation study to understand the contribution of each component in our method. Notably, when we omit the refining stage $\mathcal{R}_{\eta}$, the denoised frames tend to be slightly blurry due to the low $L$ in the feature generator $\mathcal{F}_{\theta}$. However, with the incorporation of the refining stage, our method is able to effectively recover high-frequency details, thereby underscoring its crucial role in enhancing the overall quality of the denoised frames. We show the visual comparison between each of network variants in Fig. 5. A detailed presentation of the ablation study results is provided in Table 5.