SpikeNVS: Enhancing Novel View Synthesis from Blurry Images via Spike Camera

Novel view synthesis methods, such as Neural Radiance Fields (NeRF) [18] and 3D Gaussian Splatting (3DGS) [12], have garnered significant attention due to their remarkable capability learned from multi-view images. The NeRF model predicts color and density from 3D scene coordinates and ray information, subsequently volume-rendered into an image. On the other hand, 3DGS uses the learnable 3D Gaussian to represent the scene and employs the splatting method for rendering. In both methods, achieving sharp NVS heavily relies on the quality of training images. However, RGB cameras are prone to motion blur, compromising the captured images and learning process. Previous approaches, such as NeRF-W [16] and Deblur-NeRF [15], have attempted to model image degradation factors like blur, occlusion, and illumination changes. Nevertheless, these methods fail to address the inherent limitations of RGB cameras. In contrast, neuromorphic cameras like event and spike cameras inherently capture richer temporal information than RGB cameras. Consequently, a few recent studies have proposed enhancing NVS with neuromorphic cameras.

The primary focus of event-RGB methods is to utilize event cameras for capturing dynamic objects and guiding the learning process [9, 23, 24]. However, event cameras rely on temporal differentiation, restricting their ability to capture static scenes like the background. Additionally, these approaches use an event-based loss function that requires multiple independent renderings for differentiation calculation. These methods lead to significantly higher training costs. The limitations shown in Fig. 1 (b) arise due to the trade-off between precision and computational cost in E2NeRF [23], an event-RGB NeRF method.

In contrast, the spike camera measures accumulated brightness to activate a spike pulse, perfectly avoiding the intrinsic limitations of event-based methods. This temporal integration property allows for the simultaneous capture of static and dynamic scenes, avoiding the need for independent rendering for asynchronous differentiation. With this insight, we propose a novel Texture from Spike (TfS) loss to introduce spike data to the NVS methods. Specifically, Texture from Interval (TFI) and Texture from Playback (TFP) [4] are two commonly used spike image texture reconstruction techniques. As demonstrated in Fig. 2 (a), TFI evaluates the relative trigger frequency within an interval and reconstructs the image based on the eligibility traces. Conversely, TFP utilizes a sliding time window (e.g., 32, 64 timestamps) to aggregate the spike pulses within this time window. Our TfS loss leverages the benefits of both TFP and TFI in a learnable manner, as shown in Fig. 2 (b).

We also build a new dataset with our synchronized spike-RGB camera system to evaluate our method on real-world scenes. In summary, our main contributions can be concluded as follows:

1. 1.

   We present a pioneer study that assesses the advantages of spike over event streams regarding both quality and cost for NVS. We have developed the first spike-RGB NVS technique based on this.

2. 2.

   We propose a novel Texture from Spike (TfS) loss that effectively integrates the advantages of both Texture from Interval (TFI) and Texture from Playback (TFP) results, which are commonly employed for spike reconstruction, into the learning process.

3. 3.

   We have developed a synchronized spike-RGB camera system with aligned field-of-view and trigger time. We contributed the first real-world dataset for spike-RGB NVS based on it.

We first commences with a concise introduction to NeRF, 3DGS, and spike reconstruction. Subsequently, we demonstrate of our synchronized spike-RGB camera system. Ultimately, all components are integrated to form our final pipeline.

The First section of the Appendix will provide a more comprehensive account. However, due to spatial constraints, we will present a concise overview on the generation and collection of spike data, experimental setup, and result analysis in this section to showcase our contributions.

In Fig. 8 we demonstrate additional advantages that spike streams bring to our test, including facilitating camera pose estimation and mitigating overexposure. We utilize COLMAP [26, 27] for camera pose estimation. Direct estimation of camera poses from blurry RGB inputs is notoriously difficult. However, our experiments reveal that spike reconstruction significantly enhances COLMAP’s ability to accurately estimate camera poses. The robustness of COLMAP’s model estimation algorithms, such as RANSAC and bundle adjustment, largely depends on the availability of high-quality and consistent feature points across images. Spike reconstruction contributes to this process by providing denser and clearer feature points, which are essential for precise feature matching in COLMAP.

In summary, the findings of our study showcase the superior performance of spike streams compared to event streams in the context of 3D reconstruction with blurry images task from various perspectives. Our novel ’Texture from Spike’ (TfS) loss function, coupled with an efficient end-to-end training process, demonstrates state-of-the-art performance without incurring substantial additional computational costs. Besides, the creation of the synthetic and real-world RS-NeRF datasets marks a pioneering step towards furthering research in the mutual enhancement of spike data and Implicit Neural Representation (INR) networks. Our approach opens new avenues for leveraging the high temporal resolution of spike cameras, surpassing traditional RGB cameras in capturing fast-moving scenes without motion blur and in challenging lighting conditions.

Future work could explore the extension of the SpikeNeRF framework to other domains where high-speed capture is crucial. Despite the limited availability of spike hardware in the market, we anticipate that future research will unveil further applications for spikes in diverse visual tasks, similar to event cameras but with their distinctive attributes.

Appendix - SpikeNeRF: Enhancing Learning-based 3D Reconstruction from Blurry Images using Spike Camera

This appendix of SpikeNeRF includes:

• •

  Experimental Setups: The model aligns with standard NeRF configurations in terms of depth and width parameters, trained on a single NVIDIA A100 GPU. Baseline description and detailed hyperparameters are provided, ensuring a balance between computational efficiency and performance.

• •

  Computational Complexity Analysis: A detailed comparison between the computational costs of E2NeRF and SpikeNeRF is provided, highlighting the efficiency of SpikeNeRF in resource management.

• •

  Comparison between Cameras and Future works The cost of the equipment poses a practical concern. We conducted a comparison of the cost and configuration between spike, event, and high-speed RGB cameras. Additionally, we explored the potential applications of next-generation spike cameras that incorporate an RGB channel.

Our study undertakes a detailed comparison with the E2NeRF model, building upon its framework and incorporating elements from the standard NeRF architecture as described in [23]. The training was conducted on a single NVIDIA A100 GPU, mirroring the typical resource allocation in standard NeRF implementations. Notably, our approach aligns with common NeRF configurations, specifically in terms of the ‘depth‘ and ‘width‘ parameters of the model. These parameters are crucial, as the ‘depth‘ corresponds to the number of ‘Dense‘ layers, and the ‘width‘ to the number of units in each layer, as is standard in NeRF models. We also includes the baseline description in Tab. 4

As we discussed in Sec. 1 of our main text, there exists a fundamental trade-off between event loss accuracy and computational efficiency in event-based neural rendering frameworks. To illustrate, consider two sampled timestamps $t_{0}$ and $t_{1}$, which exhibit similar brightness values. These values do not trigger an event as their difference falls within the non-activation range ($-\Theta<b_{t_{1}}-b_{t_{0}}<\Theta$), where $\Theta$ denotes the event threshold. However, introducing an intermediary timestamp $t_{\frac{1}{2}}$ reveals that an event does occur within the interval $t_{0}$ to $t_{1}$, particularly if $b_{t_{\frac{1}{2}}}-b_{t_{0}}>\Theta$.

A straightforward solution to this detection dilemma is to decrease the time sampling interval, thereby capturing more granular changes in brightness. Yet, this approach introduces a significant computational overhead. In E2NeRF [23], for instance, each additional sampled timestamp proportionally increases the computational load for event-based loss computation, as illustrated in Fig. 1 in our main text.

To quantify this complexity, we analyze the inference operations in the original NeRF network architecture, as shown in Fig. 9 (a). The computational load for each fully connected (dense) layer is determined by the number of neurons in both the current and preceding layers. Consequently, the total inference complexity, denoted as $C$, is calculated by summing the products of neuron counts across successive layers:

In the case of E2NeRF, which employs an event-based loss, each rendered event requires independent calculations, resulting in a computational cost of at least $N\times C$ if we have $N$ sampling timestamps. Conversely, SpikeNeRF primarily incurs additional computational expenses in the RGB converter layer, represented by $O(n_{L-1}\cdot n_{L})$. This distinction underscores the efficiency of SpikeNeRF in managing computational resources.

In Fig. 10 Column 3 presents a comparison between spike cameras, event cameras, and regular high-speed cameras in terms of specifications and prices. Spike cameras prove to be more cost-effective than high-speed cameras; with similar hardware costs, they yield superior reconstruction results while requiring significantly less training expenditure compared to event cameras. In column 4, next-generation spike cameras offer higher resolution and RGB content capabilities, enabling direct utilization for NeRF or 3DGS training purposes—a potential breakthrough when considering scenes involving relative movement between the camera and targets.