Spike-NeRF: Neural Radiance Field Based On Spike Camera

Neural Radiance Field (NeRF) [1] has arisen as a significant development in the field of Computer Vision and Computer Graphics, used for synthesizing novel views of a scene from a sparse set of images by combining machine learning with geometric reasoning. Various research and approaches based on NeRF have been proposed recently. For example, [18, 19, 20, 21] extend NeRF to dynamic and non-grid scenes, [2, 3] significantly improve the rendering quality of NeRF and [9, 8, 22, 23] robustly severe blurred images that could affect the rendering quality of NeRF.

Neuromorphic sensors have shown their advantages in most computer version problems including novel views synthesizing. EventNeRF [24] and Ev-nerf [25] synthesize the novel view in scenarios such as high-speed movements that would not be conceivable with a traditional camera by event supervision. Nonetheless, these works assume that event streams are temporally dense and low-noise which is inaccessible in practice. Robust e-NeRF [26] incorporates a more realistic event generation model to directly and robustly reconstruct NeRFs under various real-world conditions. DE-NeRF [27] and E2NeRF [28] extend event nerf to dynamic scenes and severely blurred images as NeRF researchers did.

As a neuromorphic sensor with high temporal resolution, spike cameras [12] offer significant advantages on many high-speed version tasks. [29] and [13] propose spike stream reconstruction methods for high-speed scenes. Subsequently, deep learning-based reconstruction frameworks [14, 15] are introduced to reconstruct spike streams robustly. Spike cameras also show its superiors on downstream tasks such as optical flow estimation [16, 30], monocular and stereo depth estimation [17, 31], Super-resolution [32] and high-speed real-time object tracking [33].

Unlike traditional cameras that record the luminance intensity of each pixel during the exposure time at a fixed frame rate, tensors on spike cameras of each pixel capture photons independently and keep recording the luminance intensity asynchronously without a dead zone.

Each pixel on the spike camera converts the light signal into a current signal. When the accumulated intensity reaches the dispatch threshold, a spike is fired and the accumulated intensity is reset. For pixel $\boldsymbol{x}=(x,y)$,this process can be expressed as

where:

Here ${A}(\boldsymbol{x},t)$ is the accumulated intensity at time $t$, ${s}(\boldsymbol{x},t)$ is the spike output at time $t$ and ${I_{in}}(\boldsymbol{x},\tau)$ is the input current at time $\tau$ (proportional to light intensity). We will directly use $I(\boldsymbol{x},t)$ to represent the luminance intensity to simplify our presentation. Further, due to the limitations of circuits, each spike is read out at discrete time $nT,n\in\mathbb{N}$ ($T$ is a micro-second level). Thus, the output of the spike camera is a spatial-temporal binary stream $S$ with $H\times W\times N$ size. Here, $H$ and $W$ are the height and width of the sensor, respectively, and $N$ is the temporal window size of the spike stream.

Neural Radiance Field (NeRF) uses a 5D vector-valued function to represent a continuous scene. The input to this function consists of a 3D location x = (x, y, z) and 2D viewing direction d =($\theta$,$\phi$), while output is an emitted color c = (r, g, b) and volume density $\sigma$. Both $\sigma$ and c are represented implicitly as multi-layer perceptrons (MLPs), written as:

Given the volume density $\sigma$ and color functions c, the rendering result $I$ of any given ray $\textbf{r}=\textbf{o}+t\textbf{d}$ passes through the scene can be computed using principles from volume rendering.

where

The function T(t) denotes the accumulated transmittance along the ray from $t_{n}$ to t. For computing reasons, rays were divided into N equally spaced bins, and a sample was uniformly drawn from each bin. Then, equation 5 can be approximated as

where:

and:

After calculating the color $I(\textbf{r})$ for each pixel, a square error photometric loss is used to optimize the MLP parameters.

Taking inspiration from NeRF, Spike-NeRF implicitly represents the static scenes as an MLP network $F_{\Theta}$ with 5D inputs:

Here, each $t_{i}$ corresponds to a frame of spike $s_{i}=\{0,1\}$ in the continuous spike stream $\mathbb{S}=\{{s}_{i}\in\mathbb{R}^{W\times H}|i=0,1,2,\dots\}$ captured by a spike camera in a very short time. Considering the difficulty of directly using spike streams for supervision, we firstly reconstruct the spike stream $\mathbb{S}$ into image sequence $\mathbb{I}=\{{im}_{i}\in\mathbb{R}^{W\times H}|i=0,1,2,\dots\}$ where ${im}_{i}$ is the reconstructed image at $t_{i}$. We use the results with inputs=$\mathbb{I}$ as our baseline. Since all methods reconstruct images from multi-frame spikes, using the reconstructed images as a supervision signal would lead to artifacts and blurring. We introduce spike masks ${M_{s}}$ to make NeRF focus on the triggered area. We also propose a spiking volume renderer based on the coding method of the spike camera to generate spike streams for novel views. We then use the g.t. spike streams constraint network directly.

The total loss used to train Spike-NeRF is given by:

$L_{recon}$ is the loss between the image rendering result and masked images which are reconstructed from g.t. spike streams. $L_{spike}$ is the loss between the spike rendering result generated by our spiking volume rendering method and g.t. spike streams.

If we introduce time $t$ into the volume rendering equation 5, the rendering results $I(\textbf{r},t)$ of any given ray $\textbf{r(t)}=\textbf{o(t)}+k\textbf{d(t)}$ at time $t$ is:

Where

Then, if we assume that for any $x$ $A(x,t_{0})=0$, equation LABEL:ax can be written as:

Here,$\phi$ is the threshold of the spike camera and N is the number of ”1” for spike streams $\mathbb{S}(x)=\{{s}_{t_{i}}|t_{i}\in(t_{0},t)\}$ and x=(x,y) is the coordinates for each pixel. For computing reasons, rays were divided into $N_{0}$ equally spaced bins, $(t_{0},t)$ were divided into $N_{1}$ equally spaced bins, and a sample was uniformly drawn from each bin. Then, equation 2 can be written as:

where:

where:

and:

However, $A(x,t_{0})$ is not equal to 0 in real situations. To address this, we introduce a random startup matrix and utilize the stable results after several frames. The above processes do not participate in backpropagation as they are not differentiable. After generating spike streams $\mathbb{S}$, we can compute:

Due to the serious lack of information in a single spike, all reconstruction methods use multi-frame spikes as input. These methods can reconstruct images with detailed textures from spike streams, but can also introduce erroneous information due to the use of preceding and following frame spikes (see Figure 4 original), which results in foggy edges in the scene learned by NeRF. We introduce spike masks ${M_{s}}$ to solve this problem.

Because of the spatial sparsity of the spike streams, using a single spike mask will lead to a large number of cavities. To address this, we use a relatively small number of multi-frame spikes to fill the cavities. Considering spike $s_{i}$ at time $t_{i}$ and reconstruction result $im_{i}$, we first choose $\mathbb{S}_{ti}=\{{s}_{j}\in\mathbb{R}^{W\times H}|j=i-n,i-n+1,\dots\ ,i,\dots\ ,i+n-1,i+n\}$ as original mask sequence. Finally, we have:

where $|$ means or. After masking image sequence $\mathbb{I}$, we can compute:

Our code is based on NeRF [1] and we train the models for $2*10^{5}$ iterations on one NVIDIA A100 GPU with the same optimizer and hyper-parameters as NeRF. Since the spiking volume renderer requires continuous multi-spikes, we select the camera pose and sampling points for spiking volume rendering determinedly rather than randomly as NeRF did. We examined our method on synthetic sequences from NeRF [1]. We examined six scenes(lego, ficus, chair, materials, hotdog and drums) which cover different conditions. We rendered all of them with a 0.025-second-long 360-degree rotation of the camera around the object resulting in 1000 views to simulate the 40000 fps spike camera and other blurred images in 1000 views to simulate the 400 fps high-speed traditional camera by Blender. Like NeRF, we directly use the corresponding camera intrinsics and extrinnsics generated by the blender.

We compare the spike NeRF results with three baselines: Spk2ImgNet+NeRF[14], the 400 fps traditional camera results on NeRF and BAD-NeRF[9](see Figure 3).To better demonstrate the adaptability of our method to spike cameras, we show both gray and RGB results. From Figure 3 we know that our method has absolute advantages over NeRF and BAD-NeRF in high-speed scenes. Compared with directly using spike reconstruction results for training, our method also has obvious supervisors. Corresponding numerical results are reported in Table I from what we can conclude our method improves more in gray space. We also compared the data sizes of two modalities: spikes and images. From Figure 5 we can conclude that spikes have less information resulting in noise and loss of detail. However, our method leverages temporal consistency (see section IV) to derive stable 3D representations from information-lacking and unstable spike streams.

Compared with baselines, our Spike-NeRF introduces two main components: spike masks and the spike volume renderer with spike loss. Next, we discuss their impact on the results.

Spike Loss: In section IV, we proposed spike loss to solve the cavities caused by the partial information loss due to spike masks and the blur caused by incorrect reconstruction. Figure 6 shows the results before and after disabling spike loss. From Figure 6 we know that after disabling spike loss, some scenes have obvious degradation in details and a large number of wrong holes. TabII shows the improvement of the spike loss.

Spike masks: Incorrect reconstruction will also lead to a large number of artifacts in NeRF results. We use spike masks to eliminate artifacts to the maximum extent (see IV). Figure 7 shows the results before and after disabling spike masks. From Figure 7 we know that after disabling spike masks, all scenes have obvious artifacts. TabII shows the improvement of the spike masks.