V2CE: Video to Continuous Events Simulator

The Motion-Aware Event Voxel prediction aims to transform the APS video into a 3D voxel grid where the video data is temporally upsampled. The temporal resolution is increased by a significant margin and the event sequence is represented in a spatio-temporal $xyt$ coordinate system. We use a simple UNet-based architecture to design this module.

However, the main challenge of this task is to preserve the temporal continuity and the microstructure compatibility of event voxels. High-fidelity event voxel reconstruction requires information about nonlinear dynamics of light intensity changes and object movements (e.g., acceleration or higher order moment). While any linear assumption invariably leads to suboptimal video-to-event conversion performance, all prior work in this space only used an adjacent frame pair to infer the events between them. Since no hint is available to infer the nonlinear dynamics, all baseline methods were essentially doing linear interpolation between the input APS frame pair.

Therefore, we advocate using longer frame sequence instead of frame pairs to serve as the input of the model, and help local temporal information flow properly during the inference. Bear these considerations in mind, we modified a 3D UNet model and use a sequence of 16 frame pairs as the input to the model.

Further complicating the task, event and APS cameras differ in dynamic ranges, which affects information compression in overexposed and underexposed areas. Additionally, both camera types have adjustable parameters such as exposure, ISO, and aperture, which can be dynamically tuned to adapt to varying environments. This renders the video-to-event voxel prediction a time-varying task, making a straightforward one-to-one mapping between APS video frames and event voxels especially challenging.

To address this complex task, a key contribution of our work is the development of a hybrid loss function composed of five distinct losses, which we briefly describe below.

Denote the input to the model as $I\in\mathcal{R}^{(B,L,2,H,W)}$, where the five dimensions represent the batch size, sequence length, and spatial resolution. Then the output event voxels satisfy $V\in\mathcal{R}^{(B,L,2\times C,H,W)}$, where $C$ represents the timebin number between two frames, and the third dimension has a shape of $2\times C$ since events of different polarities are also separated. All losses take ground truth voxels and predicted voxels as input.

The first loss to introduce is the Spatial-Temporal-Pyramid Loss (STP Loss, $\mathcal{L}_{STP}$). STP loss takes the entire concatenated voxel with a shape of $(B,L\times C,H,W)$ and applies a series of 3D Average Poolings with varying kernel sizes and strides. This produces more compact representations of both ground truth and predicted event voxels. The STP Loss encourages the model to extract multi-scale information from adjacent voxels, enhancing its robustness against noise by applying coarse supra-voxel matching. Formally, the STP Loss is defined as:

$$\mathcal{L}_{STP}=\sum_{k\in\mathcal{K},s\in\mathcal{S}}w_{k,s}\cdot\left\|\mathcal{P}^{3D}_{k,s}(V_{GT})-\mathcal{P}^{3D}_{k,s}(V_{pred})\right\|_{2}^{2}$$

(1)

Where $\mathcal{P}^{3D}_{k,s}(V)$ denotes 3D average pooling operation applied to voxel $v$ with a kernel size $k$ and stride $s$, $\mathcal{K}$ represents the set of all kernel sizes used in the pooling operations, $\mathcal{S}$ represents the set of all strides used in the pooling operations, and $w_{k,s}$ denotes the weights for each combination of kernel size $k$ and stride $s$.

Temporal-Pyramid Loss (TP Loss, $\mathcal{L}_{TP}$) is designed to prioritize neighboring events, which are crucial for voxel-level event reconstruction. We employ 1D average pooling along the time axis using varying kernel sizes and strides on both ground truth and predicted event voxels, followed by an L2 loss calculation. Formally, the TP Loss is defined similarly to STP Loss:

$$\mathcal{L}_{STP}=\sum_{k,s\in\mathcal{K,S}}w_{k,s}\cdot\left\|\mathcal{P}^{T}_{k,s}(V_{GT})-\mathcal{P}^{T}_{k,s}(V_{pred})\right\|_{2}^{2}$$

(2)

Where $\mathcal{P}^{T}_{k,s}$ denotes 1D average pooling along the time axis.

Similarly, Event Frame Loss (EF Loss, $\mathcal{L}_{EF}$) compresses the time axis by summing timebins between adjacent frames or across the entire frame sequence along the time. This addresses the issue of sparsity in voxels and ensures better and aligned information flow between generated event frames and the input frame sequence. Event frames with and without polarity consideration are separately counted in the loss calculation.

	$\displaystyle\mathcal{L}_{EF}=\left\|\mathcal{S}_{C}(V_{GT})-\mathcal{S}_{C}(V_{pred})\right\|_{2}^{2}+$		(3)
	$\displaystyle\left\|\mathcal{S}_{LC}(V_{GT})-\mathcal{S}_{LC}(V_{pred})\right\|_{2}^{2}$

Where $\mathcal{S}_{C}(\cdot)$ and $\mathcal{S}_{LC}(\cdot)$ denotes the compression operation that sums over timebins $C$ between adjacent frames and the entire frame sequence $LC$ respectively.

Adversarial Loss (ADV Loss, $\mathcal{L}_{ADV}$) aims to enhance the realness of our generated event voxels. Utilizing both ground truth and predicted voxels as real and fake samples respectively, the discriminator is trained for optimal distinction. To prevent $\mathcal{L}_{ADV}$ from becoming unbounded, the generated event voxels strive for high similarity with real voxels to effectively deceive the discriminator.

As previously noted, the relationship between APS frames’ pixel brightness and the event number between frame pairs is not static, necessitating a dynamic, semantics-based modeling of intrinsic camera parameters. This complexity arises because APS captures brightness as $\phi(I)$, while DVS records $\log(\phi(I))$, where $I$ is the scene’s absolute brightness and $\phi(I)$ represents the effect of camera parameters. Given that multiple intrinsic parameters affect $\phi$, a fixed linear mapping is untenable. To address this, we introduce Brightness-Compensation Loss (BC Loss, $\mathcal{L}_{BC}$), which compute the average brightness $I_{a}$ of voxels exceeding a threshold $\beta$, and align this $I_{a}$ with that of the ground truth voxels. we define the average brightness $I_{a}$ as:

$$I_{a}(V)=\frac{\sum_{v\in V,v>\beta}v}{|\{v\in V:v>\beta\}|}$$

(4)

Where $\beta$ serves as a threshold to consider voxels that exceed a certain brightness. Given this, the BC Loss between ground truth voxels $V_{GT}$ and predicted voxels $V_{pred}$ is:

$$\mathcal{L}_{BC}=\left\|I_{a}(V_{GT})-I_{a}(V_{pred})\right\|_{2}^{2}$$

(5)

At last, all these losses are combined together with a separate weight factor $\alpha$ (which is learned by grid search). The complete loss formula is:

	$\displaystyle\mathcal{L}=$	$\displaystyle\alpha_{STP}\mathcal{L}_{STP}+\alpha_{TP}\mathcal{L}_{TP}+$		(6)
		$\displaystyle\alpha_{EF}\mathcal{L}_{EF}+\alpha_{ADV}\mathcal{L}_{ADV}+\alpha_{BC}\mathcal{L}_{BC}$

Once trained, the motion-aware event voxel prediction generates 10 voxels per pixel between consecutive grayscale video frames at 30 fps. This effectively maps each frame with dimensions $H\times W$ to an event voxel grid of dimensions $(2\times 10)\times H\times W$. Upon evaluation with the entire MVSEC dataset [30], we found this ten-fold upscaling to be adequate. Specifically, only approximately 2.30% of the voxels are non-zero, and among these, a mere 6.66% exceed one. This validates that partitioning the time range between two frames into 10 timebins is sufficient for capturing the event stream.

This second task recovers precise event timestamps from Stage1’s event voxels. Previous research used basic sampling methods, such as random or uniform sampling. We introduce Local Dynamics-Aware Timestamp Inference (LDATI), an advanced technique considering each voxel’s event firing trends. It achieves a significantly lower error metric, which is only 3.5% of the best traditional method.

Event voxels aim to discretize temporally contiguous events into a dense tensor, suitable for deep learning inference. Rather than merely counting the event numbers in each voxel (with the temporal resolution of $\delta$), the generation of event voxels also preserves the relative temporal information of events within the timebin. Each event influences the voxel series for a short and finite duration which can be characterized by a continuous-time unit step signal (with an on-time duration same as $\delta$). The value of each voxel is determined by integrating all the step signals for all events within a voxel’s designated time range. The sum of all voxels at the same pixel location equals the total number of events occurring within that time frame. This allows the voxel to summarize the total number of events and their relative times with a single number.

Considering the inverse process of event voxel generation, let $v$ be the value of a voxel and $v^{\prime}$ be its value after removing the influences of events from the preceding voxel. If only one event is fired in the current voxel, its relative occurrence time within the voxel can be determined by $e=\lceil v^{\prime}\rceil-v^{\prime}$. This event will then exert an $e$-unit influence on subsequent voxels. Starting from the first voxel, where $v=v^{\prime}$, we can iteratively deduce the event count and their relative positions in each voxel, a process we term as Chain Decoupling (Fig. 3.a). This computation is deterministic, as it merely reconstructs the inherent temporal information in the voxels.

Voxels are Type1 if $v^{\prime}\leq 1$ and Type2 if $v^{\prime}>1$. As per Section II-A, event voxels are sparse. Type1 voxels use Chain Decoupling; Type2 requires alternatives. Motion and changes happen in a continuous manner and don’t change abruptly under natural condition. Thus, if the preceding voxel has a greater value than the succeeding one, events in the intervening voxel are more likely to be biased towards the former, and vice versa.

To accurately model this phenomenon, we assume that each voxel and its neighboring voxels conform to a slope distribution described by the Probability Density Function (PDF) $f(t)=k_{p}t+b_{p}$. Given that the event timestamp distribution within a voxel is primarily influenced by its temporal neighbors, a simpler slope distribution suffices for both accuracy and computational efficiency. This PDF allows us to estimate event timestamps while accounting for local dynamics, outperforming random sampling approaches.

Let the voxel value in three adjacent voxels be denoted as $N_{0}$, $N_{1}$, and $N_{2}$, and their sum as $M$. Our objective is to derive the expression for their PDF $f(t)$ using these known variables. If $g(t)$ represents the PDF of the event timestamp distribution conditional on $t$ being in the central voxel $v_{1}$, we have:

$\displaystyle g(t)=f_{T|V}(t|v)=\frac{f_{T,V}(t,v)}{P(V=v)}=\frac{f(t)}{P(V=v)}$

(7)

In this formulation, $T$ and $V$ denote the exact event timestamp and its corresponding timebin, respectively. The relationship among $f(t)$, $g(t)$, and $P(v)$ is visually explained in Fig. 3.b. It can be readily shown that $P(v)$ also follows a linear formula with a slope $k_{s}=\delta k_{p}=\delta k_{g}$. Given $k_{s}=\frac{N_{2}-N_{0}}{2\delta M}$, the expression for $g(t)$ can be derived accordingly.

$\displaystyle g(t)=k_{p}t+1/\delta-\delta k_{p}/2$

(8)

where $k_{p}=\frac{N_{2}-N_{0}}{2\delta^{2}M}$. While $g(t)$-based sampling could be applied individually on each voxel, this would be computationally expensive due to varying distribution formulas and event numbers. However, we can optimize this process through distribution transformation. By initially sampling from a uniform distribution with PDF $\gamma(u)=1/\delta$, and then converting it to the desired slope distribution via matrix operations, we significantly accelerate the sampling process. This conversion can be achieved using the inverse cumulative distribution function (CDF) method as follows:

$\displaystyle t=(-b_{p}+\sqrt{b_{p}^{2}+2k_{p}u})/k_{p}$

(9)

Existing video to event simulation methodologies primarily rely on qualitative assessments and consequently lack in standardized objective quantitative evaluation metrics. To address these challenges, we introduce the following novel metrics tailored for this specific task:

1) PMSE (Pooling MSE) aims to compare the predicted voxel cube with the ground truth by employing a 3D average pooling (with kernel and stride of 2 or 4) and then estimating the Mean Square Error (MSE). The 3D average pooling extracts a slightly higher-level summary of the sparse voxel cube. 2) TAcc collapses the Time axis of the 3D voxel cube to convert to a 2D frame for each polarity, apply a thresholding (with a value of 0.001) to convert it to two binary 2D frames for each polarity, and lastly estimate the accuracy by performing the binary matching. 3) TPAcc is similar to TAcc, except it further collapses the event Polarity by accumulating the voxel cubes of two polarities. 4) If no temporal or polarity-wise accumulation is performed, the metric is termed RAcc (Raw Accuracy). 5) TPF1, TF1, RF1: we follow the same procedure as TAcc, TPAcc, or RAcc except F1 scores are calculated instead. Given that event voxels are generally sparse, the F1 score is more representative. TAcc, TPAcc, TF1, and TPF1 aim to evaluate high-frequency spatial patterns, while RAcc and RF1 provide a straightforward way to check the absolute voxel occupancy condition. TPF1 and TF1 are selected as the main metrics.

To evaluate event voxel-to-event stream generation, we introduce three metrics. Mean Event Timestamp Error (METE) quantifies the average temporal discrepancy, in microseconds, between each ground truth and the nearest predicted event at the same pixel. Ground truth events without a nearby counterpart within $3\delta$ on the time axis are termed “overflow events,” and their error is capped at $3\delta$. Number of Overflowed Events (NOE) quantifies the number of such extreme errors. To mitigate potential bias from oversampling, we introduce Predicted/Ground Truth Event Number Ratio (PGER), aiming for a value as close to 1 as possible. By multiplying the PGER when it is greater than one, we can get the calibrated METE and NOE, which are called C-METE and C-NOE.

For baselines, we consider two standard techniques: random and even samplings. Let $v$ be the input voxel value. In both methods, the estimated event number $\tilde{v}$ in a voxel is computed as $\tilde{v}=\lfloor v\rfloor+\text{Bernoulli}(v-\lfloor v\rfloor)$. In random sampling, $\tilde{v}$ events are sampled randomly within the timebin. In even sampling, we find the maximum $M=\max(\tilde{v})$ across all voxels, divide the timebin into $M$ equal sub-timebins, and sequentially place $\tilde{v}$ events starting from the leftmost sub-timebin. A qualitative comparison is shown in Fig. 7.

To rigorously evaluate our LDATI sampling method, we conducted an experiment on ground truth event streams from our test set, converting them to event voxels as outlined in Section II-B. We then applied various sampling techniques for event reconstruction and computed the predefined metrics. The results, detailed in Table III, confirm a PGER metric of 1 for all methods, eliminating under- or over-sampling concerns. LDATI significantly outshines the baselines, with a C-METE metric just 3.5% that of the even sample and a NOE of zero throughout the test set. Substituting our slope distribution sampling with random sampling for Type2 events led to a 6$\times$ higher C-METE error, underscoring the efficacy of slope distribution sampling.

In our study, we implemented a two-stage evaluation and refined the metrics. Although Stage 1 introduces notable voxel-level errors, skewing the overall error relative to ground truth, our V2CE pipeline demonstrates its efficacy as shown in Table IV. Unlike prior methods that tend to oversample events substantially (up to 53.3$\times$ more), leading to significant divergence from the true event distribution, V2CE closely matches the ground truth in event count while simultaneously achieving optimal accuracy metrics.