Monocular Event-Inertial Odometry with Adaptive decay-based Time Surface and Polarity-aware Tracking

Kueng et al. [7] developed a pioneering method for event-based visual odometry that relies on feature extraction from grayscale maps and asynchronous event-based tracking, facilitating 6DoF pose estimation and mapping. However, this method is confined to feature points detected in image frames. Another stride in event-based visual odometry by Kim et al. [8] achieved real-time event-based SLAM by integrating triplet probabilistic filters for pose estimation, scene mapping, and intensity estimation, although this approach demands GPU acceleration due to its computational intensity. EVO [9] advanced the field by presenting an image-to-model alignment-based tracking approach to capture rapid camera movements, recovering semi-dense maps in parallel processing, although a lengthy start-up phase and poor accuracy hinder it. EDS [10] improved upon this with an event generation model to track camera motion, enhancing the accuracy of visual odometry but at the expense of increased computational demand and slower optimization speeds. Lastly, an innovative 6DoF motion compensation mechanism by Huang et al. [11] allowed deblurred event frame generation synchronized to RGB images using an event generation model, addressing the modality disparities between images and event data.

Geometric approaches inadequately harness event data, whereas deep learning-based event odometers introduce an innovative processing paradigm. RAMP-VO [12] is the inaugural end-to-end framework for event- and image-based visual odometry, merging asynchronous event streams with image data through a parallel encoding scheme. Alternatively, DEVO [13] constitutes the premier monocular event-only odometry system, grounded in deep learning, which trains on event voxel grids and inverse depth maps under the guidance of ground truth poses, achieving markedly enhanced accuracy over conventional techniques. Although deep learning odometry signifies a stride in precision, cost-effective accuracy enhancements on computation-constrained platforms, such as drones, may still benefit from integrating event data with IMU measurements.

Zhu et al. [14] introduced the pioneering event-based odometry by integrating an event-driven tracker with IMU data. However, its real-time application is hindered by computationally demanding feature tracking. Vidal et al. [15] advanced the field by establishing a tightly integrated framework combining events, frames, and inertial readings for enhanced state estimation. Complementarily, some research integrates events and IMU within a continuous-time schema. Mueggler et al. [16] devised a continuous-time approach for merging high-frequency event and IMU data for visual inertial odometry. Embracing a sophisticated feature tracker [17], EKLT-VIO [18] demonstrated accurate performance in Mars-like and high-dynamic-range sequences and showed good potential in vision-based exploration on Mars. Dai et al. [19] presented a comprehensive model marrying continuous-time inertial data with events, innovating an exponential decay correlation for events. Guan et al. [20] developed a uniformly distributed event corner detection algorithm for raw events, and designed two event representations to perform feature tracking and loop closure matching for a keyframe-based visual inertial system. Based on this, Guan et al. [21] utilized motion compensation for the event stream based on IMU measurements and introduced line and point feature constraints, improving odometric precision.

Although time surfaces based on exponential decay can be additionally motion compensated to enhance performance, as employed by [11, 21], they still rely on accurate sensor measurements (e.g. IMU) or accurate odometer estimates. To make the exponential decay kernel adaptable to the dynamics of the event stream, we adopted the event activity model proposed by Nunes et al. [6], and introduced an adaptive decay kernel for different event cameras and practical operating conditions. Leveraging such advancements, we propose a monocular event-inertial odometry with the adaptive decay kernel and optimize feature tracking for the problems faced by the polarity-weighted time surface-based approach, resulting in a more robust and accurate odometry.

The system overview is depicted in Fig. 2. Raw events are transformed into adaptive decay-based time surfaces (Sec. III-C) using the proposed Time-Priority strategy. Data from a monocular event camera and an IMU are fused in the Multi-State Constraint Kalman Filter (MSCKF) [22], enabling precise and low-latency pose estimation. Given the sparse event outputs from the event camera when the system is in static or slow motion, we utilize a dynamic initialization method [23] to incorporate feature observations across time surfaces and IMU measurements for system initialization. Once initialized, upon receiving a new event packet, the system leverages IMU measurements to propagate the mean and covariance of the state to the timestamp of the new event packet, and augments a cloned IMU pose to the state vector (Sec. III-B). Afterward, sparse feature observations from the proposed tracking method (Sec. III-D) are utilized to update the state (Sec. III-E). Landmarks and old cloned poses are marginalized out of the state vector for computational efficiency.

The state vector $\mathbf{x}_{j}$ at timestamp $t_{j}$ of the system is composed of the inertial state $\mathbf{x}_{I_{j}}$, inertial pose clones $\mathbf{x}_{C}$ and the inverse depths of landmarks $\mathbf{x}_{f}$, given by

where ${}^{I_{j}}_{G}\overline{q}$ denotes the unit quaternion, and the corresponding rotation matrix is denoted as $\mathbf{R}({}^{I_{j}}_{G}\overline{q})={}^{I_{j}}_{G}\mathbf{R}$. Furthermore, ${}^{G}\mathbf{p}_{I_{j}}$ and ${}^{G}\mathbf{v}_{I_{j}}$ denote the position and velocity of the IMU in the global frame at time $t_{j}$, while $\mathbf{b}_{a_{j}}$ and $\mathbf{b}_{g_{j}}$ are the biases of the accelerometer and the gyroscope, respectively.

The sliding window maintains a set of $m+1$ cloned IMU poses at the timestamps of the event packet for feature triangulation and state update like an RGB-Camera-based VIO. Stable features tracked across the entire sliding window frames are considered as SLAM features and will be augmented to the state vector for extending the time span of active constraints. In this paper, we use the inverse depth parameterization and only include maximum $s$ landmarks in the state for limiting computational complexity. The extrinsic parameters ${}^{C}_{I}\mathbf{R},{}^{C}\mathbf{p}_{I}$ between the IMU and the camera are precalibrated and assumed to be known. In our practical experiments, $m$ and $s$ are set to 10 and 50, respectively.

The number of event activities increases with event frequency. As inferred from Eq. (6), when the event frequency increases, events with the later timestamp have less decays. In other words, if the pixel value remains constant, a higher event frequency implies that the event timestamps are closer to $t$. Event information is retained for a longer period of time when the event frequency is high. Therefore, modeling the dynamics of the event stream by introducing the event activity enables adaptively adjusting the decay for the events and thus provides high-quality imaging.

When the system receives an event message, as shown in the Fig. 3, the event activity is calculated recursively, i.e., the decay $\tau_{2}(e_{k-1},t_{k})$ is calculated first from Eq. (6), and then Eq. (4) is used to calculate the event activity $\alpha(t_{k})$ at the event timestamp $t_{k}$. $n(t_{k},t_{k-1})$ is always 1 since there is only one event in adjacent moments. In this recursive manner, calculate all event activity at the moment of the event timestamp and save it with the corresponding event.

And there are two different strategies to determine the initial event timestamp $t_{init}$ and the timestamp $t_{j}$ of the adaptive decay-based time surface:

• •

  Data-Priority [6]: $t_{init}$ starts with the first event timestamp and is typically reset after obtaining a set of active events when $\nu(e_{init},t_{j})<w_{th}$. The method tends to utilize all events without duplication, but the timestamp $t_{j}$ of the time surface cannot be predicted in advance. In addition, the frequency of the time surfaces generated based on this method is sensitive to the threshold $w_{th}$.

• •

  Time-Priority: we first estimate $t_{init}$ based on the the timestamp $t_{j}$ of the time surface, event activity $\alpha(t_{j})$ and predefined threshold $w_{th}$, as follows:

  $$\nu(e_{init},t_{j})=w_{th}\implies t_{init}=t_{j}-\frac{1-w_{th}}{r\cdot\alpha(t_{j})w_{th}}\,.$$

  (8)

  This approach allows for arbitrary timestamps for time surfaces and decouples event activity and generation of the adaptive time surfaces for parallel processing.

Events that are considered active have timestamps between $t_{init}$ and $t_{j}$. We find the latest active events in each pixel, denoted as $e_{l}(\mathbf{x})$ and the pixel value is $\tau_{2}(e_{l}(\mathbf{x}),t_{j})$ or $p_{l}\cdot\tau_{2}(e_{l}(\mathbf{x}),t_{j})$ if the time surface is polarity-weighted. If the pixel has no active events, set the value of the pixel to zero. Finally, all pixel values are mapped to $[0,255]$.

To verify the performance of adaptive decay-based time surfaces in the odometry, we utilize the Fast corners [24] and track them with the LK optical flow [25] method. This approach functions properly within datasets containing event cameras. However, we observe that abrupt changes in motion direction may lead to tracking failures as they violate the assumption of brightness constancy. Diverse motion directions induce events with opposite polarities at identical scene positions, leading to fluctuations in pixel values, as illustrated in Fig. 1 and Fig. 4. Such fluctuations notably impact the accuracy of odometry estimation.

Actively inverting event polarities generates a polarity-inverted time surface map, and this map can be regarded as an approximation under the condition of maintaining event polarity, which to some extent mitigates the issue of the violated brightness constancy assumption. By simultaneously tracking both the original and polarity-inverted time surfaces and adaptively merging the tracking results, we ensure continuous and stable feature tracking. Estimating the velocity based on IMU measurements to make this determination may suffer from misjudgments or delays. We track both images separately and then compare the number of successfully tracked features. When the number of successfully tracked features on the polarity-weighted time surface is less than that on the polarity-inverted map, we merge the both tracking results. Finally, we apply the RANSAC method to eliminate outliers and obtain the final matches by estimating the fundamental matrix from the matched features.

When tracking results are available, we select features within the sliding window for triangulation and system update. The system update follows two principles. Firstly, we aim to triangulate using as much data as possible to ensure accuracy, thereby ensuring positive gains from system updates. Secondly, stable tracked points are selectively included in the state vector for continuous estimation, aiming to improve system accuracy while maintaining control over increased computation time. Based on these principles, features are categorized into SLAM and MSCKF types.

Specifically, for the current time surface feature tracking, if tracking fails or reaches the maximum length (equal to the sliding window), suggesting that landmark observations are maximized, we attempt to triangulate. For a landmark ${}^{G}\mathbf{p}_{f}$ associating with a successfully triangulated feature, we have

where $\pi(\cdot)$ represents the back projection function that transforms a pixel to the normalized image plane. The inverse depth $\rho_{l}$ is defined in the anchor frame. For instance, the SLAM feature always selects the oldest frame $\{C_{j-m}\}$ in the sliding window as the anchor frame. Consequently, the position ${}^{G}\mathbf{p}_{f}$ could be computed using the observation $e_{k}$ in the anchor frame. It’s worth noting that for the MSCKF feature, the anchor frame is determined as the first observed frame. Given the new observation (e.g. $e_{i}$) in the latest frame $\{C_{j}\}$, the nonlinear measurement model is given by:

where the measurement noise is associated with the event pixel noise and follows a white Gaussian distribution $\mathcal{N}\left(\mathbf{0},\mathbf{Q}_{e}\right)$. Features tracked throughout the entire sliding window are augmented into the state vector, continuously updating the system with subsequent landmark observations—referred to as SLAM features. Other successfully triangulated points, termed MSCKF features, are updated using the efficient MSCKF nullspace projection [22], avoiding the inclusion of landmarks in the state vector and thus reducing system complexity.

In this section, we analyze how different decay functions (exponential and adaptive decay) for time surface representations affect odometry accuracy. Specifically, we evaluate various time constants $\eta$ for the exponential decay kernel and compare them with the adaptive decay kernel (as shown in Tab. III). The experiments utilize a dataset publicly available in [20], including a DAVIS 346 event camera.

Subsequently, we investigate the effectiveness of the polarity weighting and the polarity-aware feature tracking process, as presented in Tab. III. Experimental results indicate some improvement when using polarity weighting, achieving average ATE of 0.210m. Polarity weighting enhances the distinction between brighter and darker pixels in the time surface maps, particularly accentuating nearby edges. While offering the above benefits, it also introduces challenges for feature tracking, mainly when the event camera’s motion direction changes abruptly, causing shifts in event polarity and undermining the assumption of intensity constancy. Our method incorporates the tracking results of time surfaces with inverted polarity, which somewhat mitigates the problem of tracking failures due to abrupt changes in the direction of motion, and thus achieves an improvement in the accuracy.

Finally, we conduct a detailed analysis of the time consumption for each operation in the proposed method, as outlined in Tab. IV. The analysis is conducted on a typical sequence vicon_aggressive_hdr, which has aggressive motion speed and includes high dynamic range scenes. The reported results represent the average values obtained from multiple tests. The statistical results indicate that the average time required for generating an adaptive time surface map is 5.4ms, while the adaptive tracking module is 6.7ms. The total time required for the system to complete one update is 16ms, indicating its capability for real-time performance.