Frame-Event Alignment and Fusion Network for High Frame Rate Tracking

Visual object tracking based on the conventional frame has undergone astonishing progress in recent years, which can be generally divided into two categories, i.e., correlation filter (CF) trackers [1, 4, 18, 33], and deep trackers [39, 2, 26, 55, 64, 65]. CF trackers learn a filter corresponding to the object of interest in the first frame, and this filter is used to locate the target in subsequent frames. While mainstream deep trackers estimate a general similarity map by cross-correlation between template and search images. However, limited by sensors and benchmarks, those methods are mainly applied to low frame rate (30FPS) tracking.

The high temporal resolution of event cameras allows tracking targets at a higher frame rate. Compared with conventional frame-based tracking, a few attempts have been made at event-based tracking, which can be generally classified into cluster-based and learning-based methods. Litzenberger et al. [28] assigned each new event to a cluster based on distance criteria, which is continuously updated for locating the target. Linares et al. [27] used software to initialize the size and location of clusters, then proposed an FPGA-based framework for tracking. Piatkowska et al. [41] extended the clustering method by a stochastic prediction of the objects’ states to locate multiple persons. However, these methods involve handcrafted strategies and only apply in simple situations. Based on the powerful representation ability of deep learning [30, 52], Chen et al. [6, 5] designed two different event representation algorithms based on Time Surface [25] for target location regression. Zhang et al. [59] combined Swin-Transformer [31] and spiking neural network [12, 29, 58] to extract spatial and temporal features for improving event-based tracking performance. However, these event-based trackers often fail to locate targets accurately when events are too sparse or insufficient.

To combine benefits from frame and event modalities,  [61] employed attention schemes [37, 60, 36, 43, 42] to balance the contributions of the two modalities. This work is most closely related to ours, but it does not exploit the high measurement rate of event-based cameras to accomplish a higher frame rate tracking, thus the tracking frequency is constrained by the frame rate in the frame modality. In contrast, our approach attains high frame rate tracking under various challenging conditions by aligning and fusing frame and event modalities with different measurement rates.

Alignment across multiple frames in the same sequence is essential to exploit the temporal information for video-related tasks, such as video super-resolution [50, 48] and compressed video quality enhancement [11, 63]. A line of works [44, 54, 47] performs alignment by estimating the optical flow field between the reference and its neighbouring frames. In another line of works [45, 48, 11], implicit motion compensation is accomplished by deformable convolution. Deformable convolution was first proposed in  [8], which improves the ability of convolutional layers to model geometric transformations by learning additional offsets. Although the deformable convolution has shown superiority in alignment on the conventional frame domain, aligning on the frame and event modalities brings unique challenges caused by the different styles. In this paper, we propose a novel alignment strategy to simultaneously achieve cross-modality and cross-frame-rate alignment.

Event-based cameras asynchronously capture log intensity change for each pixel. An event will be triggered when:

where $C$ denotes a certain contrast threshold; $p$ is the polarity which means the sign of bright change, with $+$1 and $-$1 representing the positive and negative events, respectively. $\Delta t$ is the time since the last event at location $(x,y)^{\top}$.

Suppose two sequential conventional frames $F_{i}$ and $F_{i+1}$ are captured at times $i$ and $i+1$, respectively. $E_{i\rightarrow i+1}=\left\{\left[x_{k},y_{k},t_{k},p_{k}\right]\right\}_{k=0}^{N-1}$ contains $N$ events triggered during the interval $[i,i+1]$. Our goal is to achieve high frame rate tracking by aligning and fusing conventional frame $F_{i}$ and $E_{i\rightarrow t}$ at any time $t\in[i,i+1]$. The apart in time between dual-modality inputs depends on their frame rates. Specifically, $t-i=\frac{n}{\gamma_{e}}$, where $n$ is an integer in $[1,\frac{\gamma_{e}}{\gamma_{f}}]$; $\gamma_{e}$ and $\gamma_{f}$ denote the frame rates of event and frame modalities, respectively. Following  [61], we represent events $E_{i\rightarrow t}$ as:

where $g(x,y)$ denotes the pixel value of aggregated events at $(x,y)^{\top}$; $\delta$ is the Dirac delta function. In this way, the asynchronous event stream $E_{i\rightarrow t}$ is accumulated to a 2D event frame, denoted $E_{t}$.

Following DiMP [3], as illustrated in Figure 2 (a), the overall architecture of our proposed AFNet contains three components: the feature extractor (i.e., backbone, ECA, and CF), the target classifier, and the bbox regressor. The feature extractors of the template branch and the search branch share the same architecture. Each branch receives an RGB image $F_{i}$ and aggregated events $E_{t}$ at different times as inputs, and corresponding features $F_{f}$ and $F_{e}$ can be extracted by the backbone network. ECA and CF are two key components of our method. The goal of ECA is to address the misalignment between the conventional frames and aggregated event frames at different moments. While CF aims to combine the strengths of both modalities by complementing one modality with information from another. Both target classifier and bbox regressor receive the fused features from feature extractors. Given a template set of fused features and corresponding target boxes, the model predictor generates the weights of the target classifier. Applying these weights to the features collected from search branch predicts the target confidence scores. The bbox regressor estimates the IoU of the groundtruth and the predicted bounding box.

The ECA module is proposed to align conventional frames to the reference aggregated events at the feature level. The key to ECA is designed based on the following challenges: (i) Cross-style alignment is a challenge. Frames and events are recorded by different sensors and thus have different styles, making alignment challenging. (ii) Cross-frame-rate alignment is another challenge. The frame rate of aggregated event frames is far higher than that of conventional images, resulting in target location ambiguity that confuses the tracker’s predictions. As shown in Figure 2 (b), ECA contains three modules: Motion Aware (MA), Style Transformer (ST), and Deformable Alignment (DA).

Our CF is proposed to robustly fuse frame and event correlations by adaptively learning a dynamic filter from one modality that contributes to the feature expression of another modality. Simply fusing frame and event modalities ignores circumstance in which one of the modalities does not provide meaningful information. In an HDR scene, for instance, the frame modality will provide no useful information, yet the event modality still exhibits strong cues. Conversely, in the absence of motion, event-based cameras cannot successfully record target-related information, while conventional frames can still deliver rich texture features. Therefore, we propose a cross-correlation scheme to complement one domain with information from another domain as shown in Figure 2 (c). Specifically, given the aligned frame feature $F_{f}^{da}$ and enhanced event feature $F_{e}^{c}$, the proposed CF first adaptively estimates a dynamic filter of high-level contextual information from one modality. Then, this dynamic filter serves to enhance the features of another modality. Formally,

where $F_{f}^{e}$ denotes the enhanced feature of the frame modality based on the dynamic filter $\mathcal{K}_{e}$ from event modality; $\circledast$ is the depthwise convolution; $\mathcal{A}$ denotes the adaptive average pooling; $\vartheta$ is the Batch Normalization (BN) followed by a ReLU activation function. Similarly, we can extract the complementary feature $F_{e}^{f}$ of the event modality based on the dynamic filter $\mathcal{K}_{f}$ from frame modality. Finally, $F_{f}^{e}$ and $F_{e}^{f}$ are concatenated to build the fused feature $F_{fu}$,

$F_{fu}$ will be fed into the classifier and regressor to locate the target. The classifier adopts an effective model initializer and a steepest descent based optimizer to predict the score map. The regressor employs the overlap maximization strategy for the task of accurate bounding box estimation. We refer to [3] for details.

We adopt the pretrained ResNet18 [17] as the backbone to extract frame and event features. Following  [3, 61], the loss function is defined as:

where $L_{cls}$ is the target classification loss which includes a hinge function to equally focus on both positive and negative samples. $L_{bb}$ is the bounding box regressor loss which estimates MSE between the predicted IoU and the groundtruth. $\beta$ is set to 100.

We implemented our approach in PyTorch [40] and trained our network for 100 epochs with a batch size of 32 using Adam optimizer with the default parameters. We set the initial learning rate of the feature extraction network, the classifier, and the regressor to 2e-4, 1e-3, 1e-3, respectively. The learning rate is adjusted by the CosineAnnealingLR strategy [32]. Our network is run on a single Nvidia RTX3090 GPU with 24G memory.

We evaluate our AFNet on two event-frame-based datasets: FE240hz [61] and VisEvent [51]. The FE240hz dataset has annotation frequencies as high as 240 Hz and consists of more than 143K images and corresponding recorded events. With this dataset, our method can accomplish a high frame rate tracking of 240Hz. Compared with FE240hz, VisEvent provides a low annotation frequency, about 25Hz. However, it contains various rigid and non-rigid targets both indoors and outdoors. Following [59], there are 205 sequences for training and 172 for testing.

To demonstrate the effectiveness of our method, we compare AFNet with the nine state-of-the-art trackers. Specifically, ATOM [9], DiMP [3], PrDiMP [10], STARKs [56], TransT [7] and ToMP [35] are conventional frame-based trackers. For a fair comparison, we extend them to multi-modality trackers via the following two fusion strategies: (i) Early Fusion (EF), we first add the aggregated events and corresponding frame as unified data, and then feed it into trackers; (ii) Middle Fusion (MF), we first use the backbone of these trackers to extract the frame and event features separately before feeding the sum of these features into the regressor. We also compared three original multi-modality methods: DeT [57], HMFT [62], and FENet [61] are frame-depth, frame-thermal, and frame-event trackers, respectively. All approaches are re-trained and tested on the FE240hz and VisEvent datasets. Following [61], we use RSR and RPR to evaluate all trackers. RSR and RPR focus on the overlap and center distance between the ground truth and the predicted bounding box, respectively.

Figure 3 (a) shows the overall evaluation results on the FE240hz [61] dataset, which demonstrates the proposed AFNet offers state-of-the-art high frame rate tracking performance and outperforms other compared approaches in terms of both precision and success rate. In particular, our proposed AFNet achieves an 87.0% overall RPR and 58.4% RSR, outperforming the runner-up by 2.7% and 2.8%, respectively. We further validate the robustness of our AFNet under five common challenging scenarios: high dynamic range (HDR), low-light (LL), fast motion (FM), no motion (NM), and severe background motion (SBM). Among them, the first three conditions present challenges for tracking in the conventional frame modality, while the last two scenarios provide difficulties for the event modality. As shown in Table 1, we can see that AFNet surpasses other approaches in all conditions. These results validate the effectiveness of our proposed approach on high frame rate object tracking. The extended multi-modal methods [9, 3, 10, 56, 7, 35] lack a well-designed fusion module, preventing them from efficiently combining the complementary information of the two domains. While original multi-modality trackers DeT [57], HMFT [62] and FENet [61] do not address the misalignment between frame and event data at different measurement rates, causing ambiguity when locating targets. Figure 4 further qualitatively shows the effectiveness of our AFNet in different challenging conditions.

Even though the VisEvent dataset [51] has a low frame rate annotation, it provides various non-rigid targets that are absent from the FE240hz dataset. Thus, we also compare our AFNet against other state-of-the-art methods on VisEvent. As shown in Figure 3 (b), our AFNet obtains 44.5% and 59.3% in terms of RSR and RPR, respectively, surpassing all previous methods. Table 2 reports the evaluation of various trackers on rigid and non-rigid targets, showing that AFNet outperforms other competing trackers on these two attributes, except the RSR on rigid targets. These results validate that our proposed multi-modality approach still remains effective for low frame rate frame-event tracking.

Ideally, the tracking frame rate of our AFNet can reach the measurement rate of an event-based camera. Constrained by the existing annotated rates, we verify the effectiveness of our proposed AFNet on FE240hz at 240Hz and VisEvent at 25Hz. Our current focus is on exploiting multi-modality alignment and fusion schemes for effective and robust high frame rate tracking in various challenging conditions. However, we have not developed a lightweight network or a simple regression mechanism to speed up the evaluation of our approach. As shown in Table 6, we report the RPR and RSR with respect to the evaluation speed of the four multi-modality approaches on the FE240hz [61] dataset. We can see that, at nearly equal assessment speeds, our AFNet offers the best tracking accuracy.