End-to-end Learning of Object Motion Estimation from Retinal Events
for Event-based Object Tracking

The $k$-th event $e_{k}$ of retinal events $\mathcal{E}$ can be represented as a quadruple:

$$e_{k}\doteq(u_{k},v_{k},p_{k},t_{k}),$$

(1)

where $u_{k}$ and $v_{k}$ are the horizontal and vertical coordinates of $e_{k}$; $p_{k}$ indicates the polarity (On or Off) of $e_{k}$, and $t_{k}$ is the timestamp of $e_{k}$. Retinal events can occur independently in an asynchronous manner, which makes it difficult for conventional computer vision algorithms to directly process the raw data. There are several attempts (?; ?; ?) that try to convert asynchronous retinal events to synchronous frames. For example, (?) adopts a hierarchical model, which is beneficial to the recognition task but containing less motion information. (?; ?) use specially designed histograms to format retinal events, which cut off the continuity of motion patterns in the temporal domain. In contrast, we prefer to create a clear and lightweight motion pattern for training our network with the help of event cameras, which allows our method to use a much smaller network structure, and maintain high precision estimation and real-time performance simultaneously.

In this work, we propose a synchronous representation of retinal events named Time-Surface with Linear Time Decay (TSLTD), as shown in Fig. 1. The TSLTD representation is based on the Time-Surface representation in (?), but with two major differences: (1) the Time-Surface representation, which is designed for object recognition, consists of clear object contours with little motion information. We replace the exponential time decay kernel in Time-Surface with a linear time decay kernel to efficiently create effective motion patterns; (2) our TSLTD representation does not need the two hyper-parameters (i.e., the size of neighborhood and the time constant of exponential kernel) in Time-Surface. Thus, TSLTD can be effectively generalized to various target objects for motion estimation.

In TSLTD, motion information, captured by an event camera, is encoded and represented as a set of TSLTD frames over every time window T. The time window T will be discussed later. Each of the TSLTD frames is initialized to a three-dimensional zero matrix $\mathcal{M}\in{\mathcal{N}^{h\times w\times 2}}$. Here $h$ and $w$ are the height and width of the event camera resolution, the third dimension indicates the polarity of events (and the information from On or Off events will be stored separately). Then asynchronous retinal events within the current time window are used to update the matrix in an ascending order in terms of their timestamps.

More specifically, supposing that we are processing a retinal event set $\mathcal{E}_{t_{s},t_{e}}$ which is collected between the start timestamp $t_{s}$ and the end timestamp $t_{e}=t_{s}+T$ of the current time window, to yield a new TSLTD frame $\mathcal{F}_{t_{s},t_{e}}$. $\mathcal{F}_{t_{s},t_{e}}$ is initialized to a 3D zero matrix $\mathcal{M}_{t_{s},t_{e}}$. During the updating process, we process ${{\cal E}_{t_{s},t_{e}}}=\left\{{{e_{i}},{e_{i+1}},\ldots,{e_{j}}}\right\}$ from ${e_{i}}$ to ${e_{j}}$, where ${e_{i}}$ is the first event and ${e_{j}}$ is the last event. Each of the events in $\mathcal{E}_{t_{s},t_{e}}$ triggers an update, which assigns a value ${g}$ to $\mathcal{M}_{t_{s},t_{e}}$ at the coordinates ${(u,v)}$ corresponding to the triggering event. The value ${g_{k}}$ for the $k$-th update caused by ${e_{k}}$ is calculated using the following equation:

$${g_{k}}=round(255*\left({{t_{k}}-{t_{s}}}\right)/T),$$

(2)

where ${t_{k}}$ is the timestamp of ${e_{k}}$. So the assigned value for each update is proportional to the timestamp of the triggered event. ${t_{k}}-{t_{s}}$ is the linear time decay, and we use $255/T$ to normalize the decay. When a pixel of an object moves to a new coordinate, an event will occur at that coordinate and the TSLTD frame records a higher value of $g$ than the previous one at that coordinate. As a result, the time-motion variation information is naturally embedded in the TSLTD frames to form intensity gradients as shown in Fig. 2, which have shown a clear pattern of the direction and magnitude of the corresponding object motion. Therefore, the TSLTD format facilitates our network to extract the motion information through intensity gradient patterns that are embedded in the TSLTD frames.

There are two main problems that are related to the time window T value, used in generating TSLTD frames. The first problem is that if T is set to a large interval (e.g., more than 16 $ms$), there are two consequences for fast moving objects. Since these objects move fast, they can return to the previous position or move far from the previous position, during a large time interval. As a result, new motion patterns overlap and contaminate the previous patterns, or become too large to recognize. On the contrary, if T is set to a small interval (e.g., less than 3 $ms$), TSLTD can only capture a very small movement, which may not be distinguished from sensor noises, and thus it may cause an ambiguous motion pattern (especially for a low-resolution event camera). After testing T with different objects and motions, we experimentally set T to be $6.6$ $ms$ (according to the sampling frequency of 150 Hz) for good generalization performance.

Nowadays, pre-trained deep models, such as VGGNet (?) and ResNet (?), are very popular among many computer vision tasks. But most of these pre-trained models were trained using three-channel RGB images, which is not the optimal option for two-channel TSLTD frames. In addition, since TSLTD frames have clear motion patterns, we do not need very deep and complex networks to extract very high-level features for general pattern recognition.

Here we design a lightweight network, named RMRNet, to learn object motions directly from TSLTD in an end-to-end manner. Between every two adjacent video frames, there are five TSLTD frames, which contain multiple objects. For individual object motion estimation, we crop object patches from the TSLTD frames. During the training stage, we crop the object patches from adjacent five TSLTD frames according to $\tau$ times of their axis-aligned bounding boxes, to preform a joint training. Here $\tau$ is a parameter that renders the cropped region slightly larger than the previous bounding box to capture the full pattern of object motion. During the test stage, we crop object patches frame-by-frame according to $\tau$ times of their axis-aligned bounding boxes. Finally, these object patches are resized to 64 $\times$ 64, and sent to the proposed RMRNet as the input.

As shown in Fig. 2, the first part of RMRNet contains four convolutional layers for feature extraction. The initial three layers share the same kernel size of 3 $\times$ 3 with stride of 2, which is similar to the VGG-Network (?). The kernel size of the final layer, which is used to reduce the feature dimensions, is 1 $\times$ 1 with stride of 1. The filter numbers of the four layers are 32, 64, 128 and 32, respectively. The four convolutional layers are followed by a batch normalization layer. A dropout layer is added in the end during the training stage. Finally, the output feature is flattened and sent to the next part. The second part of RMRNet is an LSTM module. This module contains three layers with 1568 channels in each layer. By adding the LSTM module, we can stack object patches in one regression process. Then the LSTM module can fuse the stacked motion features from the CNN part to regress an accumulated motion, which is the motion between the first object patch and the final object patch. The final part of RMRNet is a set of fully connected layers, which are used to predict 5-DoF motions. The first fully connected layer has 1568 channels. Then the following layers are divided into five branches for different components of the 5-DoF motion. Each branch contains two layers, which respectively have 512 and 128 channels. This network structure is chosen due to its desirable performance on balancing both precision and speed. The output of RMRNet is a 5-DoF transform ($e_{1}$ to $e_{5}$), as described next.

A 5-DoF transform ${\mathcal{T}_{i,j}^{o}}$ between a frame ${i}$ and the next frame ${j}$ for an object ${o}$ can be defined as a subset of the 2D affine transform on an object bounding box in the ${i}$-th frame:

$${\mathcal{T}_{i,j}^{o}}\doteq(d_{x},d_{y},\theta,s_{x},s_{y}),$$

(3)

where ${\mathcal{T}_{i,j}^{o}}$ is represented as a quintet, $d_{x}$ and $d_{y}$ are respectively the horizontal and vertical displacement factors, $s_{x}$ and $s_{y}$ are respectively the horizontal and vertical scale factors, and $\theta$ is the rotation factor. Note that the rotation factor $\theta$ and the scaling factors $s_{x}$ and $s_{y}$ in ${\mathcal{T}_{i,j}^{o}}$ are “in-place operations”, which means that we will keep center alignment before and after these two operations. The resulting coordinate transform is as follows:

$$\begin{bmatrix}{x}^{\prime}\\ {y}^{\prime}\end{bmatrix}=\begin{bmatrix}s_{x}&0\\ 0&s_{y}\end{bmatrix}\otimes\begin{bmatrix}cos\theta&-sin\theta\\ sin\theta&cos\theta\end{bmatrix}\otimes\begin{bmatrix}x+d_{x}\\ y+d_{y}\end{bmatrix}.$$

(4)

Here the original coordinates $(x,y)$ of the bounding box of object ${o}$ in previous frame ${i}$ are transformed into the new coordinates $({x}^{\prime},{y}^{\prime})$ in current frame ${j}$ through the transform ${\mathcal{T}_{i,j}^{o}}$. The operator $\otimes$ indicates an in-place operation. Note that the five parameters $e_{1}$ to $e_{5}$ predicted by RMRNet are normalized to a range of $[-1.0,1.0]$ using the Tanh activation function. Then we set five boundary parameters $p_{1}$ to $p_{5}$ for $e_{1}$ to $e_{5}$ to constrain the range of object movements. Finally, the transform ${\mathcal{T}_{i,j}^{o}}$ is calculated as follows:

$${\cal T}_{i,j}^{o}=\left\{{\begin{array}[]{*{20}{l}}{{d_{x}}={e_{1}}*{p_{1}}}\\ {{d_{y}}={e_{2}}*{p_{2}}}\\ {\theta=({e_{3}}*{p_{3}})*\pi/180}\\ {{s_{x}}=1+{e_{4}}*{p_{4}}}\\ {{s_{y}}=1+{e_{5}}*{p_{5}}}\\ \end{array}}\right..$$

(5)

In this paper, we respectively fix $p_{1}$ to $p_{5}$ to 72, 54, 30, 0.2 and 0.2, and fix $\tau$ to 1.2 according to the 240 $\times$ 180 resolution of the DAVIS camera for RMRNet. This parameter setting will allow a relatively large range for object movements. Thus, the setting is suitable for estimating the majority of object motions, which includes most of the fast movements.

There are two key points that should be mentioned in relation to the training stage. The first one is that if we only use B&W object samples (i.e., black objects with a white background) as the training data, our network can only learn some relatively simple motion patterns for the corresponding object motions. Thus, we follow the standard five-fold cross-validation protocol and use the object pairs (refer to the next section) from the shapes_6dof, poster_6dof and light_variations sequences as the training and validation data (while the other five sequences are unseen during the training stage) to train and validate the proposed RMRNet. The second point is that it is difficult to learn an object motion model from a single TSLTD frame. There are five TSLTD frames for every object pair. A single TSLTD frame usually includes only a small movement and it has only a weak motion pattern. After extensive experiments, we find that stacking five TSLTD frames in one prediction has gained the optimal performance during the training stage.

The proposed network is trained using the ADAM solver with a mini-batch size of 16. We use a fixed learning rate of 0.0001, and our loss function is the MSE loss:

$$MSE_{loss}\left({\hat{\cal T},{\cal T}}\right)=\frac{1}{{{N_{train}}}}\sum\limits_{l=1}^{{N_{train}}}{{{\left\|{{{\hat{\cal T}}_{l}}-{{\cal T}_{l}}}\right\|}^{2}}},$$

(6)

where $\hat{\cal{T}}$ is the estimated 5-DoF motion, $\cal{T}$ is the corresponding ground truth, $N_{train}$ is the number of training samples, and the subscript $l$ indicates the $l$-th sample.

For our evaluation, we use a challenging mixed event dataset including a part of the popular Event Camera Dataset (ECD) (?) and the Extreme Event Dataset (EED) (?), which were recorded using a DAVIS event camera in real-world scenes. The details of the dataset can be found in Table 1. Note that the mixed dataset contains both the event data sequences and the corresponding video sequences for every sequence. Since the ECD dataset does not provide ground truth bounding boxes for object tracking, we labeled a rotated or an axis-aligned rectangle bounding box as the ground truth for each object in the dataset to evaluate all the competing methods.

We choose five state-of-the-art object tracking methods, including SiamFC (?), ECO (?), SiamRPN++ (?), ATOM (?) and E-MS (?), as our competitors. About these competitors: SiamFC and SiamRPN++ are fast and accurate methods, which are based on Siamese networks. ECO and ATOM are state-of-the-art methods that have achieved great performance on various datasets. E-MS (?) is recently proposed for event-based target tracking based on mean-shift clustering and Kalman filter. We extend E-MS to support bounding box-based tracking, by employing the minimum enclosing rectangle of those events that belong to an identical cluster center as its estimated bounding box.

To perform ablation studies, we also compare our RMRNet with an event-based variant of ECO (called as ECO-E) and a variant of RMRNet (called as RMRNet-TS). ECO-E, which uses our proposed TSLTD frames as its inputs, is an event-based variant of ECO (?). ECO-E is used to evaluate the performance of ECO on the event data sequences. RMRNet-TS uses the classical Time-Surface frames in Hots (?) instead of using the proposed TSLTD frames as its inputs. RMRNet-TS is used to evaluate the performance of the classical Time-Surface representation and our TSLTD representation. For SiamFC, ECO, SiamRPN++ and ATOM, we use the video sequences as their inputs. For ECO-E, E-MS, RMRNet-TS and the proposed RMRNet, we use the event data sequences (in quadruple format) as their inputs. For all the five competitors, we use their released codes, default parameters and best pre-trained models.

The output of our RMRNet is a 2D 5-DoF frame-wise bounding box transform model, estimated between two adjacent frames. In order to evaluate the quality of the estimated frame-wise bounding box transform model, we compare the proposed RMRNet with all the competitors on frame-wise object tracking, that is, our evaluation on these competing methods is based on object pairs, each of which includes two object regions on two adjacent frames corresponding to an identical object. During the evaluation, we treat each of the object pairs as a tracking instance in the corresponding frame. In addition, all the competitors only estimate axis-aligned bounding boxes, whereas the proposed RMRNet can estimate both rotated and axis-aligned bounding boxes (by using the 5-DoF motion model). Therefore, we evaluate all the methods on the axis-aligned ground truth bounding boxes for a fair precision comparison.

For evaluating the precision of all the methods, we calculate the Average Overlap Rate (AOR) as follow:

$${AOR}=\frac{1}{{{N_{rep}}}}\frac{1}{{{N_{pair}}}}\sum\limits_{u=1}^{{N_{rep}}}{\sum\limits_{v=1}^{{N_{pair}}}{\frac{{O_{u,v}^{E}\cap O_{u,v}^{G}}}{{O_{u,v}^{E}\cup O_{u,v}^{G}}}}},$$

(7)

where $O_{u,v}^{E}$ is the estimated bounding box in the $u$-th round of the cross-validation for the $v$-th object pair, and $O_{u,v}^{G}$ is the corresponding ground truth. Eq. (7) shows that the AOR measure is related to the Intersection over Union (IoU). ${N_{rep}}$ is the repeat times of the cross-validation, and ${N_{pair}}$ is the number of object pairs in the current sequence. We set ${N_{rep}}$ to 5 for all the following experiments.

We also calculate the Average Robustness (AR) to measure the robustness of all the competing methods as follow:

$${AR}=\frac{1}{{{N_{rep}}}}\frac{1}{{{N_{pair}}}}\sum\limits_{u=1}^{{N_{rep}}}{\sum\limits_{v=1}^{{N_{pair}}}{succes{s_{u,v}}}},$$

(8)

where $succes{s_{u,v}}$ indicates that whether the tracking in the $u$-th round for the $v$-th pair is successful or not (0 means failure and 1 means success). If the AOR value obtained by a method for one object pair is under 0.5, we will consider it as a tracking failure case.

We use the mixed event dataset to evaluate the eight competing methods. We choose the shapes_translation, shapes_6dof, poster_6dof and slider_depth sequences from the ECD dataset (?) as the representative sequences for comparison. The first three sequences have increasing motion speeds. The fourth sequence has a constant motion speed. The object textures of these sequences vary from simple B&W shapes to complicated artifacts. For these sequences, we are mainly concerned with the performance of all methods for various motions, especially for fast 6-DoF motion, and for different object shapes and textures.

For comparison, the quantitative results are reported in Table 2. We also provide some representative qualitative results obtained by SiamFC, ECO, SiamRPN++, ATOM, E-MS and our method in the top three rows of Fig. 3. From Table 2, we can see that our method achieves the best performance on the first three sequences and it achieves the second best performance on the fourth sequence. SiamFC, ECO, SiamRPN++ and ATOM also achieve competitive results. However, as we can see in Fig. 3, our method has achieved better performance in estimating fast motion. In comparison, SiamFC, ECO, SiamRPN++ and ATOM usually lose the tracked objects, due to the influence of motion blur. Comparing with the original ECO, ECO-E has achieved inferior performance, which shows that state-of-the-art object tracking methods, like ECO, are not suitable to be directly applied to event data. The classical Time-Surface frames are designed for object recognition and detection, which contain less motion information for motion estimation and object tracking. Thus, RMRNet-TS shows much inferior results in this evaluation. By leveraging the high temporal resolution feature of the event data, E-MS can also effectively handle most fast motions. However, E-MS is less effective to handle complicated object textures and cluttered backgrounds (e.g., for the poster_6dof and slider_depth sequences). In contrast, the proposed RMRNet outperforms E-MS by a large margin, which shows the superiority of our method in handling various object textures and cluttered backgrounds.

Moreover, we also choose the EED dataset to evaluate the eight competing methods. The EED dataset (?) contains four challenging sequences: fast_drone, light_variations, occlusions and what_is_background. The first three sequences respectively record a fast moving drone under low illumination environments. The fourth sequence records a moving ball with a net as foreground. Using this dataset, we want to evaluate the competing methods in low illumination conditions and in occlusion situations.

The quantitative results and some representative qualitative results are shown in Table 3 and Fig. 3, respectively. From the results, we can see that our method achieves the best performance on most sequences except for the what_is_background sequence, on which our method has obtained low AOR and AR. This is because that the foreground net in the what_is_background sequence covers the ball, which destroys the corresponding motion patterns. In contrast, our method has achieved the highest AOR and AR on the occlusions sequence. This is because that in this sequence, the occlusion time is short and only one object pair involves the occlusion. Our method fails at tracking that object pair but it successfully estimates the other object pairs. As the competitors, SiamFC, ECO, SiamRPN++ and ATOM show their low effectiveness with fast motion and low illumination conditions. Moreover, ECO-E and RMRNet-TS are respectively inferior to ECO and RMRNet. Although SiamFC and ECO achieve relatively good results for the first two sequences, the sequences include relatively clean backgrounds, which helps the two methods to achieve the performance. However, if we add a small amount of noise around the objects, the performance of the two methods will be significantly reduced. In contrast, our method can maintain its performance even with the severe sensor noises of an event camera, as shown in the seventh column of Fig. 3. Meanwhile, the performance of E-MS is highly affected by the sensor noises in the HDR environments. Thus, E-MS show unsatisfied results on the EED dataset.

Since the proposed RMRNet is a relatively shallow network, our method has an advantage of relatively high efficiency. The proposed RMRNet is implemented using PyTorch on a PC with an Intel i7 CPU and an NVIDIA GTX 1080 GPU. For the mixed dataset, our method achieves real-time performance and the average computational time for each object pair (including five TSLTD frames) is 38.57 ms.