3D Visual Tracking Framework with Deep Learning for Asteroid Exploration

The goal of visual tracking is to precisely and robustly perceive real-time states (e.g. location, velocity and size) of an instereted target in complex and dynamic environment. As the basis for pose estimation, behavior understanding, and scene interpretation, visual object tracking has wide application in space debris removel [14, 15], space robotic inspection [16], spacecraft rendezvous and docking [17, 18].

At present, most of visual trackers in aerospace domain mainly focus on feature-based method. Huang [15] proposed a novel feature tracking algorithm. Firstly, it extracted features by SURF detector. And then, the pyramid-Kanade-Lucas-Tomasi (P-KLT) algorithm was adopted to match key-points between two adjacent frames. Finally, accurate target bounding box is obtained with Greedy Snake method. A feature-tracking scheme was also presented in [19] that combines traditional feature-point detector and frame-wise matching to track a non-cooperative and unknown satellite. Felicetti [20] put forward an active space debris visual tracking method, in which the chaser satellite can keep the moving object in field of view of optical camera by continuously pose correction. Those feature-based trackers heavily relied on the manually designed feature detector and cannot handle extreme cases in space (e.g. illumination variation, scale variation, fast motion, rotation, truncation, and background clutters).

With many challenges and benchmarks emerging, such as OTB2015 [21], YouTube-BB [22], ILSVRC-VID [23], GOT-10K [24], and Visual Object Tracking challenges [25, 26], the development of generic object tracking is far beyond imagination [27]. Especially, deep learning based trackers [28, 29, 30, 31, 32, 33, 34] have dominated the whole tracking community in recent years, because of its striking performance. Generic object trackers often follow one protocol that no prior knowledges is available. This hypothesis is naturally suitable for asteroid visual tracking, because of high uncertainty of vision tasks in space. In paper [35], most of state-of-the-art generic trackers had been evaluated on space non-cooperative object visual tracking (SNCOVT) dataset, which provides firm research foundation for our work.

However, recent visual trackers mainly cope with RGB [36, 37, 38], RGB-Thermal [39, 40, 41], and RGB-Depth [42] video sequences, which only provides poor target information in 2D space and heavily restricts practical applications of visual tracking. In contrast, 3D visual tracking is more promising and competitive. To this end, we propose a novel deep-learning based 3D tracking framework for asteroid exploration.

Although there are plenty of related works [11, 43, 12, 44, 13, 45, 8, 9, 10], the concept of 3D object tracking remains ambiguous. Hence that, we first define that 3D object tracking is to obtain 3D information of any target in real-time by leveraging various visual sensors, given initaial states at the beginning. According to the degrees of freedom of tracking result, all the 3D trackers can be concluded into three categories: 3-DoF, 4-DoF, and 6-DoF Tracker. Obviously, the more degrees of freedom, the more difficult 3D tracking task is.

3-DoF tracking means that both 3D location and size of object (i.e. 3D axis-aligned bounding-box) should be estimated. However, most of researches only considered predicting 3D object center. In paper [43], a color-augmented search algorithm was presented to obtain the position and velocity of vehicle. Asvadi et al. [44] utilized two parallel mean-shift localizers in image and pcd space, and made fusion for two localizations by Kalman filter. This algorithm can effectively achieve low 3D center error. To the best of our knowledge, the framework proposed in this paper is the first real 3-DoF tracker that can be applied for collision avoidance, 3D reconstruction, and pose estimation.

Comparing with 3-DoF methods, 4-DoF tracker often needs to predict an extra heading angle of target, which is original from 3D tracking requirement in autonomous driving. Giancola et al. [8] presented a novel 3D siamese tracker with the regularization of point cloud completion. However, the exhaustively searching for candidate shapes in this method would consume high computational cost. Qi et al. [9] also proposed point-wise tracking paradigm, P2B, which addressed 3D object tracking by potential center localization, 3D target proposal and verification. Since this method purely uses point cloud data as input, it is vulnerable to initial object point cloud and only achieves 0.562 $AO^{3d}$ at 40 FPS on KITTI tracking dataset [46].

In paper [11, 12], 6-DoF tracking task was considered as joint 2D segmentation and 3D pose estimation problem, and the method looked for the pose that best segmented the target object from the background. Crivellaro et al. [13] present a novel 6-DoF tracking framework with monocular image, including expensive 2D detector, local pose estimation and extended kalman filter. However, the non-textured 3D model of target should be given in this method, which heavily limits the scope of its application. To be honest, the asteroid tracking is one of 6-DoF tracking tasks. We think it is very hard to construct an end-to-end network that can directly and precisely predict the 6-DoF object states. Therefore, we decompose asteroid tracking problem into 3-DoF tracking and pose estimation. It is worthwhile noting that this paper simply focus on the 3-DoF tracking task of asteroid.

There is no doubt that collecting real data to create large-scale 3D asteroid tracking dataset is impractical, like KITTI dataset [46] and Princeton RGB-D Tracking dataset [42]. Meanwhile, constructing dataset by ground simulation [35] is very expensive and limited. Inspired by UAV123 dataset [47], we therefore consider to collect rich 3D tracking data by virtual physics engine, V-rep. The power and convenience of physics engine make automatic data labelling possible, which greatly reduces constructing cost and improves annotation accuracy.

The critical foundation of dataset constructing based on physics engine is 3D modeling of asteroids with diverse shapes and textures. We create three types of 3D asteroid model (i.e. Asteroid04, 05, and 06) with 6 different textures, which are illustrated at the left of Fig. 3. From point of our view, one asteroid model with diferent textures can be considered as different fine-grained categories. In addition, we introduce 9 simulated space scenes into dataset construction. All the 3D asteroid models have been controlled by scripts to carry out random 6-DoF motion in simulated scenes.

The detailed setup of vision sensors for data collection can be seen at upper right corner of Fig 3. Only two perspective vision cameras are equipmented with 0.4-meter baseline in x-axis direction on observer spacecraft, which can not only acquire binocular video sequences during simulation, but also achieve aligned depth maps and point clouds from camera buffer by the API of physics engine. The camera matrix is very vital for our 3D tracking framework in which point clouds should be projected to image plane, however, V-rep merely provides perspective angle $(\alpha_{x},\alpha_{y})$ and resolution $\left(W,H\right)$. To this end, we compute camera intrinsic matrix $M_{i}$ following Eq. 1 (more derived details are given in Appendix A):

At final, we collect 360 sequences for training and 135 sequences for testing set. Each sequence has 300 frames binocular images, depth maps, and point clouds. Following the protocol proposed in [24], there is no overlap on fine-grained categories between training and testing set. The screenshots of 3D asteroid tracking testing set with 2D and 3D annotations are shown in Fig. 3.

In actual, our framework predicts the object size class and normalized size residuals, rather outputs an axis-aligned 3D bounding-box directly. Therefore, size classes should be predefined and the number of samples in each category has better to be balanced. To this end, we make a statistic analysis for size category of axis-aligned 3D bounding-boxes in 3D asteroid tracking dataset and define 14 default size categories summarized in Table I. The final distribution of size classes can be seen in Fig. 4a.

Meanwhile, the distribution of object volumes and motion speeds are shown in Fig. 4b and 4c, respectively. Because of random asteroid rotation and manual shape modification, the volume of samples is widely distributed (16 $m^{3}$ to 1600 $m^{3}$). It is worthwhile noting that object volume metioned here is the volume of axis-aligned 3D bounding-box, instead the real one. In addition, as we mentioned above that all the 3D asteroid models are controlled by scripts to carry out random 6-DoF motion, Fig. 4c clearly demonstrates that the random translation of object obeys normal distribution.

As paper [35] had proven that all the generic trakcers are vulnerable to illumination variation, we also consider this factor into our dataset. Furthermore, we have limited the illumination of all samples at low value (see in Fig. 4d), which conforms to the reality of space environment.

The main metric utilized to analyse 2D tracking performance is average overlap, $AO$, which is also widely applied in visual tracking benchmarks like OTB[21], VOT[26], and GOT-10k [24]. In this paper, we also introduce this metric to evaluate 2D monocular tracker and 3D tracking results in bird’s eye view (i.e. $AO^{2d}$ and $AO^{bev}$). Furthermore, We extend average overlap measurement to evaluate 3D tracking accuracy, named as $AO^{3d}$:

in which, $M$ is the sequences number in 3D asteroid tracking test set, $N_{s}$ denotes the length of s-th sequence, and $\Omega^{3d}_{t}$ denotes the region overlap between axis-aligned 3D annotation $\tilde{A}_{t}^{3d}$ and 3D tracking result $A_{t}^{3d}$ at t-th frame. It is worthwhile noting that tracking restart mechanism is introduced to 3D tracking evaluation. That is 3D tracker is allowed to reinitialize, once the Intersection-over-Union (IoU) of 3D tracking result $\Omega^{3d}_{t}$ is zero.

Besides average overlap metric, we also adopt success rate, $SR$, as our indicator, which denotes the percentage of successfully tracked frames where the overlap exceeds a threshold. In this work, we take 0.5 as threshold to measure the success rate of one tracker. However, $SR$ at a specific threshold is not representative, we further introduce success plot presented by [21] into our evaluation tool.

Another intuitive technique index of tracker is location precision. Therefore, we propose 3D average center error metric ($ACE^{3d}$) that measures the mean distance between all the predicted trajectories and ground-truth trajectories:

where $\Gamma_{s}=\left\{p_{t}|t=1,\ldots,N_{s}\right\}$ and $\tilde{\Gamma}_{s}=\left\{\tilde{p}_{t}|t=1,\ldots,N_{s}\right\}$ are the predictions and ground-truths of s-th sequence, which consist of a series of object center in 3D space. $\Delta\left(\Gamma_{s},\tilde{\Gamma}_{s}\right)$ is formulated as:

in which, $\left\|\cdot\right\|_{2}$ denotes Euclidean distance.

The SiamFC algorithm utilizes full convolutional layers as an embedding layer, denoted as embedding function $\varphi$. The exemplar image $z$ and multi-scale search images $X=\left\{x_{i}|i=1,2,...,S\right\}$ are mapped to a high-dimensional feature space with $\varphi$, which can be trained by a general large-scale dataset. And then similarity score maps are generated by cross-correlation between exemplar kernal and the tensors of multi-scale search images, where the 2D tracking result can be reached after post-processing.

For this algorithm, the key is to learn discriminative embedding function $\varphi$. In this work, we train the backbone of SiamFC from scratch on ILSVRC-VID [23] with 50 epochs and further fine-tune it on our 3D asteroid tracking training set. Benefitting from the strong generalization ability of Siamese network, SiamFC can achieve a decent 2D tracking performance. Although there are multifarious deep-learning based 2D monocular trackers, like SiamRPN++ [30], Ocean [36], STARK [37], TransT [38], which greatly outperform SiamFC in tracking accuracy, we think weak dependence on high-performance 2D monocular tracker can effectively guarantee the generalization ability of our framework and improve its running speed.

Once tracking result $A_{t}$ is acquired from 2D monocular tracker, it can be used for frustum proposal extraction from raw points set $P_{\text{raw}}=\left\{(x_{i},y_{i},z_{i})\in\mathbb{R}^{3}|i=1,2,...,r\right\}$, as shown in Fig. 5. We first crop out $m$ points from raw point cloud $P_{\text{crop}}=\left\{(x_{i},y_{i},z_{i})|1\leqslant z_{i}\leqslant 45,i=1,2,...,r\right\}$, and then compute the projected point cloud in image plane $P_{\text{img}}=\left\{(u_{i},v_{i})|i=1,2,...,m\right\}$ with camera matrix $\mathbf{M}\in\mathbb{R}^{3\times 4}$. Therefore, we can achieve the frustum proposal $P_{\text{frustum}}=\left\{(x_{i},y_{i},z_{i})|i=1,2,...,n\right\}$ by the indices of points which belong to $A_{t}$ region in $P_{\text{img}}$.

In this work, we assume that there are no other objects in the range of LiDAR perception, that is, all of the points in frustum proposal belong to tracked target, which conforms to practical scenario in space. Therefore, we propose a novel light-weight amodal axis-aligned bounding-box network, A3BoxNet, which makes directly prediction on frumstum proposal and no point segmentation is considered into. Fig. 2 clearly shows that our A3BoxNet mainly consists of two modules: center regression network and amodal box estimation network. The former one is responsible for the estimation of stage-1 object center. Another is to predict object center residuals, object size category, and normalized size residuals. It is worthwhile noting that both center regression network and amodal box estimation network are only support fixed number of input points. To this end, we add random sampling at the beginning of A3BoxNet.

The architecture of center regression network and amodal box estimation network are both illustrated in Fig. 6a and 6b. It can be clearly seen that there is no significant difference between two networks which are both derived from PointNet [48]. In A3BoxNet, we use a number of MLP for high-level features extraction. And max-pooling layer is introduced to aggregate global features with symmetric property that is critical for unsorted point sets. The global feautres concatenated with one-hot vector of coarse-grained asteroid category can be further used for predicting stage-1 center, stage-2 center, and object size.

We visualize the center predction progress of A3BoxNet in Fig. 7. The frustum proposal have been normalized by substracting the centroid of point sets firstly, which can improve the translation invariance of A3BoxNet and speed up convergence rate during training. In addtion, it shows that after two stage prediction, estimated object center is very close to the ground-truth. The predicted center is formulateds as follows:

where $\bar{C}$ is the centroid of point cloud in frustum proposal, $\Delta C_{1}$ is the output of center regression network, $\Delta C_{2}$ is the center residuals predicted by amodal box estimation network.

Except for estimating the center residuals of object, our amodal box estimation network also classifies object size to 14 predefined categories (see in Table I) as well as predicts normalized size residuals ($N_{size}$ scores for size classification, $N_{size}\times 3$ for size residuals regression). At final, we remap the predicted size category and normalized size residual to original scale by multiplying the largest length of enclosing bounding-box of point clouds.

To train our A3BoxNet, we use a joint loss function $\mathcal{L}_{joint}$ to simultaneously optimize both two submodules:

in which,

more details about optimization function of A3BoxNet are given in Appendix B. In addition, A3BoxNet is trained on 3D asteroid tracking dataset with 25 epoches and 32 batch size. All the inputs are fixed number of point sets that randomly sampled from object point cloud.

We believe the fusion of 3D tracking and 2D monocular tracking results can make significant improvement on 2D tracking performance. To this end, we propose a simple and effective 2D-3D tracking fusion strategy (as shown in Fig. 8). At first, we calculate the projection of center cross-section plane of 3D bounding-box in image plane, $A^{proj}_{t}$. And then, 2D monocular tracking result $A_{t}$ is weighted with $A^{proj}_{t}$:

in this work, we set $\lambda_{1}=0.3$ and $\lambda_{2}=0.7$.

Considering that there are a few researh results in the scope of 3-DoF tracking at present, therefore, we replace the A3BoxNet of our framework with minimum enclosing 3D bounding-box algorithm (i.e. computing the length of aixs-aligned 3D bounding-box by $\max\left\{P_{\text{frustum}}\right\}-\min\left\{P_{\text{frustum}}\right\}$) to realise a simple 3D tracking baseline as comparison. Both 2D and 3D tracking evaluation results of 3D tracking baseline are summarized at 3rd row in Table II. And its success plots are shown in Fig. 9a. We find out that the 3D tracking baseline with 2D-3D fusion strategy improves 2D monocular tracking performance about 10.5%, however, it achieves quite poor performance in 3D space.

The overall performance of deep-learning based 3D tracking framework, also named as Track3D, is illustrated in Fig. 9b. It reaches excellent evaluation results on 3D asteroid tracking test set (0.669 $AO^{3d}$ and 0.570 $ACE^{3d}$) with high real-time performance (77.0 FPS). We visualize part of tracking results of Track3D in Fig. 10, which demonstrates the effectiveness of our 3D tracking method. It clearly shows that our framework can estimate accurate 3D bounding-box even under extreme truncation (see 150-th frame of sequence 0001 in Fig. 10). The 3D trajectory plots in Fig. 11 also intuitively show our framework predicts precise 3D object location, which greatly outperforms 3D tracking baseline. In addition, it can be seen in the first colum of Table II that Track3D with 2D-3D fusion strategy can also make significant improvement on 2D tracking performance.

Extensive experiments are implemented in this subsection that explore how two main modules of Track3D (i.e. 2D monocular tracker and A3BoxNet) make influences on final 3D tracking performance, which points the way to design effective 3D tracking framework for future work.

Firstly, plenty of classic monocular trackers have been evaluated on left video sequences of 3D asteroid tracking dataset, such as SiamFC [28], SiamRPN [49], ECO [50], Staple [51], KCF [52], DAT [53], BACF [54], STRCF [55], MKCFup [56], CSRDCF [57], CSK [58], and MOSSE [59]. The evaluation results are illustrated in Fig. 12 which straightforward shows the accuracy of 12 monocular trackers are distributed in a large range from 0.375 to 0.746. SiamFC adopted in our framework only achieves intermediate 2D tracking performance in both accuracy and precision metrics.

And then, we evaluate the whole framework under different monocular trackers respectively and plot 3D success and precision curves in Fig. 13a and 13b. We found that 3D evaluation curves become much dense comparing with 2D curves of Fig. 11, which denotes that 2D monocular trackers just have slight influence on 3D tracking performance of our framework. In other words, Track3D can still work even based on poor 2D monocular tracker. We further plot the relationship between 2D tracking performance and framework accuracy in Fig. 14, which intuitively demonstrates the generalization ability of Track3D.

Meanwhile, We evaluate amodal bounding-box estimation network, A3BoxNet, on 3D asteroid tracking dataset by randomly sampling 1024 points from frustum proposal. Its performance is also illustrated in Fig. 15. It can be clearly seen that our A3BoxNet predicts high accurate axis-aligned bounding-box purely with partial object points. Furthermore, our network is able to run at 281.5 FPS, which totally satisfies the requirement of application on edge computing device.

We also study how the number of input points affects the performance of amodal axis-aligned bounding-box network. We retrain the A3BoxNet by randomly sampling different numbers of point set (e.g. 512, 1024, 2048, 3072, and 4096 points) from object point clouds in 3D asteroid tracking training set. And all the models are trained from scratch with 25 epoches and 32 batch size. The evaluation results are plotted in Fig. 16, which clearly shows accuracy and precision of A3BoxNet have contradictory with the number of input points. Once the accuracy of model increasing, the corresponding precision will decrease.

Besides, the influence of object category on A3BoxNet is further studied. We remove one-hot vector of object category in center regression network and amodal box estimation network, and retrain A3BoxNet from scratch with 1024 point sets. The performance comparison between original A3BoxNet and A3BoxNet without category information is summarized in Table III, which proves the object category can make a slight improvement on the 3D accuracy of A3BoxNet.