Siamese Keypoint Prediction Network
for Visual Object Tracking

Most of the early deep tracking methods utilize deep networks just for feature extraction and rely on classical approaches for target prediction. For example, CNN-SVM [19] trains a support vector machine to classify positive and negative samples using the network outputs as appearance features. This kind of region-based classification has to resort to sparse sampling for running speed while at the price of declined performance.

A better scheme is to train correlation filters based on the deep features. For example, HCFT [34] adaptively learns correlation filters on multiple convolutional layers to encode the target appearance. C-COT [11] presents a joint learning framework to fuse deep features from different spatial pyramids. ECO [10] introduces a factorized convolution operator, a generative sample space model and a conservative model update strategy to better utilize deep features for robust and efficient tracking. UPDT [4] proposes an adaptive fusion approach to leverage deep and shallow features to improve tracking performance.

End-to-end tracking usually learns a unified network to conduct both feature extraction and target prediction. For example, DeepTrack [29] trains a simple CNN model of two convolutions and two fully-connected layers in a purely online manner for visual tracking. MDNet [35] pretrains a shared CNN model and finetunes multiple domain-specific layers during online learning. FCNT [48] employs a pretrained VGG-16 network [39] and learns two additional head networks during visual tracking. STCT [49] exploits ensemble learning to utilize different CNN feature channels. CREST [41] reformulates correlation filter as a network layer with residual learning. DSLT [33] designs a shrinkage loss to improve deep regression tracking.

Recently, ATOM [9] designs the overlap maximization based architecture to predict the intersection over union (IoU) overlap between the target and the proposal boxes. As a modification, DiMP [3] replaces the modulation module with a parametric optimizer to further improve performance. Note that, both ATOM and DiMP handle prediction modeling by leveraging the IoU-Net [21], which still belongs to the anchor-based detection paradigm. In contrast, the prediction modeling in our SiamKPN is fully based on heatmap regressions for center point, target size and offsets estimation.

Similar to [26], we employ a modified ResNet-50 to define our Siamese backbone network. To make ResNet-50 suitable for our dense prediction task, we reduce the spatial stride to maintain more features and apply dilated convolutions to increase its receptive field. In particular, the original spatial strides in conv$4\_1$ and conv$5\_1$ layers are converted to one thus yielding spatially larger feature maps. Meanwhile, the original dilation rates in conv$4$ and conv$5$ blocks are changed to $2$ and $4$ to increase the receptive field. Based on this modified ResNet-50 backbone subnetwork, we extract the outputs of conv$3\_4$, conv$4\_6$ and conv$5\_3$ layers as features, and apply $1\times 1$ convolutions to adjust channels before feeding them into the following head subnetwork.

Figure 3 illustrates the architecture of the KPN head. As show in the figure, it is constructed by three $3\times 3$ convolutions and one $5\times 5$ depth-wise cross-correlation. More specifically, each KPN head involves the following flow of operations,

where Conv and Corr are abbreviations of convolution and cross-correlation. Besides, $\mathbf{w}_{t}$ and $\mathbf{w}_{s}$ denote the parameters of the two convolutions for processing target and search feature maps respectively, $\mathbf{w}_{a}$ stands for that of an internal adjustment convolution. We have made explicit the stage number $s$ to be consistent with the following notations and $\{\mathbf{x}^{(0)},\mathbf{\tilde{y}}^{(0)}\}$ are actually the output feature maps of the Siamese backbone for the target and search images. Note that the search feature maps $\mathbf{\tilde{y}}^{(s)}$ are fed into the next stage while being further processed by another two convolutions to obtain the predicted heatmaps $\mathbf{\hat{y}}^{(s)}$.

Aside from the architecture, we would like to elaborate a little on the meanings of different channels in the predicted heatmaps. In our implementation, three types of tasks are defined which include the center point, point offsets and target size estimation respectively. In particular, we use one channel to handle the center point estimation task thus it represents the response map of the target location $\mathbf{\hat{y}}$. Meanwhile, two offsets channels are utilized to address the discretization error due to stride resulting in $\{\mathbf{\hat{o}}_{x},\mathbf{\hat{o}}_{y}\}$. Besides, another two channels are employed to estimate the target size in terms of height and width $\{\mathbf{\hat{s}}_{h},\mathbf{\hat{s}}_{w}\}$.

Unlike cascading the RPN head in [14] which requires careful adjustments on the anchor scheme sequentially, cascading the KPN head can be easily achieved by a direct repetitive stacking. In addition, the whole architecture supports refinement along the cascade by successively shrinking the variance of the label heatmap. To this end, we have the shrinking version of the Gaussian heatmap which is given as below,

where $(i,j)$ denote the coordinates of an arbitrary point in the heatmap, $(i_{c},j_{c})$ represent the coordinates of a target center point, $s=1,2,3,\dots$ stands for the stage number, $\rho\in(0,1]$ controls the shrinkage strength of the variance $\sigma$ in the Gaussian function. Hence, as the number of stage increases, the Gaussian heatmap becomes more peaked around the target center point. In other words, the supervision signal is getting stricter along the cascade.

To illustrate the effect of our proposed scheme, we compare it with a naive stacking with fixed variance. Figures 4(a) and 4(b) give an example of the sequentially evolving heatmaps of the two strategies. As shown in the figures, the stacking with fixed variance can slightly help concentrate the predicted heatmap to the target, however it also strengthens the heatmap scores for the similar distractor. In contrast, the shrinking variance scheme can well enhance the heatmap scores in the target and reduce that in the background, thus easier to distinguish the target from the similar distractor as the cascade gets deeper.

It is worth mentioning that, the above scheme can also be applied to point offsets and target size estimation. In particular, the point offsets label $\{\mathbf{o}_{x},\mathbf{o}_{y}\}$ are calculated as the discretization gap between the accurate position and a stride-clamped version, i.e., $\{\frac{i}{m}-\lfloor\frac{i}{m}\rfloor,\frac{j}{m}-\lfloor\frac{j}{m}\rfloor\}$ for position $(i,j)$ and stride $m$. Besides, we define the target size label $\{\mathbf{s}_{h},\mathbf{s}_{w}\}$ as the ground-truth height and width $\{h,w\}$ around the center point and zero otherwise.

During the offline training phase, we prepare the label heatmaps by setting $\rho=0.9$ and $\sigma=31/16$ considering that the search image feature map is in size $31\times 31$. Similar to [25, 58], we train each KPN head with a multi-task loss. In particular, the keypoint estimation channel is optimized by a weight-balanced version of the focal loss [31], while the offsets and target size estimation channels are supervised via a smoothed $\ell_{1}$ loss.

More specifically, by using Equations (1) and (4), the keypoint estimation loss between the predicted and label heatmap in our framework is defined as below

where $\mathbb{I}[\cdot]$ denotes the indicator function. Note that we have omitted the stage number $s$ for notational simplicity. We set the focal loss hyper-parameters as $\alpha=2$, $\beta=4$ and $\gamma=0.05$ in all our experiments.

Besides, the offsets and target size estimation losses are defined as follows

By combining all the losses across different stages, we obtain the overall training objective as

where $\lambda_{1}$ and $\lambda_{2}$ trade-off the balance among the keypoint, offsets and target size estimation. We set $\lambda_{1}=1$ and $\lambda_{2}=0.05$ through all our experiments.

For the online tracking phase, we first crop around the target and resize it to $127\times 127$ given its bounding box of the first frame in a video sequence. After going through the backbone and the head subnetworks, the target feature maps are adjusted to obtain several matching templates in size $5\times 5$ along the cascade and these templates are then kept unchanged during the whole tracking process. Given the predicted target location in the previous frame, we crop an approximately two-times larger image region centered on this location and resize it to $255\times 255$. After applying the $5\times 5$ templates along the cascade, we use the outmost response maps to predict the target location in the current frame.

To get the predicted bounding box, the center point response map is first transformed to the range from 0 to 1 by applying the sigmoid function. By thresholding, the points with higher scores are involved in the subsequent process while the remaining points are ignored. Based on the selected points, the candidate bounding boxes are obtained by considering the corresponding point offsets and target size. To fight against the bounding box changes in adjacent frames, we add penalties for changes in target scale and aspect ratio given by the below function

where $k$ is a hyper-parameter, $\{s_{1},s_{2},r_{1},r_{2}\}$ represent the target scales and aspect ratios of adjacent frames respectively. The score of each point is multiplied by its penalty factor $\tau_{\text{penalty}}$ to get the penalized scores. Based on the penalized scores, we introduce a Gaussian smooth function to suppress large displacements from the target to yield the final score for each candidate point. To this end, the location with the highest score corresponds to the predicted center of the target. As a post-processing step, the target size of the center point is a weighted average using two adjacent frames. Therefore, the whole process involves three hyper parameters, namely the penalty coefficient, the Gaussian smoothing window coefficient and the target size smoothing coefficient.

Network Architecture. The modified ResNet-50 backbone outputs the conv$3\_4$, conv$4\_6$ and conv$5\_3$ layer as features. Their channels are adjusted from 512, 1024 and 2048 to 256, then fed into the KPN cascade. When passing through the cascade, the channels of the target and the search feature maps are maintained at 256, while the spatial sizes are kept as $15\times 15$ and $31\times 31$ respectively. Inside of each KPN, the adjustment convolution only operates on the $5\times 5$ center locations of the $15\times 15$ target feature map. Hence, the obtained $5\times 5$ feature map plays as convolutional kernels over the search feature map during the depth-wise cross correlation. The output of each KPN is further processed by two convolutions to give the predicted heatmaps in 5 channels. A layer-wise aggregation is employed to merge the three branches to produce the final prediction of each stage.

Training Datasets. The training datasets consist of the train splits from Youtube-BB [36], LaSOT [12], GOT-10k [20] and COCO [32] datasets. In particular, Youtube-BB provides amounts of sparse-labeled videos, LaSOT and GOT-10k provides frame-by-frame labeled videos, and COCO images are used to increase class diversity. With the ratio 4:2:2:1, we sample 450000 target-search pairs similar to [27] per epoch with several data augmentation operations including random shift, random scale change, random blur, random color jitter and negative samples [59]. Each target-search pair of images are in size $127\times 127$ and $255\times 255$ which follows the same dataset curation procedure in [2].

Learning. SiamKPN is trained by the adaptive moment estimation (Adam) optimizer [23] over 5 GPUs with a total batch size of 80. The ResNet-50 backbone is pre-trained on ImageNet [38]. During fine-tuning, only the last three convolutional blocks of the backbone and the whole head subnetwork are trained. In particular, for the head subnetwork, the learning rate is step-wisely decayed from 0.005 to 0.002 for the first 5 epochs and exponentially decayed from 0.002 to 0.0005 for the last 15 epochs. As for the backbone, we set the learning rate to zero for the first 10 epochs and one tenth of that in the head subnetwork for the last 10 epochs. Finally, we set the shrinking factor $\rho$ to 0.9 for the supervision.

Inference. During tracking, we normalize the predicted center point heatmap of the final stage by sigmoid function. Based on this score map, we select top-scored points by thresholding it over 0.15 and clamping the number of points in the range from 8 to 32. As for the three hyper-parameters, a two-level grid search is employed to find the best configuration. The first-level searches each hyper-parameter from 0 to 0.9 with an interval of 0.1 for all the models of the last 10 epochs. Then, we apply the second-level search with an interval of 0.01 around the so-far best configuration.

OTB-100 contains 100 representative sequences and has eleven challenge attributes, including background clutters, scale variation, deformation and so on. There are twenty grayscale sequences among them. OTB uses one pass evaluation (OPE) to evaluate trackers with two metrics, precision and area under curve (AUC) of the success plot. The precision plot illustrates the percentage of frames whose distance between the estimated location and ground truth is within the given threshold of 20 pixels. The success plot is defined by the ratios of successful frames by varying the threshold from 0 to 1.

Figure 5 presents the comparison results between SiamKPN-3s and state-of-the-art trackers on OTB-100. As shown in the figure, SiamKPN-3s ranks first both in success plot and precision plot and is ahead of the second-place tracker UPDT by 1.7% in AUC score. Besides, among all the Siamese trackers, SiamKPN-3s outperforms the previous best SiamRPN++ with relative gains of 2.3% and 1.4% in terms of success and precision respectively. This validates that our cascade heatmap architecture can provide more accurate predictions.

VOT2018 contains 60 more challenging sequences. Different from OPE test on OTB, VOT challenge reinitializes trackers at the failure frame. It also has a burn-in period of ten frames, which means ten frames after initialization will be labeled as invalid for accuracy computation. Besides, VOT uses accuracy (A), robustness (R), and expected average overlap (EAO) metrics to evaluate trackers. Particularly, EAO score can comprehensively reflect accuracy and robustness.

We compare SiamKPN-3s with both Siamese trackers and DCF trackers on VOT2018. As shown in Table 1 and Figure 6, SiamKPN-3s ranks first according to EAO, surpassing SiamRPN++ and ATOM with a relative gain of 6.3% and 9.7% respectively. Without any online updating, our method achieves same EAO as DiMP and outperforms it on accuracy. In addition, unlike other Siamese based trackers, SiamKPN-3s improves robustness in great margin. The robustness score of SiamKPN-3s is better than all other Siamese trackers. Even more, SiamKPN-3s has higher robustness score than ATOM, which is a discriminative tracker with online updating. These results are reasonable since the variance-decay strategy can help SiamKPN-3s suppress distractors and improve robustness.

A comparison of quality and speed between SiamKPN-3s and state-of-the-art trackers on VOT2018 is shown in Figure 7. We visualize the Expected Average Overlap (EAO) with respect to the Frames-Per-Seconds (FPS). Note that the FPS axis is in the log scale and we set the real-time threshold at 20 FPS. In the figure, the top-performed CF trackers run at slow speed while Siamese trackers run faster. Our SiamKPN-3s gets best EAO score while runs at 24 FPS in real-time, achieving a better trade-off between performance and speed.

To further validate the proposed framework on a larger and more challenging dataset, we conduct experiments on LaSOT. The LaSOT dataset provides a large-scale, high-quality dense annotations with 1,400 videos in total and 280 videos in the testing set. LaSOT has 70 categories objects with each containing twenty sequences, and the average video length is 2512 frames, which is useful to evaluate long-term trackers. LaSOT adopts One-Pass Evaluation (OPE) test success and precision similar to OTB. The precision is computed by comparing distance between predicted box and ground truth bounding box in pixels. The success is computed as the Intersection Over Union (IoU) between predicted box and ground truth bounding box.

Table 2 and Figure 8 report the overall comparisons between our SiamKPN-3s tracker and other methods on LaSOT testing set. Worth noting that the robustness measure is essential on LaSOT since zero-overlapping predictions will be included if a tracker loses the target in a long-term sequence. From this point, trackers with online updating shall perform better on LaSOT. Especially, DiMP ranks first in both success and precision. However, among trackers without online updating, our SiamKPN-3s outperforms all other trackers, including SiamRPN++. In particular, SiamKPN-3s achieves a larger precision score that is 4.7% relatively higher than SiamRPN++. Very unexpectedly, SiamKPN-3s even outperforms ATOM which is a recent tracker with online updating. From the above comparison, SiamKPN-3s shows great potentiality for long-term tracking.

GOT-10k is also a large-scale dataset containing over 10000 video segments and has 180 test videos. The train and test splits have no overlap in object classes, thus overfitting on particular classes is avoided. In addition, this benchmark requires all trackers to use the train split only for model training while external datasets are forbidden. We strictly follow this protocol and retrain SiamKPN-3s solely on the train split of GOT-10k. Table 3 shows the comparison results between SiamKPN-3s and other methods on the test split of GOT-10k. Not astonishing, DiMP achieves the best performance since online updating is important for tracking unseen class object. However, for trackers without online updating, our SiamKPN-3s achieves the best $\text{SR}_{0.75}$ and AO scores with relative gains of 10% and 2.3% over SiamRPN++.

To investigate the impact of different components for our method, we conduct two ablation studies using OTB-100 and VOT2018.

Numbers of Stages: Table 4 presents the performance of SiamKPN by varying the number of stages. With more stages, SiamKPN gathers more strength from the refinement process though in a diminishing manner of the performance gains. It is also not astonishing that the tracking speed is decreased by increasing the number of stages. However, the best performing SiamKPN-3s still runs at real-time speed with 24 FPS. As a side observation, by recalling that in Figure 5 and Table 1, one can find that the basic one-stage SiamKPN achieves comparable results with SiamRPN++ on the OTB-100 and VOT2018 benchmarks.

Shrinking Variance: During training, the shrinking variance in the heatmap supervision is important for guiding our framework to gradually refine the predictions. To show its effectiveness, we compare the shrinking variance strategy using factor $\rho=0.9$ with a fixed variance strategy, i.e. $\rho=1$. Table 5 shows the comparison results of SiamKPN-3s by using the two different strategies. One can observe that, with shrinking variance, our tracker can predict more accurately on OTB-100. This observation validates the importance of applying loose-to-strict supervision signals along the cascade.