Explicit Visual Prompts for Visual Object Tracking

As shown in Fig.2, EVPTrack is a simple end-to-end tracker with Image-Prompt Encoder, Spatio-Temporal Encoder, and Prompt Generator. The Image-Prompt Encoder is used for the fusion of explicit visual prompts with image features to efficiently utilize spatio-temporal and multi-scale information. The Spatio-Temporal Encoder is utilized for the interaction among template tokens, search tokens, and spatio-temporal tokens to propagate spatio-temporal information. The Prompt Generator extracts information from template and spatio-temporal tokens to generate explicit visual prompts (i.e., multi-scale prompts and spatio-temporal prompts). EVPTrack employs image pairs and spatio-temporal tokens as input. Specifically, here the tokens come from the previous frame. One of the images is a template image $z\in\mathbb{R}^{3\times H_{z}\times W_{z}}$. The image is from the target region in the first frame of the video and contains information about the initial appearance of the target and some of the background information. The other image is a search image $x\in\mathbb{R}^{3\times H_{x}\times W_{x}}$. It comes from the region where the target may appear in the subsequent sequences of the video. To summarize, we utilize spatio-temporal and multi-scale information by providing explicit visual prompts to improve the robustness of the tracker in complex scenarios such as target appearance changes and deformation.

Image-Prompt Encoder inputs include search tokens, template tokens, and explicit visual prompts. We use explicit visual prompts to guide feature extraction and fusion without other complex operations. This way of utilizing spatio-temporal information is simple and avoids additional elaborate updated modules. Specifically, we first perform linear projection on the image pairs. Here, we utilize hierarchical patch embedding with a total downsampling stride of 16, and obtain search tokens and template tokens denoted as $f_{z}\in\mathbb{R}^{N_{z}\times D}$ and $f_{x}\in\mathbb{R}^{N_{x}\times D}$, respectively. Here, $N_{x}=H_{x}W_{x}/16^{2}$, $N_{z}=H_{z}W_{z}/16^{2}$, $D=512$. And we add learnable position encoding ($P_{z}\in\mathbb{R}^{N_{z}\times D}$, $P_{x}\in\mathbb{R}^{N_{x}\times D}$) to preserve spatial information. Then, we concatenate $f_{z},f_{x},f_{p}$, and feed them into N-layer encoder. Finally, the search features are utilized to locate the target. This process can be described by the following equations:

$$\begin{split}&f_{mp}^{0}=concat(f_{p},f_{z},f_{x}),\\ &f_{mp}^{n}=encoder(f_{mp}^{n-1}),n=1...N,\\ &f_{mp}^{N}=LN(f_{mp}^{N}),\end{split}$$

(1)

here, $f_{p}$, $LN$ denote explicit visual prompts and layer normalization, respectively. For a more detailed design of encoder and patch embedding refer to HiViT(Zhang et al. 2023).

Spatio-Temporal Encoder is a critical component of our framework for maintaining spatio-temporal tokens and thus exploiting the rich contextual information in successive frames. Its structure is shown in Fig.3, which is also based on the transformer encoder implementation. Specifically, it uses template, search, and spatio-temporal tokens as input, and obtains new spatio-temporal tokens after feature fusion by transformer encoder. This process can be described by the following equations:

$$\begin{split}&f_{in}=concat(f_{t-1},f_{z},f_{x}),\\ &f_{out}=encoder(LN(f_{in})),\\ &f_{t}=split(f_{out})[f_{z}],\end{split}$$

(2)

here, $f_{t-1}$ denotes the spatio-temporal tokens obtained at time t-1, which will be used at moment t. When t = 1, we initialize it using template tokens, i.e., $f_{0}=f_{z}$, which ensures that the initialized spatio-temporal tokens can be adapted to any video sequence and can enhance the target features. Then, we split the template tokens into $f_{out}$ as new spatio-temporal tokens. This is to avoid heavy accumulation of mistakes in the spatio-temporal tokens propagated during long-term tracking.

We propagate information between successive frames via tokens, thus effectively exploiting spatio-temporal information to capture the target appearance changes. In contrast to intermittent template updating mechanisms, we utilize a mechanism that allows for continuous propagation of spatio-temporal information. Therefore, avoiding the difficult problem of when-to-update that appears with intermittent template updates.

EVPTrack is equipped with prompt generators to obtain explicit visual prompts (i.e., spatio-temporal prompts and multi-scale prompts). On the one hand, the role of spatio-temporal prompts is to enable the tracker to capture target appearance changes. On the other hand, the multi-scale prompts provide different fine-grained features, thus maintaining excellent tracking even when variations in target scale. For multi-scale prompt generator, whose structure is shown in Fig.4(a). We first crop the template image into patches with resolution ($P,P$). Then, each patch is mapped to a 1D vector of size D using a learnable linear projection $E\in\mathbb{R}^{(3\times P^{2})\times D}$. In order to obtain features at different scales, here we use three sizes of resolution, P is set to 14,16,18 respectively. The features $f_{p14},f_{p16},f_{p18}$ are obtained corresponding to three different scales. We perform avgpooling on each feature separately and then concatenate to get $f_{ms}$. Finally, $f_{ms}$ are fed into a fully connected network to get the final multi-scale prompts $P_{ms}$. The following equations can describe this process:

$$\begin{split}&f_{ms}=concat(avg(f_{p14}),avg(f_{p16}),avg(f_{p18})),\\ &P_{ms}=FFN(f_{ms}).\end{split}$$

(3)

As shown in Fig.4(b), the spatio-temporal prompt generator uses spatio-temporal tokens as input. First, the input is average pooled by performing avgpooling operation. Then, the extracted features are fed into a fully connected network to obtain the final prompts. Although, these prompt generators are simple in design, it is shown in the experimental section that the generated explicit visual prompts are effective. Therefore, EVPTrack using a small number of explicit visual prompts reduces the computational burden and avoids introducing too much redundant information.

Our methods are implemented based on python3.8 and pytorch1.10 framework. Our trackers were trained on 4 NVIDIA A10 GPUs. During the inference phase, the trackers were tested at speed on a single NVIDIA RTX2080Ti.

Models. We train two variants of EVPTrack with different input image pair resolutions as follows:

• •

  EVPTrack-224. Template size: 112x112 pixels. Search region size: 224x224 pixels.

• •

  EVPTrack-384. Template size: 192x192 pixels. Search region size: 384x384 pixels.

Training strategy. We use HiViT-Base(Zhang et al. 2023) model as the Image-Prompt Encoder and its parameters are initialized with MAE(He et al. 2022). EVPTrack is trained on the same datasets as mainstream trackers(Ye et al. 2022), including LaSOT(Fan et al. 2019), GOT-10k (Huang, Zhao, and Huang 2021), TrackingNet(Müller et al. 2018), COCO (Lin et al. 2014). During training, data augmentation employed horizontal flip and brightness jittering, and the optimizer utilized AdamW(Loshchilov and Hutter 2019). The learning rate of backbone is set to $1\times 10^{-5}$, the learning rate decay is set to $1\times 10^{-4}$, and the learning rate of other parameters is set to $1\times 10^{-4}$. A total of 150 epochs of training, and each epoch uses 60k search images. The learning rate decreases by factor after 120 epochs. In addition, for a fair comparison of got-10k, we follow its standard protocol and train out the other two models using only the got10k dataset. The models were trained for 50 epochs and started decaying in the 40th epoch as it was trained with only one dataset.

In order to allow EVPTrack to learn the spatio-temporal information between consecutive frames, the sampling strategy during training is therefore different from the traditional sampling strategy. Specifically, we sample M videos in one iteration, each containing N search frames and one template frame. Therefore the batch size is $N*M$. Then, the N search frames are predicted sequentially in order, enabling the tracker to utilize tokens to propagate spatio-temporal information between consecutive frames. For EVPTrack-224, we set N, M to 4 and 8, respectively, with a batch size of 32. EVPTrack-224 is trained on 4 GPUs, so the total batch size is 128. Due to GPU memory constraints, for EVPTrack-384 we set N, M to 4 and 8, respectively.

LaSOT. LaSOT (Fan et al. 2019) test dataset is a challenging long-term tracking benchmark dataset containing 280 video sequences to effectively evaluate the long-term tracking performance of trackers. EVPTrack’s evaluation results are shown in Tab.1, EVPTrack outperforms existing trackers by obtaining 72.7%, 82.9% and 80.3% for AUC, P and $P_{norm}$ respectively. In addition, Fig.5 demonstrates the performance of EVPTrack-384 in different challenge scenarios (e.g., scale variation, aspect ratio change). Illustrating that our SeqTrack-B384 performs better than other competing trackers on almost all challenge scenarios.

LaSOT_ext. $\rm LaSOT_{ext}$ (Fan et al. 2021) is an expansion of LaSOT with the addition of 150 videos from 15 new categories. There are many similar interfering objects and fast-moving small objects in the added videos, so it is a challenging dataset. As shown in Tab.1, although EVPTrack-224 has only 48.7% AUC, EVPTrack-384 obtains 53.7% AUC and 61.9% P at larger input resolutions. EVPTrack-384 outperforms EVPTrack-224 by 5% while outperforming all comparative trackers. The reason for this disparity, we consider, is that the low resolution lacks enough background context to distinguish between similar objects.

TrackingNet. TrackingNet (Müller et al. 2018) is a large-scale tracking benchmark, that contains 511 video sequences, which covers diverse object classes and scenes. It does not provide ground-truth and needs to submit the prediction results to its server to get the Success(AUC) and Precision(P and $P_{norm}$). The returned results are shown in Tab. 2, our EVPTrack-384 obtains 84.4% , 89.1% and 84.2% in terms of AUC, P, and $P_{norm}$, respectively. Our EVPTrack-384 can still achieve an advanced performance.

GOT-10k. GOT-10k (Huang, Zhao, and Huang 2021) is a large-scale challenging tracking benchmark with over 10k video segments and has 180 segments for the test set. One of the challenges of this dataset is that there is no overlap between the object classes in the training and test sets, except for the general classes of motion objects and motion patterns. We follow the defined protocol proposed in (Huang, Zhao, and Huang 2021) and train our model only using the GOT-10k training set and submit the tracking output to the official evaluation server. As reported in Tab.1, EVPTrack-384 obtains 76.6% and 86.7% in terms of AO and $SR_{0.5}$, respectively. Furthermore, EVPTrack-384 achieved the top-ranked performance on AUC, reaching 76.6%, surpassing all other trackers. This strongly verifies that our method has strong generalization and is not sensitive to categories.

TNL2K. TNL2K is a recently publicly released large-scale tracking dataset, which contains 3000 challenging video sequences: 700 for testing and 2300 for training. We evaluated our tracker with a test set of 700 video sequences. From Tab. 3, we can observe that our tracker achieves a state-of-the-art performance on this dataset. In particular, EVPTrack-224 obtained 57.5% AUC, even more than SeqTrack-384 using a higher resolution.

UAV123. The UAV123 (Mueller, Smith, and Ghanem 2016) contains 123 video sequences that are captured from the UAV platform. It is a challenging aerial video dataset. The benchmarks can be used to assess whether the tracker is suitable for deployment to a UAV123 in real-time scenarios. From Tab. 1, we can observe that our tracker performs the best among all compared trackers.

We use EVPTrack-224 to validate the effectiveness of our method on the LaSOT and UAV123 benchmarks.

Baseline $v.s.$ EVPTrack. In Tab.4, we compare the performance of the baseline tracker #1 using the initial template and ours EVPTrack #2 using explicit visual prompts. The results show that EVPTrack performance improves with the addition of spatio-temporal and multi-scale explicit visual prompts. While the initial template contained reliable targets, it was unable to cope with target target appearance changes during tracking. In contrast, EVPTrack provides multi-scale target information to enhance the target’s features and spatio-temporal information to perceive target appearance changes, which helps the tracker accurately predict the bounding box.

Study on explicit visual prompts. As shown in Tab.4, #3 adds the same number of learnable tokens as explicit visual prompts, which is a type of implicit prompt. Although implicit prompts can improve performance, they are slightly lower than EVPTrack. This is because the implicit prompts does not change during the inference phase, and the tracker relies solely on initial template information and lacks exploitation of spatio-temporal information. In contrast, our display prompt is adaptively generated from templates and spatio-temporal tokens.

Moreover, we conducted experiments #4,#5,#6,#7,#8 and #9 to explore the effectiveness of the multi-scale prompts and the spatio-temporal prompts. We can observe that there is a significant improvement on the performance of multi-scale prompts relative to single-scale prompts. In experiment #4 compared to baseline #1, this improves the performance of the tracker by providing different fine-grained template features. In experiment #5, we trained EVPTrack to use only spatio-temporal prompts. This improves the performance of the tracker by exploiting spatio-temporal information between consecutive frames compared to baseline #1. Comparison of experiments #5 and #6 demonstrates the effectiveness of the spatio-temporal encoder. In Tab.4, the best performance is achieved when both prompts are employed.

Video sequence length during training. As shown in Tab.5, to explore the spatio-temporal information between consecutive frames, we performed experiments related to the length of the video sequence during training. We keep the batchsize, the learning rate, and the total amount of data constant when training the trackers. Comparing experiments #1 and #2, when the sequence length was changed from 4 to 8, the AUC increased by 0.5% on LaSOT. This demonstrates that the tracker can obtain spatio-temporal information from consecutive video frames. Furthermore, comparing experiments #2 and #3, although the sequence length changed from 8 to 16, the AUC decreased by 1.2% on LaSOT, which may be because only two videos were used in each iteration during training, leading to a lack of generalization of the trained tracker. So we can notice a limitation of our approach: the limited GPUs memory restricts the length of the training video which hinders our tracker from fully exploiting the spatio-temporal information.

Speed, FLOPs, and Params. As demonstrated in Tab.6, our EVPTrack-224 can run in real-time at over 71fps. Furthermore, the FLOPs of EVPTrack-384 are 2x lower than SeqTrack-B384, and the AUC of EVPTrack-384 on LaSOT is 1.2% higher than that of SeqTrack-B384. This demonstrates that our method can have the effective benefit of temporal and multi-scale information, as well as a good balance between efficiency and performance.

Qualitative comparison. In addition, to demonstrate the effectiveness of our method more intuitively, we provide some qualitative comparison results in Fig.6. For example, in the first sequence, compared to the other three SOTA trackers, our EVPTrack can still obtain accurate tracking results in the situation of having similar objects by utilizing explicit visual prompts.