Learning Adaptive and View-Invariant Vision Transformer with Multi-Teacher Knowledge Distillation for Real-Time UAV Tracking

There are two main types of modern UAV trackers: DCF-based and CNN-based trackers (Li et al., 2021; Liu et al., 2022a). DCF-based trackers are favored for UAV tracking due to their remarkable efficiency on CPU, but they face difficulties in maintaining robustness under complex conditions (Li et al., 2021, 2020; Huang et al., 2019). Current CNN-based trackers, like those in (Cao et al., 2021, 2022), show advancements in tracking precision and robustness for UAV tracking, but their efficiency lags behind DCF-based trackers considerably. Although model compression techniques, as seen in (Wang et al., 2022; Wu et al., 2022; Zhong et al., 2023), were utilized to enhance efficiency, these trackers still face challenges associated with unsatisfactory tracking precision. A recent trend in the visual tracking community shows an increasing preference for single-stream architectures that seamlessly integrate feature extraction and correlation using pre-trained ViT backbone networks (Xie et al., 2021; Cui et al., 2022; Ye et al., 2022; Xie et al., 2022, 2024b), demonstrating significant potential. Several representative methods, including OSTrack (Ye et al., 2022), Mixformer (Cui et al., 2022), DropMAE (Wu et al., 2023), ZoomTrack (Kou et al., 2023), AQATrack (Xie et al., 2024b), EVPTrack (Shi et al., 2024), etc., demonstrate the significant success of applying this paradigm to tracking tasks. Although these frameworks are efficient due to their compact nature, very few are based on lightweight ViTs, making them impractical for real-time UAV tracking. To address this, TATrack (Li et al., 2024a) proposes an efficient ViT-based tracking framework for real-time UAV tracking, which integrates feature learning and template-search coupling into an efficient one-stream ViT. Similarly, ETDMT (Zhang et al., 2024) builds on lightweight ViT to introduce a tracker that combines template distinction with temporal context, while Aba-ViTrack (Li et al., 2023) enhances efficiency in real-time UAV tracking using lightweight ViTs and an adaptive background-aware token computation method. However, the variable token number in (Li et al., 2023) necessitates unstructured access operations, leading to notable time costs. Recent research has focused on improving the efficiency of ViTs by balancing their representation capabilities with computational efficiency. Methods like lightweight ViTs, model compression, and hybrid designs (Wang et al., 2020; Zhang et al., 2022b; Mao et al., 2021; Chen et al., 2021b; Li et al., 2022c) have been explored, but they often sacrifice accuracy or require time-consuming fine-tuning. Recent developments in efficient ViTs with conditional computation focus on adaptive inference, dynamically adjusting computational load based on input complexity to accelerate model performance. For example, DynamicViT (Rao et al., 2021) introduces control gates to selectively process tokens, while A-ViT (Yin et al., 2022) employs an Adaptive Computation Time strategy to avoid auxiliary halting networks, achieving gains in efficiency, accuracy, and token prioritization. Aba-ViTrack (Li et al., 2023) effectively utilizes the latter, showcasing significant potential for real-time UAV tracking. In our work, we focus on improving the efficiency of ViTs for UAV tracking through more structured methods, specifically the adaptive activation of Transformer blocks for feature representation.

View-invariant feature representation has garnered significant attention in the field of computer vision and image processing. This technique aims to extract features from images or visual data that remain consistent across various viewpoints or orientations, providing robustness to changes in the camera angle or scene configuration (Kumie et al., 2024a; Bracci et al., 2018; Li et al., 2017; Rao et al., 2002). Traditional methods typically rely on handcrafted features and geometric transformations to achieve view-invariant representations (Xia et al., 2012a; Ji and Liu, 2010; Rao et al., 2002). While effective, these traditional methods are often limited to specific scenarios with relatively simple and fixed backgrounds, making them unsuited for handling the complexity and variability of real-world visual data. Recently, many researchers have been eager to apply Convolutional Neural Networks (CNNs) and other deep architectures for extracting view-invariant representations, thanks to deep learning’s remarkable ability to extract discriminative features (Kumie et al., 2024b; Men et al., 2023; Pak et al., 2023; Gao et al., 2022; Shiraga et al., 2016). For instance, Kumie et al. (2024b) propose the Dual-Attention Network (DANet) for view-invariant action recognition, incorporating relation-aware spatiotemporal self-attention and cross-attention modules to learn representative and discriminative action features. Gao et al. (2021) propose a View Transformation Network (VTN) that realizes the view normalization by transforming arbitrary-view action samples to a base view to seek a view-invariant representation. These approaches leverage the capacity of deep models to capture complex patterns and variations in visual data. By ensuring that learned features are resilient to changes in viewpoint, these methods contribute to the robustness and generalization of vision-based systems. View-invariant feature representation is helpful for a variety of vision tasks, such as action recognition (Xia et al., 2012b; Liu et al., 2018; Kumie et al., 2024a), pose estimation (Bracci et al., 2018), human re-identification (Liu et al., 2023; Perwaiz et al., 2023), and object detection (Feng et al., 2022). However, to the best of our knowledge, the integration of view-invariant representations into visual tracking frameworks remains underexplored. In our work, we make the first attempt to learn view-invariant feature representations through mutual information maximization based on ViTs, specifically tailored for UAV tracking. This marks the first instance where ViTs are employed to acquire view-invariant feature representations in the context of UAV tracking.

Multi-teacher knowledge distillation (MD) leverages knowledge from multiple off-the-shelf pre-trained teacher models to guide the student, enhancing its generalization by inheriting diverse knowledge from various teachers (Gou et al., 2021; Wang and Yoon, 2021). It has primarily been studied in the contexts of image classification (You et al., 2017; Yuan et al., 2021; Lan et al., 2024; Wen et al., 2024), action recognition (Wu et al., 2019) and visual retrieval (Ma et al., 2024). For instance, Wen et al. (2024) proposed the multi-teacher distillation method for class incremental learning, using weight permutation, feature perturbation, and diversity regularization to ensure diverse mechanisms in teachers. To better leverage the complementary knowledge of each modality, Lan et al. (2024) propose a multi-teacher multi-modal knowledge distillation framework to guide the training of the multi-modal fusion network and further improve the multi-modal feature fusion process. Whiten-MTD (Ma et al., 2024) aligns the outputs of teacher models by whitening their similarity distributions and identifies the most effective fusion strategy for their multi-teacher distillation framework through empirical analysis. To the best of our knowledge, existing knowledge distillation methods for visual tracking rely on a single-teacher framework (Li et al., 2022a; Sun et al., 2023), limiting their ability to fully utilize the diverse knowledge from multiple teachers, potentially leading to suboptimal tracking performance. In this work, we propose a simple yet effective MI maximization-based multi-teacher knowledge distillation framework, integrated into AVTrack, to develop a more efficient UAV tracker. For the teacher models, we utilize all three off-the-shelf trackers from AVTrack (i.e., AVTrack-DeiT, AVTrack-ViT, and AVTrack-EVA), offering diverse and high-quality teachers without requiring additional operations, such as those in (Wen et al., 2024), to implement varied mechanisms in the teachers. To ensure the student model captures the most relevant information from the teacher models’ representations, we propose maximizing the MI between the averaged softened feature representation of the multi-teacher models and the student model’s softened feature representation. Averaging the predictions of all teacher models is a common and effective method, as it reduces biases and provides a more objective output than any single teacher’s prediction (Buciluǎ et al., 2006; Ba and Caruana, 2014; Ilichev et al., 2021; Song and Chai, 2018).

The proposed AVTrack introduces a novel single-stream tracking framework, featuring an adaptive ViT-based backbone and a prediction head. To improve the efficiency of the ViT, each transformer block in the ViT-based backbone, except for the first $n_{f}$ blocks, incorporates an activation module (AM). This module is trained to adaptively determine whether to activate the associated transformer block. The framework takes a pair of images as input, comprising a template denoted as $Z\in\mathbb{R}^{3\times H_{z}\times W_{z}}$ and a search image denoted as $X\in\mathbb{R}^{3\times H_{x}\times W_{x}}$. These images are split into patches of size $P\times P$, and the number of patches for $Z$ and $X$ are $P_{z}=H_{z}\times W_{z}/P^{2}$ and $P_{x}=H_{x}\times W_{x}/P^{2}$, respectively. The features extracted from the backbone are fed into the prediction head to generate tracking results. To obtain view-invariant representation with ViTs, we maximize the mutual information (MI) between the feature representations of two different views of the target, i.e., the template image and the target patch in the search image. During the training phase, since the ground truth localization of the target in the search image is known, we can obtain the feature representation of the subsequent view from the representation of the search image using interpolation techniques.

Building on the proposed AVTrack, we introduce a simple yet effective MI maximization-based multi-teacher knowledge distillation (MD) framework to develop a more efficient UAV tracker, called AVTrack-MD. During the training of the AVTrack-MD model (i.e., the student models), the AVTrack’s (i.e., the teacher models) weights remain fixed while both the teacher and student models receive inputs $Z$ and $X$. Let $\mathfrak{B}_{T}$ and $\mathfrak{B}_{S}$ represent the backbones of the teacher and student, respectively. In our implementation, $\mathfrak{B}_{S}$ shares the same structure as the $\mathfrak{B}_{T}$ but uses a smaller ViT layer. Feature-based knowledge distillation transfers knowledge from the teacher models’ backbone features to the student model by maximizing the MI between the averaged softened features of the multi-teacher models and the student model’s softened features.

The details of the these design principles will be elaborated in the subsequent subsections.

The Activation Module selectively activates transformer blocks based on input, enabling adaptive adjustments to the ViT’s architecture and ensuring that only the essential blocks are activated for effective tracking. To enhance efficiency, the AM processes a subset of tokens representing both the target template and the search image. Its output determines whether the current Transformer block is activated; if not, the tokens from the preceding block are bypassed. Specifically, let’s consider the $i$-th layer ($i>n_{f}$). We denote the total number of tokens by $\mathcal{K}$, the embedding dimension of each token by $d$, and all the tokens output by the $(i-1)$-th layer by $\mathbf{t}_{1:\mathcal{K}}^{i-1}(Z,X)\in\mathbb{R}^{\mathcal{K}\times d}$. The slice of all tokens generated by the $(i-1)$-th Transformer block is expressed as $\mathbf{e}_{1}^{\textup{T}}\mathbf{t}_{1:\mathcal{K}}^{i-1}(Z,X):=\textbf{r}^{i-1}\in\mathbb{R}^{\mathcal{K}}$, where $\mathbf{e}_{1}^{\textup{T}}=[1,0,...,0]\in\mathbb{R}^{\mathcal{K}}$ is a standard unit vector in $\mathbb{R}^{\mathcal{K}}$, the linear layer is denoted by $\mathfrak{L}^{i}$. Formally, the Activation Module (AM) at layer $i$ is expressed as:

$$p^{i}=\sigma(\mathfrak{L}^{i}(\textbf{r}^{i-1})),$$

(1)

where $p^{i}\in[0,1]$ represents the activation probability of the $i$-th Transformer block, $\sigma(x)=1/(1+e^{-x})$ indicates the sigmoid function. The sigmoid function is versatile and finds applications in various machine learning tasks where non-linear transformations or probabilistic outputs are required. In our work, the activation mechanism in each Transformer block is determined by the sigmoid function, which helps introduce non-linearity into the activation module’s decision-making process. Specifically, the sigmoid function maps the output value of the linear layer to a range between 0 and 1, which represents the activation probability of the associate Transformer block. If $p^{i}>\beta$, where $\beta\in(0.5,1)$ is the activation probability threshold, the transformer block at layer $i$ will be activated; otherwise, it is deactivated and the output tokens from the $(i-1)$-th layer will be fed into the $(i+1)$-th block directly. Let $\mathcal{N}$ denote the total number of transformer blocks in the given ViT. Theoretically, deactivating all $\mathcal{N}$ blocks simultaneously would result in no correlation being computed between the template and search image. To avoid such unfavorable conditions, the first $n_{f}$ layers are mandated to remain activated. This strategy helps alleviate computational burdens associated with AM, as these initial layers are typically essential, providing foundational information on which high-level and more abstract features and representations can be built. Another extreme case occurs when all Transformer blocks are activated for any input, allowing the model to more efficiently minimize classification and regression losses, as larger models have greater fitting capacity. To address this, we introduce a block sparsity loss, $\mathcal{L}_{spar}$, which penalizes a higher mean probability across all adaptive layers, encouraging the deactivation of many blocks on average to improve efficiency. The block sparsity loss is defined as follows:

$$\begin{split}\mathcal{L}_{spar}=|\frac{1}{\mathcal{N}-n_{f}}\sum^{\mathcal{N}}_{i=n_{f}+1}p_{i}-\zeta|,\end{split}$$

(2)

where $\zeta\in[0,1]$ is a constant used with $\beta$ to control block sparsity. Generally, for a given $\beta$, a smaller $\zeta$ leads to a sparser model. When $\zeta=0$, $p_{i}$ can be viewed as the weight of block $i$, and the sparsity loss becomes proportional to the $l_{1}$ norm of the vector of these weights, which is a convex-relaxed sparsity regularization commonly used in statistical learning theory. $\zeta$ serves as a hyperparameter for finer adjustments.

We begin by introducing the concept of mutual information (MI) and establishing the relevant notations. Given two random variables: $\textbf{a}\in\mathcal{A}$ and $\textbf{b}\in\mathcal{B}$. The MI between $\mathbf{a}$ and $\mathbf{b}$, denoted as $\textup{I}(\mathbf{a},\mathbf{b})$, is expressed as follows:

$$\mathbb{E}_{p(\textbf{a},\textbf{b})}\left[log\frac{p(\textbf{a},\textbf{b})}{p(\textbf{a})p(\textbf{b})}\right]=\textup{D}_{KL}(p(\textbf{a},\textbf{b})||p(\textbf{a})p(\textbf{b})),$$

(3)

where $p(\textbf{a},\textbf{b})$ represents the joint probability distribution, while $p(\textbf{a})$ and $p(\textbf{b})$ are the marginal distributions. The symbol $\textup{D}_{KL}$ denotes the Kullback–Leibler divergence (KLD) (MacKay, 2004). Estimating MI is not an easy task in real-world situations, as we usually only have access to the samples that are readily available, not the underlying distributions (Poole et al., 2019). As a result, most existing estimators typically rely on approximating the MI between variables using observed samples. In contrast to these approaches, we employ the Deep InfoMax MI estimator (R.D. and et al., 2019), which estimates MI based on Jensen-Shannon divergence (JSD). This strategy has proven to be effective, as knowing its precise value is less critical than maximizing MI in this context. The Jensen-Shannon MI estimator, represented by $\hat{\textup{I}}_{\mathbf{\mathbf{\Theta}}}^{(JSD)}(\textbf{a},\textbf{b})$, is defined by:

$$\mathbb{E}_{p(\textbf{a},\textbf{b})}[-\alpha(-T_{\mathbf{\mathbf{\Theta}}}(\textbf{a},\textbf{b}))]-\mathbb{E}_{p(\textbf{a})p(\textbf{b})}[\alpha(T_{\mathbf{\mathbf{\Theta}}}(\textbf{a},\textbf{b}))],$$

(4)

where $T_{\mathbf{\mathbf{\mathbf{\Theta}}}}:\mathit{X}\times\mathit{Y}\rightarrow\mathbb{R}$ is a neural network parameterized by $\mathbf{\mathbf{\mathbf{\Theta}}}$, and $\alpha(z)=log(1+e^{z})$ represents the softplus function.

In our work, the proposed approach involves learning view-invariant feature representations by maximizing MI using the aforementioned Jensen-Shannon MI estimator between the feature representations of two different views of a specific target. Let $\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_{Z}\cup\mathcal{K}_{X}}^{\infty}(Z,X)$, $\mathcal{K}_{Z}\cup\mathcal{K}_{X}=[1,\mathcal{K}]$, denote the final output tokens of the ViT, where $\mathbf{t}_{\mathcal{K}_{Z}}^{\infty}$ and $\mathbf{t}_{\mathcal{K}_{X}}^{\infty}$ represent the tokens corresponding to the template and the search image, respectively. Given the ground truth localization of the target, denoted as $Z^{\prime}$ in the search image, we obtain the tokens corresponding to $Z^{\prime}$ through linear interpolation, represented by $\mathbf{t}_{\mathcal{K}_{Z^{\prime}}}^{\infty}(Z,X)\subset\mathbf{t}_{\mathcal{K}_{X}}^{\infty}(Z,X)$. The specially designed loss $\mathcal{L}_{vir}$ for learning view-invariant feature representations is formulated as follow,

$$\mathcal{L}_{vir}=-\hat{\textup{I}}^{(JSD)}_{\mathbf{\mathbf{\mathbf{\Theta}}}}(\mathbf{t}_{\mathcal{K}_{Z^{\prime}}}^{\infty}(Z,X),\mathbf{t}_{\mathcal{K}_{Z}}^{\infty}(Z,X)).$$

(5)

During the inference phase, only the sequence $[Z,X]$ is input into the VIR, and the process for learning view-invariant representations is not involved. Consequently, our method imposes no additional computational cost during the inference phase. Additionally, the proposed view-invariant representation learning is ViT-agnostic, making it easily adaptable to other tracking frameworks.

We developed AVTrack-MD with the aim of achieving performance comparable to or superior to the AVTrack while reducing computational resources and memory usage, making it a more efficient UAV tracker. To achieve this, we introduce a novel MI maximization-based multi-teacher knowledge distillation framework into the AVTrack, as illustrated in Fig. 3. Our method focuses on feature-level knowledge distillation, meaning that the student model learns from an ensemble of teacher models by concentrating on the features the teacher models have learned.

In the context of multi-teacher knowledge distillation, the selection of teachers-student architectures plays a crucial role in the knowledge distillation process, as it determines how the knowledge from the various teacher models is transferred to the student model. For the teacher models, we use all three off-the-shelf trackers from AVTrack (i.e., AVTrack-DeiT, AVTrack-ViT, and AVTrack-EVA), providing diverse and high-quality teachers without the need for additional operations to implement varied mechanisms in teachers. For the selection of student models, we opt for a self-similar architecture, where the student shares the same structure as the teacher but uses a smaller ViT backbone (i.e., with half ViT blocks). This selection offers two main advantages: 1) it eliminates the need for complex design processes, enabling straightforward implementation and training, and enhancing the interpretability of the model’s behavior; 2) it promotes modularity and scalability, reducing the difficulty of expanding or modifying the model as needed. Averaging the predictions of all teacher models is a common choice (Buciluǎ et al., 2006; Ba and Caruana, 2014; Ilichev et al., 2021; Song and Chai, 2018), as it is a straightforward and effective method. In our implementation, we average the $K$ feature representations $\mathcal{F}_{k}$ produced by the teacher models into an aggregated feature representation $\mathcal{F}=\frac{1}{K}\sum_{k=1}^{K}\mathcal{F}_{k}.$ In practice, to provide richer information during training, the model outputs are softened using a temperature $\tau$, since the original outputs are typically represented in a one-hot encoding format (You et al., 2017). Let $\mathcal{F}^{S}$ represent the feature representation of the student. The softened outputs of the student and teacher are expressed as follows:

$\displaystyle\mathcal{F}_{\tau}=softmax(\frac{\mathcal{F}}{\tau}),\mathcal{F}^{S}_{\tau}=softmax(\frac{\mathcal{F}^{S}}{\tau}),$

(6)

where $\tau$ is a constant that we set to 2. Encouraging the student network to replicate the feature representations of the teacher’s final layer network using mean squared error (MSE) loss is straightforward (Ba and Caruana, 2014; Kim et al., 2021; Hamidi et al., 2024). However, since MSE is sensitive to noise and outliers (Hinton et al., 2015), we maximize the MI between the aggregated softened feature of the teacher models and the student model’s softened feature, enhancing generalization and performance, especially in noisy conditions. Given the teachers-student architecture, we apply the Jensen-Shannon MI estimator to achieve MI maximization for multi-teacher knowledge distillation, which establishes the objective function for our MI maximization-based multi-teacher knowledge distillation approach, as expressed in the following equation:

$\displaystyle\mathcal{L}_{MD}=-\hat{\textup{I}}^{(JSD)}_{\mathbf{\mathbf{\mathbf{\Theta}}}}(\mathcal{F}_{\tau},\mathcal{F}^{S}_{\tau}),$

(7)

In distillation training, the student model is trained using a weighted sum of $\mathcal{L}_{MD}$ and the total loss function used for training the teacher model. $\tau$

Following the corner detection head in (Cui et al., 2022; Ye et al., 2022), we utilize a prediction head $\mathcal{H}$ consisting of several Conv-BN-ReLU layers to directly estimate the target’s bounding box. The output tokens associated with the search image are first reinterpreted into a 2D spatial feature map, which is then input to the prediction head. The head generates three outputs: a target classification score $\mathbf{p}\in[0,1]^{H_{x}/P\times W_{x}/P}$, a local offset $\mathbf{o}\in[0,1]^{2\times H_{x}/P\times W_{x}/P}$, and a normalized bounding box size $\mathbf{s}\in[0,1]^{2\times H_{x}/P\times W_{x}/P}$. The highest classification score is used to estimate the coarse target position, i.e., $(x_{c},y_{c})=\textup{argmax}_{(x,y)}\mathbf{p}(x,y)$, based on which the final target bounding box is determined by:

$$\{(x_{t},y_{t});(w,h)\}=\{(x_{c},y_{c})+\mathbf{o}(x_{c},y_{c});\mathbf{s}(x_{c},y_{c})\}.$$

(8)

For the tracking task, we employ the weighted focal loss (Law and Deng, 2018) for classification and a combination of $L_{1}$ loss and GIoU loss (Rezatofighi et al., 2019) for bounding box regression. The overall loss function is formulated as:

$$\mathcal{L}_{overall}=\mathcal{L}_{cls}+\lambda_{iou}\mathcal{L}_{iou}+\lambda_{L_{1}}\mathcal{L}_{L1}+\gamma\mathcal{L}_{spar}+\kappa\mathcal{L}_{vir}$$

(9)

where the constants $\lambda_{iou}=2$ and $\lambda_{L_{1}}=5$ are the same as in (Ye et al., 2022), $\gamma$ and $\kappa$ are set to $50$ and $0.0001$, respectively. Our framework is trained end-to-end with the overall loss $\mathcal{L}_{overall}$ after the pretrained weights of the ViT for image classification is loaded. After this training, we apply the proposed MI maximization-based multi-teacher knowledge distillation framework to obtain a student model that better balances performance and efficiency. Specifically, during the distillation phase, the distillation loss $\mathcal{L}_{MD}$ is added to the overall loss from the previous training stage, yielding the total loss $\mathcal{L}_{overall}^{*}$ for end-to-end distillation training as follows:

$$\mathcal{L}_{overall}^{*}=\mathcal{L}_{overall}+\eta\mathcal{L}_{MD}$$

(10)

where the weight coefficients in $\mathcal{L}_{overall}$ remain the same as those used during teacher model training, and the weight $\eta$ of $\mathcal{L}_{MD}$ is set to $0.1\times 10^{-3}$.

Model. The AVTrack utilizes three different lightweight ViTs as backbones to build three trackers for evaluation: AVTrack-ViT, AVTrack-DeiT, and AVTrack-EVA. Building on it, AVTrack-MD introduces the novel MI maximization-based multi-teacher knowledge distillation (MD) framework to develop three corresponding improved trackers: AVTrack-MD-ViT, AVTrack-MD-DeiT, and AVTrack-MD-EVA. The head and input sizes for AVTrack and AVTrack-MD are configured identically. The head consists of a stack of four Conv-BN-ReLU layers, with input sizes set to $256^{2}$ for the search image and $128^{2}$ for the template image.

Training. For training, we employ a combination of training sets from four different datasets: GOT-10k (Huang et al., 2021), LaSOT (Fan et al., 2019), COCO (Lin et al., 2014), and TrackingNet (Muller et al., 2018). It is noteworthy that all the trackers utilize the same training pipeline to ensure consistency and comparability. The batch size is set to 32. We use the AdamW optimizer to train the model, with a weight decay of $10^{-4}$ and an initial learning rate of $4\times 10^{-5}$. The total number of training epochs is fixed at 300, with 60,000 image pairs processed per epoch. The learning rate decreases by a factor of 10 after 240 epochs. During the multi-teacher distillation phase, we use all three pre-trained AVTrack-* models as teacher models. The parameters of the teacher models are frozen to guide the training of the student model with the proposed knowledge distillation approach, which follows the same training pipeline as that of the teacher models.

Inference. In line with common practices, Hanning window penalties are applied during inference to integrate positional prior on tracking (Zhang et al., 2020). Specifically, a Hanning window of the same size is multiplied by the classification map, and the location with the highest score is selected as the predicted result.

1) Overall Performance: We compare our methods with 22 SOTA lightweight trackers, inclunding KCF (Henriques et al., 2015), fDSST (Danelljan et al., 2017), ECO_HC (Danelljan et al., 2017), MCCT_H (Wang et al., 2018), STRCF (Li et al., 2018), ARCF (Huang et al., 2019), AutoTrack (Li et al., 2020), RACF (Li et al., 2022b), HiFT (Cao et al., 2021), SiamAPN (Fu et al., 2021), P-SiamFC++ (Wang et al., 2022), TCTrack (Cao et al., 2022), TCTrack++ (Cao et al., 2023), SGDViT (Yao et al., 2023), ABDNet (Zuo et al., 2023), DRCI (Zeng et al., 2023), PRL-Track (Fu et al., 2024), Aba-ViTrack (Li et al., 2023), LiteTrack (Wei et al., 2024), SMAT (Gopal and Amer, 2024), LightFC (Li et al., 2024c), and SuperSBT (Xie et al., 2024a), across five renowned UAV tracking datasets: DTB70 (Li and Yeung, 2017), UAVDT (Du et al., 2018), VisDrone2018 (Zhu et al., 2018), UAV123 (Mueller et al., 2016), and UAV123@10fps (Mueller et al., 2016). The evaluation results are presented in Table 1. From table, our AVTrack-DeiT outperforms all SOTA trackers except Aba-ViTrack (Li et al., 2023) across these five benchmarks in terms of average (Avg.) precision (Prec.) and success rate (Succ.). Specifically, RACF (Li et al., 2022b) exhibits the best performance among DCF-based trackers, with Avg. Prec. and Succ. of 74.6% and 51.2%, respectively. Among CNN-based trackers, DRCI (Zeng et al., 2023) stands out with the highest Avg. Prec. and Succ. of 79.8% and 59.1%. However, even the best methods among DCF- and CNN-based trackers fall short of the worst method among ViT-based trackers. All ViT-based trackers achieve Avg. Prec. and Succ. exceeding 80.0% and 62.0%, respectively, clearly surpassing the CNN-based methods and significantly outperforming the DCF-based ones. Regarding GPU speed, the top three trackers are our proposed methods: AVTrack-MD-EVA, AVTrack-MD-DeiT, and AVTrack-MD-ViT, achieving tracking speeds of 334.4 FPS, 310.6 FPS, and 303.1 FPS, respectively. In terms of CPU speed, all our methods deliver real-time performance on a single CPU¹¹1It is important to note that the real-time performance discussed in this work is applicable only to platforms that are similar to or more advanced than ours.. Obviously, DCF-based trackers are the most efficient UAV trackers, as all methods achieving speeds above 80 FPS belong to this category. However, these fastest DCF-based trackers are unable to achieve performance comparable to our methods. For example, the fastest KCF (Henriques et al., 2015) tracker attains only 31.6% in average success rate, which is about half that of our methods. Additionally, DCF-based methods often incur substantial costs to enhance tracking precision. Specifically, RACF exhibits the best performance among DCF-based trackers; however, it runs at only 35.6 FPS, which is significantly lower than the slowest CPU speed of our trackers. Although CNN-based trackers can achieve tracking speeds comparable to our ViT-based trackers, the latter significantly outperform the former in overall performance. Although Aba-ViTrack achieves the highest average precision of 85.3% and the highest average success rate of 64.7%., our AVTrack-DeiT secures second place with only slight gaps of 1.2% and 0.3%, respectively. Furthermore, our AVTrack-DeiT demonstrates higher speeds than Aba-ViTrack. Notably, AVTrack-DeiT runs up to 41%/18% faster than Aba-ViTrack in terms of GPU/CPU speed. When the proposed MI-based multi-teacher knowledge distillation framework (MD) is applied to AVTrack-*, the resulting student model, AVTrack-MD-*, demonstrates a smaller performance drop, with some even achieving performance improvements, all while delivering a significant speed boost. Specifically, AVTrack-MD-* models show varying trade-offs: AVTrack-MD-ViT achieves 0.5% and 0.9% gains in Avg. Prec. and Succ., respectively, with 21.1% GPU and 8.7% CPU speed improvements; AVTrack-MD-EVA gains 0.4% in precision and a minor 0.1% drop in success rate, while also improving GPU and CPU speeds by 17.9% and 6.8%; AVTrack-MD-DeiT has a slight performance drop of 0.5% in precision and 0.3% in success rate but still benefits from significant 20.9% GPU and 8.9% CPU speed gains. These results highlight the advantages of our method and justify its SOTA performance for UAV tracking.

2) Performance on WebUAV-3M (Zhang et al., 2022a): As illustrated in Table 2, we have conducted a comparison of our AVTrack-DeiT and AVTrack-MD-DeiT with ten lightweight SOTA trackers on WebUAV-3M. Our AVTrack-DeiT surpasses all other lightweight trackers in success rate, achieving a 1.1% improvement over the second-place Aba-ViTrack while maintaining a comparable precision with only a 0.4% difference. Notably, our AVTrack-MD-DeiT experiences less than a 1.0% drop in both precision and success rate, while improving AGX speed by 9.2%. These results further underscore the effectiveness of our approach.

3) Efficiency Comparison: To further demonstrate that our tracker achieves a better trade-off between accuracy and efficiency, we compare our methods against ten SOTA lightweight trackers, in terms of floating point operations (FLOPs), the number of parameters (Params.), and the speed (AGX.FPS) on the Nvidia Jetson AGX Xavier edge device. The results are shown in Table 2. Notably, since our AVTrack-DeiT tracker features adaptive architectures, the FLOPs and parameters range from minimum to maximum values. As observed, both the minimum FLOPS and Params of our AVTrack-DeiT are notably lower than those of all the SOTA trackers, and even its maximum values are lower than those of most SOTA trackers. Furthermore, our AVTrack-MD-DeiT achieves the lowest FLOPs (1.5G) and parameters (5.3M), except for AVTrack-DeiT, while also delivering the fastest speed at 46.1 AGX.FPS. This comparison in terms of computational complexity also underscores the efficiency of our methods.

4) Attribute-Based Evaluation: We also performed attribute-based evaluations to assess the robustness of our trackers (i.e., AVTrack-DeiT and AVTrack-MD-DeiT) in challenging scenarios involving viewpoint changes. This includes comparing our trackers with 16 SOTA lightweight trackers under challenges such as ‘Camera Motion’ on the DTB70 and VisDrone2018 and ‘Viewpoint Change’ on the UAV123, as shown in Fig. 4. Notably, the DTB70 and VisDrone2018 datasets do not have a dedicated subset for scenarios involving viewpoint changes. As an alternative, we used the ‘Camera Motion’ subset to evaluate performance in such scenarios, as it is a key factor contributing to viewpoint changes in visual tracking. For instance, when a camera or the object itself moves rapidly or changes direction suddenly, it can result in varying views of the target in the captured images. As observed, our tracker ranks first in ‘Camera Motion’ on VisDrone2018, and ‘Viewpoint Change’ on UAV123, exceeding the second-best tracker by 1.0%/0.5% and 1.0%/0.6% in precision/success rate, respectively, with only slight gaps of 0.1% and 0.6% to the top tracker in the ‘Camera Motion’ on DTB70. Therefore, this attribute-based evaluation validates, both directly and indirectly, the superiority and effectiveness of the proposed method in addressing the challenges associated with viewpoint change.

4) Qualitative evaluation: To intuitively showcase the tracking results in UAV tracking scenes, qualitative tracking results of our trackers (i.e., AVTrack-MD-DeiT and AVTrack-DeiT) alongside six SOTA UAV trackers are visualized in Fig. 5. Four video sequences, selected from different benchmarks and scenarios, are presented for demonstration: Surfing12 from DTB70, S1607 from UAVDT, uav0000180_00050_s from VisDrone2018, person10 from UAV123@10fps, and truck1 from UAV123. The upper right corner features a selectively zoomed and cropped view of the tracked objects within the corresponding frames for better visualization. As observed, compared to the other SOTA UAV trackers, our tracker tracks the target objects more accurately in all challenging scenes, including occlusion (i.e., S1607 and uav0000180_00050_s), low resolution (i.e., Surfing12), scale variations (i.e., in all sequences), and viewpoint change (i.e., in all sequences). In these cases, our method significantly outperforms others and offers a more visually appealing result, highlighting the effectiveness of the proposed approaches for UAV tracking.

To further validate the superiority of our trackers, we compare AVTrack-DeiT and AVTrack-MD-DeiT with 16 deep SOTA trackers, including DiMP (Bhat et al., 2019), PrDiMP (Danelljan et al., 2020), TrSiam (Wang et al., 2021), TransT (Chen et al., 2021a), AutoMatch (Zhang et al., 2021), SparseTT (Fu et al., 2022), CSWinTT (Song et al., 2022), SimTrack (Chen et al., 2022), OSTrack (Ye et al., 2022), ZoomTrack (Kou et al., 2023), SeqTrack (Chen et al., 2023), MAT (Zhao et al., 2023), ROMTrack (Cai et al., 2023), DCPT (Zhu et al., 2024), HIPTrack (Cai et al., 2024), and EVPTrack (Shi et al., 2024) ,across five UAV tracking datasets. The evaluation results are shown in Table 3, which shows precision (Prec.), success rate (Succ.), their average (Avg.), and average GPU speed (Avg.FPS). As shown, our AVTrack-MD-DeiT and AVTrack-DeiT excel by achieving the fastest and second-fastest speeds while maintaining high performance comparable to the SOTA deep trackers. On average, although AVTrack-MD-DeiT and AVTrack-DeiT exhibit slight gaps in average precision and success rate compared to HIPTrack (Cai et al., 2024), EVPTrack (Shi et al., 2024), and ROMTrack (Cai et al., 2023), they are noticeably slower than our methods in terms of GPU speed. Specifically, our AVTrack-MD-DeiT is 9.7, 11.9, and 5.9 times faster than HIPTrack, EVPTrack, and ROMTrack, respectively. These results indicate that our method delivers both high precision and speed, validating its suitability for real-time UAV tracking that prioritizes efficiency as well as accuracy.

Impact of the Number of Teachers: More teachers may provide more diverse knowledge. Table 4 summarizes the performance of the teacher models and distillation models with various teacher combinations, while Table 5 presents their model complexity and average GPU speed. As shown, single-teacher distillation can significantly reduce model complexity during inference and achieve a notable speedup of 17%. However, it leads to a considerable decline in performance compared to the corresponding teacher model, with decreases of over 2.0% in average precision and success rate. In contrast, multi-teacher distillation methods, including double-teacher and triple-teacher distillation, not only reduce model complexity and provide a significant speedup but also result in a smaller performance decrease, with all drops remaining below 1.6% in average precision and success rate, and some even surpassing the performance of the teacher models. In the triple-teacher distillation, the student models (i.e., AVTrack-MD-ViT and AVTrack-MD-EVA) exhibit superior performance when compared to their corresponding teacher models with the same backbone. Specifically, AVTrack-MD-ViT achieves gains of 0.5% in average precision and 0.9% in success rate, while AVTrack-MD-EVA shows a 0.4% improvement in average precision, with only a 0.1% drop in success rate. Although AVTrack-MD-DeiT does not outperform AVTrack-DeiT, it experiences only a slight performance drop, with both average precision and success rate decreasing by less than 0.5%. Notably, in practice, the model complexity of AVTrack is higher than that of AVTrack-MD, as the activated blocks in AVTrack typically exceed half of the total blocks in the backbone. As a result, AVTrack is slower than AVTrack-MD when both use the same ViT backbone. These experimental results provide evidence for the effectiveness of our proposed multi-teacher knowledge distillation based on MI Maximization, with the performance gain attributed to the complementarity of multiple teacher models.

Impact of varying numbers of ViT blocks in student models: To closely investigate the impact of varying numbers of ViT blocks on performance in student models, we train the AVTrack-MD-DeiT student model using a range of block counts from 4 to 8. The evaluation results are shown in Table 6. As observed, the number of ViT blocks in the student model directly affects both tracking performance and speed. On average, increasing the number of ViT blocks leads to an upward trend in accuracy and model complexity but a decline in tracking speed. As the number of blocks increases from 4 to 6, each additional block results in a greater than 1.0% improvement in both average precision and average success rate, an increase of more than 0.4M in parameters, and a rise of over 0.1G in FLOPs, accompanied by an approximately 10% decrease in speed. When the number of blocks exceeds 6, further increases do not lead to significant performance gains, while model complexity continues to rise substantially and speed continues to decrease significantly. Considering the balance between performance and speed, we have chosen to set the default number of ViT blocks in the student model to 6 in our implementation, which is half that of the teacher models.

Impact of Weighting the MI maximization-based multi-teacher knowledge distillation: To obtain the optimal weight $\eta$ for the proposed MI maximization-based multi-teacher knowledge distillation, we trained AVTrack-MD-DeiT using varied values of $\eta$ ranging from $0.1\times 10^{-1}$ to $0.1\times 10^{-6}$ with with a scale factor of 0.1. Evaluation results are shown in Table 7. As shown, our tracker achieves best performance when the loss weight ($\eta$) is set to $0.1\times 10^{-3}$. Additionally, we have observed that the second- and third-best performances across these datasets are scattered both above and below the value of $0.1\times 10^{-3}$, without any apparent patterns. This variation may be attributed to the inherent differences between these datasets. Setting a value of $0.1\times 10^{-1}$ for the loss weight ($\eta$) results in an average maximal difference of Avg. Prec. of 1.8% and a maximal difference of Avg. Succ. of 1.3%. These significant differences clearly demonstrate that the choice of weight a has a considerable impact on the tracking performance. To be more specific, when the proposed loss is appropriately weighted, it can enhance the tracking performance. However, if not properly weighted, it may have detrimental effects on the training of the tracking task.

Impact of Multi-Teacher Knowledge Distillation Based on MI Maximization: To demonstrate the superiority of the proposed multi-teacher knowledge distillation based on the MI maximization, we train the model using MSE loss and the proposed MI-based loss for multi-teacher knowledge distillation, respectively. The evaluation results across five datasets are presented in Table 8. As observed, using the MSE loss results in greater performance loss compared to our method. On average, employing the proposed $L_{MI}$ results in a notable enhancement over the MSE loss, with improvements of 1.9% in precision and 1.7% in success rate, respectively. Additionally, compared to the teacher model, our method incurs only a slight decrease of 0.5% in average precision and 0.3% in average success rate, while achieving a notable speedup of 20.9%. These comparisons show that the benefit of our multi-teacher knowledge distillation based on the MI maximization is that it can provide a more thorough measure of the relationship between features through MI-based loss, which enables the student model to learn a more effective representation from the teacher model. On the other hand, our MI-based loss is less sensitive to noise and outliers compared to MSE, making it particularly effective in noisy environments.

The proposed AVTrack enhances the robustness of Vision Transformers (ViTs) to viewpoint changes by enforcing invariance in the target’s feature representation. Building upon this, we present an improved tracker, dubbed AVTrack-MD, which integrates a novel multi-teacher knowledge distillation (MD) framework based on MI maximization. The extensive experimental results above show that the proposed MD framework not only greatly improves model efficiency but also enables the AVTrack-MD (i.e., the student models) to achieve performance comparable to, or even exceeding, that of AVTrack (i.e., the teacher models). To intuitively demonstrate the effectiveness of the proposed components, we visualize the feature maps of several examples in Fig. 6. These samples, generated by AVTrack-DeiT with and without the proposed distillation framework, feature targets undergoing viewpoint changes.

Visualization of feature maps: In Fig. 6, the first row presents the original target images from UAV123@10fps (Mueller et al., 2016), while the corresponding feature maps generated by AVTrack-DeiT*, AVTrack-DeiT, and AVTrack-MD-DeiT are displayed in the second, third, and fourth rows, respectively. Note that AVTrack-DeiT* refers to AVTrack-DeiT without the integration of the proposed VIR and AM components. As observed, the feature maps generated by AVTrack-DeiT and AVTrack-MD-DeiT demonstrate greater consistency with changes in viewpoint, while the feature maps from AVTrack-DeiT* exhibit more significant variations, particularly at different viewing angles. This suggests that the VIR component enhances the tracker’s ability to focus on targets experiencing changes in viewpoint, thereby improving overall tracking performance. Furthermore, the proposed MD allows AVTrack-MD-DeiT to inherit this capability from AVTrack-DeiT, enabling it to maintain similar robustness to the teacher models while also enhancing model efficiency. These qualitative results offer visual evidence of our method’s effectiveness in learning view-invariant feature representations using Vision Transformers (ViTs).

To showcase the applicability of AVTrack-MD-DeiT for UAV tracking, we performed two real-world tests on the NVIDIA Jetson AGX Xavier 32GB. As shown in Fig. 7, the proposed tracker performs well in real-time despite challenges like camera motion, occlusion, and viewpoint changes, maintaining a Center Location Error (CLE) of under 20 pixels across all test frames, highlighting its high tracking precision. Furthermore, AVTrack-MD-DeiT consistently achieves an average real-time speed of 46 FPS in tests, demonstrating its robustness and efficiency, making it ideal for UAV applications requiring high performance and speed.