Optimized Information Flow for Transformer Tracking

Until recently, two-stream tracking pipeline-based approaches have shown state-of-the-art performances in VOT with high computational efficiency. In this pipeline of trackers, initial target template and search region features are independently extracted using a neural network model, and then the target is located in the search region by finding the similarity of target features in the search region features.

Transformers were initially introduced into the two-stream tracking pipeline as a feature fusion module (Chen et al., 2021; Gao et al., 2022; Song et al., 2022; Yan et al., 2021; Yu et al., 2021), replacing the correlation modules of CNN-based Siamese trackers. In these CNN-Transformer-based two-stream trackers, a CNN backbone is employed to extract the features of both target template and search region. The extracted CNN features are then flattened and concatenated before being fed to a Transformer to capture feature dependencies using the attention mechanism. Finally, the enhanced search region features are input to a prediction head to locate the target. While some early CNN-Transformer trackers adopted Transformer architectures, such as DETR (Carion et al., 2020) in STARK (Yan et al., 2021) tracker, from the object detection task with minor modifications, recent approaches have recognized issues specific to Transformer-based tracking and subsequently adjusted their architectures. CNN-Transformer based trackers have demonstrated superior performance compared to CNN-based trackers by employing a trainable Transformer instead of the linear cross-correlation operation.

Some of the recent two-stream Transformer trackers (Lin et al., 2022; Fu et al., 2022) replaced the CNN backbone with two individual but identical Transformers to extract the features of both the target template and search region. Since these trackers utilized the global feature representations of the Transformers in both feature extraction and future fusion they outperformed the CNN-Transformer based trackers in terms of accuracy.

Although the two-stream Transformer trackers showed better performance than the CNN-based trackers due to the learnable feature fusion modules and the global feature learning capability of the Transformers, they extracted the target template and search region features individually. Therefore, the search region features are extracted without knowledge of the target cues, and as a result, the importance of target-specific features in the search region fails to be considered. To address this limitation, recent Transformer trackers enable information flow between target template and search region tokens from the early encoder layers, and these trackers are known as one-stream Transformer trackers.

One-stream Transformer trackers (Chen et al., 2022; Cui et al., 2022; Ye et al., 2022; Lan et al., 2023; Xie et al., 2023; Gao et al., 2023; Wu et al., 2023; Zhao et al., 2023b; Wei et al., 2023; Chen et al., 2023) combine the feature extraction and future fusion processes and enable the bidirectional information flow between target template and search region tokens. In one-stream Transformer trackers, a set of encoder layers is utilized with the self-attention mechanism to jointly extract the features of the target and template tokens. Finally , the features of the search region tokens from the last encoder layer are fed to a prediction head to locate the target. As these trackers enhance target-specific feature extraction, they improve the discriminative capability of the tracker, resulting in outstanding performances compared to both CNN-based trackers and two-stream Transformer based trackers.

In the last two years, a considerable number of one-stream Transformer trackers have been proposed, incorporating bidirectional information flow between target template and search region tokens. OSTrack (Ye et al., 2022) is well-known as one such tracker due to its outstanding performances in benchmark datasets and highly parallelized architecture. It utilized a Masked Auto Encoder (MAE) (He et al., 2022) based pre-trained model to initialize the encoder layers of the tracker, resulting in a remarkable performance boost. In addition, a candidate elimination module is included in OSTrack to remove background tokens from search regions, thereby increasing the computational efficiency of the tracker. Due to OSTrack’s outstanding tracking performance in many benchmark datasets, several follow-up approaches (Lan et al., 2023; Xie et al., 2023; Gao et al., 2023) have been proposed within the one-stream tracking pipeline. Although OSTrack and follow-up approaches showed better performance than the two-stream Transformer trackers, they allowed bidirectional information flow between all target and search region tokens without evaluating the influence of tokens on discriminative target objects from the surroundings. As a result, their tracking performances are considerably limited. Moreover, since OSTrack and some of the follow-up approaches fail to consider temporal information, their tracking performances are considerably poor in long tracking sequences, especially when the target undergoes severe appearance changes.

A few recent approaches (Cui et al., 2022; Lan et al., 2023; Xie et al., 2023; Chen et al., 2023) have enhanced the tracking performances of one-stream Transformer trackers by incorporating the temporal cues of the target. In these trackers, dynamic target tokens are extracted from an intermediate frame with high-confidence detection and then concatenated with the initial target template and search region tokens. Some of these trackers follow special mechanisms to select the dynamic target template, such as the MixFormer (Cui et al., 2022) tracker, which used a learnable score prediction module to select reliable dynamic templates based on predicted confidence scores. Although these trackers enhance the tracking performance by incorporating dynamic tokens, they allow bidirectional information flow between all initial target, dynamic target, and search region tokens without considering their discriminative capability. In addition, most of these trackers fail to consider dynamic background cues, which would allow the tracker to incorporate surrounding information of the target for more accurate localization in the search frame.

In the proposed work, instead of allowing free bidirectional information flow between all target and search region tokens as in previous one-stream Transformer approaches, unnecessary information flow between tokens is blocked to enhance the discriminative capability of the tracker. Additionally, surrounding information of the dynamic target is included in the proposed work to accurately locate the target and avoid distractor objects in the search frame.

All of the one-stream Transformer trackers freely allowed bidirectional information flow between target and search region tokens in encoder layers without any restrictions. However, recently, a few trackers have controlled the information flow between tokens for various purposes. The recently proposed SeqTrack (Chen et al., 2023) and ARTrack (Wei et al., 2023) approaches restrict the information flow between tokens in decoder layers to prevent tokens from attending to subsequent tokens. However, in the encoder layer of these trackers, attention features are extracted by freely allowing bidirectional information flow between target and search region tokens.

The GRM tracker (Gao et al., 2023) restricts the information flow between the template and search tokens by allowing only a subset of search region tokens, determined by the selection of a learnable adaptive token division module, to interact with template tokens. The GRM approach performs as a two-stream tracker in some encoder layers, as all search tokens are rejected by the division module for interaction with template tokens in those layers, and performs as a one-stream tracker in some other layers, as all search tokens are selected for interaction. While functioning as a two-stream tracker in certain encoder layers, the GRM tracker individually extracts attention features from the target and search region tokens, which may result in the failure to extract target cues from the search tokens in those layers, potentially leading to a decrease in tracking performance.

Recently proposed, the F-BDMTrack approach (Yang et al., 2023a) blocks the information flow between the foreground and background tokens of the target template using a fore-background agent learning module. Additionally, the information flow between the foreground and background tokens of the search region is also restricted using that module to enhance the discriminative capability of the tracker. After the fore-background module, a distribution-aware attention module is used to prevent the incorrect interaction between the foreground and background tokens by modeling foreground-background distribution similarities. In this tracker, ground-truth bounding boxes are used to identify the foreground and background tokens of the template, while a pseudo bounding box generation technique is employed to identify the foreground in the search region.

Although the GRM and F-BDMTrack approaches block information flow between different groups of tokens for various purposes, they employ a learnable module for token grouping from early encoder layers. Therefore, a search token may be identified as a potential target token in one layer and as a background token in another, due to variations in the knowledge level of the learnable module between early and deeper layers, that may lead to poor tracking performance. In contrast to previous approaches, the proposed OIFTrack approach is carefully designed with a simple and neat architecture. Additionally, by conducting the search token partitioning only in deeper encoder layers using the learned knowledge from the middle encoder layers and blocking the information flow accordingly, it achieves more accurate identification of target and non-target tokens from the search region in a simple and fast manner.

In this section, we explore the baseline tracking model used in one-stream Transformer trackers. All of the previous approaches (Ye et al., 2022; Chen et al., 2022; Cui et al., 2022) employed this baseline model to establish their tracking methodologies.

In the one-stream Transformer tracking paradigm, a tracker takes the initial target template $Z\in\mathbb{R}^{H_{Z}\times W_{Z}\times 3}$ and search region $X\in\mathbb{R}^{H_{X}\times W_{X}\times 3}$ as inputs, and then produces the location of the target in the search region as the output. As the first step, template and search region images are divided into non-overlapping patches, each of size $P\times P$. The total number of target template patches ($N_{Z}$) and search region patches ($N_{X}$) can be calculated as follows:

After partitioning the patches, a learnable linear projection layer is employed to generate template and search patch embeddings from their respective patches. Subsequently, the spatial information of the template and search patches is individually embedded using a learnable position embedding scheme. The outcome of the embeddings is known as tokens, and we refer to the target template and search region tokens as $E_{Z}$ and $E_{X}$, respectively. Before being fed to the encoder layers of the Transformer, these tokens are concatenated to generate a sequence of tokens [$E_{Z}$ ; $E_{X}$] with a total length of $N_{Z}+N_{X}$.

In the next phase of the one-stream Transformer tracking pipeline, attention features for the concatenated tokens are computed through a multi-head attention (MHA) block and a then fed to a feed-forward network (FFN) in each encoder layer. The self-attention of $i^{th}$ head in an encoder layer is computed as follows:

where, Query ($Q_{i}$), Key ($K_{i}$) and Value ($V_{i}$) are the three linear projections of the input matrix and $d_{k_{i}}$ represents the feature dimension of $K_{i}$. The MHA block of the one-stream tracker captures a wide range of relationships and dependencies between input tokens through linear projection while concurrently processing them with multiple independent attention heads. The process of MHA can be formulated as follows:

where $h$ is the number of attention heads in the tracker and $W_{O}$ denotes the output projection matrix.

In the $n^{th}$ encoder layer of the baseline tracking model, the concatenated target and search region tokens ($[E_{Z}^{n};E_{X}^{n}]$) are obtained from previous layer and then considered as the input tokens. After a normalization operation, the attention features of these tokens are computed by the MAH block and then added to the input tokens. Following another normalization operation, the combined features are fed into the FFN block of the encoder layer to obtain the output tokens. In summary, the operations of the $n^{th}$ encoder layer can be expressed as:

where $[E_{Z}^{n+1};E_{X}^{n+1}]$ is the output tokens of the $n^{th}$ encoder layer.

From the output tokens ($[E_{Z}^{l+1};E_{X}^{l+1}]$) of the last encoder layer of the baseline tracking model, the search region tokens ($[E_{X}^{l+1}]$) are extracted and fed into a prediction head to locate the target within the search region.

Most one-stream Transformer tracking approaches (Ye et al., 2022; Chen et al., 2022; Wu et al., 2023; Zhao et al., 2023a) only utilize the baseline tracking model with free bidirectional information flow. In contrast, the proposed OIFTrack approach goes further by enhancing the tracking performance of the baseline tracker through the incorporation of temporal cues and the optimization of information flow between the tokens.

Capturing and utilizing the temporal information is important in VOT, especially in long-term tracking scenarios where the appearance of the target may undergo significant changes. While many one-stream Transformer trackers (Ye et al., 2022; Chen et al., 2022; Wu et al., 2023; Zhao et al., 2023a) have shown better performance on benchmark datasets, their effectiveness is considerably limited by their failure to incorporate temporal cues in tracking, relying solely on the initial target template to locate the target in long-term sequences. Although some recent approaches (Lan et al., 2023; Cui et al., 2022; Xie et al., 2023) have included temporal cues by incorporating dynamic templates, our focus is on further enriching the utilization of temporal cues within the one-stream Transformer tracking paradigm.

Background cues are important in VOT as they are used to capture the surrounding information of the target object. Some Transformer tracking approaches (Mayer et al., 2022; Wang et al., 2021; He et al., 2023) expand the size of the target template to match the size of the search region, even though it increases the computational complexity of the tracker, aiming to incorporate the surrounding background information of the target. However, it is observed that the surrounding information of the initial target template becomes less useful for locating the target after a set of frames, especially in long-term tracking scenarios. Moreover, these approaches extract features from both the actual target region and a large portion of background regions together in the template, resulting in the distraction of target-specific features. Therefore, in the proposed approach, we have utilized the background cues from the dynamic template since they are very close to the surrounding region of the target in the search region while keeping the initial template without any additional background patches.

In the proposed OIFTrack approach, we include the dynamic target template from an intermediate frame in which the target object was located with high confidence. In addition to the dynamic target template, as shown in Fig. 2, we extract a few additional patches from the surrounding region of the dynamic target and consider them as dynamic background patches. Since nearby similar objects are also identified as dynamic background patches, the proposed tracker is able to successfully avoid distractor objects in the search region. In addition, unlike previous approaches, the proposed tracker restricts the information flow from dynamic background patches to the initial target template patches, ensuring that the target-specific features of the initial template remain undistracted.

In the proposed tracker, similar to the initial target template and search region patches, the dynamic target template image with background is divided into non-overlapping patches, each of size $P\times P$. After the linear projection and positional embedding, dynamic target template tokens and dynamic background tokens are generated; we refer to them as $E_{DT}$ and $E_{DB}$, respectively. Before being fed to the encoder layers of the Transformer, these tokens are concatenated with the initial target template and search region tokens to generate a sequence of input tokens [$E_{Z}$; $E_{DT}$; $E_{DB}$; $E_{X}$]. In the first frame of a tracking sequence, the dynamic target and dynamic background tokens are initialized by incorporating the target and surrounding cues in the initial frame. In every fixed frame interval, based on the highest confidence score, these tokens are updated to obtain the temporal cues.

We have conducted a comprehensive analysis of the information flow between target template and search tokens, focusing on enhancing the performance of one-stream tracking further, and have discovered several informative findings. In the search region tokens, the number of non-target tokens is higher compared to those containing the target cues, as all of these trackers search for the target in a large region. In many frames, most of the non-target search tokens contain background cues, while a few others contain cues of distractor objects. We have found that the information flow from the large number of background search tokens to the fewer number of target template tokens gives more influence to the background rather than the target, thereby reducing the importance of target-specific features in the template. In addition, information flow from search tokens with distractor cues reduces the originality of target-specific cues in the template, leading to drifting problems. However, we have also found that bidirectional interaction between target template tokens and search tokens containing target cues is crucial, as it is used to enrich the target-specific feature extraction, enabling the tracker to capture the appearance changes of the target.

We have expanded the comprehensive analysis of information flow between tokens by incorporating dynamic target tokens and dynamic background tokens. Consistent with previous findings, the information flow from dynamic background tokens to the initial target template tokens reduces the importance of target-specific features, while bidirectional information flow between dynamic target template and initial template tokens strengthens target-specific feature extraction.

Based on the findings, the proposed OIFTrack approach, instead of employing the free bidirectional information flow used in previous approaches, optimizes the information flow by blocking certain unnecessary token interactions that might distract the discriminative capability of the tracker. In the early encoder layer of the OIFTrack approach, the concatenated tokens are categorized as initial target template tokens, dynamic target template tokens, dynamic background tokens, and search tokens, and then the unnecessary information flows between these categories are blocked. The search tokens are maintained as a single category in the early layers because these layers lack sufficient knowledge about target-specific cues to perform partitioning. As shown in Fig. 3, in the deeper encoder layer of the proposed work, the number of token categories is increased by partitioning the search region tokens into non-target tokens and tokens with target cues, and then unnecessary information flows between these categories are blocked. We blocked the unnecessary information flow between tokens by utilizing the attention masking technique. In this technique, some attention scores in the matrix are manually set to negative infinity to prevent the interaction between corresponding tokens. The attention score matrix ($S$) is computed as follows:

since the Softmax operation is used to calculate the attention features by using the score matrix, as given in Equation 2, the information flow between the selected tokens is completely ignored in the proposed tracker.

The proposed OIFTrack approach obtained the initial target ([$E_{Z}$]), dynamic target ([$E_{DT}$]), dynamic background ([$E_{DB}$]), and search region ($E_{X}$) tokens and then concatenated them as a sequence of input tokens ([$E_{Z}$; $E_{DT}$; $E_{DB}$; $E_{X}$]). In an early encoder layer $n$, the attention features of the initial target template tokens are computed by obtaining the Query ($Q_{Z}$) from the initial template tokens and the Key ($K_{Z}$) and Value ($V_{Z}$) from both the initial target template and dynamic target template tokens, while ignoring the information flow from both the search tokens and dynamic background tokens. The attention feature extraction process of initial target template tokens in the $n^{th}$ layer can be expressed as follows:

In that $n^{th}$ early encoder layer, attention features of dynamic tokens is computed by obtaining the Queries ($Q_{DT}$ and $Q_{DB}$) from corresponding dynamic tokens and the Keys ($K_{DT}$ and $K_{DB}$) and Values ($V_{DT}$ and $V_{DB}$) from initial target template, dynamic target template, and dynamic background tokens, while ignoring the information flow from the search tokens. The attention feature extraction process of dynamic template and background tokens in an early layer can be expressed as follows:

In that $n^{th}$ early encoder layer, attention feature extraction process of search tokens can be expressed as follows:

As the last step of the feature extraction in the $n^{th}$ early encoder layer, the extracted attention features are concatenated and then fed to the FFN block. This process can be expressed as:

where $[E_{Z}^{{n}+1};E_{DT}^{{n}+1};E_{DB}^{{n}+1};E_{X}^{{n}+1}]$ are the output tokens of the $n^{th}$ encoder layer.

In the deeper encoder layers of the proposed approach, search region tokens are partitioned into target search tokens and non-target search tokens. This partition is carried out based on the attention scores of the search region tokens in relation to the initial target template and dynamic target template tokens. As the initial step in this partitioning process, attention score weights of all search region tokens are computed as follows:

We have only used the center region tokens of the initial target and dynamic target in the above computation to accurately identify the target search tokens. Since the obtained weighted score matrix ($\omega$) represents the relevance of search region tokens to the initial target and dynamic target template tokens, we sorted the scores and selected the corresponding top-K tokens as search tokens with target cues, while the remaining were considered as non-target search tokens.

After the search token partitioning, a deeper encoder layer $m$ receives the initial target ([$E_{Z}$]), dynamic target ([$E_{DT}$]), dynamic background ([$E_{DB}$]), target search ([$E_{XT}$]), and non-target search ([$E_{XB}$]) tokens from previous layer and then concatenated them as a sequence of tokens ([$E_{Z}$; $E_{DT}$; $E_{DB}$; $E_{XT}$; $E_{XB}$]). The attention features of the initial target tokens are extracted by blocking the information flow from both the dynamic background tokens and the non-target search tokens, as described in the following equation:

In that deeper encoder layer, the attention features of the dynamic target tokens are computed by blocking the information flow from non-target search tokens, and this process is expressed as follows:

Additionally, the attention features of the dynamic background tokens are extracted by restricting the information flow from all search region tokens, as follows:

The attention features of the search tokens ($E_{X}=[E_{XT};E_{XB}]$) are obtained by allowing interaction with all other tokens, and this process can be written as:

As the last step of the feature extraction in the $m^{th}$ deeper encoder layer, the extracted attention features are concatenated and then fed to the FFN block. This process can be expressed as:

Finally, from the output tokens of the last encoder layer of the proposed tracker, search region tokens are separated and then fed to a prediction head to locate the target. The overall tracking framework of the proposed OIFTrack approach is depicted in Fig. 3. It illustrates the attention feature extraction mechanism of the deeper encoder layers on the left side while demonstrating the difference in information flow between early and deeper encoder layers on the right side. Since the proposed optimized information flow mechanism enables bidirectional information flow among all tokens extracted from the initial target, dynamic target, and target search regions, the proposed tracker can effectively capture the appearance changes of the target object compared to previous approaches. Moreover, as the proposed optimized information flow mechanism blocks information flow from tokens containing non-target cues to tokens containing target cues, target-specific features are extracted more accurately, thereby enriching the discriminative capability of the tracker.

While the proposed OIFTrack approach enhances tracking accuracy, its computational efficiency is reduced due to the increase in the total number of input tokens resulting from the incorporation of tokens from the dynamic template. To address this issue, we eliminate a set of non-target search tokens in some encoder layer of the proposed tracker.

In one-stream Transformer tracking, a large portion of input tokens comes from the search region, with many of them containing only background cues. Therefore, eliminating the background tokens from the concatenated input tokens in deeper layers is one of the strategies to reduce the number of computations. Although the background token elimination mechanism is utilized in some previous approaches (Ye et al., 2022; Lan et al., 2023), we incorporated dynamic target cues to accurately identify the search tokens containing only background cues. Similar to the search token partitioning, Equation 10 is used to identify the search region tokens that have background cues, based on their lowest attention scores relative to the initial and dynamic template tokens. Then, among the tokens with the lowest scores, $P$ number of tokens are selected based on their poor attention scores and subsequently eliminated from the input token set.

The prediction head of the proposed tracker obtains the search region features from the final encoder layer and then reshapes them back into two dimensions with the size of $(W_{x}\times H_{x})$ to train the convolution-based heads. Similar to other single-stream Transformer trackers (Ye et al., 2022; Lan et al., 2023), the prediction head of the proposed OIFTrack consists of three components: a classification head, an offset head, and a size head. These heads are used to generate classification scores, correct discretization errors, and predict the width and height of the target, respectively. The target state in the search region is determined by selecting the position with the highest confidence classification score and incorporating the corresponding offset and size coordinates to calculate the size of the bounding box at that position.

The classification and regression losses of the prediction head components are used to train the proposed tracker. The overall loss function is calculated as a weighted combination of the losses of components: a focal loss (Law & Deng, 2018) for the classification head ($L_{cls}$), a generalized IoU loss ($L_{giou}$) (Rezatofighi et al., 2019) for the offset head, and an $L_{1}$ loss for the size head. It can be expressed as:

where $\lambda_{iou}$ = 2 and $\lambda_{L_{1}}$ = 5 are the regularization parameters in our experiments as in (Yan et al., 2021).

The proposed OIFTrack approach is implemented in Python using the PyTorch framework, and both training and evaluation are conducted on a Tesla P100 GPU with 16 GB of memory.

The architecture of the proposed tracker, except for the prediction head, is similar to the encoder part of ViT-B (Dosovitskiy et al., 2021). Similar to many other one-stream trackers, we utilize the pre-trained MAE (He et al., 2022) model as the backbone to initialize the Transformer architecture of the proposed approach. In our tracker, the search and target template sizes are set as $256\times 256$ pixels and $128\times 128$ pixels, respectively. Both are divided into non-overlapping patches, each with a size of $16\times 16$ pixels. Although some one-stream trackers trained their model in two variants with search region sizes of $256\times 256$ and $384\times 384$ pixels, we did not train the other variant due to limitations in our GPU memory. In an intermediate frame of the proposed tracker, the dynamic region is cropped, resized to $192\times 192$ pixels, with the central $128\times 128$ pixels assigned for the dynamic target, and the remaining surrounding pixels assigned for the dynamic background. In addition, in the proposed tracker, the token partitioning for the search tokens was conducted from the $10^{th}$ encoder layer.

The proposed OIFTrack approach was trained on the training sets of LaSOT (Fan et al., 2019), TrackingNet (Muller et al., 2018), GOT-10k (Huang et al., 2019) and COCO 2017 (Lin et al., 2014) benchmarks to evaluate the performance on the testset of LaSOT, UAV123 and TrackingNet. In addition, our approach was exclusively trained on the GOT-10k training set to evaluate its performance on the test set of that dataset, following the protocol described in (Huang et al., 2019). During training, we employ a few data augmentation techniques such as brightness jittering and horizontal flipping to enhance the generality of the tracker. The AdamW (Loshchilov & Hutter, 2019) optimizer is used to train the model. The proposed tracker is trained for 300 epochs, with 60k image pairs used in each epoch. The model was trained with a learning rate of 1e-4, and it was decreased by a factor of 10 after 240 epochs. For the GOT-10k dataset, training was conducted for 100 epochs, and the learning rate was decreased by a factor of 10 after 80 epochs.

Similar to other VOT approaches, we have utilized the percentage of average overlap (AO), Success Rate at a 0.5 threshold (SR_0.5), and the Success Rate at a 0.75 threshold (SR_0.75) to measure tracking performances on the GOT-10k dataset. For other benchmarks, we use the percentage of Area Under Curve (AUC) score, precision (P), and normalized precision (P_n) to measure performances.

The tracking performance of the proposed OIFTrack approach was evaluated on the GOT-10K (Huang et al., 2019), LaSOT (Fan et al., 2019), TrackingNet (Muller et al., 2018), and UAV123 (Mueller et al., 2016) benchmarks and the reported results of the state-of-the-art trackers are used for comparison. We have compared the performances of the proposed tracker with Transformer-based trackers: GRM (Gao et al., 2023), TATrack (He et al., 2023), ROMTrack (Cai et al., 2023), CTTrack (Song et al., 2023), MATTrack (Zhao et al., 2023a), HiT (Kang et al., 2023), OSTrack (Ye et al., 2022), MixFormer (Cui et al., 2022), SimTrack (Chen et al., 2022), SwinTrack (Lin et al., 2022), and SparseTT (Fu et al., 2022) CNN-Transformer based trackers: SMAT (Gopal & Amer, 2024), BANDT (Yang et al., 2023c), AiATrack (Gao et al., 2022), CSWinTT (Song et al., 2022), ToMP (Mayer et al., 2022), DTT (Yu et al., 2021), STARK (Yan et al., 2021), TransT (Chen et al., 2021), and TrDiMP (Wang et al., 2021), and CNN-based trackers: GdaTFT (Liang et al., 2023), SiamGAT (Guo et al., 2021), SiamR-CNN (Voigtlaender et al., 2020), PrDiMP (Danelljan et al., 2020), Ocean (Zhang et al., 2020), DiMP (Bhat et al., 2019), and SiamRPN++ (Li et al., 2019). While some trackers trained and evaluated their models on both small and large search region sizes of $256\times 256$ and $384\times 384$ respectively, we only consider the reported results of $256\times 256$ search region models to compare performances without bias. The tracking robustness comparison of the proposed tracker for the GOT-10K, LaSOT, and TrackingNet datasets is provided in Table 5.

In this study, we propose a novel one-stream Transformer tracker that blocks unnecessary information flow between tokens, rather than relying on the free bidirectional information flow used in previous approaches. In addition, we have utilized dynamic temporal cues to capture surrounding information about the target, which is then leveraged to mitigate the influence of distractor objects in the search region. To demonstrate the effectiveness of the proposed OIFTrack approach, the heatmap of the classification scores for a search region is visualized and shown in Fig. 7. Based on the figure, it is evident that in the heatmap of the baseline tracker, the target and similar distractor objects have equal scores, as the information flow from a large number of background tokens in search region distracts the target-specific features. Although the classification scores of the distractor are reduced in the heatmap of Model A by blocking unnecessary flow from search to target, they are entirely diminished in the heatmap of Model C after capturing the surrounding information of the target from dynamic background cues. Furthermore, the classification scores of the target became stronger in Model D after allowing the interaction of target search tokens with other tokens that have target cues.

In addition to the overall performance, our tracker demonstrated better performance in many tracking scenarios compared to other Transformer-based trackers, as shown in Fig. 8 through attribute-wise comparison. Since our OIFTrack approach captures the appearance changes of the target from the interaction between the initial target, dynamic target, and target search tokens, it exhibited superior performance in rotation, scale variation, deformation, as well as full and partial occlusion scenarios compared to other one-stream Transformer trackers. Furthermore, our tracker performs well in background clutter and camera motion scenarios due to its ability to capture the surrounding information of the target.

Our OIFTrack runs at an average speed of 21 FPS on a single core of a Tesla P100 GPU and it contains 92.63 million parameters and 33.82 giga floating point operations (FLOPS). Based on computational efficiency, our tracker is slightly lower than similar approaches such as OSTrack and GRM, due to our incorporation of dynamic target and background cues into the tracking process. As another limitation, although our tracker showed better performance in long-term tracking according to the results on the LaSOT benchmark, the difference between our performance and that of competitors is marginal. Since the probability of wrong identification of the dynamic template in long-term sequences is high, that may be the reason for this performance degradation.