Unified Single-Stage Transformer Network for Efficient RGB-T Tracking

The input of USTrack is a pair of target template images and a pair of search region images, consisting of a total of four images, namely, the RGB template image ${\boldsymbol{z}_{image}^{rgb}\in\mathbb{R}^{H_{z}\times W_{z}\times 3}}$, the RGB search region image ${\boldsymbol{x}_{image}^{rgb}\in\mathbb{R}^{H_{x}\times W_{x}\times 3}}$, the thermal template image ${\boldsymbol{z}_{image}^{t}\in\mathbb{R}^{H_{z}\times W_{z}\times 3}}$ and the thermal search region image ${\boldsymbol{x}_{image}^{t}\in\mathbb{R}^{H_{x}\times W_{x}\times 3}}$. They are first split and flattened into sequences of patches ${\boldsymbol{z}_{rgb},\boldsymbol{z}_{t}\in\mathbb{R}^{N_{z}\times(3P^{2})}}$ and ${\boldsymbol{x}_{rgb},\boldsymbol{x}_{t}\in\mathbb{R}^{N_{x}\times(3P^{2})}}$, where ${P\times P}$ is the resolution of each patch, and ${N_{z}=\frac{H_{z}W_{z}}{P^{2}}}$, ${N_{x}=\frac{H_{x}W_{x}}{P^{2}}}$ are the number of patches of template and search region respectively. Then, two trainable linear projection layers with parameters ${\boldsymbol{E}_{rgb}\in\mathbb{R}^{(3P^{2})\times D}}$ and ${\boldsymbol{E}_{t}\in\mathbb{R}^{(3P^{2})\times D}}$ are used to project ${\boldsymbol{z}_{rgb}}$, ${\boldsymbol{x}_{rgb}}$ and ${\boldsymbol{z}_{t}}$, ${\boldsymbol{x}_{t}}$ into ${D}$ dimension latent space. The output of this projection are four patch embeddings ${\hat{\boldsymbol{z}}_{rgb}}$, ${\hat{\boldsymbol{x}}_{rgb}}$ and ${\hat{\boldsymbol{z}}_{t}}$, ${\hat{\boldsymbol{x}}_{t}}$. Learnable ${1D}$ position embeddings ${\boldsymbol{P}_{z}}$ and ${\boldsymbol{P}_{x}}$ are added to the template patch embeddings ${\hat{\boldsymbol{z}}_{rgb}}$, ${\hat{\boldsymbol{z}}_{t}}$ and search region patch embeddings ${\hat{\boldsymbol{x}}_{rgb}}$, ${\hat{\boldsymbol{x}}_{t}}$ separately, and Learnable ${1D}$ modality embeddings ${\boldsymbol{M}_{rgb}}$ and ${\boldsymbol{M}_{t}}$ are added to the RGB patch embeddings ${\hat{\boldsymbol{z}}_{rgb}}$, ${\hat{\boldsymbol{x}}_{rgb}}$ and thermal patch embeddings ${\hat{\boldsymbol{z}}_{t}}$, ${\hat{\boldsymbol{x}}_{t}}$ separately. The patch embeddings after adding position and modality embeddings are final features called token embeddings. The above operations can be represented as follows:

$$\hat{\boldsymbol{z}}_{rgb}=\left[\boldsymbol{z}_{rgb}^{1}\boldsymbol{E}_{rgb};\boldsymbol{z}_{rgb}^{2}\boldsymbol{E}_{rgb};...;\boldsymbol{z}_{rgb}^{N_{z}}\boldsymbol{E}_{rgb}\right]+\boldsymbol{P}_{z}+\boldsymbol{M}_{rgb},$$

(1)

$$\hat{\boldsymbol{z}}_{t}=\left[\boldsymbol{z}_{t}^{1}\boldsymbol{E}_{t};\boldsymbol{z}_{t}^{2}\boldsymbol{E}_{t};...;\boldsymbol{z}_{t}^{N_{z}}\boldsymbol{E}_{t}\right]+\boldsymbol{P}_{z}+\boldsymbol{M}_{t},$$

(2)

$$\hat{\boldsymbol{x}}_{rgb}=[\boldsymbol{x}_{rgb}^{1}\boldsymbol{E}_{rgb};\boldsymbol{x}_{rgb}^{2}\boldsymbol{E}_{rgb};...;\boldsymbol{x}_{rgb}^{N_{x}}\boldsymbol{E}_{rgb}]+\boldsymbol{P}_{x}+\boldsymbol{M}_{rgb},$$

(3)

$$\hat{\boldsymbol{x}}_{t}=[\boldsymbol{x}_{t}^{1}\boldsymbol{E}_{t};\boldsymbol{x}_{t}^{2}\boldsymbol{E}_{t};...;\boldsymbol{x}_{t}^{N_{x}}\boldsymbol{E}_{t}]+\boldsymbol{P}_{x}+\boldsymbol{M}_{t}.$$

(4)

After passing the dual embedding layer, RGB template token embeddings ${\hat{\boldsymbol{z}}_{rgb}}$, thermal template token embeddings ${\hat{\boldsymbol{z}}_{t}}$, RGB search region token embeddings ${\hat{\boldsymbol{x}}_{rgb}}$ and thermal search region token embeddings ${\hat{\boldsymbol{x}}_{t}}$ will be input into the backbone for subsequent processing.

The self-attention mechanism is the core component of the ViT, and it is also the key to performing joint feature extraction, feature fusion and relation modeling in a single ViT backbone. From the perspective of the self-attention mechanism, we take the RGB search region token embeddings ${\hat{\boldsymbol{x}}_{rgb}}$ as an example to further analyze the intrinsic reasons why the proposed network is able to realize simultaneous feature extraction, feature fusion and relation modeling.

In the attention layer, the token sequences ${\hat{\boldsymbol{x}}_{rgb}}$, ${\hat{\boldsymbol{x}}_{t}}$, ${\hat{\boldsymbol{z}}_{rgb}}$, ${\hat{\boldsymbol{z}}_{t}}$ from dual embedding layers are concatenated as ${\boldsymbol{H}=\begin{bmatrix}\widehat{\boldsymbol{x}}_{rgb};\widehat{\boldsymbol{x}}_{t};\widehat{\boldsymbol{z}}_{rgb};\widehat{\boldsymbol{z}}_{t}\end{bmatrix}\in\mathbb{R}^{(2N_{x}+2N_{t})\times D}}$. Then Self-attention operation is performed on ${\boldsymbol{H}}$ as follows:

$$\boldsymbol{M}=\boldsymbol{A}\cdot\boldsymbol{V}=softmax\left(\frac{\boldsymbol{Q}\boldsymbol{K}^{T}}{\sqrt{d_{k}}}\right)\cdot\boldsymbol{V},$$

(5)

$$\boldsymbol{Q}\boldsymbol{K}^{T}=[\boldsymbol{Q}^{x}_{rgb};\boldsymbol{Q}^{x}_{t};\boldsymbol{Q}^{z}_{rgb};\boldsymbol{Q}^{z}_{t}][\boldsymbol{K}^{x}_{rgb};\boldsymbol{K}^{x}_{t};\boldsymbol{K}^{z}_{rgb};\boldsymbol{K}^{z}_{t}]^{T},$$

(6)

$$\boldsymbol{V}=\left[\boldsymbol{V}^{x}_{rgb};\boldsymbol{V}^{x}_{t};\boldsymbol{V}^{z}_{rgb};\boldsymbol{V}^{z}_{t}\right],$$

(7)

where ${\boldsymbol{M}}$ is the output of self-attention operation. ${\boldsymbol{A}}$ is the attention weight. ${\boldsymbol{Q}}$, ${\boldsymbol{K}}$, and ${\boldsymbol{V}}$ are query, key and value matrices separately. The superscripts ${z}$ and ${x}$ denote matrix items belonging to the template and search region. The subscripts ${rgb}$ and ${t}$ denote matrix items belonging to the RGB modality and thermal modality. The calculation of attention weights in Eq. (6) can be expanded to follows:

	$\displaystyle\boldsymbol{Q}\boldsymbol{K}^{T}$	$\displaystyle=[\boldsymbol{Q}^{x}_{rgb}{\boldsymbol{K}^{x}_{rgb}}^{T},\boldsymbol{Q}^{x}_{rgb}{\boldsymbol{K}^{x}_{t}}^{T},\boldsymbol{Q}^{x}_{rgb}{\boldsymbol{K}^{z}_{rgb}}^{T},\boldsymbol{Q}^{x}_{rgb}{\boldsymbol{K}^{z}_{t}}^{T};...]$		(8)
		$\displaystyle=[\boldsymbol{W}^{x_{rgb}}_{x_{rgb}},\boldsymbol{W}^{x_{rgb}}_{x_{t}},\boldsymbol{W}^{x_{rgb}}_{z_{rgb}},\boldsymbol{W}^{x_{rgb}}_{z_{t}};...],$

where the left part of Eq. (8) represents the calculation and the output of attention weights between the RGB search region tokens and the other inputs. the output of self-attention operation can be further written as follows:

	$\displaystyle\boldsymbol{M}=\left[{\boldsymbol{W}^{x_{rgb}}_{x_{rgb}}\boldsymbol{V}^{x}_{rgb}+\boldsymbol{W}^{x_{rgb}}_{x_{t}}\boldsymbol{V}^{x}_{t}}\right.$		(9)
	$\displaystyle\left.{+\boldsymbol{W}^{x_{rgb}}_{z_{rgb}}\boldsymbol{V}^{z}_{t}+\boldsymbol{W}^{x_{rgb}}_{z_{t}}\boldsymbol{V}^{z}_{t};...}\right],$

where the left part of Eq. (9) is the output corresponding to the RGB search region tokens after the self-attention operation. ${\boldsymbol{W}^{x_{rgb}}_{x_{rgb}}\boldsymbol{V}^{x}_{rgb}}$ is responsible for aggregating the RGB search region image feature (RGB modality feature extraction). ${\boldsymbol{W}^{x_{rgb}}_{x_{t}}\boldsymbol{V}^{x}_{t}}$ is responsible for aggregating the thermal modality-specific information based on semantic similarity between two modalities features (feature fusion and modality features interaction). The attention weights can intuitively measure the semantic similarity between modalities. Network can model modality-sharing information based on this similarity. The aggregation of complementary information enables the network to promptly adjust the subsequent extraction of features in RGB search region image. ${\boldsymbol{W}^{x_{rgb}}_{z_{rgb}}\boldsymbol{V}^{z}_{t}}$ is responsible for aggregating RGB template image feature to further obtain the relation information between the RGB template and the RGB search region (relation modeling based on modality-specific information). ${\boldsymbol{W}^{x_{rgb}}_{z_{t}}\boldsymbol{V}^{z}_{t}}$ is responsible for aggregating thermal template image feature to further obtain the relation information between the thermal template and the RGB search region (relation modeling based on modality-sharing information). The RGB search region fusion features, which contains relation information, can be used for prediction. Therefore, with the global perception ability of the self-attention, we seamlessly unify feature extraction, feature fusion, and relation modeling into a single ViT backbone. The network can directly extract fusion features of the template and search region under the mutual interaction of modalities, and simultaneously performs relation modeling between fusion features of the template and search region. This alleviates the lack of interaction and guidance between modalities during the feature extraction stage, as well as the problem of additional fusion modules significantly affecting the inference speed of the RGB-T tracking network. Additionally, by inheriting the advantages of relation modeling which is performed by the self-attention, the network can extract more target-specific search region fusion features for prediction under the guidance of two templates.

After passing the ViT backbone, two search region fusion features can be obtained for final prediction: Thermal-assisted RGB fusion features based on the RGB search region image, and RGB-assisted thermal fusion features based on the thermal search region image. Both fusion features contain the fusion information of modalities and the relation information between the template and the search region, which can be directly used for target position prediction. To fully exploit the modality-sharing information and the modality-specific information of the modality that is more suitable for the current scene. As shown in Fig. 3(b), during the training phase, we equip each fusion feature with a prediction module and a reliability evaluation head. We set the same loss for each prediction head and let each reliability evaluation module output a weight as the adaptive reliabilty for the loss of the corresponding prediction head. Then, they are combined into a final total loss function for training. This method allows the modality that is not suitable for the current scene to produce inferior results, and then guides the modality reliability evaluation module to assign smaller weights to its loss by minimizing the overall loss function. Conversely, for fusion features that are more suitable for the current scene, it assigns larger weights. During the testing phase, the network will simultaneously output two results and evaluate the reliability of both modalities. Based on the reliability ${R_{RGB}}$ and ${R_{T}}$, we select the predicted results with higher reliability scores as the final output.

We adopt the prediction head of OSTrack (Ye et al. 2022) directly as our prediction head. The detailed information and corresponding settings can be found in OSTrack. The loss corresponding to the two prediction heads are set as follows:

$${\mathcal{L}_{RGB}=\mathcal{L}_{cls_{RGB}}+\lambda_{giou}\mathcal{L}_{giou_{RGB}}+\lambda_{L_{1}}\mathcal{L}_{1_{RGB}}},$$

(10)

$${\mathcal{L}_{T}=\mathcal{L}_{cls_{T}}+\lambda_{giou}\mathcal{L}_{giou_{T}}+\lambda_{L_{1}}\mathcal{L}_{1_{T}}},$$

(11)

where ${\mathcal{L}_{RGB}}$ and ${\mathcal{L}_{T}}$ are the overall loss function for each prediction head, ${\mathcal{L}_{cls_{RGB}}}$ and ${\mathcal{L}_{cls_{T}}}$ are the weighted focal loss for classification, ${\mathcal{L}_{1_{RGB}}}$ and ${\mathcal{L}_{1_{T}}}$ are the ${\mathcal{L}_{1}}$ loss, ${\mathcal{L}_{giou_{RGB}}}$ and ${\mathcal{L}_{giou_{T}}}$ are the generalized IoU loss, and ${\lambda_{giou}}$ and ${\lambda_{L_{1}}}$ are the regularization parameters. On the basis, a modality reliability evaluation module is added to each search region fusion features. The evaluation module is a fully convolutional neural network, which consists of several stacked Conv-BN-ReLU layers. Two modality reliability evaluation modules will output the reliability scores corresponding to two search region fusion features ${R_{RGB}}$, ${R_{T}\in\mathbb{R}}$ respectively. In order to prevent the model from directly making the weight ${R_{RGB}}$ and ${R_{T}}$ zero to minimize the overall loss during the training process, we softmax the reliability scores to obtains the adaptive weight ${\lambda_{RGB}}$ and ${\lambda_{T}}$ and overall loss as follows:

$${\lambda_{RGB},\lambda_{T}=softmax(R_{RGB},R_{T})},$$

(12)

$${\mathcal{L}_{total}=\lambda_{RGB}\mathcal{L}_{RGB}+\lambda_{T}\mathcal{L}_{T}},$$

(13)

where ${\lambda_{RGB}}$ and ${\lambda_{T}}$ are used as the adaptive weights of the loss of the two prediction heads, and the two losses are combined together as the overall loss to train the model.

We compare our method with previous state-of-the-art RGB-T tracking methods on three benchmarks including VTUAV, RGBT234, and GTOT. GTOT and RGBT234 use success rate SR and precision rate PR as evaluation metrics. To mitigate small alignment errors, VTUAV use Maximum Precision Rate MPR and Maximum Success Rate MSR as evaluation metrics. SR measures the ratio of tracked frames, determined by the Interaction-over-Union (IoU) between tracking result and ground truth. With different overlap thresholds, a success plot (SP) can be obtained, and SR is calculated as the area under curve of SP. MSR adopts the maximum overlap in frame level as the final score. PR measures the percentage of frames whose distance between the predicted position and the ground truth is less than a certain threshold $\tau$. Similar to MSR, MPR adopt the smaller center distance as the final score. $\tau$ is set to 20 in our experiment.

Our model is implemented based on Python 3.8, PyTorch 2.0.0. All experiments are conducted on one NVIDIA RTX3090 GPU. We adopt AdamW as the optimizer with 1e-4 weight decay. The learning rate is set as 4e-5 for the backbone and 4e-4 for other parameters. The search regions are resized to 256×256 and templates are resized to 128×128. Each batch size is set to 24, and each epoch contains 30k image pairs. In order to fairly compare our method with other SOTA methods and demonstrate its effectiveness, we aligned our experimental conditions with other methods. We pretrained the network on RGB tracking datasets such as COCO (Lin et al. 2014), LaSOT (Fan et al. 2018), GOT-10k (Huang, Zhao, and Huang 2018), and TrackingNet (Muller et al. 2018). When testing on the GTOT and RGBT234, we only used LasHeR as the training set. When testing on the short-term and long-term testing sets of VTUAV, we only use the training set of VTUAV for training.

We test our network USTrack on three popular RGB-T tracking benchmarks, comparing performance and speed with the SOTA trackers, such as FSRPN, mfDimp, DAFNet , DAPNet, MANet, CAT, CMPP, JMMAC, MANet++, ADRNet, SiamCDA, M5LNet, TFNet, DMCNet, MFGNet, APFNet, HMFT, MIRNet, ECMD, ViPT and TBSI, to validate the effectiveness of our method. The test results on three datasets show that our method has achieved significant improvements in both performance and inference speed.

It is worth mentioning that, to our knowledge, HMFT-LT is currently the only long-term RGB-T tracking method that utilizes the local-to-global tracking strategy. We have achieved significantly better tracking performance on long-term datasets compared to HMFT-LT at ten times the speed of HMFT-LT. Furthermore, we have achieved the best results on all subsets of challenge attributes in VTUAV, especially those that will lead to continuous changes in the state of modality appearance, such as occlusion, extreme lighting, etc. Detailed test results are provided in the supplementary materials. Above results demonstrate the excellent adaptability of our method to the diverse dual-modality appearances of targets and the dynamic relationship between modalities, fully demonstrating the effectiveness and efficiency of USTrack, which can directly extract dynamic fusion features of templates and search regions under the interaction of modalities, and simultaneously perform the relation modeling to further improve the tracking speed.