All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment

In recent years, the two-stream framework [2, 4, 8, 5] has emerged as a dominant VL tracking paradigm (see Fig. 1(a)). They first extract features using two independent unimodal feature extractors, and then model relation of visual features and language features in sequential manner by a lightweight [5] or relatively heavy [9] network. Early work [3] contains a visual specification network and a lingual specification network, and further selectively focuses on parts of language prompts using a lingual specification attention network. Later, GTI [16] and [2] decompose the VL tracking problem into three sub-tasks of visual tracking, grounding and integration. VLT_TT [5] suggests to learn VL representations through an asymmetrical modeling architecture. JointNLT [9] introduces a joint visual grounding and tracking framework by localizing the referred object based on the visual-language references. However, these works rely on separate visual and language encoders to extract multi-modal features, leading to limited information interaction. We note that several works [10, 11, 14] introduce one-stream framework for visual object tracking. Different from them, we extend the one-stream framework to multi-modal VL tracking by training jointly on videos and language prompts. As shown in Fig. 1(b), for the first time we seamlessly integrate feature extraction and interaction into a unified backbone architecture for VL tracking. The proposed framework not only enables information flow from language to vision, but also allows bidirectional integration of information between visual and language features.

Thanks to the scalability to very large-scale model and capability to handle sequential and non-sequential data, transformer has become a prevailing architecture in both natural language [17, 18] and computer vision [19, 20] communities. Following ViT [19], a series of variants of ViT have been developed to improve the performance on vision tasks, including reducing computational cost [21, 20], and architecture design [22, 23]. Additionally, transformer has extensively used in various multi-modal tasks [24, 25, 26].

In consideration of the capacity of the unified transformer model to handle unimodal input or multi-modal input with a shared encoder, a few pioneering works have tried to explore unified multi-modal encoders [27, 28, 29]. For instance, ViLT [28] proposed a vision-language transformer without using regional features or deep convolutional visual embeddings. VATT [29] developed a video-audio-text transformer using multi-modal self-supervised pre-training to improve the performance of video action recognition and audio event classification. In this paper, we follow the trend of unified architecture for multi-modal data. To the best of our knowledge, the proposed All-in-One transformer is the first unified backbone architecture for multi-modal VL tracking.

Recently, a potential multi-modal learning paradigm is to adopt transformer to process and relate information from multiple modalities [30, 13, 31, 32]. CLIP [13] applied language prompts as supervisory signals to learn better visual representations. VisualBERT [31], VilBERT [33], and Unicoder-VL [32] combined visual and textual features into transformers to capture cross-modal relationships. However, previous works mainly focus on how to learn multi-modal representations by exploiting the complementary advantages of multiple modalities, or to fuse multi-modal features for prediction. Multi-modal alignment, discovering relationships and correspondences between fine-grained elements (e.g., objects and words) of instances (e.g., images and languages) from two or more modalities, has been rarely explored. In this work, we propose the MMA module with CMA and IMA based on self-supervised CL [15] to explore efficient multi-modal learning for VL tracking.

Before detailing the architecture of our All-in-One framework, we briefly review the transformer tracking [11, 34, 10, 35, 14], which achieves remarkable tracking performance. Given a video with a pair of visual template and visual search region $\mathcal{X}_{xz}$, an initial target box $\mathcal{B}_{0}$, the transformer tracking can be formulated as $F_{trans}:\{\mathcal{X}_{xz},\mathcal{B}_{0}\}\rightarrow\mathcal{B}$, where $F_{trans}$ is the transformer tracker, $\mathcal{B}$ is the predicted box of the target in all subsequent search frames. In general, the transformer tracker $F_{trans}$ can be decomposed into $\Phi\circ f$, where $f:\{\mathcal{X}_{xz},\mathcal{B}_{0}\}\rightarrow\mathcal{H}$ denotes the backbone (e.g., ViT [19]) for feature extraction and interaction function, $\mathcal{H}$ represents the output features of visual search region, and the tracking head $\Phi:\mathcal{H}\rightarrow\mathcal{B}$ is adopted to predict the target box.

Specifically, a pair of images, namely visual search region $\mathbf{x}\in\mathbb{R}^{3\times H_{x}\times W_{x}}$ and visual template $\mathbf{z}\in\mathbb{R}^{3\times H_{z}\times W_{z}}$ are divided into $N_{x}$ and $N_{z}$ non-overlapping image patches of resolution $P\times P$, where $N_{x}=H_{x}W_{x}/P^{2}$ and $N_{z}=H_{z}W_{z}/P^{2}$ are the number of patches of search region and template, respectively. Then, a linear projection is applied to these image patches to generate 1D tokens $\mathcal{H}_{x}\in\mathbb{R}^{N_{x}\times D}$ and $\mathcal{H}_{z}\in\mathbb{R}^{N_{z}\times D}$, where $D$ is the token dimension. Two learnable positional embeddings are added to $\mathcal{H}_{x}$ and $\mathcal{H}_{z}$ to retain the position information. After that, these tokens are concatenated as a sequence $\mathcal{H}_{xz}^{0}=[\mathcal{H}_{x};\mathcal{H}_{z}]$ and fed to a $L$-layer transformer encoder. Here, we represent $\mathcal{H}_{xz}^{l-1}$ as inputs to the $l$-th encoder layer $E^{l}$. Formally, the forward operation of the $l$-th encoder layer can be written as:

where each transformer encoder layer contains a multi-head self-attention (MHSA), and a feed-forward network (FFN). Each sub-layer is constructed as a residual connection, where layer normalization (LN) is followed by the residual connection. The visual search region tokens $\mathcal{H}^{L}_{x}$ of the last transformer encoder layer is taken as the input of tracking head $\Phi$ for target box prediction.

For VL tracking [1, 36, 2], it introduces an extra language prompt $\mathcal{T}$ for each video to express the attribute, behavior, position (relative location), and surroundings of the target. Accordingly, VL tracking can be formulated as $F_{VL}:\{\mathcal{X}_{xz},\mathcal{B}_{0},\mathcal{T}\}\rightarrow\mathcal{B}$, where $F_{VL}$ is the VL tracker. Similarly, the VL tracker $F_{VL}$ can also be decomposed into $\Phi\circ f^{*}$, where $\Phi$ is the tracking head, and $f^{*}$ represents the proposed unified backbone architecture in this work.

Fig. 2 gives an overview of our All-in-One framework for VL tracking. To optimize the VL tracker $F_{VL}$, a pair of visual template and visual search region $\mathcal{X}_{xz}=\{\mathcal{X}_{x},\mathcal{X}_{z}\}$, and an extra language prompt $\mathcal{T}$ are first fed to a patch embed (i.e., a linear projection) and a text tokenizer [18], respectively. They are mapped and flattened into $D$-dimension embeddings, where $D=768$. We denote them as vision tokens (i.e., $\mathcal{H}^{0}_{x}$ and $\mathcal{H}^{0}_{z}$), where $\mathcal{H}^{0}_{x}\in\mathbb{R}^{N_{x}\times D}$ and $\mathcal{H}^{0}_{z}\in\mathbb{R}^{N_{z}\times D}$ are visual search region tokens and visual template tokens, and language tokens $\mathcal{H}^{0}_{t}\in\mathbb{R}^{N_{t}\times D}$, where $N_{t}$ is the number of language tokens. Following [18], a special classification token ([CLS]) is attached at the beginning of the language tokens. Then, $\mathcal{H}^{0}_{x}$, $\mathcal{H}^{0}_{z}$, and $\mathcal{H}^{0}_{t}$ are aligned with the multi-modal alignment module (see section III-C) in the embedding space. It is worth noting that the well aligned vision embeddings and language embeddings can facilitate multi-modal representation learning and interaction [30]. Here, we still refer to the aligned vision embeddings and language embeddings as $\mathcal{H}^{0}_{x}$, $\mathcal{H}^{0}_{z}$, and $\mathcal{H}^{0}_{t}$, respectively. Afterwards, we perform a modal mixup operation [5] between the aligned vision embeddings and language embeddings as follows:

where $\odot$ represents the Hadamard product, $Linear(\cdot)$ is a linear projection layer. In this way, the language information is injected into vision embeddings via the modal mixup operation. Moreover, Eqs. (3)-(4) also construct a bidirectional information flow between vision and language modalities that allows mutual guidance for multi-modal feature extraction and interaction. By establishing a bidirectional information flow between well aligned visual and language signals as early as possible via a unified transformer backbone, we can avoid the loss of discriminative information and thus make the extracted features highly target-aware [10].

Formally, the operations of the $l$-th encoder of our All-in-One transformer backbone can be expressed as:

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ represent the query, key, and value embeddings, $[;]$ denotes the concatenation operation, $\mathbf{F}^{l}_{x}$ and $\mathbf{F}^{l}_{z}$ are the input embeddings of the $l$-th transformer encoder. Therefore, the language information-injected vision embeddings are jointly processed by the transformer encoder, enabling seamless multi-modal feature extraction and integration. Finally, the visual search region embeddings of the last layer of the transformer encoder are reshaped into a 2D feature map. The feature map is fed into the tracking head to predict the location of the target.

To model the interaction between language and vision features, recent VL trackers [9, 5] adopted a customized fusion model to directly serialize vision and language embeddings into sequences to learn a joint multi-modal embedding. Although, our All-in-One transformer backbone using the pretrained ViT  [19] has the ability to model long-range dependencies of sequential data, alleviating the negative effects of modal differences for multi-modal learning, the vision embeddings and language embeddings lying in different feature spaces is still challenging for the transformer encoder to learn their interactions [37, 38]. To tackle this limitation, we further propose a self-supervised MMA module, which is used before feature extraction and integration. The MMA module includes CMA and IMA, which can efficiently learn more reasonable feature distributions as shown in Fig. 3.

Intuitively, by optimizing the CMA loss, vision and language embeddings can be well aligned in the feature space as in Fig. 3. However, the CMA ignores the significant intra-modal supervisory signals (i.e., visual template and visual search region) for learning desired multi-modal features. Aligning the visual template with the visual search region enables learning temporal-invariant features [39, 40, 41], which are crucial to enhance the discriminative ability of tracking models. To this end, we further propose the IMA to fully utilize the intra-modal temporal supervision information.

The IMA loss encourages learning representations by aligning temporal-invariant positive pairs within visual modality. Importantly, it enforces the uniformity of vision and language, resulting in a uniform distribution across the whole feature space [42, 38]. Therefore, CMA and IMA have complementary advantages in multi-modal learning: on the one hand, CMA encourages matched vision and language embeddings close in the feature space. On the other hand, IMA maximizes the temporal-invariant features between visual tokens, and make the multi-modal features evenly distributed in the feature space. As shown in Fig. 3, combining them makes the learned representations more reasonable, and further facilitates joint multi-modal feature learning and interaction.

Following [10], the tracking head is decomposed into two branches of classification and bounding box regression. As shown in Fig. 2, the learned visual search region tokens are first reshaped into a 2D feature map according to the original spatial resolution, followed by a 4-layer fully convolutional network to predict the target classification score map. In the classification branch, a weighted focal loss $\mathcal{L}_{cls}$ [43], is adopted to enhance the model’s ability to distinguish objects from background. The bounding box regression branch is used to predict the center coordinate offset of the object and the size of the object. To regress the center coordinate offset and size of objects, we combine the $\ell_{1}$ loss and the generalized IoU loss $\mathcal{L}_{giou}$ [44]. The regression loss is calculated as $\mathcal{L}_{reg}=\lambda_{giou}\mathcal{L}_{giou}+\lambda_{{1}}\mathcal{L}_{1}$, where $\lambda_{giou}$ and $\lambda_{{1}}$are two hyper-parameters.

To train our model in an end-to-end manner, we convert it into a multi-task optimization problem [45], simultaneously optimizing classification loss, regression loss, CMA loss, and IMA loss. Finally, the overall loss function for our model is defined as:

where $\lambda_{cma}$ and $\lambda_{ima}$ are trade-off weights to balance the multi-task optimization problem.

We adopt ViT-Base [19] as the architecture of All-in-One transformer backbone. It is stacked by $L$ (i.e., 12) transformer encoder layers, and each layer contains two sub-layers, i.e., a multi-head self-attention layer and a feed-forward network. Each sub-layer is a residual connection structure, followed by a layer normalization. To accelerate convergence, we initialize our backbone with MAE-pretrained weights [12]. The visual template and visual search region are $2^{2}$ times and $4^{2}$ times of the target bounding box, and then resized to $128\times 128$ and $256\times 256$, respectively. We use bert-base-uncased tokenizer [18] to tokenize language prompts.

Our experiments are conducted on an Ubuntu server with two NVIDIA RTX 3090 GPUs. The training data includes training splits of LaSOT [36], GOT-10k [6], TrackingNet [47], COCO [48], OTB99-L [3], TNL2K [2], WebUAV-3M [1], and VisualGenome [49]. For several datasets (i.e.,  [6, 47, 48]) without natural language prompts, we use class names as pseudo language labels similar to [5]. The tracker is optimized using an AadmW optimizer [50] with initial learning rate $4\times 10^{-4}$. The total epoch is 300. The weight decay factor is $1\times 10^{-4}$ after 240 epochs. Following [10], the hyper-parameters $\lambda_{{giou}}$ and $\lambda_{1}$ are set to 2 and 5. While $\lambda_{cma}$ and $\lambda_{ima}$ are set to 1 and 1 without parameter optimization. We set the temperature parameter $\tau=0.5$. The batch size $N$ is 32. Following [7, 1], we adopt the one-pass evaluation with five metrics, i.e., precision ($P$), normalized precision ($P_{norm}$), success rate (AUC), complete success rate (cAUC), and accuracy (ACC) to measure the tracking performance.

We first conduct ablation experiments trained on the LaSOT training set and evaluated on the LaSOT test set to validate different components of our approach.

As shown in Fig. 8, we compare All-in-One with three SOTA trackers (i.e., VLT_TT [5], TransT [35], and SiamRPN++ [57]) on three videos from the LaSOT test set, in which the main challenges include similar distractors, severe viewpoint changes, background clutter, appearance variation, occlusion and extreme illumination. We can see that All-in-One is more robust than other methods. For instance, the prior most competitive VL tracker VLT_TT gradually loses the target as the target appearance varies in the video of Sepia-16 (the second video in Fig. 8). By contrast, All-in-One provides accurate and stable prediction results, demonstrating the effectiveness of our unified framework in complex environments.