Collaborative Tracking Learning for Frame-Rate-Insensitive
Multi-Object Tracking

As shown in Fig. 2, the proposed collaborative tracking learning model (ColTrack) for frame-rate-insensitive MOT is built on the encoder-decoder Transformer [32] architecture. Given a sequence of video frames $\{\mathbf{I}^{1},\mathbf{I}^{2},\cdots,\mathbf{I}^{T}\}$, ColTrack tracks objects of interest in each frame and predicts their class and bounding boxes. For the video frame $\mathbf{I}^{t}$, the CNN model extracts its features, which are then fed to the transformer encoder to provide feature maps and $N_{\mathrm{det}}$ candidate target anchors. Each anchor $\hat{\mathbf{b}}^{t}_{j}$ is the predicted bounding box of one object. These anchors together with a set of learnable content features constitute $N_{\mathrm{det}}$ detection queries $\mathbf{D}^{t}=\{\mathbf{d}^{t}_{j}\}_{j=1:{N_{\mathrm{det}}}}$. Detection queries are used to detect new objects appearing in the current frame.

Similar to the existing methods [16, 30, 6], ColTrack uses the queries output from previous video frames as the tracking queries to track the tracked targets. But unlike these methods that only construct one tracking query for each tracked target, ColTrack utilizes multiple historical features of each tracked target to construct collaborative tracking queries for collaborative tracking in the current frame. The tracking queries $\mathbf{X}^{t}$ and detection queries $\mathbf{D}^{t}$ are combined and fed to subsequent multiple temporal blocking decoders to iteratively refine features and bounding boxes. An information refinement module (IRM) is inserted between two adjacent decoders to allow information fusion of historical features belonging to the same target.

The output queries of each decoder are sent to the bipartite matching loss and the tracking object consistency loss to guide the training of the model (in Fig. 2, the losses of the middle decoders are not drawn for brevity). The queries of the matched targets output by the last decoder are sent to subsequent frames as new historical queries. After sequentially processing each frame, ColTrack obtains the tracking results of the entire video.

To enhance the model’s tracking at a low frame rate, we propose a collaborative tracking method based on historical features. For the ${i}^{\mathrm{th}}$ tracked object, we store its historical features $\mathbf{F}^{t}=\{\mathbf{f}^{t-\hat{t}}_{i}\}_{\hat{t}=1:N_{i}^{t}},N_{i}^{t}\leq N_{\mathrm{max}}$. $N_{\mathrm{max}}$ is the max memory size and $N_{i}^{t}$ is the number of historical features. $\mathbf{f}^{t-\hat{t}}_{i}\in{\mathbb{R}}^{d}$ is the output content feature of the corresponding target in ${t-\hat{t}}$ frame. Considering that usually the latest location prediction $\hat{\mathbf{b}}^{t-1}_{i}$ is closer to the location of the target in the current frame than $\{\hat{\mathbf{b}}^{t-\hat{t}}_{i}\}_{\hat{t}=2}^{N_{i}^{t}}$, we adopt $\hat{\mathbf{b}}^{t-1}_{i}$ as the target anchor, which is combined with each historical feature $\mathbf{f}^{t-\hat{t}}_{i}$ to construct the tracking query $\mathbf{x}^{t-\hat{t}}_{i}$. The $N_{i}^{t}$ queries $\mathbf{X}^{t}_{i}=\{\mathbf{x}^{t-\hat{t}}_{i}\}_{\hat{t}=1:N_{i}^{t}}$ are considered as collaborative tracking queries and jointly track the same target in the current frame. They are initialized with different historical features and the same anchor $\hat{\mathbf{b}}^{t-1}_{i}$.

The benefits of the introduction of collaborative tracking queries are manifold. First, this approach provides more abundant descriptions of objects to alleviate the impact of unreliable features in low-frame-rate videos. Multiple historical queries containing different temporal clues directly participate in the tracking of the target in the current frame. During the iterative refinement of multiple decoders and IRM modules, each historical query adaptively collects valuable clues based on the tracking results of other historical queries to obtain a more accurate feature description for better tracking in the next decoder. The direct participation of historical queries enables the contained valuable information to be mined better.

Second, this approach is more conducive to training a frame-rate-insensitive MOT model. ColTrack requires historical features of the same target with different time spans from the current frame to track the target correctly. This is equivalent to training the model to track targets at different frame rates. Older features usually have a larger appearance difference from the current target. This requires ColTrack to be able to extract more robust features, and the refinement module IRM should be able to better integrate effective information from other historical features to deal with the impact of frame rate.

As demonstrated in [8], benefit from the self-attention mechanism over the activations in decoders, the DETR-like detection model [8, 39, 32] can discard the non-maximum suppression (NMS) post-processing. This means that the decoder suppresses the case where multiple queries detect the same object to inhibit duplicate predictions. But in our method, collaborative tracking queries of the same object have to track the same object, which is against the design of the decoder. To address this issue, we propose temporal blocking decoders, which avoid the interaction between historical tracking queries of the same target by modifying the attention mask of the multi-head self-attention module in the decoder.

The temporal interaction blocking capability provided by the temporal blocking decoder avoids mutual inhibition between collaborative tracking queries of the same target, but it also makes it impossible to exchange information between collaborative tracking queries to refine themselves. To solve this problem, we devise a new information refinement module (IRM) to allow the interaction between collaborative tracking queries and refine them. As shown in Fig. 3, the multi-head self-attention layer is the main component of the IRM. The input of IRM is the $N^{t}$ collaborative tracking queries of all the tracked objects, where $N^{t}=\sum_{i}{N^{t}_{i}}$. The output is the refined features. We modify the attention mask to avoid the interaction of features from different objects.

Since the main contribution of IRM is to allow collaborative tracking queries belonging to the same target to interact and update information, then for each query, it is necessary to decide how to remove old information and what new information to add. Inspired by this, the IRM we devised mainly consists of two branches: an information removal branch and an information addition branch. Each branch contains two parts, i.e. the temporal clues collection part and the action part.

A single multi-head self-attention layer of the transformer is not able to compute any cross-correlations between the queries [8]. This is because a single self-attention layer can only organize information once based on the similarities of features. The model cannot see the global information to decide how to output. Therefore, we add a multi-head self-attention module as the temporal clues collection part to collect global temporal clues for each tracking query.

For the information addition branch, the action part uses the collected global temporal clues as queries and uses the original content features as keys and values to generate the content features $\mathbf{F}^{t}_{\mathrm{add}}\in{\mathbb{R}}^{N^{t}\times d}$ that needs to be added. Compared with the information addition branch, the information removal branch has one more fully connected layer followed by a sigmoid layer to map the embedding to a gating vector $\mathbf{z}\in{\mathbb{R}}^{N^{t}\times d_{\mathrm{head}}}$. $d_{\mathrm{head}}$ is the head number of the multi-head self-attention module. We divide each content feature into $d_{\mathrm{head}}$ groups, and use the gating vector $\mathbf{z}$ to control the degree of deletion of each group of features. Then, we formulate the refinement of content features as

$$\mathbf{F}^{t}_{r}=\mathrm{LN}(2\mathbf{F}^{t}\times(1-\mathbf{z})+\mathbf{F}^{t}_{\mathrm{add}})\,,$$

(1)

where $\mathrm{LN}(\cdot)$ is the layer normalization [1]. $\mathbf{F}^{t}_{r}\in{\mathbb{R}}^{N^{t}\times d}$ is the refined content features. $\mathbf{F}^{t}\times(1-\mathbf{z})$ denotes the reserved features. We double it to increase its weight. The refined content features $\mathbf{F}^{t}_{r}$ are combined with the corresponding anchors to form the refined tracking queries, which are sent to the next temporal blocking decoder. The combination of the temporal blocking decoders and IRM avoids the duplicate predictions problem and ensures effective information interaction between historical tracking queries belonging to the same target.

Besides, we insert IRM between every two decoders, which allows multiple interactions between historical queries. After each decoder, each query completes a target detection in the current frame and obtains new features and position estimation. During multiple interactions through IRMs, collaborative tracking queries continuously exchange varied new observations to obtain a more comprehensive description of the target. Collaborative tracking based on multiple information interactions is very important for stable tracking at a low frame rate.

In the existing transformer-based end-to-end MOT methods [30, 6, 16], the bipartite matching loss [8] is adopted to train the network. In our method, for each tracked target, due to the introduction of collaborative tracking queries, predictions of multiple queries belonging to the same target are matched with the same ground truth. This prevents us from directly using the one-to-one matching strategy to calculate the bipartite matching loss. Therefore, we propose a tracking object consistency loss (TOCLoss) to handle the training of the collaborative tracking queries. Then, the bipartite matching loss $\mathcal{L}^{t}_{\mathrm{bip}}$ and the TOCLoss $\mathcal{L}^{t}_{\mathrm{toc}}$ make up the overall training objective $\mathcal{L}$, which is formulated as

$$\mathcal{L}=\sum\nolimits_{t=1}^{T}(\mathcal{L}^{t}_{\mathrm{bip}}+\mathcal{L}^{t}_{\mathrm{toc}})\,.$$

(2)

For collaborative tracking queries $\mathbf{X}^{t}_{i}=\{\mathbf{x}^{t-\hat{t}}_{i}\}_{\hat{t}=1:N_{i}^{t}}$ of one target, we use $\{\hat{\mathbf{y}}^{t-\hat{t}}_{i}\}_{\hat{t}=1:N_{i}^{t}}$ to denote their predictions in the current frame. Each prediction $\hat{\mathbf{y}}^{t-\hat{t}}_{i}$ contains the predicted class probabilities $\hat{\mathbf{p}}^{t-\hat{t}}_{i}$ and box prediction $\hat{\mathbf{b}}^{t-\hat{t}}_{i}$. We use $\psi^{t}$ to denote the identity set of all tracked objects.

As shown in Fig. 2, the tracking predictions $\{\hat{\mathbf{y}}^{t-1}_{i}\}_{i\in\psi^{t}}$ of the latest tracking queries of all tracks are combined with the predictions of the detection queries to participate in the bipartite matching. The mapping $\pi^{t}$ between predictions and ground truth objects is determined either via track identities or costs based on object class and bounding box similarity [16]. Then, we calculate the bipartite matching loss for them by mapping $\pi^{t}$, which is denoted as $\mathcal{L}^{t}_{\mathrm{bip}}$. Since the additional predictions $\{\hat{\mathbf{y}}^{t-\hat{t}}_{i}\}_{\hat{t}=2:N_{i}^{t}}$ of each track are separated from $\mathcal{L}^{t}_{\mathrm{bip}}$, $\mathcal{L}^{t}_{\mathrm{bip}}$ satisfies the one-to-one matching requirement.

As for the remaining predictions $\{\hat{\mathbf{y}}^{t-\hat{t}}_{i}\}_{\hat{t}=2:N_{i}^{t}}$ of the $i^{\mathrm{th}}$ track, since they and $\hat{\mathbf{y}}^{t-1}_{i}$ have the same identity, they can share the mapping $\pi^{t}$ between them and the ground truth objects. Then, we define the loss of each prediction as

$$\mathcal{L}^{t}_{i}(\hat{\mathbf{y}}^{t-\hat{t}}_{i},\pi^{t})\!\!=\!\!\left\{\begin{array}[]{l}\!\!\!\!-\log{\hat{\mathbf{p}}^{t-\hat{t}}_{i}}(\pi^{t}(i))\!+\!\mathcal{L}_{\mathrm{b}}(\hat{\mathbf{b}}^{t-\hat{t}}_{i},\pi^{t})\,\,\,\,\,\,\mathrm{if}\,i\in\pi^{t}\,,\\ \!\!\!\!-\log{\hat{\mathbf{p}}^{t-\hat{t}}_{i}}(0)\quad\quad\quad\quad\quad\quad\quad\ \,\,\,\,\,\,\mathrm{otherwise}\,,\end{array}\right.$$

(3)

where $\hat{\mathbf{p}}^{t-\hat{t}}_{i}(\pi^{t}(i))$ is the predicted probability of the assigned class obtained from mapping $\pi^{t}$. $\hat{\mathbf{p}}^{t-\hat{t}}_{i}(0)$ is probability of background class. $\mathcal{L}_{\mathrm{b}}(\hat{\mathbf{b}}^{t-\hat{t}}_{i},\pi^{t})$ is the bounding box loss [32]. Then, the tracking object consistency loss $\mathcal{L}^{t}_{\mathrm{toc}}$ of the remaining historical tracking queries is formulated as

$$\mathcal{L}^{t}_{\mathrm{toc}}=\frac{\sum_{i\in\psi^{t}}\sum\nolimits_{\hat{t}=2}^{N^{t}_{i}}\mathcal{L}^{t}_{i}(\hat{\mathbf{y}}^{t-\hat{t}}_{i},\pi^{t})}{\sum\nolimits_{t=1}^{T}N^{t}_{\mathrm{his}}}\,,$$

(4)

where $N^{t}_{\mathrm{his}}$ is the number of queries that are assigned ground truth objects by mapping $\pi^{t}(i)$ among the remaining historical tracking queries, which is formulated as

$$N^{t}_{\mathrm{his}}=\sum_{i\in\pi^{t}}\lvert\{\hat{\mathbf{y}}^{t-\hat{t}}_{i}\}_{\hat{t}=2:N_{i}^{t}}\rvert\,.$$

(5)

In the inference stage, for a track, given the predictions of all collaborative tracking queries, only the output $\hat{\mathbf{y}}^{t-1}_{i}$ of query $\mathbf{x}^{t-1}_{i}$ is adopted as the predicted location and score of the target in the current frame. The remaining historical queries provide temporal cues to assist tracking and their final output is ignored. We consider the object to appear when the score is greater than a threshold $\sigma$, otherwise, the object is lost. The queries of lost targets are kept for up to $N_{\mathrm{keep}}$ frames.

Datasets. We evaluate our method on three popular multi-object tracking datasets, including MOT17 [11], Dancetrack [21] and BDD100K [29]. We use the private detection protocol for MOT17. Following  [33, 34, 30], the CrowdHuman dataset [20] is added to build the joint dataset when training on the MOT17 and Dancetrack. MOT17 does not provide a validation set. In the ablation studies part, we use the first half of each video in the training set for training and the last half for validation following [33, 34, 30].

Evaluation Protocols. We follow the standard evaluation protocols including CLEAR metrics (MOTA, IDs) [3], Identity F1 Score (IDF1) [19], and HOTA [15]. MOTA focuses more on the detection performance, while IDF1 focuses more on the association performance. HOTA [15] balances the impact of detection and data association well. Therefore, we take HOTA as the main metric.

Implementation Details. ColTrack extends the DETR-like deformable detection model DINO [32] and takes ResNet50 [13] as the CNN feature extractor. The tracking-by-detection version of the baseline method is represented as Baseline+Bytetrack, which directly trains a detection model and uses Bytetrack [33] to associate objects. The model is trained for 40 epochs. The end-to-end (E2E) version of the baseline method is denoted as Baseline+E2E, which directly takes the tracking results of the last frame as the tracking queries [16, 30] in the current frame. To reduce the GPU memory usage and increase the video clip length during training, we use the detection model trained by Baseline+Bytetrack to initialize the CNN and encoders of Baseline+E2E and ColTrack, whose parameters are frozen. The decoders and 300 learnable query embeddings are trained from scratch. The model is trained 60 epochs on MOT17, 40 epochs on Dancetrack and 20 epochs on BDD100K.

Following [33], the input frames are resized to $1440\times 800$. The data augmentation includes multi-scale training, Mosaic [5] and Mixup [31]. All the models are trained by AdamW algorithm with an initial learning rate of $1\times 10^{-4}$ and weight decay of $1\times 10^{-4}$. The learning rate is scaled by $\times 0.1$ during the last 10 epochs. The batch size is 8 video clips and each has 4 frames. $N_{\mathrm{det}}=300,d_{\mathrm{head}}=8,\sigma=0.6,N_{\mathrm{keep}}=5,N_{\mathrm{max}}=3$. Following [33], we measure FPS with FP16-precision and batch size of 1 on a single V100 GPU.

To obtain videos under different frame rates, following [36], we sample frames at a fixed interval $n$ from the original dataset to obtain low-frame-rate videos. $n$ varies from 1 to 30. The larger $n$, the lower the frame rate, and the larger difference between adjacent frames.

Performance at low frame rates. In Fig. 4(a), we compare the performance of ColTrack with existing methods [33, 34, 30] at different frame rates on MOT17 dataset.

MOTR [30] has stable HOTA when $n$ is less than 6. The HOTA of ColTrack is stable when $n$ is varied from 1 to 10. This is benefited from the deformable attention-based end-to-end architecture, which relies on content features to track the target. This approach has more advantageous in a low-frame-rate situation that has large displacements.

Compared to MOTR, the HOTA performance and frame rate robustness of ColTrack are significantly better, which benefits from the introduction of collaborative tracking queries and the exquisite design of each module. They provide more temporal cues for the model to obtain more comprehensive and accurate descriptions of objects.

For YOLOX+Bytetrack [33] and ReID-based FairMOT [34], the ablation models use the officially provided weight. Both of them show a rapid performance drop when the frame rate is lowered. This is because these methods all rely on the Kalman filter and IOU matching, which are unreliable for fast-moving objects.

Equivalent FPS. The HOTA accuracy of ColTrack when $n=3$ is higher than that of YOLOX+Bytetrack when $n=1$. Therefore, as shown in Table. 1, although ColTrack doesn’t have the highest FPS, it achieves a higher equivalent FPS by reducing the frame rate requirement. This allows it to process a video in a shorter time while ensuring high accuracy. Baseline+E2E has an 11.3 FPS running speed. Comparing the FPS of Baseline+E2E and ColTrack, it can be seen that the calculation overhead caused by the introduction of historical queries and several IRM modules is small. This is because these only slightly affect the calculation of the decoder part, and the calculations of CNN and encoders are not affected.

Verification under various scenarios. Further, in Table. 2, we compare the performances of different methods in videos at varied frame rates on three datasets. Compared with APPTracker[36] devised for MOT in low-frame-rate videos, ColTrack achieves higher performance on both low-frame-rate videos as well as high-frame-rate videos. APPTracker pays more attention to the detection and association of emerging targets and does not fully solve the problems in the follow-up tracking of fast-moving targets.

Compared with Baseline+Bytetrack and Baseline+E2E, ColTrack has a significant performance improvement. The erratic movements and similar appearance of the dancers in Dancetrack make tracking difficult for the classical association method ByteTrack [33] based on Kalman filtering. Meanwhile, the large driving dataset BDD100K with multiple categories contains more fast-moving objects. This makes object tracking very challenging in its low-frame-rate videos. Benefiting from the introduction of collaborative tracking queries, ColTrack still has satisfactory performance when $n$ is varied from 1 to 10 on BDD100K.

In Table. 3, we analyze the impact of each module on the model performance under low frame rates ($n=6$). In BL+E2E+CTQ, although collaborative tracking queries are used, there is no interaction between historical queries of the same track. They can only assist the model training by interacting with queries of other tracks as negative samples through temporal blocking decoders. Therefore, the improvement BL+E2E+CTQ brings is very small.

When IRM is adopted, the performance of the model is significantly improved. This is because IRM enables collaborative tracking queries to interact with each other and refine themselves with temporal clues. We also analyze the impact of the information removal branch and the information addition branch in IRM on the model performance. Experimental results show that having both information removal and information addition capabilities helps the model refine features better.

After adding TOCLoss, the performance of the model is further improved. TOCLoss forces collaborative tracking queries to refine themselves with IRM to better track corresponding targets. Then, they provide better temporal clues during iterative refinement in the following decoders.

In Fig. 4(b) we analyze the effect of the max number of collaborative tracking queries $N_{\mathrm{max}}$ on the tracking performance of ColTrack. The results indicate that ColTrack achieves best performances when $N_{\mathrm{max}}\!=\!3$. A too large $N_{\mathrm{max}}$ results in too many historical features being included, which introduces too much noise.

Analysis of the location and number of IRMs. We also compare the performance of ColTrack with the different number of IRM modules in Fig. 4(c). 6 decoders can add up to 6 IRM modules. We gradually remove the corresponding IRM in the shallow decoder layers. The experimental results indicate that removing the IRM before the first decoder layer and adding an IRM before each of the following decoder layers achieves the best performance. This is because queries have not interacted with the features of the current frame before passing through the first decoding layer, which makes tracking queries unable to adjust the fusion of temporal information according to the current frame. After the first decoder, more IRM modules allow queries to have more opportunities to communicate new observations with each other to gather more valuable information.

In Table. 4, Table. 5 and Table. 6, we compare ColTrack with the state-of-the-art methods on three datasets. MOT17 is a small dataset containing only 7 training videos and 7 testing videos. Although such a small amount of data is difficult to train an end-to-end model to learn temporal relationship modeling, ColTrack still outperforms existing end-to-end methods and achieves comparable performance to existing tracking-by-detection methods. Dancetrack containing 100 videos is a large challenging dataset due to the irregular movements, similar clothing, and severe occlusions of the dancers. ColTrack outperforms all methods on Dancetrack. This is benefited by our introduction of collaborative tracking queries, which provide more abundant descriptions of targets and makes ColTrack less susceptible to similar appearances and occlusions. BDD100K containing 2000 driving videos is also a challenging dataset due to complex scenes and more fast-moving objects. ColTrack also achieves the best performance on BDD100K, especially for the IDF1 metric that focuses more on association performance.

These sufficient experimental results show that on more challenging datasets, our method not only achieves higher performance than existing methods under high frame rates but also better tracks objects in low-frame-rate videos. This fully proves that ColTrack is a frame-rate-insensitive model. It achieves faster processing speed while ensuring high performance at different frame rates.