Towards Frame Rate Agnostic Multi-Object Tracking

Even for applications that require a detailed path of an object, a high perception frequency may not always be necessary. For instance, in large-scale video surveillance databases, it may be sufficient to determine the location of a specific target every few seconds rather than obtaining 25 locations per second. Another example could be in sports analysis, where the movement of players and the ball needs to be tracked. However, not every frame of the video needs to be analyzed, and a lower frame rate may suffice to capture the key moments of the game.

The variation in frame rate results in changes in the motions of objects. The challenge of handling different frame rates ultimately lies in managing different motion-appearance relationships. In some cases, the frame rate may not accurately reflect the motion distribution. For instance, a 25 fps video of a highway may have a similar motion distribution to a 1 fps pedestrian video. In such situations, using motion descriptors instead of relying solely on the frame rate may be more effective. However, in some cases, the motion descriptors may not be meaningful, such as when there are only a few objects with large covariance of motions, and the exact frame rate number may provide precise information. Therefore, we believe that an MOT solution with frame rate intelligence should be developed under two different modes: known frame rate mode and unknown frame rate mode. These modes can address the issues of different motion-appearance relationships and allow for more effective tracking.

Most methods for MOT follow the tracking-by-detection (TBD) paradigm (Wojke et al, 2017; Tang et al, 2017; Milan et al, 2017; Sadeghian et al, 2017; Chu et al, 2017; Xu et al, 2019; Zhou et al, 2020; Zhang et al, 2021). Based on the TBD paradigm, recent MOT methods could be categorized into some different groups.

Motion Tracking. Some methods (Milan et al, 2017; Saleh et al, 2021) focused on motion tracking. Milan et al (2017) proposed to model the target motion states using RNNs and predict target motions in future frames. Saleh et al (2021) proposed a motion model to score the tracklets, and was able to improve the performance by tracklet inpainting.

Complex Feature Learning. Other than motion features, learning appearance features is also an important part of modern MOT methods (Wojke et al, 2017; Sadeghian et al, 2017; Chu et al, 2017). Wojke et al (2017) proposed to borrow techniques from the Re-ID task to aid the MOT task. Sadeghian et al (2017) proposed to encode motion, appearance, and interaction information for data association. Chu et al (2017) proposed to use target-specific trackers to model the appearance of each target.

Joint Detection and Tracking. Recently joint learning of detection and tracking is also a trending branch of MOT (Zhou et al, 2020; Zhang et al, 2021). Zhou et al (2020) proposed a point-based framework for joint detection and tracking. Zhang et al (2021) proposed to fairly balance the detection and tracking part in the joint task. Some other methods focus on the optimization of the association of trajectories.

Association as Graph Partition. While the Hungarian Algorithm is commonly used in online MOT data association, some other graph-based association methods are developed for near-online or offline MOT. For example, Tang et al (2016, 2017); Hornakova et al (2020) developed different graph partition methods for offline data association. Yoon et al (2019) further proposed to add structural constraints on the data association. Hu et al (2020) formulates the data association as a rank-1 tensor approximation problem.

Cross-Camera MOT. Besides single-camera MOT, Wen et al (2017); Ma et al (2021) contributed to the cross-camera MOT problem. The cross-camera MOT problem focuses on building up relationships between different cameras so that identities can be associated among multiple different cameras.

Nevertheless, all these methods have the common assumption of a fixed input frame rate, which makes the methods vulnerable in some industrial scenarios where input frame rate sampling strategies may be various. Our proposed approaches aim to establish a frame rate agnostic MOT solution that is robust to frame rate changing via unified frame rate inferring and encoding mechanisms.

There are mainly two types of MOT training, i.e., with the association in end-to-end training and without association in end-to-end training. Most recent research on joint detection and tracking separates the feature training and the association training. Typically, Wojke et al (2017); Zhang et al (2021) utilized deep models to perform detection and generate appearance features for tracking, and the tracker itself was not data-driven. Some hand-crafted rules were used for target association. These rules might work well in some situations, but would be difficult to be transferred to new environments. Brasó and Leal-Taixé (2020); Li et al (2020); Sadeghian et al (2017); Chu and Ling (2019) presented their methods with association in end-to-end training. However, most methods still had a post-processing stage, with hand-crafted target management. These post-processing steps were not considered in the association training. It might have minor impacts when the streaming frame rate is high and the movements of targets are small, but we found that it actually has large impacts when running a mixed frame rate training. This problem leads to a gap between association training and inference, which is even enlarged in FraMOT. The post-processing including target management and smoothing techniques like Kalman Filter also helps reduce the training difficulties. If we still follow the traditional training scheme and ignore the impacts of the post-processing, we can hardly achieve a promising tracking performance. Therefore, in this paper, we propose a new training scheme that includes the impacts from post-processing into the training stage of association, in a way of periodically updating. We thus make the training and inference environment almost the same and reduce the challenge of mixed frame rate training. Maksai and Fua (2019) also proposed to train a tracker in an iterative way, which aims to enrich the training data by adding wrongly tracked samples of the previous checkpoint. Our method differs from this work in two aspects: first, our method is for the online FraMOT problem and improvements are mainly from low frame rate scenarios, while  Maksai and Fua (2019) works on normal frame rate multiple hypothesis tracking (MHT), which is for a totally different purpose; second, our approach is not enriching the training data but doing the opposite, aiming to reduce the gap and reduce unnecessary challenges in frame rate agnostic tracking. Our work is technically complementary to this work.

CLEAR metrics (Bernardin and Stiefelhagen, 2008), the most popular MOT evaluation metrics up to now, propose to use Multi-Object-Tracking Accuracy (MOTA) as the performance indicator for MOT. MOTA considers the False Positive (FP), False Negative (FN), and IDentity SWitch (IDSW) as three critical terms for MOT. However, the IDSW number is usually far smaller than the FN number, which makes detection quality dominate the task. Later then, the Identity metrics (Ristani et al, 2016) and Higher Order Tracking Accuracy (HOTA) (Luiten et al, 2021) metrics are proposed to solve such a problem. The core metric of Identity metrics, IDF1 score, is designed to evaluate MOT in the multi-camera MOT task. IDF1 basically indicates the largest ratio of consistent tracking. Due to the optimization problem in Identity metrics, IDF1 takes a long time to compute, especially when the video to be evaluated contains thousands of frames and hundreds of objects. HOTA is then proposed to solve these drawbacks of MOTA and IDF1. In the experiment of the HOTA metric, surveys were conducted and it shows that HOTA can reflect the judgments of human beings about tracking quality in a more accurate way.

Although these metrics provide a general definition of tracking quality, it is still difficult to predict the actual performance in some complicated scenarios, e.g., a so-called better tracker may become much worse when the input frame rate changes. Unlike the change of resolution size, changing the frame rate leads to a more complicated setting which usually involves dozens of modifications of the method details. We believe that a robust tracker should have the capability of handling different frame rates. As current evaluation methods are not sufficient for frame rate agnostic MOT evaluation, we attempt to propose new evaluation methods and provide a wider analysis of tracking quality in application scenarios.

There are three different modules in the framework, i.e., Joint Extractor Module (JEM), Association Module (AM), and Trajectory Management Module (TM). The JEM generates the detection results and corresponding appearance feature embeddings from raw images. The AM associates the new detection results with existing trajectories. The TM determines the initiation and termination of all trajectories, making them smoother and handling their status.

The core module of the proposed framework is the Association Module. We design a new Frame Rate Agnostic Association Module with mechanisms to encode frame rate information, providing the capability to handle complex motion-appearance relations of the various frame rates. At the same time, the framework is trained with the proposed Periodic Training Scheme (PTS), which takes all post-processing steps into consideration, providing a simulation of the inference stage environment and thus reducing the gap of data association between training and inference.

Fig. 2 illustrates the overview of the proposed framework. The training pipeline follows the proposed PTS consisting of several training periods, each period contains two stages, i.e., tracking patterns generation stage, and tracking module training stage. Specifically, the tracking patterns generation stage conducts a forward pass using the model checkpoints from the last period and generates the tracking patterns. The tracking patterns include some inference stage information that helps simulate the inference run-time (details in 4.2). In the module training stage, the Association Model takes the output of the JEM and the tracking patterns as input, generates the affinity features, predicts the association scores using the proposed Frame Rate Agnostic Association Module (FAAM), and is supervised by the corresponding association ground-truth signals, as shown in Fig. 3. Especially, during training, instead of passing the input data to the FAAM directly, we design an affinity feature generation module to adjust the affinity features utilizing the generated tracking patterns via pattern matching and fusion. Then the adjusted affinity features will be passed through the FAAM. The FAAM networks utilize the frame rate information to infer the frame rate aware attention and enhance the association prediction. During Inference, the model checkpoints of the last period will be used. The association model only takes the output of the JEM as input, the tracking patterns are no longer needed and the pattern matching and fusion step is removed. The inference pipeline is the same as the tracking patterns generation pipeline.

As mentioned before, one of the challenges that FraMOT brings to us is that the diverse video frame rates enlarge the gap between frame-pair association training and actual inference run-time. The most straightforward solution is adding these inference stage post-processing procedures to the training stage, which is hard to implement since most of these procedures can not be backward propagated. To address this issue, we propose to reflect the inference time post-processing steps during training by considering the tracking patterns in the association network (detailed in Section  4.3.1), which is generated via the Periodic Training Scheme.

Here, the tracking patterns are some records that can reflect the inference time patterns, which depend on the post-processing steps of the tracker. In this paper, the tracking patterns $P_{t}=\{(p^{loc}_{i},p^{pred}_{i},p^{apr}_{i},p^{lvl}_{i})\}$, where $i=1,2,\ldots,N_{t}$, include the location of each tracked target at every frame (denoted by $p^{loc}_{i}$), the Kalman-Filter-predicted movement based on the previous trajectory (denoted by $p^{pred}_{i}$), the cached appearance feature embedding (denoted by $p^{apr}_{i}$), as well as the level index (denoted by $p^{lvl}_{i}$) of the two-stage association strategy because we use a cascade association strategy as some previous works  (Yu et al, 2016; Zhang et al, 2021). Other patterns such as the tracklet occlusion status can also be included if the tracker does have a carefully-designed occlusion management.

The Periodic Training Scheme aims at generating these tracking patterns in a periodic manner considering that the inference stage tracking behaviors are not going to change significantly within a short training period. As shown in Fig. 2, we set up multiple training periods and re-sample the tracking patterns before each period starts. In the tracking pattern generation stage of a new period, we utilize the trained checkpoints of the previous period to build an MOT tracker, conduct tracking on full training videos, and generate the tracking patterns that will be used in the affinity feature generation pipeline (see Section 4.3.1). Then we train all the models using the training data and tracking patterns. When a training period is done, we then have the updated checkpoints to generate tracking patterns for the next period. For the first period, we use a pre-trained joint extractor and an IoU-based hand-crafted association module to generate the patterns instead of using randomly initialized checkpoints. For example, in this paper, we use the pre-trained YOLOX detection framework (Ge et al, 2021) with appearance embedding branch as the JEM, a linear combination of the spatial distances and appearance similarities as the AM, and the commonly used Hungarian Algorithm and Kalman Filter with thresholding strategies for target initiation and termination as the TM (detailed in Section 4.4). For the later periods, we use the proposed Frame Rate Agnostic Association Module (FAAM), and the architecture of JEM and TM remains the same.

The whole pipeline is shown in Procedure 1. $L(\mathcal{X},P_{t}\mathchar 24635\relax\;\mathcal{N}_{t})$ is the overall loss which combines detection losses and id loss from the JEM, and association loss from the AM (see Section  5). $\lambda_{E}$ and $\lambda_{A}$ are learning rates.

The Association Model (AM) takes the detection results and corresponding appearance feature embeddings from the JEM and the tracking patterns (for training only) provided by the PTS as inputs and generates the affinity scores supervised by ground-truth labels. As shown in Fig. 3, there are two parts in the AM, i.e., Affinity Feature Generation Module and Frame Rate Agnostic Association Module.

For online tracking, we adopt a simple but effective pipeline that consists of steps including detection filtering, Kalman-Filter-based movement prediction, and a two-stage association strategy. The target management cycle is stated in Procedure 2. We first filter out background objects with a threshold $\lambda_{low}$, then predict the movement for each tracked target using Kalman-Filter. In the next, we perform two different stages of association, which is a commonly used technique to reduce fragments (Yu et al, 2016; Zhang et al, 2021). For the first stage, only detection results of high confidence (confidence score greater than $\lambda_{high}$) will be put into the matching pool. For the second stage, the remaining unmatched tracked targets will be again put into the matching pool and matched with the less confident detection results (confidence score less than $\lambda_{high}$ but greater than $\lambda_{low}$). The association step will find out the best matching pairs minimizing the sum of pair distances (or maximizing the sum of pair scores). This can be easily calculated by the Hungarian Algorithm. Finally, the remaining unmatched detection results of high confidence (confidence score greater than $\lambda_{high}$) will be appended to the tracked target list. If a target in the tracked target list is unmatched for a long interval $\lambda_{i}$, then it will be dropped from the list.

Table 2 shows the overall results of the recent open-source state-of-the-art MOT methods and our method on the challenging MOT17 and MOT20 datasets, with the aforementioned multiple frame rate simulation. ByteTrack (Zhang et al, 2022), FairMOT (Zhang et al, 2021), and GSDT (Wang et al, 2021) have design strategies using the provided frame rate to control cache length. CenterTrack and TraDeS do not design any frame-rate-relevant procedures so their known frame rate mode results are absent. Our method outperforms all other methods with all metrics in terms of frame rate agnostic tracking. In the known frame rate mode, we are 4.5% and 6.8% higher than the runner-up methods in MOT17 and MOT20 for mHOTA, respectively. For mMOTA, we are 5.6% and 10.7% higher. For mIDF1, we are 6.0% and 6.6% higher. Besides, we have the least Vulnerable Rate (VR), which means our performance is the least influenced by the change of frame rate. In unknown frame rate mode, our method still outperforms other methods. The improvement in this mode is smaller than those of the known frame rate mode due to the difficulty of obtaining accurate motion distribution. We have 2.8% and 5.1% improvement in the term of mHOTA for MOT17 and MOT20, respectively. We also have higher mHOTA and mIDF1 scores in this mode, and the Vulnerable Rate is also lower than other methods. The results show that our proposed method is effective for frame rate agnostic tracking.

Fig. 6 further shows the HOTA and MOTA curves of these methods w.r.t the sampling factor $k$ (larger $k$ means lower frame rate) on MOT20 FraMOT simulation datasets. Among these methods, ByteTrack is the first method making use of the YOLO-X detection baseline, and thus has more accurate detection results and outperforms all other trackers in normal frame rate setting ($k=1$). To compare with ByteTrack fairly, we also develop our method based on YOLO-X baseline. However, ByteTrack does not use any appearance features for tracking, while our method has an extra tracking branch for exploiting appearance features. As can be observed in Fig. 6, our method is slightly lower than ByteTrack at $k\leq 4$ in terms of HOTA, which confirms that appearance features do not help improve and may harm the tracking performance with high frame rates. However, when the frame rates are low (e.g., with larger $k$), the performance of ByteTrack drops significantly due to less reliable motion. At the same time, the performance of our method also drops slowly but is much higher than all compared methods. It shows that the tracking becomes more difficult when the frame rate is lowered, but our method has better capability to handle these complicated scenarios. Nevertheless, the overall performance of our method is the best, which reveals our design of FAAM and PTS training is effective for FraMOT.

Other than the MOTChallenge benchmarks, we also conducted experiments using the recent SOMPT22 dataset (simsek2023sompt22), and the results are presented in the applendix B.

To better understand the proposed methods, we further conduct quantitative experiments on the MOT20 validation set (which is split from the original training set and is not included in the real training set).