PD-SORT: Occlusion-Robust Multi-Object Tracking Using Pseudo-Depth Cues

To achieve reliable association, most MOT methods that follow the TBD paradigm leverage the target’s motion consistency [6, 7, 8, 23, 9, 10]. The pioneering work SORT [6] employs the Kalman filter (KF) [16] to model the target motion: at the beginning of each frame, the motion states of the targets are predicted by the KF using the linear motion assumption. Then, the IoU similarities between the predictions and the detections are calculated and used in the cost matrix for matching by the Hungarian algorithm [24]. After being successfully matched, the corresponding new detections are used to update the tracklets’ KF parameters. This association pipeline of SORT is followed and improved by later TBD methods [8, 9, 10]. To alleviate the high ID switch of SORT under occlusion, DeepSORT [8] introduces ReID-based appearance similarity in the cost matrix. Also, it proposes an association strategy that prioritizes the tracklets with more recent successful associations. To effectively integrate appearance cues, SAT [25] explores a deep Siamese network to extract instance-level appearance features. The obtained features are then used for similarity computation in the association stage. Besides, appearance features extracted by deep appearance models [26, 27] provide effective discriminating cues that benefit tracking quality, which are exploited by later works [28, 29, 30, 31]. To realize more reliable association, BoT-SORT [18] modifies the KF model and uses the camera motion compensation technique to generate more accurate KF predictions while combining motion and appearance cues. Due to factors like occlusion and motion blur, low-confidence detections can also indicate the existence of targets. However, both SORT and DeepSORT perform associations for high-confidence detection results only. Therefore, ByteTrack [9] proposes a new matching cascade strategy: Once high-confidence detections have been matched, low-confidence detections and tracklets not matched with high-confidence detections are also matched. By considering all the detections, ByteTrack effectively improves the association performance of the SORT-like method. But it still has limitations when dealing with nonlinear motions and occlusions. When interruptions happen, the parameters of the Kalman filter cannot be updated due to the absence of new observations. And the KF prediction error will accumulate over time. On the other hand, the error of the observations (detections) depends on the detector, which is stable and smaller than the KF errors. Therefore, OC-SORT [10] uses the tracklets’ historical observations to compute the velocity-direction consistency with the new detections as well as to recover the interrupted tracklets. Also, after the target is reappeared, the observations before and after interruption are used to interpolate a virtual trajectory, which is then used to update the KF. Generally, the main challenge of TBD methods is the association under complex scenes, including dense objects, heavy occlusions, and nonlinear motions.

In instance-level object identification tasks, effectively leveraging scene context information can enhance the model’s ability to distinguish targets [32]. For the MOT task, exploring richer scene context can contribute to more robust object association. As an effective form of spatial context, depth information can refine the motion modeling of targets, thereby improving the tracker’s localization and discrimination capabilities. In 3D MOT, AB3DMOT [33] obtains detections with depth information from a LiDAR point cloud and extends the KF to be 3D. CenterPoint [34] detects object centers using a keypoint detector and estimates attributes like 3D size, orientation, and velocity. It refines these estimates using point features and simplifies the tracking to greedy closest-point matching. To obtain a comprehensive understanding of the scene, EagerMOT [35] fuses object observations from both 3D and 2D object detectors. However, in mobile device applications (e.g., smartphones), deploying depth sensors brings additional costs. Meanwhile, actual depth data obtained from depth sensors is often limited by their perception range, resulting in reduced tracking performance for distant targets. In fact, as a projection of the 3D scene, a 2D image also implies certain depth information. In 2D MOT, previous work have attempted to enhance tracking performance by incorporating pseudo-depth extracted from the image signal [36, 37, 15]. QuoVadis [36] combines the 2D detector with a monocular depth estimator and a segmentation network to achieve trajectory forecasting from a Bird’s-Eye View (BEV). However, this method has a high model complexity. On the other hand, DP-MOT [37] uses a geometry-based approach to estimate the depth for detected objects. Then, tracking is performed by joint use of the depth-aware motion cue and the appearance cue. Similarly, SparseTrack [15] proposes a projection rule-based method for obtaining the relative depth of targets from 2D images, which does not require training any additional networks. Based on this pseudo-depth, the tracklets and detections are divided into subsets. Eventually, cascaded matching is performed on tracklets and detections that are at the same depth level. However, the aforementioned 2D MOT methods treat pseudo-depth as an auxiliary cue for constructing BEV, complementing it with appearance features, or partitioning object subsets. In contrast, we propose to integrate pseudo-depth directly into the target’s motion model as a reliable motion state, aiming at enhancing the tracker’s robustness in complex scenarios with dense occlusions.

In 2D MOT, the robustness of association relies on the estimation of the object’s position, which is highly susceptible to nonlinear motions and occlusions. On the other hand, by expanding the spatial information of the object, 3D tracking that includes depth information can effectively improve the accuracy of object localization and robustness to occlusions. Meanwhile, the effectiveness of projection-based pseudo-depth in MOT tasks has been verified in previous work [15]. However, to the best of our knowledge, there’s no work that incorporates pseudo-depth as a state into the motion model for pure motion-based 2D MOT. A key challenge lies in maintaining the accuracy of depth estimation when handling difficult targets like boundary objects. Reliable pseudo-depth estimation is essential, as it underpins the effectiveness of subsequent similarity computation modules. Moreover, appropriate pseudo-depth-based motion states to be integrated into the Kalman filter are required to ensure the discrimination ability of the motion predictions.

This revelation leads us to extend the MOT motion model by introducing pseudo-depth and its velocity, which in turn extends the 2D MOT to 3D for better processing. For the definition of pseudo-depth, as in SparseTrack [15], we first used the projection of depth given by the distance from the target bounding box to the bottom of the image view. Such projection-based pseudo-depth estimation relies on the assumptions that the image capture device is above the ground plane and all objects in the scene are on the same plane. In practical tracking applications in terms of mobile device capturing, pedestrian monitoring, and in-car camera sensing, these assumptions are typically satisfied, enabling pseudo-depth estimation to provide effective guidance. However, considering that the target bounding box may move to the boundary of the view during the tracking process, the pseudo-depth of the object may become a negative value or zero, which cannot correctly reflect the depth of the target for modules using depth values directly, influencing the subsequent pseudo-depth-based calculation.

Therefore, we propose a novel pseudo-depth based on the complementary view. By expanding a complementary view of the same size and below the real image view, we define the pseudo-depth as the distance from the bottom of the target bounding box to the bottom of the complementary view, and our pseudo-depth $pd$ is computed as in Eq. 1.

Here, ${IMG}_{h}$ is the height of the real view, $Y_{b}$ is the coordinate value of the bottom of the target bounding box along the y-axis. The visualization of the ground plane real depth $depth$ and our pseudo-depth $pd$ is shown in Fig. 4.

For objects whose size is within the real view, such pseudo-depth using complementary view can correctly reflect the depth information.

Based on the proposed pseudo-depth, we extend the standard KF in SORT with two additional states: the target’s pseudo depth $pd$ and its velocity component $v_{pd}$. The standard Kalman filter states in SORT are shown in Eq. 2.

Here, $(x_{c},\ y_{c})$ is the coordinate of the target’s bounding box center, s and r are the area and aspect ratio of the target’s bounding box. $v_{x},\ v_{y},\ v_{s}$ are the velocity components for $x_{c},\ y_{c},\ s$ respectively. By introducing two new states, $pd$ and $v_{pd}$, the KF state is revised to be as in Eq. 3.

To utilize the depth information in location consistency evaluation, we extend the 2D IoU similarity to 3D by introducing the concept of depth volume. Given two object observations $b^{1}=(x_{1}^{1},\ y_{1}^{1},\ x_{2}^{1},\ y_{2}^{1},\ {pd}^{1})$ and $b^{2}=(x_{1}^{2},\ y_{1}^{2},\ x_{2}^{2},\ y_{2}^{2},\ {pd}^{2})$, where $(x_{1}^{1/2},\ y_{1}^{1/2})$, $(x_{2}^{1/2},\ y_{2}^{1/2})$, and $pd^{1/2}$ represent the top-left corner, bottom right corner, and the pseudo-depth, respectively. We give the definition of the depth volume of the intersection between the two objects, $V^{\text{inter}}$, as in Eq. 4.

Here, $w^{inter}$ and $h^{inter}$ are the width and height of the intersection box area. Meanwhile, we define the pseudo-depth of the intersection, $pd^{inter}$, as the smaller value of the pseudo-depths of the two objects. Similarly, we can obtain the depth volumes of two objects, $V^{1}$ and $V^{2}$, as in Eq. 5.

Furthermore, to achieve more robust distinguishing between objects, we introduce depth volume IoU (DVIoU) by using the volume metric, as shown in Eq. 6.

The comparison between standard IoU and DVIoU is visually represented in Fig. 5. By integrating the depth to modulate the IoU similarity, not only the robustness of target location consistency measurement is improved, but also the extra discrimination information provided by the depth cue benefits the overall association accuracy.

Occlusions can harm the reliability of the pseudo-depth, which in turn leads to a decrease in tracking accuracy. On the other hand, in successive frames, the relative depth of the object with respect to other objects fluctuate only in narrow intervals. Therefore, we propose a quantized pseudo-depth cost to better utilize the pseudo-depth to guide the association.

For each frame, we find the minimum pseudo-depth value of all detected objects in the frame $pd_{min}$ and the maximum value $pd_{max}$. Then, divide the interval $[pd_{min},\ pd_{max}]$ into $interval_{num}$ sub-intervals uniformly; each sub-interval is assigned with an interval depth (in this paper, the interval depth is defined as the upper limit of the sub-interval after min-max normalization). After that, the interval depths are assigned to the objects according to the sub-intervals they are in. The interval depth for the $i^{th}$ $(i=0,\ 1,\ …,\ {interval}_{num}-1)$ sub-interval is computed as in Eq. 7.

Next, the interval depth is computed for the last historical observation of each tracklet in the same manner. Finally, the quantized pseudo-depth cost $C_{QPD}$ is computed as the absolute difference between the interval depths of the new detections, $interdepth_{dets}$, and the interval depths of the tracklets, $interdepth_{tracks}$, as shown in Eq. 8.

Then the pseudo-depth difference between the tracklets and new detections can be evaluated by their interval depth values. Compared to directly calculating the difference in pseudo-depth, using the proposed interval depth between detections and tracklets reduces the depth estimation error caused by partial occlusions, thus improving the robustness of pseudo-depth utilization. Meanwhile, interval depth-based cost computation helps to alleviate the association error caused by the velocity direction consistency evaluation when the object is steering, which further improves the algorithm’s performance against nonlinear motions. Finally, the pseudo-code of the QPDM algorithm is given in Algorithm 1.

In our association method, the motion information is consisting of 3 parts: the DVIoU similarity, the OCM velocity-direction consistency, and the quantized pseudo-depth loss. Among them, both DVIoU and OCM are sensitive to the position information of the target. For example, for the DVIoU, the depth volume is the product of pseudo-depth and 2D box area. Here, the pseudo-depth is a relative position information robust to camera motion, but the 2D bounding box overlap is sensitive to position drift. Once the position of either the previous observation or the current detection drifts, the overlap area will change largely and can lead to incorrect association. Meanwhile, OCM relies on the center point coordinates of historical observations to calculate the velocity direction, which is also sensitive to the offset of the target center point. Thus, the accuracy of the target position is essential for association quality.

However, when the camera moves, the position of the target in the view will also shift, which affects the association result. To this end, we introduce CMC before KF’s prediction step for more robust tracklet-detection association in the coming frame. Specifically, we use the OpenCV [38] implementation of the Video Stabilization module with affine transformation to generate transforms using key point extraction [39], sparse optical flow [40], and RANSAC [41], as in previous work [18]. Given a scale and rotation matrix $M\in R^{2\times 2}$ and a translation $T\in R^{2\times 1}$, we correct the camera motion of the KF state and the target historical observation as follows.

For new detections in each frame, OC-SORT performs a two-stage association: the first stage of regular association using the IoU and the velocity consistency (OCM), followed by a second stage to recover the lost tracklets using the IoU only (OCR). PD-SORT follows the association flow of OC-SORT and additionally adds pseudo-depth cues to the associations. First, the QPDM module, which directly leverages pseudo-depth, is introduced into the regular association. Meanwhile, the conventional IoU similarities used in both rounds of associations are replaced with the proposed DVIoU, which also uses pseudo-depth. Eventually, the composition of the final cost matrix is shown in Eq. 11.

Here, $C_{DVIoU}$ is the opposite of the DVIoU between KF predictions and the detections. $C_{QPD}$ is the QPDM cost. $C_{OCM}$ is inherent from OC-SORT, which is the velocity direction consistency difference between historical observations and new detections. $\lambda_{1}$ and $\lambda_{2}$ are two weighting factors. The detailed pseudo-code for PD-SORT is shown in Algorithm 2.

We compare our PD-SORT with state-of-the-art trackers on the test sets of DanceTrack, MOT17, and MOT20, as shown in Tables I, II, and III, respectively. Note that all of the test results are evaluated on official websites.

The performance comparisons between the classical 2D tracker (OC-SORT) and our proposed approach (PD-SORT) utilizing pseudo-depth on DanceTrack are shown in Fig. 7. From the visualized results, our method can handle identity consistency problems well in challenging scenes with occlusions and nonlinear object motions, thus leading to a robust association. Specifically, PD-SORT can handle three typical kinds of occlusion-induced ID problems, namely the ID replacement of the foreground object by the occluded object, the ID reinitialization of the occluded object after reappearance, and the ID swap of objects under occlusion and trajectory intersection. In such cases, the depth of the object provides discriminative information that fixes the association failure of pure 2D information.

Our experiments reveal several limitations of PD-SORT. One concern is its association ability against long-term occlusion. In such cases, if the occluded object is in quick motion, the motion consistency of the object can fail to match the reappeared object’s previous trajectory. This is a common problem with motion-based MOT trackers. To solve such problems, incorporating appearance models or using learnable association matchers can be effective. Another concern is that our projection-based pseudo-depth estimation is performed at the instance level, without generating a full depth map of the entire image. This limits the full use of depth information. Besides, in highly crowded environments, the presence of numerous targets with similar pseudo-depth values and significant overlap between objects can reduce the discriminative power of pseudo-depth cues. Similarly, rapid motion changes will challenge the tracker’s ability to maintain accurate pseudo-depth estimates, potentially affecting association precision. Exploring network-based depth estimators and incorporating context-aware techniques could be potential solutions for the above issues. In addition, although our method performs well on the HOTA metric, the performance gain on the MOTA metric is not significant and even has a slightly lower MOTA than the baseline on the MOT20 test set. This may be due to the missing of low-confidence detection results, which may be solved using an adaptive detection threshold strategy. Future work is needed to incorporate appearance cues and develop more comprehensive strategies to exploit all possible targets.