PatchTrack: Multiple Object Tracking Using Frame Patches

Object detection concerns locating and/or classifying objects of interest in a single image. As the preliminary to object tracking, there is a close connection between the two. Many popular object detection methods generate detections from hypothesis of object locations, including regional proposals [12, 11, 31, 4] and anchors/object centers [30, 22, 54]. On the other hand, there is an increasing number of object tracking systems that utilize Transformer [39], which has shown success in object detection [5, 25, 56, 6] before. Transformer-based object-detection methods encode the CNN [13] feature of images and decodes learned object queries to obtain detections. Aside from architecture adjustment [25, 56] from the original DETR [5], we also see modification to object queries [6] using image patches to facilitate detection. Inspired by the usage of regional proposal and image patches, our proposed method uses frame patches, which can be considered as our initial guess of track locations and appearance.

One major paradigm in MOT is tracking-by-detection, where the MOT systems [5, 51, 44] first obtain detections for each frame and then associate them across frames to form tracks. Since the object detection is a standalone step in the tracking process, one benefit of tracking-by-detection methods is the flexibility to pair different object detection models [31, 30, 5] with different association strategies, thus be benefited directly from advancement in the area of object detection. On the other hand, the object detection step omits information across frames as each of them is processed separately by the detector.

Object motion and appearance may only be considered as part of the detection association strategy for these methods [51, 34]. For object motion, Kalman filter [43] is one of the most popular algorithm used to propagate detections in previous frame to predict their location in its future frame. Combined with Hungarian algorithm [16] and intersection-over-union (IOU) metrics, it has proven to be an effective tracking mechanism [3]. Object appearance information like Re-ID features [44, 28, 51] are also commonly used as similarity measures.

The other popular paradigm in MOT is joint-detection-and-tracking, where the object detection and object tracking are performed simultaneously [53, 35]. One advantage of joint-detection-and-tracking methods is the accessibility to information across frames. For instance, features of multiple frames can be used at once [53, 35, 45] to facilitate detection and/or tracking. For Transformer-based joint-detection-and-tracking methods, both the encoder and the decoder may take additional information from previous frames to infer predictions of the current frame of interest [35, 24, 50]. Specifically, recent works have introduced track queries [35, 24], which come from the output embeddings when processing previous frames. Depends on the design, the track queries may be decoded to bounding boxes separately from the object queries [35] and matched together to predict new tracks, or processed together to form new tracks directly [24].

PatchTrack is a Transformer-based joint-detection-and-tracking system. The Transformer encoder takes in CNN features of a consecutive frame pair. The Transformer decoder takes queries as input and output bounding boxes. PatchTrack deals with four types of queries: object queries, track queries, patch queries, and patch-track queries. Depending on the source of the queries, the predicted bounding boxes may correspond to either tracks associated with existing tracking IDs or detections that need to be assigned with new tracking IDs.

Object tracking for the first frame $f_{1}$ is equivalent to object detection, where each predicted detection can be arbitrarily assigned to a unique tracking ID to form tracks. Frame $f_{1}$ is sent to the CNN backbone that outputs the corresponding frame feature. This feature is stacked with itself [35] and sent to the Transformer encoder. Since there are no existing tracks to form non-object queries, the Transformer decoder only takes object queries as input and produces embeddings. The output embeddings that result in the non-background bounding boxes are the predicted detection in $f_{1}$, each of which is assigned to a unique tracking ID to form tracks. These embeddings are also used as the track queries for the next frame.

For frame $f_{k}$ ($k>1$), there exists $f_{k-1}$ with a set of tracks $T_{k-1}$. We can propagate these tracks using a motion model and infer tracks in $f_{k}$ (Algorithm 1).

Here we use the Kalman filter [43] as our motion model to predict a set of track candidates for $f_{k}$, namely $\widehat{T}_{k}$. The reason we call them track candidates is because there are several problems if we use them directly as tracks in $f_{k}$. First of all, since the tracks in $\widehat{T}_{k}$ are mapped one-to-one with the ones in $T_{k-1}$, they only include objects that have appeared in $f_{k-1}$. Secondly, although Kalman filter and other motion models have shown effectiveness in many cases [3, 40, 51], their predicted bounding boxes are not accurate enough in terms of locating objects. This is the reason why motion models are often used to process existing tracks, and IoU is introduced to match processed tracks with new detections to form new tracks. In the paradigm of joint-object-detection-and-tracking, our architecture is designed to refine these track candidates to more accurate tracks.

To tackle the above problems, we take inspiration from UP-DETR [6] where its Transformer decoder is pre-trained to detect locations of random image patches using their corresponding CNN features. Our proposed PatchTrack takes patches of frame $f_{k}$ as additional visual information besides the entire $f_{k}$ to perform object tracking. Specifically, for each track candidate $\widehat{t}\in\widehat{T}_{k}$, we crop the frame using its bounding box and send the resulting patch to the CNN backbone to get the corresponding patch feature. We use a fully-connected (FC) layer followed by global average pooling (GAP) to process all patch features to patch queries that align with track queries (Figure 2). Each patch query is added to the track query from the same tracking ID to form a patch-track query. The patch-track queries are sent to the Transformer decoder alone with the initial object queries, both of which are processed jointly. Output embedding decoded from each patch-track query may either correspond to the refined location of the corresponding track candidate, or the background if the object has left $f_{k}$. On the other hand, the embeddings decoded from object queries that result in non-background detections locate new objects entering $f_{k}$, which are assigned with new tracking IDs to form new tracks. All embeddings that contribute tracks in $f_{k}$ form the track queries for $f_{k+1}$ (Figure 2).

To obtain track queries for frame $f_{k+1}$ from the track queries for $f_{k}$, embeddings corresponding to the new detections are added and track queries corresponding to background class are removed (Figure 2). A problem with this mechanism is that it is not robust to long-range tracking: if one object is not successfully detected, it can only be assigned to a new tracking ID when it is detected again, which causes fragmented trajectories. To tackle this problem, we adopt the track re-identification strategy from TrackFormer [24] and store these originally removed patch-track queries to an inactive query set. Queries in this set are included in the list of patch-track queries and sent to the decoder for at most $P$ consecutive frames. If the queries can be decoded to non-background bounding boxes during this process, these queries are re-activated with their original tracking IDs, otherwise they will be removed.

As shown in the model architecture 2, PatchTrack processes a frame pair $f_{k-1}$ and $f_{k}$ iteratively, and there are two steps involved. The first step is performing object detection on $f_{k-1}$ in order to initialize track queries for processing $f_{k}$ later. The second step is to perform object tracking on $f_{k}$ using previously generated track queries. Since the second steps involves detecting new objects, which is the same as the first step, as well as tracking existing object with tracking IDs associated with track queries, we use two set prediction loss [5], one for detection new objects and the other for tracking objects existing in $f_{k-1}$.

Let us denote $T_{k-1}$ and $T_{k}$ as the tracks for $f_{k-1}$ and $f_{k}$ respectively. In the case of detecting new objects, we are looking at any track $t\in T_{k}\setminus T_{k-1}$, which corresponds to new objects in $f_{k}$ but not $f_{k-1}$. We adopt object detection set prediction loss following the matching cost in TransTrack [35] and DETR [5]:

where $\mathcal{L}_{det\_cls}$ is the focal loss [20] between predicted class labels and the ground truth, $\mathcal{L}_{det\_L1}$ and $\mathcal{L}_{det\_IoU}$ are L1 loss and generalized IoU loss [32] between the normalized center and sides of the predicted bounding boxes and ground truth, while $\lambda_{cls}$, $\lambda_{L1}$ and $\lambda_{IoU}$ are their weights respectively. Predictions generated from decoding object queries are compared with the ground truth $t\in T_{k}\setminus T_{k-1}$, so $\mathcal{L}_{det}$ handles new object detection.

Similarly, our object tracking set prediction loss is as follows:

where $\mathcal{L}_{trk\_cls}$, $\mathcal{L}_{trk\_L1}$, and $\mathcal{L}_{trk\_IoU}$ are calculated between predictions generated from decoding patch-track queries and the ground truth $t\in T_{k}\cap T_{k-1}$, so $\mathcal{L}_{trk}$ handles tracking objects in $f_{k-1}$ and predict their new locations in $f_{k}$.

Our final loss function is simply the sum of object detection set prediction loss and object tracking set prediction loss: $\mathcal{L}=\mathcal{L}_{det}+\mathcal{L}_{trk}$.

Given the architecture of PatchTrack (Figure 2), we need two consecutive frames to train the model. Although we could simply take frames pairs, predict track candidates from tracks of the previous frame using Kalman filter [43] as shown in the architecture, Kalman filter would not be able to provide high quality predictions due to high uncertainty in the early stage when there is a lack of prior information, which will in turn degrades the performance of decoder since the patch queries do not serve as good guesses to where existing tracks may be in the current frame.

To simulate the role of Kalman filter [43] and generate track candidates for training, we propose the following augmentation strategy. Given a frame pair $f_{k-1}$ and $f_{k}$. We first randomly shift and reshape each track bounding box in frame $f_{k-1}$ within a pre-defined domain. We ensure that the IoU between each augmented bounding box and the track bounding box in frame $f_{k}$ with the same tracking ID, if exists, is at least 0.5. This is to align with commonly used IoU threshold value in detection association [44, 3, 51]. These augmented tracks are the track candidates to our system during training.

We also adapt the track augmentation strategy from Trackformer [24], where we introduce false negatives by removing some queries associated with tracks that exist in both $f_{k-1}$ and $f_{k}$ from the input. The objective of the system is to detect the corresponding objects as new objects using object queries. On the other end, we sample output embeddings (generated from performing object detection on $f_{k-1}$) that map to background bounding boxes. They are included in the track queries as false positives when performing object tracking on $f_{k}$. To obtain their corresponding patch queries, we get their respective bounding boxes and augment them in the same manner as track candidates generation. We ensure that the IoU of each augmented bounding box is below 0.5 with ground truth tracks in $f_{k}$. For each patch-track queries generated from the above procedure, our system should decode them and get background objects.

Frame pairs are selected from two sources. The first one is video data from MOT benchmarks [26], where we take two video clips within a certain range from each other in the same video. This gives us more variety in terms of camera motion. The second one is image data from CrowdHuman [33], where we augment a single image through random scaling and translating to obtain a frame pair. For each selected frame pair, we perform the aforementioned steps to generate track candidates and modify the ground truth corresponding to false positives/negatives we inserted manually. PatchTrack is optimized towards the modified ground truth during training.

The Kalman filter [43] following a constant velocity model is used to predict track candidates. PatchTrack uses ResNet-50[15] pre-trained on ImageNet [8] as its CNN backbone and Deformable DETR [56] for the Transformer encoder-decoder framework. The number of object queries is set to be 500. Inactive track queries will be kept for 30 frames for track re-birth.

We adopt the training procedure from TransTrack [35] as follows. The optimizer is AdamW with $\beta_{1}=0.9,\beta_{2}=0.999$ and initial learning rate $2\mathrm{e}{-4}$. We use 8 NVIDIA Tesla V100 GPUs with batch size 16. PatchTrack is first pre-trained on CrowdHuman [33] for 150 epochs with the learning rate dropped to $2\mathrm{e}{-5}$ after the first 100 epochs. Then, PatchTrack is trained on both CrowdHuman and MOT17 [26] for another 20 epochs. Lastly, it is evaluated on MOT16 and MOT17 [26] test sets.

The ablation study is performed on the MOT17 [26] validation set. The original MOT17 training set is split to a new training set and validation set, each consisting of the first half and the second half of training videos. After pre-training PatchTrack on CrowdHuman [33], the system is fine-tuned on the both CrowdHuman and the new MOT17 training set and evaluated on the validation set.