Semi-TCL: Semi-Supervised Track Contrastive Representation Learning

Existing separate and joint visual embedding learning methods mostly start from an image-level instance matching problem. That is, they try to learn an embedding function $f(\cdot)$ that maps each image $I$ to a $C$-dimensional vector $f(I)$ with a certain distance metric, which is usually the $\ell_{2}$ distance. Given two images or image crops depicting the appearance of two object instances, $I_{1}$ and $I_{2}$, we expect $f(I_{1})$ and $f(I_{2})$ to have a small distance when they are showing the same object and otherwise a large distance. Traditionally, learning the embedding function is achieved by comparing each image to other images of the same or different object. One can use either a classification loss

or the margin loss [19, 9]

Here $y_{i}$ denotes an instance’s identity label and $\widehat{p}_{y_{i}}$ is the classification output probability of $I_{i}$ being to identity class $y_{i}$ out of the $K$ identity classes. For example in [42], Eq. 2 is used to classify one detected instance to $K$ potential classes, the annotations of which are obtain by labeling all tracks in all videos across all training datasets.

Now consider the case of using the learned visual embedding in online tracking. At each time-step $t$, a newly detected object instance needs to be matched to a set of existing tracks. But each track $\mathbf{T}_{j}$ usually contains multiple instances of the tracked object accumulated over time. An additional aggregation function $G(\cdot)$ has to be introduced to make this matching possible. Thus the matching is actually between $f(I_{i})$ and the aggregate track-level $G(I_{j}^{t_{0}},\ldots,I_{j}^{t-1})$, where $I_{j}^{t}$ denotes the instance of the object depicted by track $T_{j}$ at time $t$. The added aggregation function is apparently not addressed in the original learning objective of image level matching, as in Eq. 2 or Eq. 3. Thus, using the visual embedding learned by either one for the matching in online track could be sub-optimal.

To address this discrepancy, our learning objective should be directly built on the aforementioned instance-to-track matching task. Formally, for a temporally ordered set of object instances $T_{j}=\{I_{j}^{0},\ldots,I_{j}^{n}\}$ that belong to the same object $j$, we defined the aggregation function $G(\cdot)$ that maps the set of features $\{f(I_{j}^{0}),\ldots,f(I_{j}^{n})\}$ to a single vector $\mathbf{g}_{j}$. We learn the embedding function $f(\cdot)$ and the aggregation function $G(\cdot)$ so that the object to track distance

is small when $I_{i}$ and $T_{j}$ are depicting the same object and large otherwise. Explicitly incorporating the aggregation function into the learning objective has two advantages: 1) it makes the learning objective close to the actual tracking scenario, which enables the embedding learning to benefit from the temporal information in videos; 2) as we shall see later, it make it easier to extend the learning objective to videos with partial or without track level annotations.

Given one object instance $I_{i}$, there is a track $T_{i}$ that this instance belongs to. $T_{i}$ contains multiple instances $G(I_{i}^{0},\ldots,I_{i}^{l})$, where $l$ is the length of the track. We can generate random sub-tracks $S(i)=\{\tilde{T}_{i}^{j}\}$ of $T_{i}$ by sampling random subsets of instances $T_{i}$ . These sub-tracks resemble the actual partial tracks occur in online tracking. That is, at a given time-step during online tracking, we can only observe a portion of the complete track that have already been shown in the video. For a batch of input videos $V$, we can sample a set of object instances $\mathbf{I}=\{I_{0},\ldots,I_{N}\}$ belonging to their corresponding sub-tracks $\mathbf{\tilde{T}}=\{\tilde{T}_{0},\ldots,\tilde{T}_{L}\}$. With these $N$ instances and $L$ subtracks, we can implement the instance-to-track matching objective in the contrastive loss form

Here $S(i)$ denotes all sub-tracks that are sampled from the tracks that $I_{i}$ belongs to. $\mathbf{\tilde{g}}_{j}$ is the aggregated visual feature of a sub-track $\tilde{T}_{j}$. We assume the feature vectors are all $\ell_{2}$normalized and the temperature parameter $\tau$ controls the scaling of the cosine similarities between vectors. We use $\tau=0.07$ following the general practices for contrastive losses.

We call the proposed method of learning visual features in a tracker with Eq. 5 as tracklet contrastive learning (TCL). Compared with instance-level contrastive learning [38, 13, 11] which compares one image to another image, the instance to track loss has two different concepts in the comparison: the object instances and the sub-tracks. Because this type of comparison is close to the actual use case in tracking, we expect the learned visual features to be more suitable to the ReID task during online tracking. In this work, we use a simple aggregation function $G$ that averages all input feature vectors, which we empirically found to give satisfying visual embeddings. But TCL does not inhibit the uses of more advanced aggregation functions which could be developed in future.

Learning with the instance-to-track matching objective also enables us to extend the learning task to videos without human annotated track labels. In Eq. 5, we notice that only the sampled sub-tracks, instead of the complete tracks, are used in training. On the other hand, when we apply a certain primitive multiple object tracker that relies on motion prediction to videos without track-level annotations, we can obtain a large amount of potentially incomplete tracks. The generation of these incomplete tracks can be viewed as sampling sub-tracks from the complete, annotated tracks, which have not been annotated. This means the seemingly unusable unlabelled videos have now become a potential source for mining useful sub-tracks in TCL. In particular, for videos with no track annotation, we can apply a motion prediction based tracker [29] and obtain a set of predicted tracks. These tracks are treated as the pseudo labels of these videos. We can then train our tracker using these pseudo-labeled videos together with the annotated videos.

Formally, we obtain a track annotated video set $\mathbf{V}_{A}$ and an unlabeled video set $\mathbf{V}_{U}$ for learning with Semi-TCL. Usually the unlabeled video set is much larger than the labeled set but may contain segments that has very few objects of interest, thus has less value for learning. We apply a primitive tracker such as [29] on $\mathbf{V}_{U}$ to obtain predicted tracks for each video. Then we rank the unlabeled videos in $\mathbf{V}_{U}$ by the number of produced tracks in them. To mine potentially useful videos for Semi-TCL, we simply take the top-$K$ videos based on tracklet density in the rank and produce a refined video set $\mathbf{V}_{R}$. This $\mathbf{V}_{R}$ is used together with $\mathbf{V}_{L}$ in training. We split each video in both $\mathbf{V}_{A}$ and $\mathbf{V}_{R}$ into segments of $32$ consecutive frames. We randomly sample $2$ segments from $\mathbf{V}_{A}$ and $2$ segments from $\mathbf{V}_{R}$ in each training step to form one training mini-batch. From these $4$ segments, we can obtain $M$ tracks, either annotated or produced by the primitive tracker. We perform another round of sampling on these tracks so that for each track we can obtain $3$ sub-tracks, meaning $L=3M$. This ensures that each instance is exposed to multiple sub-tracks of the same track. We extract $N$ object instances from these sampled sub-tracks. These samples are then used for calculating the loss in Eq. 5. This process is illustrated in Fig. 2. The loss function in Eq. 5 is differentiable and easy to empirically optimize. Thus models with Semi-TCL can be learned with backprogagation in an end-to-end manner.

In the Semi-TCL experiments, three types of dataset are used. They are image detection dataset for pre-training, labeled video tracking dataset for supervised joint tracking and embedding learning, unlabeled video dataset for Semi-supervised learning.

We report IDF1, MOTA, MT, ML, IDS on MOT series test benchmarks. Among the metrics, we prioritize IDF1 and MOTA over other metrics as it corresponds closely with the embedding learning. On the test benchmarks, we report our results on the private session based on our results obtained from the MOT challenge server. On our ablation studies, we report IDF1, MOTA and IDS to compare the impact of different components.

A 8 NVIDIA Tesla V-100 GPUs machine is used to train the Semi-TCL models. We select 144 as our batch size and the starting learning rate is 1e-3. Person detection dataset is first employed as pre-training, and then Semi-TCL training is conducted on the joint set of labeled and unlabeled videos. We train the Semi-TCL model for 200 epochs before dropping learning rate to 1e-4, and for another 20 epochs until the training fully converges. For the unlabeled video, we use Center Track [29] preprocessed 20k videos from AVA-Kinetics and 15 MEVA videos, tracking threshold is set to be 0.3 to process all the videos. From the 20k processed AVA videos, we select 100/200/300 videos based on a tracklet density based mining strategy. To make sure unlabeled data not dominate the training, we applied a balanced sampling strategy based on the method in 3.3.

Semi-TCL is trained on the joint labeled and unlabeled video dataset, while tested on MOT15, MOT16, MOT17, MOT20 benchmarks. With the MOT benchmarks test annotations unavailable, we submit our test prediction results to the MOT server and obtain our benchmarks results. Table 1 shows the benchmark results of Semi-TCL as well as other SOTA approaches. Since our work focuses on ReID embedding learning for tracking, the primary metric for us is the IDF1. Based on Table 1, our method consistently outperforms the other the state of the art approaches in all MOT benchmarks. Specially, on MOT16 and MOT17, Semi-TCL is able to have 1% and 1.1% increase under the IDF1 metric. On MOT20 where the dataset tends to have very crowded scenes and ReID is highly relied to match trackletd, our method improves the SOTA IDF1 score from 67.5% to 70.1% with a 2.5% improvement. It is also worth noticing that, in all four MOT benchmarks, we have the best score in IDF1, which highlights the quality of the ReID embedding. The comparison of the test results with other SOTA approaches shows the superiority and robustness of Semi-TCL.

As the core component of this work, TCL based a instance to tracklet matching scheme instead of the widely used instance to instance matching during contrastive pair building. To show the effectiveness of the work, we start the ablation study with the comparison of TCL with other instance matching based approach. All the comparison experiments are using half of MOT17 as labeled tracking training data and the other half for validation.

Impact of batch size on training Larger batch sizes tend to be useful in image embedding contrastive learning tasks[4]. We would like to see if this also holds in the scenario of tracking embedding learning. We use 3 batch sizes for comparison, 32, 96, 144. We keep the training setting same as the main experiment and only use batch size as variable. Evaluation results can be found in 3. We find that while increasing the batch size from 32 to 96 and 144, the MOTA and IDF1 have a 0.7%, 2.7% and 1.1%,1.5% improvement respectively. This means larger batch sizes, or more contrastive learning pairs, are helpful to the tracking embedding learning.

Pre-training comparison Static image pre-training is proved to be useful in the joint tracking and ReID learning [42]. We also applied contrastive learning based pre-training in our approach. To see whether CE or contrastive learning can have better pre-training quality, we use Crowdhuman dataset [26] for training and evaluation is conducted on MOT17 validation. Table 3 demonstrates the benchmark results, we can see the SCL based approach can outperform the CE based approach significantly in IDF1 with a 5.4% gap. In MOTA the SCL is behind for 1.8%, which shows that the SCL help learned better embedding quality.

We demonstrate qualitative results of the Semi-TCL on MOT test samples. In Figure 4(b), we show a positive sample in the first row and two error samples in second and third row. In the first row, we find the person with track #255 can be correctly re-identified after being occluded for one frame. In the second row, the region is extremely blurred which deteriorates the visual repsentation quality. As a result, Track #1452 is first assigned to a person in black coat then matched with a person in yellow. Example in the third row shows a case where a person is occluded for a extended period of time and thus can not be correctly associated with his previous track. The error sampls shows though we have achieved good improvement in tracking accuracy, there still exist several challenging situations that remains to be tackled in future research works.