Unsupervised Lightweight Single Object Tracking with UHP-SOT++

Two representative single object trackers are reviewed and compared here. They are the DCF tracker and the siamese network tracker as shown in Fig. 1. The former is an unsupervised one while the latter is a supervised one based on DL. Both of them conduct template matching within a search region to generate the response map for object location. The matched template in the next frame is centered at the location that has the highest response.

In the DCF, the template size is the same as that of the search region so that the Fast Fourier Transform (FFT) could be used to speed up the correlation process. To learn the template, a DCF uses the initial object patch to obtain a linear template via regression in the Fourier domain:

where $\mathbf{f}$ is the template to be determined, $\mathbf{x}\in\mathbb{R}^{N_{x}\times N_{y}\times D}$ is the spatial map of $D$ features extracted from the object patch, $*$ is the feature-wise spatial convolution, and $\mathbf{y}\in\mathbb{R}^{N_{x}\times N_{y}}$ is a centered Gaussian-shaped map that serves as the regression label. Templates in DCFs tend to contain some background information. Furthermore, there exists boundary distortion caused by the 2D Fourier transform. To alleviate these side effects, it is often to weigh the template with a window function to suppress background and image discontinuity.

In contrast, the template size in a siamese networks is more flexible and significantly smaller than the search region. Usually, the object box of the first frame serves as the template for the search in all later frames. The correlation is typically implemented by the convolution which runs fast on GPU. The shape and size of the predicted object bounding box are determined by the regional proposal network inside the siamese network.

STRCF is a DCF-based tracker. It has an improved regression objective function using spatial-temporal regularization. The template is initialized at the first frame. Suppose that the object appearance at frame $t$ is modeled by a template, denoted by $\mathbf{f}_{t}$, which will be used for similarity matching at frame $(t+1)$. By modifying Eq. (1), STRCF updates its template at frame $t$ by solving the following regression equation:

where $\mathbf{w}$ is the spatial weight on the template, $\mathbf{f}_{t-1}$ is the template obtained from time $t-1$, and $\mu$ is a constant regularization coefficient. We can interpret the three terms in Eq. (2) as follows. The first term is the standard regression objective function of a DCF. The second term imposes the spatial regularization. It gives more weights to features in the center region of a template in the matching process. The third term imposes temporal regularization for smooth appearance change.

To search for the box in frame $(t+1)$, STRCF correlates template $\mathbf{f}_{t}$ with the search region and determines the new box location by finding the location that gives the highest response. Although STRCF can model the appearance change for general sequences, it suffers from overfitting. That is, it is not able to adapt to largely deformed objects quickly. Furthermore, it cannot recover from tracking loss. The template model, $\mathbf{f}$, is updated at every frame with a fixed regularization coefficient, $\mu$, in standard STRCF.

Our UHP-SOT++ adopts STRCF as a building module. To address the above-mentioned shortcomings, we have some modification in our implementation. First, we skip updating $\mathbf{f}$ if no obvious motion is observed. Second, a smaller $\mu$ is used when all modules agree with each other in prediction so that $\mathbf{f}$ can adapt to the new appearance of largely deformed objects faster.

There are three main challenges in SOT:

1. 1.

   significant change of object appearance,

2. 2.

   loss of tracking,

3. 3.

   rapid variation of object’s location and/or shape.

We propose a new tracker, UHP-SOT++, to address these challenges, As shown in Fig. 2, it consists of three modules:

1. 1.

   appearance model update,

2. 2.

   background motion modeling,

3. 3.

   trajectory-based box prediction.

UHP-SOT++ follows the classic tracking-by-detection paradigm where the object is detected within a region centered at its last predicted location at each frame. The histogram of gradients (HOG) features as well as the color name (CN) [39] features are extracted to yield the feature map. We choose the STRCF tracker [32] as the baseline because of its efficient and effective appearance modeling and update. Yet, STRCF cannot handle the second and the third challenges well because it only focuses on the modeling of object appearance which could vary a lot across different frames. Generally, the high variety of object appearance is difficult to capture using a single model. Thus, we propose the second and the third modules in UHP-SOT++ to enhance its tracking accuracy. UHP-SOT++ operates in the following fashion. The baseline tracker gets initialized at the first frame. For the following frames, UHP-SOT++ gets proposals from all three modules and merges them into the final prediction based on a fusion strategy.

The STRCF tracker was already discussed in Sec. II-B. For the rest of this section, we will examine the background motion modeling module and the trajectory-based box prediction module in Secs. III-B and III-C, respectively. Finally, we will elaborate on the fusion strategy in Sec. III-D. Note that the fusion strategies of UHP-SOT and UHP-SOT++ are completely different.

We decompose the pixel displacement between adjacent frames (also called optical flow) into two types: object motion and background motion. Background motion is usually simpler, and it may be fit by a parametric model. Background motion estimation [40, 41] finds applications in video stabilization, coding and visual tracking. Here, we propose a 6-parameter model in form of

where $(x_{t+1},y_{t+1})$ and $(x_{t},y_{t})$ are corresponding background points in frames $(t+1)$ and $t$, respectively, and $\alpha_{i}$ and $\beta_{i}$, $i=0,1,2$ are model parameters. With more than three pairs of corresponding points, we can determine the model parameters using the linear least-squares method. Usually, we choose a few salient points (e.g., corners) to build the correspondence. We apply the background model to the grayscale image $I_{t}(x,y)$ of frame $t$ to find the estimated $\hat{I}_{t+1}(x,y)$ of frame $(t+1)$. Then, we can compute the difference map $\Delta I$:

which is expected to have small and large absolute values in the background and foreground regions, respectively. Thus, we can determine potential object locations. While DCF trackers exploit foreground correlation to locate the object, background modeling uses background correlation to eliminate background influence in object tracking. They complement each other. DCF trackers cannot recover from tracking loss easily since it does not have a global view of the scene. In contrast, our background modeling can find potential object locations by removing the background.

Given predicted box centers of the object of the last $N$ frames, $\{(x_{t-N},y_{t-N}),\cdots,(x_{t-1},y_{t-1})\}$, we calculate $N-1$ displacement vectors $\{(\Delta x_{t-N+1},\Delta y_{t-N+1}),\cdots,(\Delta x_{t-1},\Delta y_{t-1})\}$ and apply the principal component analysis (PCA) to them. To predict the displacement at frame $t$, we fit the first principal component using a line and set the second principal component to zero to remove noise. Then, the center location of the box at frame $t$ can be written as

Similarly, we can estimate the width and the height of the box at frame $t$, denoted by $(\hat{w}_{t},\hat{h}_{t})$. Typically, the physical motion of an object has an inertia in motion trajectory and its size, and the box prediction process attempts to maintain the inertia. It contributes to better tracking performance in two ways. First, it removes small fluctuation of the box in its location and size. Second, when there is a rapid deformation of the target object, the appearance model alone cannot capture the shape change effectively. In contrast, the combination of background motion modeling and the trajectory-based box prediction can offer a more satisfactory solution. For example, Fig. 3, shows a frame of the diving sequence in the upper-left subfigure, where the green and the magenta boxes are the ground truth and the result of UHP-SOT++, respectively. Although a DCF tracker can detect the size change by comparing correlation scores at five image resolutions, it cannot estimate the aspect ratio change properly. In contrast, as shown in the lower-left subfigure, the residual image after background removal in UHP-SOT++ reveals the object shape. By summing up absolute pixel values of the residual image horizontally and vertically and using a threshold to determine two ends of the box, we have

Note that raw estimates may not be stable across different frames. Estimates that deviate much from the trajectory of $(\Delta w_{t},\Delta h_{t})$ are rejected to yield a robust and deformable box proposal.

We have three box proposals for the target object at frame $t$: 1) $B_{\mathrm{app}}$ from the baseline STRCF tracker to capture appearance change, 2) $B_{\mathrm{bgd}}$ from the background motion predictor to eliminate unlikely object regions, and 3) $B_{\mathrm{trj}}$ from the trajectory predictor to maintain the inertia of the box position and size. A fusion strategy is needed to yield the final box location and size. We consider a couple of factors for its design.

To show the performance of UHP-SOT++, we compare it with several state-of-the-art unsupervised and supervised trackers on four single object tracking datasets. They are OTB2015 [38], TC128 [42], UAV123 [43] and LaSOT [44]. OTB2015 (also named OTB in short) and TC128, which contain 100 and 128 color or grayscale video sequences, respectively, are two widely used small-scale datasets. UAV123 is a larger one, which has 123 video sequences with more than 110K frames in total. Videos in UAV123 are captured by low-altitude drones. They are useful in the tracking test of small objects with a rapid change of viewpoints. LaSOT is the largest single object tracking dataset that targets at diversified object classes and flexible motion trajectories in longer sequences. It has one training set with dense annotation for supervised trackers to learn and another test set for performance evaluation. The test set contains 280 videos of around 685K frames.

Performance evaluation is conducted using the “One Pass Evaluation (OPE)” protocol. The metrics include the precision plot (i.e., the distance of the predicted and actual box centers) and the success plot (i.e., overlapping ratios at various thresholds). The distance precision (DP) is measured at the 20-pixel threshold to rank different methods. The overlap precision is measured by the area-under-curve (AUC) score. We use the same hyperparameters as those in STRCF except for regularization coefficient, $\mu$. If the appearance box is not chosen, STRCF sets $\mu=15$ while UHP-SOT++ selects $\mu\in\{15,10,5,0\}$. The smaller $\mu$ is, the stronger the update is. The number of previous frames for trajectory prediction is $N=20$. The cutting threshold along the horizontal or vertical direction is set 0.1. The threshold for good similarity score is 0.08, and a threshold of 0.5 for IoU is adopted. UHP-SOT++ runs at 20 frames per second (FPS) on a PC equipped with an Intel(R) Core(TM) i5-9400F CPU. The speed data of other trackers are either from their original papers or benchmarks. Since no code optimization is conducted, all reported speed data should be viewed as lower bounds for the corresponding trackers.

We compare different configurations of UHP-SOT++ on the TC128 dataset to investigate contributions from each module in Fig. 6. As compared with UHP-SOT, improvements on both DP and AUC in UHP-SOT++ come from the new fusion strategy. Under this strategy, the background motion modeling plays an more important role and it has comparable performance even without the trajectory prediction. Although the trajectory prediction module is simple, it contributes a lot to higher tracking accuracy and robustness as revealed by the performance improvement over the baseline STRCF.

More performance comparison between UHP-SOT++, UHP-SOT and STRCF is presented in Table II. As compared with STRCF, UHP-SOT++ achieves 1.8%, 6.2%, 6.7% and 6.8% gains in the success rate on OTB, TC128, UAV123 and LaSOT, respectively. As to the mean precision, it has an improvement of 1.2%, 6.9%, 7.2% and 10.4%, respectively. Except for OTB, UHP-SOT++ outperforms UHP-SOT in both the success rate and the precision. This is especially obvious for large-scale datasets. Generally, UHP-SOT++ has better tracking capability than UHP-SOT. Its performance drop in OTB is due to the tracking loss in three sequences; namely, Bird2, Coupon and Freeman4. They have complicated appearance changes such as severe rotation, background clutter and heavy occlusion. As shown in Fig. 7, errors at some key frames lead to total loss of the object, and the lost object cannot be easily recovered from motion. The trivial fusion strategy based on appearance similarity in UHP-SOT seems to work well on their key frames while the fusion strategy of UHP-SOT++ does not suppress wrong proposals properly since background clutters have stable motion and trajectories as well.

We compare the performance of UHP-SOT++ and several unsupervised trackers for the LaSOT dataset in Fig. 8. The list of benchmarking methods includes ECO-HC [8], STRCF [32], CSR-DCF [45], SRDCF [33], Staple [30], KCF [27], DSST [29]. UHP-SOT++ outperforms other unsupervised methods by a large margin, which is larger than 0.02 in the mean scores of the success rate and the precision. Besides, its running speed is 20 FPS, which is comparable with that of the second runner STRCF (24 FPS) and the third runner (42 FPS) based on experiments in OTB. With a small increase in computational and memory resources, UHP-SOT++ gains in tracking performance by adding object box trajectory and background motion modeling modules. Object boxes of three leading unsupervised trackers are visualized in Fig. 9 for qualitative performance comparison. As compared with other methods, the proposals of UHP-SOT++ offer a robust and flexible box prediction. They follow tightly with the object in both location and shape even under challenging scenarios such as motion blur and rapid shape change.

We compare the success rates of UHP-SOT++ and several supervised and unsupervised trackers against all four datasets in Fig. 10. Note that there are more benchmarking methods for OTB but fewer for TC128, UA123 and LaSOT since OTB is an earlier dataset. The supervised deep trackers under consideration include SiamRPN++ [14], ECO [8], C-COT [9], DeepSRDCF [33], HDT [5], SiamFC_3s [19], CFNet [31] and LCT [46]. Other deep trackers that have leading performance but are not likely to be used on resource-limited devices due to their extremely high complexity, such as transformer-based trackers [24, 25], are not included here. Although the performance of a tracker may vary from one dataset to the other due to different video sequences collected by each dataset, UHP-SOT++ is among the top three in all four datasets. This demonstrates the generalization capability of UHP-SOT++. Its better performance than ECO on LaSOT indicates a robust and effective update of the object model. Otherwise, it would degrade quickly with worse performance because of longer LaSOT sequences. Besides, its tracking speed of 20 FPS on CPU is faster than many deep trackers such as ECO (10 FPS), DeepSRDCF (0.2 FPS), C-COT (0.8 FPS) and HDT (2.7 FPS).

In Table II, we further compare UHP-SOT++ with state-of-the-art unsupervised deep trackers UDT+ [34], LUDT [35] and ResPUL [36] in their AUC and DP values, running speeds and model sizes. Two leading supervised trackers SiamRPN++ and ECO in Fig. 10 are also included. It shows that UHP-SOT++ outperforms recent unsupervised deep trackers by a large margin. We should emphasize that deep trackers demand pre-training on offline datasets while UHP-SOT++ does not. In addition, UHP-SOT++ is attractive because of its lower memory requirement and near real-time running speed on CPUs. Although ECO-HC also provides a light-weight solution, there is a performance gap between UHP-SOT++ and ECO-HC. SiamRPN++ has the best tracking performance among all trackers, due to the merit of end-to-end optimized network with auxiliary modules such as classification head and the region proposal network. Yet, its large model size and GPU hardware requirement limit its applicability in resource-limited devices such as mobile phones or drones. In addition, as an end-to-end optimized deep tracker, SiamRPN++ has the interpretability issue to be discussed later.

To better understand the capability of different trackers, we analyze the performance variation under various challenging tracking conditions. These conditions can be classified into the following attributes: aspect ratio change (ARC), background clutter (BC), camera motion (CM), deformation (DEF), fast motion (FM), full occlusion (FOC), in-plane rotation (IPR), illumination variation (IV), low resolution (LR), motion blur (MB), occlusion (OCC), out-of-plane rotation (OPR), out-of-view (OV), partial occlusion (POC), scale variation (SV) and viewpoint change (VC). We compare the AUC scores of supervised trackers (e.g., SiamRPN++ and ECO) and unsupervised trackers (e.g., UHP-SOT++, ECO-HC, UDT+ and STRCF) under these attributes in Fig. 11.

We have the following observations. First, among unsupervised trackers, UHP-SOT++ has leading performance in all attributes, which reveals improved robustness from its basic modules and fusion strategy. Second, although ECO utilizes deep features, it is weak in flexible box regression and, as a result, it is outperformed by UHP-SOT++ in handling such deformation and shape changes against LaSOT. In contrast, SiamRPN++ is better than other trackers especially in DEF (deformation), ROT (rotation)and VC (viewpoint change). The superior performance of SiamRPN++ demonstrates the power of its region proposal network (RPN) in generating tight boxes. The RPN inside SiamRPN++ not only improves IoU score but also has the long-term benefit by excluding noisy information. Fourth, supervised trackers perform better in IV (illumination variation) and LR (low resolution) than unsupervised trackers in general. This can be explained by the fact that unsupervised trackers adopt HOG, CN features or other shallow features which do not work well under these attributes. They focus on local structures of the appearance and tend to fail to capture the object when the local gradient or color information is not stable. Finally, even with the feature limitations, UHP-SOT++ still runs second in many attributes against LaSOT because of the stability offered by trajectory prediction and its capability to recover from tracking loss via background motion modeling.