Spatio-Temporal Matching for Siamese Visual Tracking

In view of the aforementioned issues, we leverage the 4-D information of height, width, channel and time for comprehensive matching, called Spatio-Temporal Matching.

In spatial matching, we increase the exploration of channel-wise matching and design a space-variant channel-guided correlation (SVC-Corr) that can emphasize target-aware responses and weaken interference responses. The innovation of SVC-Corr is two-fold: 1) It establishes various channel associations between the template and each subwindow in the search area. This allows an independent and accurate recalibration of the response to a specific instance at each location without semantic disturbance from other spatial locations. 2) It is a learnable correlation module that can benefit from large-scale offline training, rather than a handcrafted parameter-free metric such as cross correlation. Figure 1b shows the sparse channel responses using SVC-Corr. The generated target-aware responses are more effective in separating semantic objects than the DW-XCorr features.

In temporal matching, we propose an aberrance repressed module (ARM) to investigate the hidden information propagated in the interframe response maps. Specifically, an efficient optimization function is added to restrict the abrupt alteration of the interframe response maps in both training and testing stages. It takes into account the variation of target and background simultaneously in the time domain, and adjust the response map flexibly for different tracking scenarios. As shown in Figure 1b, the heatmap of frame 5 imposes a strong constraint on the subsequent heatmaps, thus avoiding the drifting. Importantly, our ARM brings negligible additional computation cost.

Using the introduced SVC-Corr and ARM, we present a novel anchor-free tracking framework, termed Siamese Spatio-Temporal Matching (SiamSTM) tracking network. The effectiveness of the framework is verified on five benchmarks: OTB100[51], VOT2018[24], VOT2020[23], GOT-10k[21] and LaSOT[14]. Our approach achieves leading performance, while running at 66 FPS.

The basic Siamese tracker is composed of feature extraction subnetwork and target localization subnetwork, as shown in Figure 2. In the feature extraction subnetwork, we modify the ResNet50[18] according to SiamRPN++[28] to make it more suitable for the tracking task. Moreover, we cut off the $conv5$ of the ResNet50 and only retain the $conv1-conv4$ to reduce computations.

In the target localization subnetwork, considering that anchor-based trackers rely on a large number of heuristic hyper-parameters[54] and lack the competence to amend the weak predictions[58], we leverage an anchor-free structure. The right part of Figure 2 illustrates its architecture that represents the object with object center, object size and local offset[60]. Let $\hat{Y}\in\mathbb{R}^{H\times W\times 1}$ be an output center heatmap of height $H$ and width $W$. For an arbitrary position $(x,y)$ in the heatmap, a prediction $\hat{Y}_{x,y}=1$ corresponds to the tracking object center, while $\hat{Y}_{x,y}=0$ is the background. The classification label $Y\in\mathbb{R}^{H\times W\times 1}$ adopts a Gaussian function $Y_{x,y}=exp\left(-\frac{\left(x-x_{c}\right)^{2}+\left(y-y_{c}\right)^{2}}{2\sigma_{p}^{2}}\right)$, where $(x_{c},y_{c})$ denotes the coordinate of the target center point in the heatmap, and $\sigma_{p}$ is an object size-adaptive standard deviation[27]. The training objective is a penalty-reduced pixel-wise logistic regression with focal loss[34]. We refer readers to [27, 60] for more details.

To eliminate the discretization gap caused by the spatial stride $R=8$, an additional local offset $\hat{O}\in\mathbb{R}^{H\times W\times 2}$ is calculated for each center point. For the object size, we directly predict a $\hat{S}\in\mathbb{R}^{H\times W\times 2}$ to represent the height and width in each location. The supervisions of local offset and object size only act at the unique target location in the tracking task, and other locations are ignored. In particular, the offset label $\{o_{x},o_{y}\}$ is $\left\{\frac{i}{R}-\lfloor\frac{i}{R}\rfloor,\frac{j}{R}-\lfloor\frac{j}{R}\rfloor\right\}$ for the target position $(i,j)$ in the original image, and the size label $\{s_{w},s_{h}\}$ is the groundtruth width and height after downsampling. Both of these are trained with L1 loss. The overall training objective of the basic tracker is:

where $L_{off},L_{size}$ are the loss functions of local offset and object size. $\lambda_{off},\lambda_{size}$ are the trade-off hyper-parameters.

We argue that our center-based structure is more suitable for object tracking than the currently popular FCOS-based structure. The Gaussian label-assign method in the center-based structure is equivalent to unifying the classification score and localization quality in the FCOS-based structure into a joint and single representation. Thus, it eliminates the training-test inconsistency mentioned in [31] and enables a stronger correlation between classification and localization quality. Additionally, the ideal prediction heatmap should be a Gaussian-like probability distribution, and we can easily constrain its alteration in the time domain as detailed in Sec. 3.3.

The difference between 2-D image matching and 3-D deep feature matching, namely modeling of the associations between the channels, has been analyzed above. The convolutional network learns specific semantic information mainly based on a few channels. When applied to the tracking task, we can obtain the target-aware feature responses by identifying channels that are active to the target and are inactive to the interference.

For this purpose, we introduce a space-variant channel-guided correlation (SVC-Corr) to establish channel correspondence between the template and the search area on the basis of traditional 2-D correlation. The red dotted box in Figure 2 illustrates its structure, including two parallel branches. Given the template feature $z\in\mathbb{R}^{H_{z}\times W_{z}\times C}$ and the search feature $x\in\mathbb{R}^{H_{x}\times W_{x}\times C}$, we first employ a DW-XCorr to obtain a spatial response, i.e., $f_{sa}(z,x)=z\star x$, where $\star$ denotes DW-XCorr and $f_{sa}(z,x)\in\mathbb{R}^{(H_{x}-H_{z}+1)\times(W_{x}-W_{z}+1)\times C}$ is the correlation result.

The DW-XCorr branch embeds the template feature into the search features along height and width, while the other branch aims to channel-wise embedding. The core step is the extraction of channel descriptors for template and search area, meanwhile modeling the interdependencies between the channels. This is implemented by the Channel Transform (Ch Trans) in Figure 3. Specifically, after a $3\times 3$ convolution $\varphi_{1}$, we squeeze the spatial dimension of the feature through max pooling $f_{k}^{max}$ and average pooling $f_{k}^{avg}$ to gather different clues about the channel importance[49]. Note that the pooling kernel size $k$ is equal to the template feature size (i.e., $f_{k}^{max,avg}(z)\in\mathbb{R}^{1\times 1\times C},f_{k}^{max,avg}(x)\in\mathbb{R}^{(H_{x}-H_{z}+1)\times(W_{x}-W_{z}+1)\times C}$), hence we can obtain various channel descriptors from each search subwindow (a sliding window of the same size as template feature size in the search region) and associate them with the template respectively. The pooled descriptors are further fed into a shared multiple fully connected ($FC$) layer to capture the channel-wise interdependencies. Considering that there are multiple channel descriptors on the search feature, we can efficiently implement all $FC$ operations in parallel with $1\times 1$ convolution. The learned descriptors are merged by element-wise summation. In short, the Ch Trans is computed as:

where $\omega\in\{z,x\}$ denotes template or search features, $\oplus$ is element-wise summation.

With the channel descriptors of the template and search area, we adopt broadcasting element-wise summation followed by $1\times 1$ convolution $\varphi_{2}$ to generate space-variant weights for different channels at each location. Finally, these weights recalibrate channel-wise feature responses of the previous DW-XCorr using element-wise summation. The whole process of SVC-Corr can be expressed as:

where $f_{ca}$ and $ca\_corr$ denote the channel weights and SVC-Corr result, respectively.

Our SVC-Corr has two prominent advantages. 1) Unlike channel attention that focuses on all semantic categories, our SVC-Corr is tailored to different instances. For each subwindow in the search area, SVC-Corr associates it with the template independently in channel dimension, i.e., $f_{ca}$ has a total of $(H_{x}-H_{z}+1)\times(W_{x}-W_{z}+1)$ space-variant channel weights. This makes the channel information of various spatial locations not interfere with each other, and thus is more beneficial for the generation of discriminative target-aware responses. Figure 4 shows a comparison between SVC-Corr and DW-XCorr. 2) All parameters in Eq. (2)- (4) are learnable rather than handcrafted and can benefit from the large-scale offline training.

After fully exploiting the 3-D information of deep features for single-frame matching, we further extend matching to time-dimension to explore the information hidden across multiple frames. Figure 5 indicates our temporal matching method, aberrance repressed module (ARM).

Instead of raw image patches, center heatmaps of the adjacent frames are used as the input data for our ARM. Based on the above analysis and existing works[22, 32], we observe two criteria: 1) An ideal heatmap should be unimodal with narrow peak. 2) The distribution of heatmaps between the adjacent frames should be as similar as possible after alignment. Consequently, the purpose of ARM is to maximize the similarity of the peak-aligned heatmaps between the adjacent frames and minimize their errors with Gaussian labels. We propose to train the objective function with KL divergence:

where $\hat{Y}_{i},\hat{Y}_{i+k},Y_{i},Y_{i+k}$ denote heatmaps and Gaussian labels for adjacent frame $i$ and $i+k$, as detailed in Sec. 3.1. $p,q$ represent peak locations of $\hat{Y}_{i},\hat{Y}_{i+k}$, and $\Delta_{p,q}$ indicates the circular shifting operation to coincide peak $p$ with peak $q$, while $\Delta_{q,p}$ indicates the opposite. $\otimes$ is element-wise multiplication. The KL divergence is calculated as $KL(y,x)=\int y\log(y)-\int ylog(x)$.

In the first line of Eq. 5, the $\hat{Y}_{i}[\Delta_{p,q}]\otimes\hat{Y}_{i+k}$ term fuses the two peak-aligned heatmaps, the distribution of which can reflect the alteration between two frames. A sharp unimodal distribution indicates that the two heatmaps are similar, and the loss between it and the Gaussian label is low. When an aberrance occurs, this distribution will change suddenly, e.g., the peak value decreases or additional interference peaks appear, causing a high loss. Therefore, the minimization of this loss function can restrict the alteration of heatmaps to suppress aberrances. The second line of Eq. 5 is symmetrical with the first line, but the Gaussian label is replaced by that of another frame. This makes the most of the monitoring information of each frame and leads to a more reliable restriction.

The ARM realizes the temporal matching without extra parameters and can be trained end-to-end in conjunction with the above basic tracker. The overall loss function is:

where $L_{base}$ comes from the basic tracker loss in Eq. 1 and $\lambda_{arm}$ is a hyper-parameter.

The online tracking procedure is summarised in Algorithm 1. After initialization, the basic tracker with SVC-Corr first generates center heatmap $\hat{Y}_{t}$, offset prediction $\hat{O}_{t}$ and size prediction $\hat{S}_{t}$ (line 6). Lines 7-16 indicate the inference process of ARM. Specifically, we align the label $Y_{last}$ and the heatmap $\hat{Y}_{last}$ of the previous frame with the top $K$ response locations of $\hat{Y}_{t}$ respectively, and then calculate the KL divergence $L_{arm}^{k}$ for each pair. Note that only the first term in Eq. 5 is calculated here, since we only have the label of the previous frame. The $q_{\hat{k}}$ that minimize $L_{arm}^{k}$ can be considered as the optimal peak position with the temporal constraint. If $q_{\hat{k}}$ is different from the original peak position $q_{1}$ in $\hat{Y}_{t}$, we add $\hat{Y}_{last}[\Delta_{p_{last},q_{\hat{k}}}]$ as a weight on $\hat{Y}_{t}$ (line 14) and update related variables (line 15). Finally, the maximum response location on the weighted heatmap $\hat{Y}_{t}$ is extracted as the target center (line 18), combining with the offset prediction and the size prediction on this point to produce the bounding box (lines 19-20). Our ARM utilizes temporal information during both raining and inference. Compared with linear updating, it benefits from large-scale offline training and considers both the target and the background. Consequently, ARM is more flexible in adjusting each position of the response for different tracking situations. Meanwhile, it is much more efficient than online classifiers and achieves a tracking speed of 66 FPS.

The proposed SiamSTM is trained on ImageNet VID[43], ImageNet DET[43], Youtube-BB[41], COCO[35] and GOT-10k[21] for experiments on OTB100[51], VOT2018[24] and VOT2020[23]. In addition, for experiments on GOT-10k[21] and LaSOT[14], the model is only trained with their official training set for a fair comparison. To train ARM, the training input is a triplet containing a template image and two search images with an interval less than 100 frames. The template size is $127\times 127$ pixels, while the search image is $255\times 255$ pixels. The training batch size is set to 96 and a total of 20 epochs are trained with SGD on 4 TITANV. We use a learning rate that linearly increased from 0.001 to 0.005 for the first 5 warmup epochs and then exponentially decayed to 0.0005 for the rest of 15 epochs. For the first 10 epochs, we freeze the parameters of the backbone. For the remaining epochs, the backbone network is unfrozen, and the whole network is trained end-to-end. The hyper-parameters $\lambda_{off},\lambda_{size}$ and $\lambda_{arm}$ in Eq. 1 and Eq. 6 are set to 1, 0.1 and 0.5, respectively. The $K$ in Algorithm 1 is set to 3.

To extensively evaluate the proposed method, we compare it with state-of-the-art trackers on five challenging tracking datasets. The selected comparison methods include XCorr based tracker[2, 46], DW-XCorr based trackers[28, 48, 54, 15, 6, 58], pixel-wise correlation based tracker[33], correlation filter based trackers[53, 4, 11], meta-learning based trackers[10, 3, 8, 56], multi-domain learning based trackers[39, 45] and others[25, 47].

Sequences in VOT2018 are annotated by the following visual attributes: occlusion, illumination change, motion change, size change and camera motion. Frames that do not correspond to any of the five attributes are denoted as unassigned. We compare the EAO of the above methods on these visual attributes in Figure 6. Our SiamSTM ranks first on almost all of the attributes except for unassigned and outperforms the second place by a wide margin in the challenging occlusion and camera motion.

To further verify the efficacy of the proposed method, we perform an ablation study on the GOT-10k test-set, as presented in Table 4. Note that triplet input is only required when training ARM, otherwise the training input is still a pair of a template image and a search image.