Graph Attention Tracking

Existing correlation based information embedding methods [2, 16, 15] take the whole target feature as a unity to match with the search features. As this operation neglects the part-level correspondence between the target and the search regions, the matching is inaccurate under shape-and-pose variances of targets. Besides, this global matching manner may greatly compress the target information propagating to the search feature. In order to address these problems, we establish the part-to-part correspondence between the target template and the search region with a complete bipartite graph.

Given two images of a template patch $T$ and a search region $S$, we first employ a Siamese feature extraction network to obtain two feature maps $F_{t}$ and $F_{s}$. To generate a graph, we consider each $1\times 1\times c$ grid of the feature map as a node (part), where $c$ represents the number of feature channels. Let $V_{t}$ be a node set including all nodes of $F_{t}$, and let $V_{s}$ be another node set of $F_{s}$. Inspired by the graph attention networks [28], we use a complete bipartite graph $G=(V,E)$ to model the part-level relations between the target and search region, where $V=V_{s}\cup V_{t}$ and $E=\{(u,v)|\forall u\in V_{s},\forall v\in V_{t}\}$. We further define two sub-graphs of $G$ by $G_{t}=(V_{t},\emptyset)$ and $G_{s}=(V_{s},\emptyset)$.

For each $(i,j)\in E$, let $e_{ij}$ denote the correlation score of node $i\in V_{s}$ and node $j\in V_{t}$:

where $\mathbf{h}_{s}^{i}\in\mathbb{R}^{c}$ and $\mathbf{h}_{t}^{j}\in\mathbb{R}^{c}$ are feature vectors of node $i$ and node $j$. Since the more similar is a location in the search region to the local features of the template, the more likely it is the foreground, and more target information should be passed to there. For this reason, we hope that the score $e_{ij}$ is proportional to the similarity of the two node features. We can simply use the inner product between features as the similarity measurement. In order to adaptively learn a better representation between the nodes, we first apply linear transformations to the node features and then take the inner product between transformed feature vectors to calculate the correlation score. Formally,

where $W_{s}$ and $W_{t}$ are the linear transformation matrices.

In order to balance the amount of information sent to the search region, we normalize $e_{ij}$ with the softmax function:

Intuitively, $a_{ij}$ measures how much attention the tracker should pay to part $i$, according to the viewpoint of part $j$.

Leveraging the attentions that passed from all nodes in $G_{t}$ to the $i$-th node in $G_{s}$, we compute the aggregated representation for node $i$ with

where $W_{v}$ is a matrix for linear transformation.

Finally, we can fuse the aggregated feature with the node feature $\mathbf{h}_{s}^{i}$ to obtain a more powerful feature representation empowered by target information:

where $\|$ represents vector concatenation.

We compute all $\hat{\mathbf{h}}_{s}^{i}~{}\forall i\in V_{s}$ in parallel, which yields a response map for subsequent task.

We have presented the graph attention module (GAM) to realize the part-to-part information propagating. Before achieving target-aware visual tracking, we need to tackle another challenge. That is, how to produce a variable template which adaptively fits different object scales and aspect-ratios.

Traditional cross-correlation based methods simply crop the center region of template $F_{t}$ as the target feature to match with the search region $F_{s}$, which delivers much background information to the response map, especially when the template target is given in extreme aspect ratios. To address the problem, we investigate a target-aware template-feature-area selection mechanism under the supervision of labeled bounding box $B_{t}$ in the template patch. By projecting $B_{t}$ onto the feature map $F_{t}$, we can attain a region of interest $R_{t}$. Only the pixels in $R_{t}$ are taken as the template feature:

Through this simple operation, the obtained feature map $\hat{F}_{t}$ is a tensor of dimensions $(w,h,c)$, where $w$ and $h$ correspond to the width and height of the template bounding box $B_{t}$, and $c$ is the number of channels of $F_{t}$.

Each element $\hat{F}_{t}(i,j,:)$ is considered as a node in the template subgraph $G_{t}$. Meanwhile, each element $F_{s}(m,n,:)$ is considered as a node in the search subgraph $G_{s}$. These two subgraphs serve as inputs to the Graph Attention Module for information embedding. As elements in $G_{t}$ are arranged in a grid pattern on the feature map $\hat{F}_{t}$, we can implement the linear transformations in Section 3.1 with $1\times 1$ convolutions. Then all correlation scores could be calculated by matrix multiplication, which is expected to greatly improve the efficiency.

In experiments, we observe that applying a batch normalization after each convolution can effectively improve the performance. However, the dimensions $w$ and $h$ corresponding to different tracking objects cannot be pre-determined, thus we cannot directly apply the batch normalization operation with the scale variable $\hat{F}_{t}$. To solve the problem, we recompute $\hat{F}_{t}$ as follows:

Besides keeping the scale invariant, this target-aware idea renders the proposed method extendable to tasks which require non-rectangular ROIs (e.g., instance-segmentation in videos) .

Now we can construct our tracking network with the proposed GAM for effective information embedding. As shown in Figure 3(a), our SiamGAT simply consists of three blocks: a Siamese network for feature extraction, a tracking head for target bounding box prediction and a GAM block to bridge them.

Numerous works demonstrated that trackers can greatly benefit from better feature extraction method [20, 15]. By replacing the classical HOG features and color features with deep CNN features, tracking accuracy has seen significant improvement [20]. Later, deepening the backbone networks and fusing features of multiple layers have further improved the tracking performance [15]. Since GoogLeNet [26] is able to learn multi-scale feature representations with much fewer parameter and faster reasonging speed, here we adopt GoogLeNet as our backbone (an ablation is performed as well to study the performance of SiamGAT using different backbones).

Encouraged by the success of anchor-free trackers, we leverage the classification-regression head network from SiamCAR [10] to be the tracking head. It contains two branches: a classification branch predicting the category information for each location, and a regression branch computing the target bounding box at this location. The two branches share the same response map output by GAM.

The proposed SiamGAT is implemented in Pytorch on 4 RTX-2080Ti cards. Unless specified, the modified GoogLeNet( Inception v3) [26] is adopted as the backbone network for feature extraction. The backbone is initialized with the weights that pretrained on ImageNet [25]. The training batch size is set as 76 and totally 20 epochs are trained with stochastic gradient descent (SGD). We use a learning rate that linearly increased from 0.005 to 0.01 for the first 5 warmup epoches and then exponentially decayed to 0.0005 for the rest 15 epoches. For the first 10 epoches, we freeze the parameters in the backbone to train the graph attention network and the head network. For the rest 10 epoches, we freeze stage 1 and 2 of GoogLeNet, and fine-tune stage 3 and 4.

We adopt COCO [19], ImageNet DET [25], ImageNet VID [25], YouTube-BB [23] and GOT-10k [14] as the training set for experiments on OTB100 [31] and UAV123 [21]. Specifically, for experiments on GOT-10k [14] and LaSOT [7], the model is respectively trained with only the specified training set provided by their official websites for fair comparison. In both training and testing processes, we use pre-fixed scales with $127\times 127$ pixels for the template patch and $287\times 287$ pixels for search regions. During testing, only the object in the initial frame of a sequence is adopted as the template patch and fixed for the whole tracking period of this sequence. The search region in the current frame is adopted as the input of the search branch.

Backbone architecture. We evaluate our network with both shallow and deep backbone architectures for visual tracking. Table 1 shows the tracking performance with AlexNet and GoogLeNet as backbones. Different backbones greatly affect the speed and performance of the tracker. By replacing AlexNet with GoogLeNet, the success is improved by $5.4\%$ from $59.2\%$ to $64.6\%$, the precision is increased by $6.4\%$ from $77.9\%$ to $84.3\%$. While the tracking speed decreases from 165 FPS to 70 FPS, which still meets the real-time requirement. It is worth pointing out that, the SiamGAT using AlexNet as the backbone also achieves a competitive performance while its precision and success are $1.1\%$ and $3.5\%$ higher than SiamRPN [16], whose results are shown in Figure 4. Clearly, the proposed approach can achieve a trade-off between accuracy and efficiency with different backbones.

Target-aware vs.  pre-fixed template area selection. To investigate the impact of template area selection, we train two models with GAM on GoogLeNet. One is trained with the traditional fixed-region cropping target features, and another is trained with the target-aware selected features. As shown in Table 1, the proposed target-aware feature area selecting mechanism brings $2.0\%$ and $2.1\%$ performance gains respectively on success and precision. The main reason is that the target-aware mechanism is able to effectively eliminate the background information and enhance the foreground representation, which helps to obtain more accurate target feature area than fixed-region cropping.

Comparison with DW-Xcorr. To conduct a comparison with cross-correlation based methods, here we replace the target-aware GAM with the popular DW-Xcorr layer [15], which achieves the best performance among cross-correlation based methods. As shown in Table 1, compared with DW-Xcorr, the GAM with target-aware mechanism brings $3.1\%$ and $2.8\%$ performance gains respectively on success and precision, while the GAM with pre-fixed region only brings $1.1\%$ and $0.7\%$ performance gains. The results further demonstrate that the pre-fixed region of target features has become the bottleneck for accurate target-information-passing. Benefiting from the GAM architecture, our method enables the target-aware region, which is adaptive to different aspect ratios of objects.

The UAV123 dataset contains a total of 123 video sequences and all sequences are fully annotated with upright bounding boxes. Objects in the dataset suffer from occlusions, fast motion, illumination and large scale variations, which pose challenges to the trackers. A comparison with state-of-the-art trackers is shown in Figure 4 in terms of the precision and success plots of OPE. Our tracker outperforms all other trackers for both metrics. Compared with the baseline SiamCAR, our tracker improves the performance by $3.0\%$ in precision and $2.3\%$ in success.

To evaluate the generalization of our tracker, we test it on the GOT-10k (Generic Object Tracking Benchmark) and compare it with state-of-the-art trackers. GOT-10k is a challenging large-scale dataset which contains more than 10,000 videos of moving objects in real-world. It is also challenging in terms of zero-class-overlap between the provided training subset and testing subset. For fair comparison, we follow the the protocol of GOT-10k that only training our model with its training subset. We evaluate SiamGAT on GOT-10k and compare it with state-of-the-art trackers including SiamCAR [10], Ocean [36], SiamFC++ [33], D3S [1], SiamRPN++ [15], SPM [29], SiamRPN [16] and other baselines. As shown in Table 2, the proposed SiamGAT performs best in term of all metrics. Compared with the baseline SiamCAR, our tracker improves by $4.8\%$, $6.6\%$ and $5.1\%$ respectively in terms of $AO$, $SR_{0.5}$ and $SR_{0.75}$. Impressively, it even outperforms the online update tracker ‘Ocean’ and improves the scores respectively by $1.6\%$, $2.2\%$ and $1.5\%$ with a much simple network architecture, which validates the generalization ability of our tracker on unseen classes. Figure 5 shows a comparison on success plots. Some qualitative results and comparisons are provided by Figure 1, which demonstrates that our SiamGAT is able to predict more accurate bounding boxes of targets.

OTB-100 is one of the most classical benchmarks that provides a fair test-bed on robustness. All sequences in the dataset are labeled with 11 interference attributes, including illumination variation (IV), scale variation (SV), occlusion (OCC), deformation (DEF), motion blur (MB), fast motion (FM), in-plane rotation (IPR), out-of-plane rotation (OPR), out-of-view (OV), low resolution (LR) and background clutter (BC). A comparison with state-of-the-art trackers is shown in Figure 6 in terms of success plots of OPE. Our SiamGAT reaches a success score of $71.0\%$ that surpasses all other trackers. An evaluation on different attributes is shown by Figure 7. Our tracker can better handle the challenges like deformation (DEF), out-of-plane rotation (OPR), occlusion (OCC), illumination variation (IV), in-plane rotation (IPR) and scale variation (SV), which may cause large shape and pose variations of the object. Regarding to fast motion (FM), out-of-view (OV), low resolution (LR) which may cause extreme appearance variations, the proposed tracker obtains a lower score than the baseline SiamCAR. The results demonstrate that the proposed tracker can achieve robust performance against shape and pose variations.

To further evaluate the proposed approach on a more challenging dataset, we conduct experiments on LaSOT [7], which is a large-scale, high-quality, and densely annotated dataset for long-term tracking. To mitigate potential class bias, it provides the same number of sequences for each category. The results on LaSOT are shown in Figure 8. Our SiamGAT is the second best only behind the online tracker Ocean-online [36] but surpasses the long-term tracker GlobalTrack [17] by $3.6\%$ in normalized precision, $0.2\%$ in precision and $1.8\%$ in success. Compared with Ocean-offline [36] which is much more complex than SiamGAT in terms of network architecture, SiamGAT performs $2.3$ points better in normalized precision, $0.4$ in precision and $1.3$ in success. The results indicate that the proposed tracker is competitive for long-term tracking tasks. Moreover, up to our investigation, both attentive and target-aware properties of the proposed tracker allow more efficient online tracking without model updating. Online tracking modules can be easily integrated in future.