Continuity-Discrimination Convolutional Neural Network for Visual Object Tracking

In general, tracking performance highly depends upon the robustness of target appearance representation. Zou et al. [8] has pointed out that with temporal slowness constraints, the learned features manifest robustness and even invariance to challenging variations. Inspired by their views, we mathematically define temporal appearance continuity of the target between frames based on temporal slowness. By doing so, we make use of inherent temporal continuity to learn appearance representations that are robust to variations.

Formally, given a feature mapping $\Phi:\mathbb{R}^{r}\mapsto\chi\subset\mathbb{R}^{n}$, where $r$ is the dimensionality of the raw pixel space and $\chi$ is the feature space, the temporal appearance continuity of a target patch is formalized by Lipchitz continuity of $\Phi$ at any dimension with respect to time $t$:

where $\Phi_{i}(P_{*}^{(t)})$ is the $i$th component of the $n$-dimensional feature vector of the ground-truth patch $P_{*}^{(t)}$ in the $t$th frame and $K$ is a positive constant. We use Euclidean norm to measure the distance between features. As $\Delta t\to 0$, we have

Consequently, if the frame switch of a given video sequence is sufficiently smooth, the Euclidean distance between high-level features is bounded by some small $\epsilon:=nK\Delta t$.

Within the Bayesian tracking framework, candidate bounding boxes are drawn from a certain probability distribution in the online tracking phase. To obtain accurate target localization, our aim is to select the candidate with the whole target located in the center of the bounding box and as less background left as possible. Apparently, such selection depends largely on specific training data with which a discriminative classifier is trained. A candidate patch that contains object semantics is of objectness. However, satisfying objectness is not sufficient for accurate target localization. It is necessary to search for a candidate that tightly envelops the entire target, leaving it located in the center of the patch (positional centroid). Those that are of objectness and positional centroid are literally defined to satisfy object-centroid. Figure 1 illustrates some positive samples that satisfy object-centroid and negative samples with loss of object-centroid. These properties can be exploited to train a discriminative binary classifier so as to filter distracting samples that contain redundant background or partial target.

Specifically, training data for object-centroid discrimination are generated using a particular sampling protocol: the positive samples are one or two pixels shifted from the groundtruths, and the negative are randomly drawn from a distribution under the constraint that the intersection-over-union (IoU) between a negative sample and its corresponding groundtruth is subject to $0<lo<\text{IoU}\leq hi$, for some pre-specified $lo$ and $hi$. Note that the non-zero lower threshold $lo$ is an essential factor to avoid object-centroid inconsistency. For example, a sample of zero IoU with the corresponding ground-truth might also satisfy the object-centroid property (i.e., it tightly contains similar distracting target in its center), thus should be considered as a positive sample instead.

Suppose the training set is given by $\mathbb{S}=\{P_{j}^{(t)},N_{l}^{(t)}|j=1,...,m_{p}^{(t)},l=1,...,m_{n}^{(t)},t=1,...,T\}$, where $P_{j}^{(t)}$ is the $j$th positive sample satisfying the object-centroid property and $N_{l}^{(t)}$ is the $l$th negative sample drawn from the $t$th frame. Before being fed into CD-CNN, all samples are resized into 224-by-224 and normalized. In the offline training stage, temporal appearance continuity and object-centroid discrimination are jointly utilized to learn a deep discriminative model for generic object tracking. To this end, we design Continuity-Discrimination CNN (CD-CNN), in which temporal appearance continuity and object-centroid are formulated as specific loss functions, respectively.

More specifically, with $\Delta t$ set to 1 (unit time), the loss function corresponding to temporal appearance continuity is defined as, for all $j,k,t$,

where $\Phi(\cdot)$ is the nonlinear mapping from the sample space to the feature space $\chi$.

To encourage object-centroid discriminability, we define the following loss function, for $\beta>0$ and all $j,l,t$,

In addition, to directly quantify object-centroid, the output of $\Phi(\cdot)$ is fed into a nonlinear binary classifier which indicates the probability of being object-centroid, $p(s)$, for all $s\in\mathbb{S}$. We define the binary classification loss as the soft-max cross entropy loss, for all $j,l,t$,

Formally, the optimization objective of the offline joint learning task is formulated as

where

Here, $\lambda>0$ and $\mu>0$ are trade-off coefficients, and $W$ denotes all the learnable parameters.

In practice, we choose PVANET [9] as the backend of CD-CNN. The overall architecture of our network is illustrated in Figure 2. As is shown in Figure 2, the network takes a triplet of RGB image patches ${P_{j}^{(t)},P_{k}^{(t+1)},N_{l}^{(t)}}$ as input. On top of PVANET pool5, two fully-connected layers (fc1 and fc2) with learnable parameters $\{W_{1},W_{2}\}$ are added in form of a nonlinear mapping from the sample space to the feature space $\chi$. Besides, after batch concatenation, features are fed into the nonlinear binary classifier in form of three fully-connected layers (fc3, fc4 and fc5) with learnable parameters $\{W_{3},W_{4},W_{5}\}$. In experiments, we will show the necessity of all these loss functions (3)(4)(5) as basic components of the overall loss function (7).

The overall online tracking algorithm is summarized in Algorithm 1 in the Appendix A. Through offline training, our CD-CNN has learned temporal appearance continuity for generic objects. Then, in online tracking, this property can be transferred to any specific object to be tracked. In order to focus on a single specific target, it is necessary to finetune the network with the initial testing frame. Specifically, all model parameters are updated by minimizing Eqn. (6) as in offline training. The only difference is that the temporal appearance continuity loss, Eqn. (3), is replaced by the Euclidean distance in $\chi$ between positive samples in the same frame, i.e., $\mathcal{L}^{P}(P_{j}^{(t)},P_{k}^{(t)})={\lVert\Phi(P_{j}^{(t)})-\Phi(P_{k}^{(t)})\rVert}_{2}^{2}$. In this way, we can not only maintain the transferred continuity but also ensure similar feature representations for positive samples.

After finetuning CD-CNN, for each subsequent frame of the given tracking sequence, our tracker draws candidates from a Gaussian distribution, performs one-pass forward for each and selects top five candidates with the highest object-centroid scores $p(\cdot)$. The final bounding box in the $t$th frame is determined by averaging their sizes and locations.

The target appearance representation error incurred by our tracker can be defined as the Euclidean distance $\mathcal{E}:={\lVert\Phi({\hat{P}}_{*}^{(t+1)})-\Phi(P_{*}^{(t+1)})\rVert}_{2}$, where $P_{*}^{(t)}$ is the ground-truth and ${\hat{P}}_{*}^{(t)}$ is the predicted target patch in the $t$th frame.

The proof of Theorem 1 is presented in Appendix B. According to Theorem 1, for a tight upper bound, $\delta$ and $\Delta t$ should be $O\left({1\over n^{1+\alpha}}\right)$ for some small $\alpha>0$. Thus, with high probability, the number of samples drawn from each frame is $m=O\left(n^{3+\alpha}\right)$. In other words, with $O\left(n^{3+\alpha}\right)$ samples, the representation error can be upper-bounded by a small value with high probability, if $\sum_{j}{\sqrt{\hat{\mathcal{L}_{j}^{C}}}}/m$ converges.

The training video sequences are selected from ALOV [2], Deform-SOT [10] and VOT [11] without overlapping with the benchmarks [12, 13]. Since the ground-truths in ALOV were only annotated every five frames, we annotate ground-truths for the rest of the frames. In order to jointly learn temporal appearance continuity and object-centroid discrimination, the training data are generated from two consecutive frames with non-occluded targets that highlight object-centroid. Specifically, positive samples are generated with one or two pixels shifted from the ground-truth while negative samples are randomly drawn under the constraint of $0.2\leq\text{IoU}\leq 0.6$.

During offline training phase, we set $\lambda=\mu=10$ and $\beta=1$ and train CD-CNN for 20K iterations using ADAM. The initial learning rate is 0.001 for the fully connected layers (fc1$\sim$5). The convolutional layers are initialized by PVANET model pre-trained on ImageNet. In the initial frame of a test sequence, the fully-connected layers are fine-tuned for 300 iterations using SGD with learning rate 0.001. In each of the subsequent frames, 800 candidates ($m=800$) are sampled and evaluated. In handling appearance variations, we decide to update our tracking model according to the object-centroid score every five frames. The model is updated by finetuning the fc layers using samples drawn within a local window centered at the previously predicted target location. The local window is twice the size of the previously predicted bounding box. These samples are labeled according to IoU thresholds, 0.9 for the positive and 0.6 for the negative.

Figure 4 and Figure 9 illustrate the overall quantitative performance comparison on OTB2015 and OTB2013, respectively. Our tracker achieves 0.600 AUC value on OTB2015 (ranking 1st) and 0.627 on OTB2013 (ranking 2nd), outperforms most of the state-of-the-art trackers in both metrics and demonstrates very promising tracking performance. This validates the effectiveness of introducing temporal appearance continuity and object-centroid into tracking.