A Bayesian Detect to Track System for Robust Visual Object Tracking and Semi-Supervised Model Learning

Visual object detection and tracking covers a large spectrum of computer vision applications such as video surveillance, motion analysis, action recognition, autonomous driving and medical operation studies. The emergence of deep Convolutional Neural Networks (CNN) [11] makes a tremendous progress on the visual object detection and tracking performances. CNN is widely utilized for theses tasks for two reasons. Firstly, CNN learns robust object features cross total variations among the whole training dataset.

Secondly, CNN’s shift covariance properties allow generating regional proposals from the area of maximum responses on object specific detection filters.

A common deep CNN-based video object tracking system consists of two parts of object detection and object displacement networks. R-CNN [15] network is the most commonly used backbone structures for object detection. Generally, a R-CNN network consists of two stages. In the first stage, a RPN network generates object likeness score and bounding box offset coordinates from a fixed number of predefined anchors on the output feature maps. In the second stage, plausible candidate regions of high object likeness score are pooled on another feature maps for refined coordinates offsets and object class score. For object displacement predictions, a correlation layer takes the features from Siamese network on reference frames and prediction frames for object displacement predictions. Several works [10, 6, 22] refine the results by proposing multi-stages of regional proposal detection, feature propagation, objects tube linking and post-processing.

The main challenge for robust tracking includes motion blur, partial occlusions and background clutters. This poses challenges for developing robust tracking algorithms by adding uncertainties for object state estimations. Here we show some typical challenges for robust detection and tracking in Fig. 1 In response to the increasing concerns about model robustness and generalizations without introducing extra cost, we see a resurgence of Bayesian models in recent years. Bayesian approaches use a probabilistic treatment of object appearances and states to deal with the uncertainties in prediction from models. However there are several challenges for taking existing works off-the-shelf for learning and making inference on network model outputs. Firstly, there are multiple objects appear and disappear in consecutive frames. Secondly, different objects linking possibilities exists for tracking objects across different frames. Finally, modeling the uncertainties of dynamic number of appeared object’s states from a fixed R-CNN structure’s outputs is still not well solved. In our work, we propose to address the first and the second problem by formulating a joint object dynamic over the distributions of a cascaded event of object appearing/disappearing, new object arriving and object associations. And we address the third problem by considering the R-CNN outputs as a clustered emission distributions from object’s ground states over appearance scores, classification scores and location coordinates.

In this paper, we formulate the problem of multi-object detection and tracking in a fully probabilistic way. Our model is parameterized by tracking and detection neural networks. We take the original network structure from the original detect to track paper [2] with minimum modification in notations and definitions. Our formulation consists of a transition models for objects dynamics and emission model for object detection. We take neural network’s outputs as our transition and emission distribution’s parameters. Our probabilistic model is capable of handling the multi-object appearing and disappearing problem by incorporating an object appearance and association prior. In our model, we see tracking and detection as inferring objects states posteriors from network outputs. And the posterior inference is proposed by our particle filter based sampling algorithm. We use our particle filter algorithm to generate samples from an approximated family of distributions and later re-weigh and re-sample in accordance with our model formulations. Our sampled trajectories is capable of taking the relations cross all visible frames into account. Finally, we present a Variation Sequential Monte Carlo (VSMC) [14] method to learn our model from intermittent labeled frames. Our VSMC method trains on unlabeled consecutive frames by optimizing a tractable evidence lower bound (ELBO). The ELBO is approximated from sampled labels generated by our particle filter algorithm. Our contribution in this paper are three folds.

• •

  We give a deep neural network parameterized Bayesian formulation of a detection and tracking system.

• •

  We propose a particle filter sampling algorithm to make robust estimation on tracking object states from neural network outputs.

• •

  We present a weakly-supervised training algorithm for our model by VSMC methods. Our VSMC algorithm trains on both labeled frames and unlabeled consecutive frames with the pseudo-label generated by our sampling algorithm.

A comprehensive review of tracking and detection algorithm is beyond the scope of this paper. We review the works that are most related to our study.

Object Detection and Tracking Currently, state-of-art object detection and tracking systems consist of multiple stages of regional proposal detection, feature propagation, object tube linking and post-processing. In the first stage is to extract regional proposal candidates. Representing works include 2D R-CNN network in frame-level box proposal [6], 3D R-CNN network in video-level tube proposal [10] and feature propagation [22]. In the second stage, detected objects are linked together to make a tracking prediction. Either direct detection box [20] or tracking displacement [2] offset box are linked by bipartite graph algorithm [4] or Viterbi algorithm [3]. In the final stage, suppression methods are used for removing duplications and false positives.

Correlation Filter and Siamese Network Correlation filters have recently been introduced into visual tracking and shown to achieve high speed as well as robust performance [1, 17]. Siamese network with triplet loss has also shown advantages in clustering similar object samples, which has been commonly used in representation learning[8]. Bo [13] proposes a Siamese-RPN network that are trained end-to-end to regress tracking template locations. Bo [12] improves the performance by using a ResNet-50 backboned CNN and taking RPN predictions by aggregating multi-layer outputs. Zhipeng [21] proposes a deeper and wider Siamese network by adding multiple crops to remove padding effects. Instead of cropping the objects on template images, Feichtenhofer [2] uses ROI poolings on correlation feature maps for object displacement prediction from template to target.

Bayesian Model for Tracking and Detection Bayesian methods have been applied to object detection and tracking for its robust performance to deal with object state uncertainties. Tianzhu [19] combines the power of particle filter and correlation filter for robust tracking objects on image and video frames. They solve a correlation filter in dual space by accelerated proximal gradient method. And then they propose a particle filter tracker to generate particles using transition model, apply proposed correlation filter to shift it to a stable location and reweights the sample using the filtered responses. However their correlation filter is based on the shallow features in Fourier domains. Ali [7] combines the power of Bayesian inference and CNN for dealing with uncertainties of deep neural network’s observations. They assume a Gaussian prior on object location and Dirichlet prior on object categories. Xinshuo [18] proposes 3D object detection baseline. They use Gaussian distribution to formulate the object location uncertainties and use Kalman Filter to infer the trajectories of each single objects.

In this section, we give a bayesian formulation of object detection and tracking, We consider the joint process of object detection and tracking as a HMM, where the distribution of object states cross frames are modeled as hidden states. We view tracking as the transition between neighboring hidden states and detection as emission from hidden states to a noisy visible states observed by R-FCN network outputs. We would show that in the case of supervised learning, the traditional tracking and detection loss takes a similar form of the maximum likelihood function in our formulated model.

With our formulation in previous section, our robust tracking is by inferring the posterior of all objects states and their associations cross frames from the observations on R-FCN and tracking dynamics on tracking network. As our probabilistic formulation is straightforward for joint probability verification but less straightforward for sampling posterior, directly inferring the joint states of objects and their associations would be intractable. An analytical solution would require traversing all the possible object appearing and association states which is impractical for implementation. Here we give a particle filter based sampling solution by sampling from an approximated family of straightforward sampling distributions.

Initial Sample on R-FCN Detectors    Assume that R-FCN outputs M anchored prediction of bounding box state $\{\mathcal{\hat{B}}_{1,t}...\mathcal{\hat{B}}_{M,t}\}$ at frame $t$. And we sample $\{\mathcal{B}_{1,t}...\mathcal{B}_{K_{t},t}\}$ according the the posterior

And we sample $r_{1}(\{\mathcal{B}_{1,t}...\mathcal{B}_{K_{t},t}\})$in the following way(the proof would be given in the later part of this section). First, per-anchor outputs from the neural network are clustered in a similar way with NMS. Greedy clustering is performed using the output category score by choosing the anchor with the highest non-background score as the cluster center, adding any anchor with an intersection over union(IOU) greater than 0.5 to the cluster, and eliminating all members in the cluster from the original updated anchor set. This process is repeated until all the anchors are assigned to clusters. By seeing each cluster as a object candidate, we have candidates objects of $\{\tilde{\mathcal{B}}_{1,t}...\tilde{\mathcal{B}}_{K_{t},t}\}$. Then we sample the objects are their states in the following way.

1. Step 1:

   Include object $\tilde{\mathcal{B}}_{i,t}$ with probability $p_{i}$

   $$p_{i}=\prod_{j\in(u_{j}=i)}(\hat{e}_{j})^{\alpha}/(\prod_{j\in(u_{j}=i)}(\hat{e}_{j})^{\alpha}+\prod_{j\in u_{j}=i}(1-\hat{e}_{j})^{\alpha})$$

2. Step 2:

   Assign initial box state $\mu(\mathcal{B}_{i,t})$ by Eq. (15)

3. Step 3:

   Assign a new tracking id for each $\mathcal{B}_{i,t}$.

4. Step 4:

   Sample object class $\mathcal{C}_{i}\sim\text{Categorical}(c_{i})$

   $$c_{i}=\prod_{j\in(u_{j}=i)}\mathcal{K}_{j,k}^{\alpha}/\sum_{k^{\prime}}\prod_{j\in(u_{j}=i)}\mathcal{K}_{j,k^{\prime}}^{\alpha}$$

Data Association    After we include our sampled objects and their states, we associate our generated initial objects $\{\mathcal{B}_{1,t}...\mathcal{B}_{\tilde{K}_{t},t}\}^{l}$ to the objects in their ancestor particles $\{\mathcal{B}_{1,t-1}...\mathcal{B}_{K_{t-1},t-1}\}^{a_{t-1}^{l}}$. To match these two sets of objects, we first construct a affinity matrix ${J}$ with dimension of $\tilde{K}_{t}\times K_{t-1}$. We assign a score to the elements of $J$ using the summation of IoU between objects and their class scores

Then we apply Hungarian algorithm to solve the bipartite graph matching problem in polynomial time. In addition, we reject matching objects $\mathcal{B}_{i,t-1}$ with $\mathcal{B}_{j,t-1}$ when $\mathcal{B}_{j,t}$ when their IoU is below then a given threshold of $IoU_{min}$. Without loss of generality, we assume to output a set of matched objects $\{\hat{\mathcal{B}}_{1,t}...\mathcal{B}_{\hat{K}_{t},t}\}^{l}$ in frame $t$ with their ancestor $\{\hat{\mathcal{B}}_{r_{1},t-1}...\mathcal{B}_{r_{\hat{K}_{t}},t-1}\}^{a_{t-1}^{l}}$ in frame $t-1$, along with the unmatched objects $\{\mathcal{B}_{\hat{K}_{t}+1,t}...\mathcal{B}_{\tilde{K}_{t},t}\}^{l}$.

Independent Proposal Update for Tracking Object After we associate our sampled objects with their ancestors, we update the state of each sampled objects independently by sampling their posterior conditioned on their ancestors. Prior on the transition probability of matched objects, we sample each object’s bounding box state from their corresponding posterior. We use this proposal distribution to minimise the variance of the importance weights

where the sufficient statistics of proposal in Eq. (19) can be estimated in closed form as:

where $M_{i}$ is the number of anchored observations for object $\mathcal{B}_{i,t}$ in R-FCN.

Similarly, for unmatched objects $i^{\prime}$, $\mu^{\prime}_{0}(\mathcal{B}_{i^{\prime},t}),\Sigma^{\prime}_{0}(\mathcal{B}_{i^{\prime},t})$ could be given by

Finally, we keep the sampled class $\mathcal{C}_{i^{\prime}}$ for unmatched objects ${i^{\prime}}$. And we discard the sampled class for matched object $i$, and assign $\mathcal{C}_{i}$ with the class of its matched ancestor objects $\mathcal{C}_{r^{-1}_{i}}^{a_{t-1}^{l}}$. Similarly, we keep the tracking id for unmatched objects and assign matched objects $\mathcal{B}_{i,t}$ with an id of $id(\mathcal{B}_{r_{i}^{-1},t-1}^{a_{t-1}^{l}})$

In this section, we describe our model training algorithm in the case of semi-supervised settings. In such settings, we only have the labeled bounding box on frame $t$. In the neighboring frames $\{t-T:t-1\}$ and $\{t+1:t+T\}$, only visual input is given without labeled bounding box. Different from our supervised learning algorithm in sectionD, training on the joint distribution for the unlabled frames is not feasible. Instead, our objective function is set as log likelihood of the marginal distributions where the unseen annotations are marginalized.

Eq. (5) is derived from the property of HMM. The total loss objective could be decomposed into three terms. The first term $\log p(\{\mathcal{\hat{B}}_{1:M,t}\},\{\mathcal{B}_{1:K_{t},t}\})$ is the supervised detection loss on the same form as the traditional R-FCN detection loss. And the second term is the marginal likelihood of forward predictions on frame $\{t+1:t+T\}$. The third term is the marginal likelihood of backward prediction on frame $\{t-T:t-1\}$. Both marginal likelihood for forward and backward predictions are conditioned on frame $t$’s annotation $\{\mathcal{B}_{1:K_{t},t}\}$ as a weakly supervised training signal for the unlabeled neighbor frames. As the backward likelihood could be factorized in the same structure of forward likelihood. We only derive the variational bound for forward likelihood in later part of the section. For the semi-supervised term, training directly is intractable, we use the following surrogate ELBO as a substitution

For convenience of understanding the surrogate ELBO loss for semi-supervised learning of our tracking and detection model. We link the surrogate ELBO loss to our widely adopted tracking and detection loss. By taking the gradient, the surrogate loss could be rewritten as

where $\hat{w}_{t+t^{\prime}}^{l}=w_{t+t^{\prime}}^{l}/\sum_{l}w_{t+t^{\prime}}^{l}$. The surrogate loss could be nicely decomposed into the weighted sum of tracking and detection loss over frame $\{t+1:t+T\}$ minus the log density of sampling distributions on sampled particle’s importance weight. Built on the our particle filter sampling algorithm, our semi-supervised learning algorithm could be easily implemented on traditional tracking and detection loss. The only modification is that we include a set of sampled particles as a proxy for object annotation and take the weighted sum of their supervised loss by their importance weight. And this training step is iterated in the whole training process.To avoid introducing additional variations, we omit gradients of $\log r(\mathcal{B}_{1:K_{t^{\prime}+1},t^{\prime}+1};\lambda)$. The detailed training loss is shown in Fig. 4.

To show the effectiveness of incorporating uncertainty treatment in tracking and detection comparing with traditional methods, we compare our performance with non-Bayesian baselines. Our evaluation is on two commonly used datasets.

ImageNet Video Object Detection Dataset (ILSVRC) [16] contains 30 classes in 3862 training and 555 validation videos. The objects have ground truth annotations of their bounding boxes and track IDs.

M2Cai16-Tool-Locations Dataset [9] extends the M2Cai16-tool dataset. It contains 15 videos record at 25 fps of cholecystectomy procedures at the University Hospital of Strasbourg in France. Among those videos, 2532 frames are labeled under the supervision and spot-checking from a surgeon with medical devices including Grasper, Bipolar, Hook, Scissors, Clipper, Irrigator and Specimen Bag.

To have a fair comparison of the Bayesian approach with the baseline, all the methods use the same structure of training and inference network and sharing training configurations in a much similar way. We only introduce an additional loss term for object transition’s covariance matrix (only diagonal elements) in Eq. (17).

In this paper, we present our Bayesian model for multi-object detection and tracking in videos. Our method has shown the potential of formulating neural network model in a probabilistic way, especially for tasks that need to infer under uncertainties.