Fast Video Object Segmentation with Temporal Aggregation Network and
Dynamic Template Matching

The goal of video object segmentation (VOS) is similar to multi-object tracking (MOT), which is to predict the trajectories of multiple objects at a fine-grained level over time. We denote the set of trajectories ${\bf{T}}={\left\{{{{\bf{T}}_{i}}}\right\}_{i=1}^{N}}$, where the trajectory of the $i^{\text{th}}$ object can be represented by a series of bounding boxes together with the masks within, denoted by ${{\bf{T}}_{i}}={\left\{{{\bf{b}}_{i}^{t},\,{\bf{m}}_{i}^{t}}\right\}_{t=1}^{T}}$, ${\bf{b}}_{i}^{t}=\left[{x_{i}^{t},y_{i}^{t},w_{i}^{t},h_{i}^{t}}\right]$; $x_{i}^{t}$ and $y_{i}^{t}$ denote the center location of the target $i$ at frame $t$; $w_{i}^{t}$ and $h_{i}^{t}$ denote respectively the width and height of the target object $i$; ${{\bf{m}}_{i}^{t}}$ denotes the corresponding foreground segmentation mask within corresponding bounding box. In MOT usually only ${\{{{\bf{b}}_{i}^{t}}\}_{t=1}^{T}}$ is needed, while VOS is more fine-grained and should produce ${\{{{\bf{m}}_{i}^{t}}\}_{t=1}^{T}}$. Bounding box representation is popular in recent detection pipelines [46, 24] while some non-box based methods [12] have also been proposed. We mainly adopt box based detection pipelines in this paper but note that our method is also compatible with non-box based pipelines.

To output segmentation mask, a light-weight Fully Convolutional Network is attached to the detector; foreground/background segmentation is performed on each detected bounding boxes. For simplicity, we ignore the ${\{{{\bf{m}}_{i}^{t}}\}_{t=1}^{T}}$ which can be easily generated from ${\{{{\bf{b}}_{i}^{t}}\}_{t=1}^{T}}$. Under semi-supervised setting of DAVIS Benchmark [44, 45, 5], the tracking targets are given in the first frame. We denote tracking targets as ${\{{\bf{t}}_{i}\}}_{i=1}^{N}$.

Our method follows the online tracking-by-detection model [54], which first detects multiple objects in each frame and then associates their identities across frames. Following [4, 29] we first train the whole detector offline with the training set in order to adapt the detector to the DAVIS dataset domain. Before inference, one-shot learning is performed with the given ground-truth segmentation for a few iterations to reduce false alarm. Note that one advantage of our training strategy is that no tracking-specific training is required, which implies that our method can be readily extended to various applications.

During inference, different from [2] and for simplicity we do not exploit bounding box regressor for tracking. Instead, simple Intersection over Union (IoU) metric is adopted. For segmentation based approaches, IoU is hard to be incorporated into the pipeline since there is no explicit bounding box or instance. Our method first detects multiple objects in each frame and then associates their identities across frames. The detected boxes at frame $t$ with confidence score greater than $\sigma_{\text{det}}$ are denoted as $\{{\bf{b}}^{t}_{j}\}_{j=1}^{N_{t}}=D_{t}$. Note that $N_{t}$ is not necessarily equal to $N$ due to false positives and false negatives output of the detector. At $t=0$, our tracker initializes the tracklet from the first frame bounding boxes ${\{{\bf{t}}_{i}\}}_{i=1}^{N}=\{{\bf{b}}^{0}_{j}\}_{j=1}^{N_{0}}=D_{0}$. At frame $t$, a bipartite graph is constructed based on location similarity IoU, the weight $\omega^{\text{loc}}_{ij}$ between $i$th target and $j$th detected object in the current frame is defined as the IoU of ${\bf{t}}_{i}$ and ${\bf{b}}^{t}_{j}$. After Hungarian matching [40] is performed, the target ${\bf{t}_{i}}$ will be updated with ${\bf{b}}^{t}_{k}$ and added to ${\bf{T}}_{i}$ if matched. When running the assignment process in a frame-by-frame manner the object trajectories are produced.

This naive tracking-by-detection pipeline under the setting of semi-supervised video object segmentation will be extended in following sections.

Figure 3 illustrates the Temporal Aggregation Network (TAN) with details to be described in the following.

Many methods in object tracking have exploited temporal information to facilitate tracking. However, most of them require optical flows to obtain pixel-wise alignment [64, 63] which can be quite computationally extensive. Inspired by the recent progress in large-scale action recognition benchmarks [7, 22], we propose a novel Temporal Aggregation Network to incorporate backbones in image classification and video recognition, which align well with the tracking-by-detection model as well as detection in image and tracking in video.

Image Backbone Most object detection methods have exploited ImageNet [17] pretrained backbone for faster convergence as studied in [23]. We use ResNet [25] as the standard setting similarly done in other detection approaches. We denote the outputs of stage 3, 4, 5 as c3, c4, c5 respectively. The input of Image Backbone is the key frame where objects are to be detected.

Video Backbone Inspired by recent progress in Human Action Recognition [7, 51, 18], we believe 3D convolution is effective in utilizing temporal information across consecutive frames, although no previous works have attempted to utilize this technique to tackle VOS task. Note that while optical flow approaches in [38, 32] utilize explicit correspondence, 3D convolution network can directly learn temporal patterns from an RGB stream. According to [7], I3D performs best against other methods such as CNN+LSTM counterpart. Therefore, we adopt I3D in our video backbone and at the same time, similar to [51], $\text{I3D}_{3\times 1\times 1}$ and $\text{I3D}_{1\times 3\times 3}$ are used in order to reduce the computation cost of 3D convolution. We denote the outputs of stage 3, 4, 5 as i3, i4, i5 respectively. The previous $\alpha$ frames and current key frame are concatenated along the temporal axis resulting in a 3D tensor of size $(\alpha+1)\times H\times W$ which is then fed into the video backbone.

Temporal Aggregation Fusion Inspired by SlowFast [18], the separation and fusion of fast/slow video stream is a good practice to make use of temporal information while preserving key frame feature. As conv-add style in [18], the new feature map $\text{f}_{i}$ is computed by $\text{c}_{i}+\mathcal{N}(\text{i}_{i})$ for stages 3, 4, 5, where $\mathcal{N}$ denotes a small CNN to fuse features with Max Pooling to compress information along the temporal dimension so that the two branches have the same size at each stage. $\text{f}_{i}$ will be used in the subsequent detection networks. Unlike RNN based methods [32, 47] which require sophisticated training schedule, our network is feed-forward and can be jointly trained. As shown in Figure 5 in the experimental section, by incorporating temporal information, our system can better handle occlusion.

Obviously the naive IoU based tracking algorithm is not robust to large camera movement or object deformation. As pointed out in [2, 38], re-identification is also essential to object tracking. Thus we propose the novel Dynamic Time-evolving Template Matching (DTTM) to tackle the re-identification problem. Figure 4 summarizes the method.

Instead of cropping bounding box regions from RGB image space, we prefer the feature produced by the backbone since it is encoded with high-level semantic meaning. Let $\mathcal{A}({\bf{b}}^{t})$ denote the appearance feature in bounding box $\bf{b}$ extracted from the backbone feature map of frame $t$ which is a high dimensional vector. The appearance similarity weight term is defined as

$${\omega^{\text{app}}_{ij}}=\frac{\mathcal{A}({\bf{t}}_{i})\cdot\mathcal{A}({\bf{b}}^{t}_{j})}{\left\lVert\mathcal{A}({\bf{t}}_{i})\right\rVert\cdot\left\lVert\mathcal{A}({\bf{b}}^{t}_{j})\right\rVert}$$

(1)

As pointed out in [47, 56], location cue and appearance cue are essential features in multiple object tracking. Therefore the bipartite graph is constructed using $\omega_{ij}=\omega^{\text{loc}}_{ij}+\omega^{\text{app}}_{ij}$ in order to take both cues into account, where $\omega_{ij}^{\text{loc}}$ has been defined in Section 3.1.

In ideal scenarios, the learned appearance feature from the first frame is sufficient for tracking the rest. However, a constant template clearly does not work in practice especially in long-term tracking, since the target objects may undertake many appearance changes due to deformation, occlusion, etc. So a proper design for time-evolving template is essential, which leads to our online updating template appearance features during the tracking progress.

Moving average is one of the widely adopted approaches for template update across temporal axis. The single hyper parameter $\mathit{momentum}$ controls the proportion of features to be updated or preserved, which is extremely sensitive to the frame rates of different videos. The static averaging process is also obviously sub-optimal due to accumulation error and feature blurring. We propose to update the template feature more dynamically in a discrete manner.

DTTM Full algorithm detailing our matching strategy in pseudocodes can be found in the supplementary material. Initially, the template bank for each target $i$ is constructed as $\mathit{bank}_{i}=\{{\bf{t}}_{i}\}$. The matching result is obtained by performing linear assignment between ${\bf{t}}_{i}\in\cup_{i=1}^{N}\mathit{bank}_{i}$ and ${\bf{b}}^{t}_{j}\in\{{\bf{b}}^{t}_{j}\}_{j=1}^{N_{t}}$ at each frame $t$ by computing $\omega^{\text{loc}}_{ij}+\omega^{\text{app}}_{ij}$. Given foreground confidence score $\text{conf}({\bf{b}}^{t}_{j})$ greater than threshold $\sigma_{\text{conf}}$, when ${\bf{t}}_{i}$ and ${\bf{b}}^{t}_{j}$ is a match but their appearances are not similar $\omega^{\text{app}}_{ij}<\sigma_{\text{app}}$ , which indicates a drastic appearance change and thus a very likely situation of losing track in the upcoming frames, our DTTM will initialize a new template from the latest matched detection ${\bf{b}}_{j}^{t}$ for the target object ${\bf{t}}_{k}$ and add it into corresponding feature bank, which results in an extended template bank for the future assignment.

To avoid overflow, the least frequently used template will be removed from the $\mathit{\mathit{bank}}_{i}$ when the number of templates is larger than some threshold. Both the initial target feature and updated feature can be considered as templates during later matching process. Our system can thus detect potential accumulation error early and thus prevent its adverse effect by updating with high confidence new template feature. Figure 4 illustrates how DTTM can effectively address the deformation problem in VOS, where not only the latest but also previous templates are considered so that abrupt deformation can be well handled. Note that this matching mechanism is entirely online and time-evolving unlike [38], namely, in our DTTM no future information is required and the template bank keeps evolving over time.

We use DAVIS benchmarks [44, 45, 5] to evaluate our method. These benchmarks are challenging, with large variety in scenes, multiple heterogeneous objects and occlusion, and are widely used in evaluating video object segmentation methods.

DAVIS 2016 DAVIS 2016 comprises of a total of 50 sequences, 3455 annotated frames, all of which were captured at 24fps and Full HD 1080p spatial resolution. Since computational complexity is a major bottleneck in video processing, each sequence has a short temporal extent (about 2–4 seconds) while including all major challenges typically found in longer video sequences

DAVIS 2017 After releasing DAVIS 2016 [44], several strong methods have been proposed making the top performance on DAVIS 2016 saturated. DAVIS 2017 was released which is an extension of the former version and a more challenging benchmark. Overall, the new dataset consists of 150 sequences, totaling 10459 annotated frames and 376 objects. One major difference from DAVIS 2016 is that in DAVIS 2017, multiple object tracking is introduced in video object segmentation. In DAVIS 2016, only single objects are annotated for each frame while in DAVIS 2017, multiple object masks in the same frame are annotated, which makes the task more challenging as complex interaction among multiple target objects can cause occlusion and thus change of topology and appearance. Moreover, as Figure 1 shows, many target objects are in the same category and have very similar appearance. Another important challenge in DAVIS 2017 is that target object category in one video can become background in another video, thus demanding high discriminative ability of the tracker to adapt to different target objects given only the first frame annotation. Note that DAVIS 2017 is also the dataset for DAVIS 2019 Challenge, which we will use in the following unless otherwise specified.

We use Faster R-CNN [46] as the detector and ResNet-50 [25] as our default backbone unless otherwise specified. The spatial branch backbone is initialized with weights given by ImageNet [17] classification and the detector is pretrained on COCO [36] as done in [38]. As for the temporal aggregation branch, we use 8-frame input I3D baseline provided by Nonlocal Network [51]. In order to incorporate multiple scales during tracking, feature pyramid network (FPN) [34] is adopted in the backbone to merge high-level semantic information from deeper coarse feature map with shallow fine-grained feature map. Following [34], the RPN anchors span 5 scales (feature is extracted from different levels of FPN outputs) and 3 aspect ratios [0.5, 1.0, 2.0]. Since category classification is unavailable in the DAVIS dataset, the last linear classification layer of R-CNN is replaced by a simple fully connected layer for foreground/background classification. This class agnostic R-CNN is more suitable for semi-supervised setting and more general for real applications. Vanilla Fully Convolution Network [37] is adopted as the segmentation head as default. As in Fast R-CNN [21], an RoI is considered positive if its IoU with the ground-truth box is at least 0.5 and negative otherwise. The segmentation is only performed on positive RoIs during both training and testing.

Training To overcome the domain gap between the pretrained dataset and DAVIS, we first train the entire detection network on DAVIS training set. The input images are resized such that their shorter side is 800p which is similarly done in [46] unless otherwise specified. We train on 8 GPUs with 2 images per GPU (effective mini batch size of 16). All layers of backbone except for c1 and c2 of backbone are jointly fine-tuned with detection annotation. Due to the limited batch size, all batch normalization layers are frozen so that the statistics of mean and variance are unchanged during training. Unlike stage-wise training with respect to RPN in [24], end-to-end training as in [34] is adopted in our implementation which yields better results. All models are trained for 100k iterations using synchronized SGD with a weight decay of 0.0001 and momentum of 0.9. The learning rate is initialized to 0.002, which decays by a factor of 10 after 70k iterations and 90k iterations. Other choices of hyper-parameters also follow the setting in [46].

Inference At test time, we mostly follow the setting of Faster-RCNN in [34]. The box prediction is done on RPN proposals, followed by non-maximal suppression [19]. The segmentation head is then applied to the detected bounding boxes above threshold $\sigma_{\text{conf}}$ only. The instance segmentation results are merged into a single segmentation map by ranking corresponding bounding box confidence scores.

The ablation study is performed on the DAVIS 2017 dataset. The outputs are resized to 480p for evaluation.

Table 5 tabulates the quantitative results in comparison with leading methods with qualitative results presented in Figure 7. We achieve state-of-the-art single model results in both speed and accuracy on DAVIS Benchmark with stronger backbone ResNeXt-101 [55] and deformable convolution [16], both contribute to further improving overall performance, which again justifies the advantage of our simple and easily extensible pipeline. Despite DAVIS dataset, many leading methods exploit YouTube-VOS [57] as pretraining. In Table 6 we demonstrated that our method achieved higher performance without computational demanding YouTube-VOS pretraining. The speed (t/s) reported in Table 6 considers that there are more than 2 objects on average on DAVIS-2017 dataset. STM [42] needs to handle them individually, while we could detect and track all the targets in one pass. We also reported a reasonable result on YouTube-VOS, $73.5\%$ $\mathcal{J}\&\mathcal{F}$-Mean without carefully selecting hyperparameter.