Exploring the Semi-supervised Video Object Segmentation Problem from a Cyclic Perspective

Semi-supervised video object segmentation has been widely researched in recent years with the rapid development of deep learning techniques. Depending on the presence of a learning process during inference stage, the segmentation algorithms can be generally divided into online methods and offline methods. OVOS Cae_OVOS_17 is the first online approach to exploit deep learning for the VOS problem, where a multi-stage training strategy is design to gradually shrink the focus of network from general objects to the one in reference masks. Subsequently, OnAVOS voigtlaender17BMVC improved the online learning process with an adaptive mechanism. MaskTrack  Perazzi_2017_CVPR introduced extra static image data with mask annotation and employed data synthesized through affine transformation, to fine-tune the network before inference. All of these online methods require explicit parameter updating during inference. Although high performance can be achieved, these methods are usually time-consuming with a real-time FPS of less than 1, rendering them unfeasible for practical deployment.

On the other hand, there are a number of offline methods that are deliberately designed to learn generalized correspondence feature and they do not require necessary online learning process during inference time. RGMP Oh_2018_CVPR designed an hourglass structure with skip connections to predict the objective mask based on the current frame and previous information. S2S Xu_2018_S2S_ECCV proposed to model video object segmentation as a sequence-to-sequence problem and proceeds to exploit a temporal modeling module to enhance the temporal coherency of mask propagation. Other works like Zeng_2019_ICCV ; luiten2018premvos resorted to using state-of-the-art instance segmentation or tracking pipeline He2018Mask ; peng2020ctracker while attempting to design matching strategies to associate the mask over time. A few recent methods FEELVOS Voigtlaender_2019_CVPR and AGSS-VOS Lin_2019_ICCV mainly exploited the guidance from the initial reference and the last previous frame to enhance the segmentation accuracy with deliberately designed feature matching scheme or attention mechanism. STM Oh_2019_ICCV further optimized the feature matching process with external feature memory and an attention-based matching strategy, such memorial structure is further optimized by involving global context GCVOS2020 , adaptive gaussian kernel KMN and structure of vision transformer AOT2021 . Compared with online methods, these offline approaches are more efficient. However, to learn more general and robust pixel-wise feature correspondence, these data-hungry methods may require backbones pretrained on large amounts of extra data with mask annotations from other tasks such as instance segmentation COCO ; Everingham15 or saliency detection CSSD . Without these auxiliary help, the methods might well be disrupted by distractions from similar objects in the video, which then propagates erroneous mask information to future frames.

All the approaches mentioned above follow a standard sequential processing order from start to end of the video and can not ensure the predicted mask is closely related to the initial reference guidance at first frame. In contrast, by embedding the cyclic mechanism into training stage, our method explicitly impose the constraint of reference mask in learning process. Besides, the online extension of gradient correction does not update the pretrained model parameters as other online learning methods, but dynamically refine the output according to the modification information from the reliable initial reference mask.

Cycle consistency is widely researched in unsupervised and semi-supervised representation learning, where a transformation and its inverse operation are applied sequentially on input data, the consistency requires that the output representation should be close to the original input data in feature space. With this property, cycle consistency can be applied to different types of correspondence-related tasks. trackcolor is a classical technique for correspondence learning, which treat the learning problem as video colorization and impose the consistency on natural color space to force embeddings of the same semantics to be closer in feature space. In the work of cycle3d , a multiple step cycle-loop is build to connect correspondence relationship between real images and rendered shots from 3D models.  CVPR2019_CycleTime combined patch-wise consistency with a weak tracker to construct a forward-backward data loop and this guides the network to learn representative feature across different time spans, jabri2020walk further extend such cyclic data loop to a random walk process. Meister:2018:UUL exploited the cycle consistency in unsupervised optical flow estimation by designing a bidirectional consensus loss during training. On the other hand Cycle-GAN CycleGAN2017 and Recycle-GAN Bansal2018Recycle and other popular examples of how cyclic training can be utilized to learn non-trivial cross-domain mapping, yielding reliable image-to-image transformation across different domains.

Our method with cyclic mechanism is different from the works mentioned above in following aspects. First, the motivation to exploit cycle consistency in our work is to explicitly regularize the predicted mask to be accurate for backward reference in cyclic loop, while in unsupervised methods like CVPR2019_CycleTime , the cycle consistency is an approach to obtain correspondence ground-truth. Further, the learning objective of our work is still a fully supervised segmentation task, with high-level and clear semantic information (the object is taken as foreground while the others are background). In contrast, methods with unsupervised cycle consistency for correspondence learning construct their objective by self-mimic in low-level semantics (e.g. the color space trackcolor , spatial position CVPR2019_CycleTime or transformation flow cycle3d ), thus the learned embeddings are easy to correspond to distractors with similar low-level expression if there is not sufficient context provided. Consequently large amount of data are required to train these unsupervised methods to learn to catch the context relationship. Finally, our cyclic structure is not only applicable during training, but also useful in the inference stage. By measuring the consistency between initial reference mask and predicted results, we can refine the output on current frame to obtain more accurate guidance for future prediction.

Given a video of length $T$, $X_{t}$ is the $t$-th frame ($t\in[1,T]$) in temporal sequential order, and $Y_{t}$ is its corresponding annotation mask. $\mathcal{S}_{\theta}$ is an object segmentation network parameterized by learnable weights $\theta$. In terms of the sequential processing order of the video, the segmentation network should achieve the function as in Equation (1) below:

where $\widehat{Y}_{t}$ denotes the predicted object mask at $t$-th frame. $\mathcal{X}_{t-1}\subset\{X_{i}|i\in[1,t-1]\}$ is the reference frame set, which is a subset of all frames appearing before objective frame $X_{t}$. Similarly, $\mathcal{Y}_{t-1}$ is a set containing reference object masks corresponding to the reference frames in $\mathcal{X}_{t-1}$. However, in the semi-supervised setting, only the initial reference mask at the first frame is available. Therefore, in the inference stage, the corresponding predicted mask $\widehat{Y}_{t}$ is taken as the approximation of the reference mask. Hence, we have $\mathcal{Y}_{t-1}\subset\{Y_{1}\}\bigcup\{\widehat{Y}_{i}|i\in[2,t-1]\}$.

For the sake of mitigating error propagation during training, we incorporate the cyclical process into the offline training process to explicitly bridge the relationship between the initial reference and predicted masks. To be specific, as illustrated in Figure 2, after obtaining the predicted output mask $\widehat{Y}_{t}$ at frame $t$, we construct a cyclic reference set for frames and mask set, respectively denoted as $\widehat{\mathcal{X}}_{t}\subset\{X_{i}|i\in[2,t]\}$, $\widehat{\mathcal{Y}}_{t}\subset\{\widehat{Y}_{i}|i\in[2,t]\}$.

With the cyclic reference set, we can obtain the prediction for the initial reference mask in the same manner as sequential processing:

Consequently, we apply mask reconstruction loss (in Equation 3) during supervision, optimizing on both the output mask of the $t$-th frame $\widehat{Y}_{t}$ and the backward prediction $\widehat{Y}_{1}$.

In implementation, we utilize the combination of cross-entropy loss and mask IOU loss as supervision at both sides of the cyclic loop, which can be formulated as,

where $\Omega$ denotes the set of all pixel coordinates in the mask while $Y_{t,u}$ and $\widehat{Y}_{t,u}$ are the normalized pixel values at coordinate $u$ of the masks, $\gamma$ is a hyperparameter to balance between the two loss terms. It should also be noted that the cyclic mechanism in Figure 2 indirectly applies data augmentation on the training data by reversing the input clips in temporal order, helping the segmentation network to learn more general feature correspondences.

Cyclic refinement process. After training with the cyclic loss as Equation (3), we can directly apply the offline model in the inference stage. However, inspired by the cyclic structure in training process, we can take the accurate initial reference mask as a measurement to evaluate the segmentation quality of current frame and proceed to refine the output results based on the evaluation results. In this way, we can explicitly reduce the effect of error propagation during inference time to keep our trained model from disturbances by natural or adversarial noises.

To achieve this goal, we design a gradient correction block to update segmentation results iteratively as illustrated in Figure 2. Since only the initial mask $Y_{1}$ is available in inference stage, we apply the predicted mask $\widehat{Y}_{t}$ to infer the initial reference mask in the same manner as Equation (2), and then evaluate the segmentation quality of $\widehat{Y}_{t}$ with the loss function in Equation (4). Intuitively, when more accurate prediction mask $\widehat{Y}_{t}$ are taken as reference, smaller reconstruction error for $Y_{1}$ will be yielded; therefore, during the gradient correction, our algorithm is focusing on minimizing the reconstruction error of the reference mask

Where the loss term $\mathcal{L}(\cdot,\cdot)$ adopts the same formulation as Equation (4) The gradient descent method is adopted to solve the reconstruction problem in Equation (7) so as to refine the mask $\widehat{Y}_{t}$. To be specific, we start from an output mask $\widehat{Y}^{0}_{t}=\widehat{Y}_{t}$, and then update the mask for $N$ iterations:

where $\alpha$ is a predefined correction rate for mask update and $N$ is the iteration times. With this iterative refinement, we naturally extend the offline model to an online inference algorithm. However, the gradient correction approach can be time-consuming since it requires multiple times of network forward-backward pass. Due to this reason, we only apply gradient correction once per $K$ frames to achieve good performance-runtime trade-off.

Interpretation in the frequency domain. Empirically, with the gradient correction process in Equation (8), the details of output mask can be better handled. This claim can be demonstrated from the aspect of frequency domain, where we find that the gradient correction module empirically acts a high-frequency amplifier to polish the output mask in fine-grained details. To analyze from the aspect of frequency, we take the gradient correction module as a black box system and calculate its frequency response by computing the averaged ratio between output and input amplitude-frequency characteristics,

where $\mathcal{FFT}(\cdot)$ denotes 2D Fast Fourier transform. We visualized the 2D frequency-domain intensity of the input mask, output mask of gradient correction and the frequency-domain response $\mathcal{AF}_{GC}$ in the form of amplitude-frequency characteristic contour in Figure 3. We observe that compared with the input mask, the output mask manifests stronger intensity in high-frequency component (the four corner of the contour figure), the frequency response of gradient correction module also highlights and amplifies the part corresponding high-frequency harmonic. This observation provides the explanation of higher boundary accuracy improvement from the perspective of frequency since the high-frequency component usually supplements some detailed information of output masks.

Anti-noise regularization. However, intuitively, amplifying the high-frequency component will not only append details but also results in noise and artifacts around the object boundaries. To overcome such artifacts from gradient-correction, we augment the reconstruction error with a regularization term to suppress potential noise after refinement, denoted as

where $\mathcal{L}_{smooth}\left(\widehat{Y}_{t}\right)$ is a spatial smooth term to avoid exaggerating some spot-like area in output masks with $\lambda$ as its corresponding weight parameter.

In instantiation, the Sobel operators $\nabla_{x},\nabla_{y}$ are first applied on the mask $\widehat{Y}_{t}$ along the horizontal and vertical directions to calculate the spatial gradient, then areas with large spatial gradient norm are penalized. The augmented reconstruction objective can still be optimized according to the manner of Equation (8).

The cyclic mechanism with gradient update in Equation (8) is not only helpful for the output mask refinement, but it also offers a new aspect of analyzing the region of interests of specific objects segmented by the pretrained network. In detail, we construct a reference set, $\mathcal{X}_{l}=\{X_{l}\}$ and $\mathcal{Y}_{l}=\{\mathbf{0}$} as the guidance, where $\mathbf{0}$ denotes an empty mask of the same size as $X_{l}$ but is filled with zeros. We take these references to predict objects at the $t$-th frame $\widehat{Y}_{t}$. To this end, we can obtain the prediction loss $\mathcal{L}(\widehat{Y}_{t},Y_{t})$. To minimize this loss, we conduct the gradient correction process as in Equation (8) to gradually update the empty mask for $M$ iterations. Finally, we take the ReLU function to preserve the positively activated areas of the objective mask as our final cycle-ERF representation, the resulting receptive field can be expressed as

As we will show in our experiments, the cycle-ERF is capable of properly reflecting the support region of specific objects for the segmentation task. Through this analysis, the pretrained model can be shown to be particularly concentrated on certain objects in videos, making the learned segmentation model more interpretable.

Datasets. We train and evaluate our method on three widely used benchmarks for semi-supervised video object segmentation, DAVIS16 Perazzi2016 , DAVIS17 Pont-Tuset_arXiv_2017 , and Youtube-VOS Xu_2018_S2S_ECCV . DAVIS16 contains 50 videos in total, where 30 sequences are used for training and the others are taken as validation. In this benchmark, each video only covers a single reference mask. DAVIS17 is a multi-object extension of DAVIS16 and contains 120 video sequences in total with at most 10 objects in a video. The dataset is split into 60 sequences for training, 30 for validation, and the other 30 for test. The Youtube-VOS is larger in scale and contains more object categories. There are a total of 3,471 video sequences for training and 474 videos for validation in this dataset with at most 12 objects in a video, it also contains videos where objects appear from intermediate frames. Following the training procedure in Oh_2019_ICCV ; Voigtlaender_2019_CVPR , we construct a hybrid training set by mixing the data from all training sequences.

Metrics. For evaluation on DAVIS16, DAVIS17 validation, and test set, we adopt the metric following standard DAVIS evaluation protocol Pont-Tuset_arXiv_2017 . The Jaccard overlap $\mathcal{J}$ is adopted to evaluate the mean IOU between predicted and groundtruth masks. The contour F-score $\mathcal{F}$ computes the F-measurement in terms of the contour based precision and recall rate. The final score is obtained from the average value of $\mathcal{J}$ and $\mathcal{F}$. The evaluation on Youtube-VOS follows the same protocol except that the two metrics are computed on seen and unseen objects respectively and averaged together.

Baselines. We take three recent methods as our base model for further analysis of our proposed cyclic mechanism in the following experiments.

• •

  Space Time Memory Network (STM) Oh_2019_ICCV is a widely used pipeline for fast and accurate semi-supervised video object segmentation, and the memory mechanism makes it flexible in adjusting reference sets $\mathcal{X}_{t}$ and $\mathcal{Y}_{t}$, which is suitable as comparison to our reference-based approach.

• •

  Kernelized Memory Network (KMN) KMN is the kernelized version of STM network, where a Gaussian kernel is dynamically calculated before merging the knowledge from memory into current query feature.

• •

  Global Context Memory Network (GCM) GCVOS2020 extends the STM-like structure with a global temporal span, where the spatially global context of each reference frame is stored and dynamically updated in the memory.

• •

  Associate Objects with Transformer (AOT) AOT2021 is a more advanced and efficient video object segmentation framework, where a long-short term cross attention structure is designed to help with parallel video object segmentation on multiple objects.

Since there is no public training code of Oh_2019_ICCV ; KMN ; GCVOS2020 , we implement them by ourselves, and for  AOT2021 , we directly implement our cycle mechanism from the public code from the author. For STM, in order to adapt to the time-consuming gradient correction process, we take the lightweight design by reducing the intermediate memory feature dimension, resizing the input resolution for inference to $240\times 427$, which is $1/4$ of the size in original work Oh_2019_ICCV ($480\times 854$), and then upsampling the output to original resolution by nearest interpolation. For KMN, we replace the argmax operation in original paper with soft-argmax to make sure the network is end-to-end trainable. For AOT, we modify the one-hot encode into soft labels to ensure the cycle loop is end-to-end differentiable. For ease of representation, we denote the models trained with cyclic scheme with a “-cycle” suffix. It should be mentioned that the adopted baseline models from the original papers involved different external static data from various segmentation datasets COCO ; Everingham15 , resulting in unfair and inconsistent comparisons here. To enable fairer and more consistent comparison, for most of our analysis, we re-train our implemented models using only the training data in DAVIS and Youtube-VOS. Nevertheless, for more comprehensive comparison, we still provide results of STM and AOT with the same setting as Oh_2019_ICCV , where the model is pretrained purely on COCO COCO and predict the object mask at the resolution of ($480\times 854$), which is the most commonly adopted experimental setting.

Implementation details. The training and inference procedures are deployed on an NVIDIA TITAN Xp GPU. Within an epoch, for each video sequence, we randomly sample 3 frames as the training samples – the frame with the smallest timestamp is regarded as the initial reference frame. Similar to Oh_2019_ICCV , the maximum temporal interval of sampling increases by 5 every 20 training epochs. We set the hyperparameters as $\gamma=1.0$, $\lambda=0.75$, $N=10$, $K=5$, and $M=50$. During training, we adopt a bootstrapping strategy for the cross entropy loss, where only the top $40\%$ pixels with maximum training loss are taken into account. The ResNet series of models He_2016_CVPR pretrained on ImageNet imagenet_cvpr09 are adopted as our backbone for baseline. The network is trained with a batch size of 4 for 240 epochs in total and is optimized by the Adam optimizer adam2014method of learning rate $10^{-5}$ and $\beta_{1}=0.9,\beta_{2}=0.999$. In both training and inference stages, the input frames are resized to the resolution of $240\times 427$. The final output is upsampled to the original resolution by nearest interpolation. For simplicity, we directly use $X_{t}$ and $\widehat{Y}_{t}$ to construct the cyclic reference sets. For the case of multiple objects, we adopt the soft aggregation method in Lin_2019_ICCV ; Oh_2019_ICCV to normalize the probability at each pixel of output masks.

Data augmentation. We apply common augmentation operations including random horizontal flipping, additive Gaussian noise and contrast enhancement. Additionally, we also adopt random crop strategy and fixed affine transformation (sheer, resize, rotation) to each training sample. We note that frames selected from the same video will share the same transformation parameters. When exploiting COCO to synthesize data, we follow stm-coco to take random affine and copy-paste trick to generate pseudo sequences.

In this section, we first report the comparison results between our cyclic model and other methods, where our full model is measured with configuration of different backbone and cyclic schemes.

DAVIS. The evaluation results on DAVIS16 validation set, DAVIS17 validation and test-dev set are reported in Table 1. From this table, we observe that our model trained with cyclic loss outperforms most of the offline methods and even performs better than the method with online learning voigtlaender17BMVC on DAVIS17 benchmark. When combined with the online gradient correction process, our method gets further improvement. It should also be noticeable that although standard STM-cycle do not require additional training data other than DAVIS and Youtube-VOS, it outperforms some other state-of-the-art pipelines highly dependent on additional training data Lin_2019_ICCV ; Oh_2018_CVPR ; Voigtlaender_2019_CVPR . In terms of the runtime speed, although gradient correction increases the computation cost, our method still runs at a speed comparable to other offline methods Lin_2019_ICCV due to our efficient implementation. When we replace the backbone network from ResNet50 to more lightweight ResNet18, our trained model can run faster with still competitive performance. Although there is a performance gap between our approach that is trained from scratch and the state-of-the-art online learning method, our method is far more efficient and it does not requires collecting extra data from instance segmentation tasks as training samples. It is also noticeable that when adding COCO into training set, cyclic version of STM can achieve state-of-the-art performance on all benchmarks of DAVIS, obtaining consistent improvement over original STM Oh_2019_ICCV , which shares the same backbone but requires more data besides COCO.

Furthermore, we also try to combine the gradient correction process with an existing open source model of AGSS-VOS luiten2018premvos ²²2https://github.com/Jia-Research-Lab/AGSS-VOS to test inference performance gain on state-of-the-art method with purely gradient correction. This appears to bring improvement on the overall segmentation quality, demonstrating that cyclic consistency is helpful even in different segmentation pipelines.

Youtube-VOS. The evaluation results on Youtube-VOS validation set are reported in Table 2. On this benchmark, our model also outperforms some offline methods and their online learning counterparts Xu_2018_S2S_ECCV ; Zeng_2019_ICCV . It is also noticeable that compared with the performance on seen objects, the one on unseen objects has improved more using our gradient correction strategy. Further, we observe that with ResNet-50 backbone and gradient correction technique, our final pipeline can achieve similar segmentation accuracy and runtime speed to some state-of-the-art methods  Lin_2019_ICCV on Youtube-VOS even without extra data for pre-training. Finally, as consistent with the results on DAVIS series, when combined with COCO pretraining, our STM-cycle model can achieves better performance than other state-of-the-art models with pretrained Oh_2019_ICCV ; Lin_2019_ICCV ; Oh_2018_CVPR

In this section, we conduct a series of experiments to analyze the cyclic property of our trained models, with all the results evaluated on the DAVIS16 and DAVIS17 validation set and ResNet50 as backbone network.

In Figure 11, we take the cycle-ERF as a tool to analysis the focusing area of different baseline models, STM Oh_2019_ICCV , GCM GCVOS2020 and KMN KMN on DAVIS16. By comparison, we find STM shows overally stronger response on focusing area around the foreground object. In contrast, GCM highlight more background context around the object, since the memorial mechanism in this method always squeeze the context into the cache and focus more on global relationship. On the other hand, cycle-ERF from KMN yields response focusing on more specific and small part of objects and less intensity in background or context, this is in consistency with the design insight behind this method, where a dynamic gaussian kernel is applied to suppress the interaction between less related areas. These comparisons with Cycle-ERF manifest that such visualization methods can be a helpful tool to provide interpretability of existing models.