UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model

UVOS is designed to segment salient objects in a class-agnostic manner, eliminating the need for manual guidance during inference. This approach reduces the time and cost of Video Object Segmentation (VOS), thus paving the way for a variety of real-time applications. Early techniques (Fragkiadaki et al. 2015; Tokmakov, Alahari, and Schmid 2017; Yang et al. 2021a) for detecting salient objects assumed that object pixels exhibited the same motion patterns across successive frames, employing motion cues such as optical flow to facilitate the segmentation process. However, these methods have a notable limitation: their performance is heavily influenced by the optical flow’s quality, especially in situations involving occlusions and rapid motion. Several studies (Zhou et al. 2020; Zhang et al. 2021; Ren et al. 2021; Yang et al. 2021c; Cho et al. 2023) have also investigated the use of learning to focus on object appearance features. For example, MATNet (Zhou et al. 2020) employs the MAT-block to convert appearance features into motion-attentive representations, while RTNet (Ren et al. 2021) mutually transforms appearance and motion features to identify primary objects. Moreover, AMC-Net (Yang et al. 2021c) implements a multi-modality co-attention gate to foster a deep collaboration between appearance and motion data for precise segmentation. DPA (Cho et al. 2022) leverages inter-modality attention to densely incorporate context information from both motion and appearance. Lastly, PMN (Lee et al. 2023) efficiently extracts appearance and motion data by deriving superpixel-based component prototypes.

In the realm of unsupervised multiple object segmentation, the majority of methods employ the track-by-detect paradigm. This involves generating object proposals using instance segmentation models, which are subsequently tracked by re-identification models. The Target-Aware model (Zhou et al. 2021), for instance, uses a target-aware adaptive tracking framework to associate proposals across frames, yielding robust matching results. Similarly, INO (Pan et al. 2022) enhances the association of proposals by incorporating inter-frame consistency through the affinity matrix. On the other hand, MCMPG (Yuan et al. 2023) builds a graph using object proposals generated from several frames around the key frame, and then propagates them to the key frame to reason about key-frame objects. Unlike these methods that rely on costly annotation-intensive segmentation datasets for training, our mask-free UVOSAM model achieves accurate segmentation by providing simple prompts to the vision foundation model SAM (Kirillov et al. 2023). This approach makes UVOS more accessible.

Vision foundation models, which have shown remarkable adaptability across a range of tasks, have recently been attracting significant attention. CLIP (Radford et al. 2021), a groundbreaking model trained on web-scale image-text pairs, demonstrates impressive zero-shot capabilities and has been widely implemented in numerous multi-modal tasks for feature alignment. Similarly, BLIP (Li et al. 2022, 2023) utilizes a dataset bootstrapped from large-scale noisy data for multi-modal pre-training, enhancing zero-shot performance in text-to-video retrieval tasks. In a bid to develop a universal detection framework, UniDetector (Wang et al. 2023b) uses images from a variety of sources and diverse label spaces for training, achieving zero-shot performance that significantly outperforms supervised methods.

In the realm of segmentation tasks, SAM (Kirillov et al. 2023), the first promptable foundation model, has been recognized for its ability to segment any object in any context using various prompts, such as boxes and points. Models such as SegGPT (Wang et al. 2023a) and SEEM (Zou et al. 2023), which share a similar idea with SAM, were proposed as generalist models to perform arbitrary segmentation tasks with different prompt types. Some contemporaneous works have extended SAM to video segmentation tasks. For instance, TAM (Yang et al. 2023) and SAM-Track (Cheng et al. 2023) utilize cutting-edge mask trackers to interactively segment video objects, while SAM-PT (Rajivc et al. 2023) employs sparse points input for mask generation. In this paper, our focus is on the UVOS task, and we optimize the box prompts of SAM to significantly reduce the need for intensive video mask annotation. Moreover, our approach inherently possesses SAM’s robust zero-shot capability, making it suitable for handling weakly labeled video segmentation datasets.

Multi-object tracking aims to accurately detect multiple objects while providing continuous trajectories for each object. The dominant paradigm in this field is Tracking-By-Detection, which consists of two tasks: detection and association. To better predict the next position of objects, SORT (Bewley et al. 2016) first uses the Kalman filter (Welch, Bishop et al. 1995) and then matches predicted boxes with detected boxes using the Hungarian algorithm (Kuhn 1955). DeepSORT (Wojke, Bewley, and Paulus 2017) effectively mitigates occlusion problems by extracting ReID features of bounding boxes with a separate CNN model. TransTrack (Sun et al. 2020) obtains bounding boxes and ReID features through the vision transformer. ByteTrack (Zhang et al. 2022) adopts YOLOX (Ge et al. 2021) as the detector and tracks bounding boxes with scores below the threshold. StrongSORT (Du et al. 2023) designs an appearance-free link model and comprehensively optimizes DeepSORT, achieving a balance between inference speed and accuracy. OCSORT (Cao et al. 2023) designs a cumulative error correction module and utilizes a learnable motion model to improve tracking performance under occlusion and non-linear motion conditions. DEEPOCSORT (Maggiolino et al. 2023) adaptively integrates ReID features with OCSORT and achieves promising tracking results. Given that the aforementioned approaches have shown outstanding performance in identity association, our proposed UVOSAM leverages the association methods of DeepOCSORT, primarily focusing on enhancing the detection quality of box prompts in terms of Intersection over Union (IoU) between predicted boxes and object boundaries.

SAM is a segmentation foundation model that has been trained using over one billion masks, demonstrating remarkable zero-shot proficiency. Its primary components consist of an image encoder denoted as $E_{i}$, a prompt encoder denoted as $E_{p}$, and a lightweight mask decoder denoted as $D_{m}$. To elaborate further, SAM takes an image $I\in\mathbb{R}^{H\times W\times 3}$ and its corresponding prompts $P$ (such as points, bounding boxes, text, or masks) as inputs. These two input streams are then embedded separately by the image encoder and prompt encoder, which can be formulated as follows:

$\displaystyle F_{i}=E_{i}(I)\in\mathbb{R}^{h\times w\times c}~{}~{}~{}F_{p}=E_{p}(P)\in\mathbb{R}^{k\times c},$

(1)

where $h$, $w$ and $c$ represent the height, width, and channel number of the feature map, and $k$ denotes the length of the prompt tokens. Following that, $F_{i}$ and $F_{p}$ are passed to $D_{m}$. In addition, there are multiple learnable tokens $F_{t}$ concatenated to $F_{p}$ as a prefix to enable flexible prompting. These two embeddings are then interacted through cross-attention, and the zero-shot mask is generated by decoding the interactive features. This process can be described as follows:

$\displaystyle M=D_{m}(~{}Attn(F_{i},concat(F_{p},F_{t}))~{})\in\mathbb{R}^{H\times W}$

(2)

As mentioned previously, SAM is trained for general segmentation tasks rather than specific ones. Consequently, it may encounter difficulties in accurately handling segmentation boundaries, especially when prompt information is limited. For instance, a single prompt such as a single point prompt can lead to segmentation ambiguity issues, where SAM cannot differentiate which mask the prompt corresponds to. Hence, the objective of this paper is to investigate methods for optimizing the quality of box prompts. By generating high-quality box prompts, we can potentially achieve promising outcomes in UVOS tasks.

UVOSAM consists of two components: a STD-Net and a SAM. As shown in Figure 2, let $\left\{I_{i}\right\}_{i=t-d}^{t+d}$ represent the input clip centered at time $t$, which includes the $t$-th frame and its $2d$ neighboring frames. The encoder of STD-Net initially extracts enriched spatial-temporal features $E$ using a novel spatial-temporal decoupled deformable attention (STD-DA) module. These features $E$ are then utilized to refine the object queries and obtain discriminative instance embeddings, which are subsequently decoded into the category, location, and ReID features of each object through three output heads. These outputs are further combined to generate the object trajectories. Subsequently, the object trajectories are utilized as box prompts in SAM to obtain refined video segmentation results. The entire SAM is kept frozen to retain its pre-trained knowledge, and only the STD-Net is trained using category and bounding box annotations.

Our STD-Net is composed of a vision backbone, a deformable transformer, and three output heads. In order to leverage the temporal information among frames for precise object discovery, we introduce a novel STD-DA module in the transformer encoder to effectively capture spatial-temporal correlations. Furthermore, we employ contrastive learning, following the approach (Wu et al. 2022b), to obtain ReID features. The key design principles will be explained in detail in the following paragraphs.

STD-DA. The proposed STD-DA module initially utilizes multi-scale deformable attention separately in the spatial and temporal domains, and subsequently combines them to generate dynamically enriched spatial-temporal features. The STD-DA architecture consists of three sub-modules: spatial multi-scale deformable attention (S-MSDA), temporal multi-scale deformable attention (T-MSDA), and dynamic attention fusion.

The S-MSDA module, proposed in this work, calculates attention on points sampled from the multi-scale feature maps of the current frame. Let the $l\text{-}th$ feature map from frame $t$ be denoted as $x^{l}_{t}\in\mathbb{R}^{C\times H_{l}\times W_{l}}$, and the normalized coordinates of the reference point for $q$ as $\hat{p}_{q}\in{[0,1]}^{2}$. With these definitions, the S-MSDA module can be described as follows, given the query feature $z_{q}$:

	$\displaystyle\mathcal{S\text{-}MSDA}$	$\displaystyle(z_{q},\hat{p}_{q},x_{t}^{l})=\sum_{m=1}^{M}W_{m}\cdot[\sum_{l=1}^{L}\sum_{k=1}^{K_{intra}}A_{mlqk}$		(3)
		$\displaystyle\cdot W_{m}^{{}^{\prime}}x^{l}_{t}(\phi_{l}(\hat{p}_{q})+\Delta p_{mlqk})],$

where $m$ denotes the index of multiple attention heads, $K_{intra}$ is the total sampled number from each feature level. $\Delta p_{mlqk}$ and $A_{mlqk}$ represent the pixel offset and attention weight of the $k\text{-}th$ sampling point in the $l\text{-}th$ feature level. Both of them are obtained via linear projection over $z_{q}$. Function $\phi_{l}$ re-scales the normalized coordinates $\hat{p}_{q}$ to the input feature map of the $l\text{-}th$ level.

We further expand the S-MSDA module to the temporal domain and introduce T-MSDA. The key concept behind T-MSDA is that, for each query in the current frame, it attends to a subset of key points sampled from multi-scale feature maps of neighboring frames. This allows our method to gather enriched temporal representations and preserve temporal consistency of the foreground instances. The formulation of T-MSDA can be expressed as follows:

	$\displaystyle\mathcal{T\text{-}MSDA}$	$\displaystyle(z_{q},\hat{p}_{q},x_{t^{\prime}}^{l})=\sum_{m=1}^{M}W_{m}\cdot[\sum_{t^{\prime}=t-d\atop t^{\prime}!=t}^{t+d}\sum_{l=1}^{L}\sum_{k=1}^{K_{inter}}$		(4)
		$\displaystyle A_{mltqk}\cdot W_{m}^{{}^{\prime}}x_{t^{\prime}}^{l}(\phi_{l}(\hat{p}_{q})+\Delta p_{mtlqk})],$

where $t^{\prime}$ denotes the neighboring frame with frame $t$. $K_{inter}$ represents the number of sampled points in each feature level across neighboring frames.

Let $E_{t}^{intra}$ and $E_{t}^{inter}$ represent the outputs for frame $t$ obtained by $\mathcal{S\text{-}MSDA}$ and $\mathcal{T\text{-}MSDA}$, respectively. According to our findings, $E_{t}^{intra}$ contains more salient instance information about the current frame, which is crucial for distinguishing the instance from the background. On the other hand, $E_{t}^{inter}$ contains more discriminative representations to detect changes in the instance’s appearance. To effectively combine the strengths of both modules, we propose a dynamic attention fusion module that uses channel-wise attention to highlight the semantic information. We aggregate the two outputs using element-wise addition and then apply a fully connected layer to adaptively select from $E^{\prime}$. Subsequently, we use two additional fully connected layers to generate gate vectors $\left\{g_{1},g_{2}\right\}$, which control the flow of information from the attention maps $\left\{E_{t}^{intra},E_{t}^{inter}\right\}$. To generate adaptive weights $\left\{w_{1},w_{2}\right\}$, we utilize the softmax function along the channel dimension. Finally, we obtain the enriched spatial-temporal features of frame $t$ by performing a weighted addition:

$\displaystyle E_{t}=E_{t}^{intra}\odot w_{1}+E_{t}^{inter}\odot w_{2}.$

(5)

ReID embeddings. To obtain ReID embeddings based on DeformableDETR (Zhu et al. 2020), we introduce a temporal contrastive loss that maps different object queries to unique ReID representations. For each video clip, we consider queries belonging to the same object as positive samples and all other queries as negative samples.

Here define $B_{t}={\left\{b_{t}^{q}\right\}}_{q=1}^{Q}$ be the object queries of frame $t$, where $b_{t}^{q}\in R^{C}$ and $Q$ is the total number of queries. Automatically, ${\left\{b_{t}^{i},b_{t^{\prime}}^{i}\right\}}$ from frame $t$ and its neighboring frame $t^{\prime}$ can be treated as the different views of the $i\text{-}th$ instance in terms of temporal domain. We first adopt two FC layers as the ReID head to generate the ReID representations of the queries. Then, InfoNCE (Oord, Li, and Vinyals 2018)  is employed between two frames, which can be described as:

$\displaystyle\mathcal{L}_{N}(B_{t},B_{t^{\prime}})=-\frac{1}{Q}\sum_{i}^{Q}log\frac{exp(s(b_{t}^{i},b_{t^{\prime}}^{i})/\tau)}{\sum_{j=1}^{Q}exp(s(b_{t}^{i},b_{t^{\prime}}^{j})/\tau)},$

(6)

where $\tau$ represents the temperature hyper-parameter, and $s$ denotes the cosine similarity. During inference, the ReID features of the queries are utilized to measure visual similarity. In summary, the total losses of STD-Net can be defined as follows:

$\displaystyle\mathcal{L}_{total}=\lambda_{cls}\cdot\mathcal{L}_{cls}+\mathcal{L}_{box}+\mathcal{L}_{cl},$

(7)

where $\mathcal{L}_{cls}$ and $\mathcal{L}_{box}$ represent classification loss and bounding box loss. Herein, we use focal loss (Lin et al. 2017) as the classification loss. The box loss consists of L1 loss and GIou loss (Rezatofighi et al. 2019): $\mathcal{L}_{box}=\lambda_{L1}\cdot\mathcal{L}_{L1}+\lambda_{giou}\cdot\mathcal{L}_{giou}$.

Datasets. We evaluate UVOSAM on three popular video segmentation datasets: DAVIS2017-unsupervised (Pont-Tuset et al. 2017) and Youtube-VIS 2019&2021 (Yang, Fan, and Xu 2019). Video instance segmentation builds upon unsupervised video object segmentation by incorporating object category recognition. Therefore, the corresponding dataset of YouTubeVIS can also be used for experiments. The DAVIS2017-unsupervised dataset consists of 120 high-quality videos in total, which are further split into 60 for training, 30 for validation, and 30 for test-dev. The YoutubeVIS-2019 dataset comprises 2,238 training, 302 validation, and 343 test videos. The semantic category number is 40. The YoutubeVIS-2021 dataset adds more samples based on YoutubeVIS-2019, with 3,859 high-quality videos, while the number of annotations is also doubled.

Evaluation Metrics. For DAVIS2017-unsupervised, we employ the official evaluation measures including region similarity $\mathcal{J}$, boundary accuracy $\mathcal{F}$ and the overall metric $\mathcal{J}\&\mathcal{F}$. For Youtube-VIS dataset, we adopt Average Precision (AP) and Average Recall (AR).

Implements Details. For STD-Net, we select ResNet-50 (He et al. 2016) as the default backbone network. All hyper-parameters related to the network architecture are the same as those in the official DeformableDETR. For DAVIS2017-unsupervised, we modify the class head to a binary classification output, identifying only whether the object is foreground. We use 4 NVIDIA A100 GPUs with a global batch size of 16. We first pre-train STD-Net on MSCOCO (Lin et al. 2014) for 120,000 steps and then train on DAVIS2017-unsupervised and Youtube-VIS for 2,0000 steps and 80,000 steps, respectively. The AdamW optimizer (Loshchilov and Hutter 2019) is adopted with a base learning rate of $1\times 10^{-4}$ and $5\times 10^{-5}$ for the two-stage training. For SAM, we employ the pre-trained model with the ViT-H (Dosovitskiy et al. 2020) image encoder as the base segmentation model. Following mainstream methods, we resize the frame to a shorter size of 360 and 480 pixels on Youtube-VIS and DAVIS2017-unsupervised during inference, respectively.

We compare UVOSAM with the state-of-the-art methods on DAVIS2017-unsupervised and YouTube-VIS 2019 & 2021. Among these competitive methods, our UVOSAM sets new state-of-the-art results, using only the training data of boxes.

DAVIS2017-unsupervised. Table 1 illustrates the results of all compared methods on DAVIS2017-unsupervised validation set. Our UVOSAM achieves $\mathcal{J}\&\mathcal{F}$ scores of 80.9$\%$, which outperforms current mask-supervised methods by a large margin. Specifically, UVOSAM makes 2.5 $\%$ absolute improvements over the best competitor MCMPG+STCN, which utilizes a more complex ResNeXt-101 backbone.

YoutubeVIS-2019&2021. We present our comparisons of YoutubeVIS-2019&2021 in  Table 2. When using the ResNet-50 backbone, our UVOSAM sets a new state-of-the-art with AP of 52.4$\%$ and 48.4$\%$, respectively. The higher AP and AR values demonstrate that our method can effectively leverage enriched spatial-temporal information to optimize segmentation results. Notably, without using any masks for training, our UVOSAM achieves an absolute gain of 1.1$\%$ &1.3$\%$ AP over the previous best method, GenVIS. Additionally, it is worth mentioning that UVOSAM outperforms GenVIS by approximately 2.3$\%$ in the overlap threshold of $\rm AP_{75}$, indicating that UVOSAM is capable of optimizing the non-overlapping cases.

Figure 3 visualizes the results of UVOSAM in four challenging scenarios. It shows UVOSAM can generate high quality masks and maintain stable tracking trajectories.

In this section, we conduct ablation studies to figure out the contribution of different designs. All the ablation experiments are conducted on the DAVIS2017-unsupervised, using the same implementation details as previously described.

The impact of the proposed modules on the quality of prompt. We verify the effectiveness of the proposed modules for improving the quality of box prompts. Firstly, we train the baseline DeepOCSORT (Maggiolino et al. 2023) with the detector of DeformableDETR. As shown in  Table 3, it only achieves $\mathcal{J}\&\mathcal{F}$ of 78.1 $\%$. Compared to the baseline model, the introduction of temporal contrastive learning provides an obvious gain of 1.0 $\%$. By performing STD-DA in the deformable transformer, it further brings the $\mathcal{J}\&\mathcal{F}$ to 80.9$\%$. In summary, our STD-Net outperforms baseline by 2.8$\%$. Moreover, we additionally offer box annotations as prompts for SAM, which are specifically denoted as ”Humans”. The superior UVOS results produced by SAM are evident when the prompts are sufficiently accurate.

Bidirectional neighboring window size of STD-DA. In Table 4, we study the impact of the bidirectional neighboring window size $d$ on overall performance. By increasing $d$ from 1 to 3, it brings the $\mathcal{J}\&\mathcal{F}$ to $79.7\%$ , $80.4\%$ and $80.9\%$, respectively. It is evident that a larger window size can utilize richer temporal information from multiple frames, which facilitates better understanding of the context of instances. To achieve a balance between computation and precision, we set the value of $d$ to 3 for all experiments. Nonetheless, we hypothesize that our performance will continue to improve as $d$ increases.

Number of temporal sampling points of STD-DA. Table 5 shows there is a large performance improvement when $K_{inter}$ changes from 1 to 4. It can be seen that when $K_{inter}$ further increases, the performance is basically unchanged. Hence, we choose $K_{inter}=4$ as our default settings.

Runtime Analysis. In terms of speed, UVOSAM achieves $11.7$ FPS on DAVIS2017, which is comparable to MCMPG (about 10.5 FPS).

The Generalization ability to weakly-annotated video data. To assess the transfer ability of UVOSAM, we additionally conduct experiments on the MOT20 (Dendorfer et al. 2020), which only provides box annotations. As depicted in  Figure 4, the existing UVOS methods fail to produce satisfactory segmentation results due to the lack of training data. In contrast, UVOSAM, leveraging the weak annotations of boxes, demonstrates exceptional segmentation performance.