LSVOS Challenge 3rd Place Report: SAM2 and Cutie based VOS

In our inference pipeline, we utilized the SAM2 model, specifically the sam2-hiera-large variant, which balances model size and speed effectively with a size of 224.4M and a frame rate of 30.2 FPS. To optimize the model’s performance, we compiled the image encoder by setting compile_image_encoder: True in the configuration. This allowed for efficient processing of high-resolution input images with a size of 1024x1024 pixels.

The configuration included several key settings designed to enhance the model’s segmentation capabilities. We utilized a num_maskmem of 7, which refers to the number of memory tokens used in the mask memory. We also applied a scaled sigmoid function on the mask logits for the memory encoder, with parameters sigmoid_scale_for_mem_enc: 20.0 and sigmoid_bias_for_mem_enc: -10.0, to adjust the memory encoding process. Additionally, by setting use-mask-input-as-output-without-sam: true, the model directly outputs the input mask as the final mask in scenarios without SAM.

For memory management, the configuration directly_add_no_mem_embed: true ensures that new frames are directly added to the memory without additional embedding. We enabled use_high_res_features_in_sam: true to incorporate high-resolution features into the SAM mask decoder, which improves the accuracy of mask predictions. Moreover, multimask_output_in_sam: true allowed the model to output three masks upon the first interaction, enhancing the initial conditioning of frames.

The model also leverages advanced object tracking and occlusion prediction strategies. For instance, by setting use_obj_ptrs_in_encoder: true, we enabled cross-attention to object pointers from other frames during the encoding process. This, combined with pred_obj_scores: true and fixed_no_obj_ptr: true, facilitated robust object occlusion prediction and tracking across the video sequence. Furthermore, we adopted multimask tracking settings, such as multimask_output_for_tracking: true, to refine the segmentation accuracy over time.

In parallel, the Cutie framework was configured with a focus on efficient video segmentation at a standard testing resolution of 720p. In the context of the memory frame encoding, we update both the pixel memory and the object memory every $r$-th frame. The default value of $r$ is set to 3, following the same configuration used in the XMem framework [2].For subsequent memory frames, we employ a First-In-First-Out (FIFO) strategy, which ensures that the most recent information is retained while older data is gradually phased out. This approach is designed to keep the memory footprint manageable and focused on the most relevant frames.The choice of a predefined limit of $T_{\max}=10$ for the total number of memory frames is a practical compromise. This value balances the need to avoid excessive memory usage and maintain real-time performance while still capturing sufficient temporal evolution of the scene. Maintaining a history of 15 frames is generally adequate for effectively exploiting temporal correlations in VOS tasks. This enhances segmentation accuracy by providing enough context for object tracking and appearance prediction without imposing excessive computational overhead or compromising system responsiveness. Extending this limit further could lead to diminishing returns, as the additional frames may not significantly improve performance and could increase computational load unnecessarily. In the final testing phase, we employed Test-Time Augmentation (TTA) to enhance the model’s robustness and accuracy. Specifically, we utilized flip-based augmentation, which involves horizontally flipping the input frames during inference. This simple yet effective technique helped mitigate potential overfitting and improved the model’s generalization by allowing it to account for possible variations in object orientation. In the dynamic nature of video data, where frames may exhibit significant variations due to camera and object movement, flip-based TTA provided a more consistent and reliable segmentation across the video sequence.

To evaluate the performance of our model, we compute the Jaccard value (J), the F-Measure (F), and the mean of J and F.

Jaccard Value (J). The Jaccard value, also known as Intersection over Union (IoU), measures the similarity between two sets. For a predicted segmentation mask $P$ and a ground truth segmentation mask $G$, the Jaccard value is defined as:

$$J=\frac{|P\cap G|}{|P\cup G|}=\frac{\sum_{i}P_{i}\cdot G_{i}}{\sum_{i}P_{i}+\sum_{i}G_{i}-\sum_{i}P_{i}\cdot G_{i}},$$

(1)

where $P_{i}$ and $G_{i}$ denote the value of the $i$-th pixel in the predicted and ground truth masks, respectively. The Jaccard value ranges from $0$ to $1$, with higher values indicating better performance.

F-Measure (F). The F-Measure is a metric that combines Precision and Recall, commonly used to evaluate the performance of binary classification models. It is calculated as follows:

$$F=\frac{2\cdot\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}},$$

(2)

where

$$\text{Precision}=\frac{|P\cap G|}{|P|}=\frac{\sum_{i}P_{i}\cdot G_{i}}{\sum_{i}P_{i}},$$

(3)

and

$$\text{Recall}=\frac{|P\cap G|}{|G|}=\frac{\sum_{i}P_{i}\cdot G_{i}}{\sum_{i}G_{i}}.$$

(4)

The F-Measure also ranges from $0$ to $1$, with higher values indicating better model performance in handling positive and negative samples.

Mean of J and F. To comprehensively evaluate the model’s performance, we compute the mean of the Jaccard value (J) and the F-Measure (F):

$$\text{Mean(J, F)}=\frac{J+F}{2}.$$

(5)

These metrics together provide a robust assessment of the segmentation model’s accuracy and consistency, offering insights into its performance in predicting segmentation masks.

In the 6th Large-Scale Video Object Segmentation (LSVOS) challenge, our method (Xy-unu) demonstrated significant performance improvements in both the development and test phases. The leaderboards for these phases are presented in Tables 1, respectively. Our method achieved Jaccard values (J) and F-Measures (F) that outperformed most other participants. Specifically, in the test phase, our method attained a Jaccard value of 0.7952 and an F-Measure of 0.7516, resulting in a combined J&F score of 0.8388. These results highlight the effectiveness and robustness of our approach.

Moreover, we present some of our quantitative results in Fig4. The results clearly demonstrate that our proposed method is capable of accurately segmenting small targets and differentiating between similar objects in challenging scenarios. These scenarios include significant variations in object appearance and instances where multiple similar objects or small objects cause confusion.