1^st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR’24 PVUW) Challenge in Video Panoptic Segmentation
and Best Long Video Consistency of Video Semantic Segmentation.

The backbone network is crucial to the performance for Video Panoptic Segmentation. we started with different backbones and investigated their performance on the VIPSeg validation dataset. In particular, we showed performance on VIPSeg validation dataset using various backbones as shown in Table 1 including lightweight backbone MobileViTv2 [13], medium size backbone Swin-L [12], DINOv2-L [16] and large visual foundation backbone DINOv2-g [16]. we also considered a even larger size backbone such as the InternViT-6B. However, for the challenge we limited our submissions to results using the DINOv2 ViT-g foundational model as the backbone. We can observe that With large visual foundation backbone DINOv2-g, we can achieve the state-of-the-art performance 60.42% VPQ on VIPSeg validation dataset.

Current large visual foundation models are normally plain ViT, that is pretrained on massive scale dataset to learn semantic-rich representations. However, unlike specifically designed backbone like Swin, the plain ViT normally has inferior performance on dense prediction tasks due to the lack of multiple scale features that are very important to performance.

We reference the design of the ViT-Adapter [2] used in DVIS++  [22], as demonstrated in Figure 1. We adapt the embedding from the large foundation model through the ViT-Adapter, that is implemented as an additional branch that interacts with the plain ViT blocks with a spatial prior module and several Extractor modules. Note that original ViT-Adapter [2] also has several Injector modules, which we removed, since, in our approach, the large visual foundational backbone ViT-g is frozen during training.

The input image is input into the Spatial Prior module, where multiple scales of feature maps (e.g. 1/4, 1/8, 1/16, and 1/32) will be extracted and concatenated as input for Extractors. The role of the ViT-Adapter becomes evident, as we then obtain multiple-scale features by splitting the ViT-Adapter output and providing them as the input for the pixel decoder. The ViT-Adapter is trained together with the pixel decoder and transformer decoder in an end to end fashion, while freezing the DINOv2 ViT-g (DINOv2-g) backbone.

In our experiments, the DINOv2-g consists of 40 blocks, we divide them evenly into four stages by indices [[0, 9], [10, 19], [20, 29], [30, 39]], where the output of each stage will interact with the ViT-Adapter by using multi-scale deformable attention, where we set the number of heads as 24.

We also considered another design choice that we call ViT-CoMer-Adapter, which is a GPU memory friendly combined version of ViT-CoMer [19] and ViT-Adapter, where the Multi-Receptive Field Feature Pyramid (MRFP) module [19] is applied before each Extractor module for better multiple scale feature interactions. However, in our preliminary experiments, ViT-CoMer-Adapter achieved similar performance as ViT-Adapter shown in Table 2

To test the generalization capability, we tested the model pretrained on VIPSeg dataset for the VPS task directly on the VSPW dataset for the VSS task without any finetuning. For this, we simply merged all the predicted instances from the same class into a single class segmentation mask for the VSS task. Next, we also finetuned the model pretrained on VIPSeg dataset on VSPW dataset. It is interesting to observe that the pretrained model achieved better temporal consistency VC8 and VC16 on VSPW test dataset without any finetuning, while the finetuned model achieved better mIoU, as shown in Table 3. This can be attributed to the fact that video panoptic segmentation puts more emphasis on video consistency by the extensive training for accurately tracking multiple instances across frames. However, video semantic segmentation training puts more emphasis on the boundaries of each class, hence improves the per-frame mean intersection over union (mIoU) metric.

The VIPseg and VSPW datasets are used for the Video Panoptic Segmentation track and Video Semantic Segmentation track in PVUW challenge 2024.

VIPSeg VIPSeg [15] dataset is a large-scale dataset for Video Panoptic Segmentation dataset in the wild. There are 124 classes including 58 things’ and 66 stuff’s classes. It consists of 3,536 videos and 84,750 frames, where 2806/343/387 videos are for training/validation/testing, respectively.

VSPW VSPW [14] dataset has the same set of images for training/validation/testing as VIPSeg dataset, while the annotation is just for video semantic segmentation.

For VPS, we follow the same training pipeline as the Decoupled Video Instance Segmentation (DVIS) frameowrk. The segmenter, referring tracker, and temporal refiner are trained separately. We first pretrained the Mask2former model [5] as our segmenter on MSCOCO panoptic segmentation dataset [11] using batch size 16 and learning rate 1e-4 for 50 epochs. Since we freeze the weights of the large visual foundation model DINOv2-g, as the backbone for feature embeddding, we only train the ViT-adapter part and the transformer decoder part in this stage. Then, we finetuned the segmenter using image-level annotations from the training set of VIPSeg, following the training strategy of MinVIS [7] with batch size 16 and learning rate 1e-4 for 20000 iterations, with a learning-rate decay rate of 0.1 at 14000 iteration. With the initialization of segmenter, we further trained the DVIS online tracker with batch size 8 and learning rate 1e-4 for 40000 iterations while reducing the learning rate by half at 28000 iteration. In the last training stage, the DVIS offline temporal refiner is trained using the same setting on VIPSeg training datasets.

We also explored to increase the crop size of the DVIS offline training from (604, 604) to (840, 840), so we finetuned our model using batch size 8 and learning rate 2.5e-5 for 10000 iterations and decayed at 7000 iteration using the model trained with crop size (604, 604) as initialization.

All the trainings are conducted on single machine of 8x A100 Nvidia GPUs. The DVIS online model used a consecutive 5-frame clip from the video as input. The DVIS offline model used a window size of 21 frames as input. Multi-scale training is used to randomly scale the short side of input video clips from 480 to 1080 during training. By default, a random crop size (604, 604) is used except the last stage of finetuning using crop size (840, 840).

For VSS, we used the best model pretrained on VIPSeg dataset as initialization and finetuned it on the VSPW dataset using a batch size of 8 and learning rate 2.5e-5 for 10000 iterations and decayed at 7000 iteration.

In the VPS track of PVUW Challenge 2024, we ranked first in the test phase. The leaderboards for the development and test phases are displayed in Table 4 and Table 5, respectively. Our method achieved a VPQ of 55.69 in the development phase and 58.26 in the test phase.

With the help of large visual foundation model as the backbone, our method surpassed all other methods in all metrics in the test phase. Additionally, our method also showed its tracking stability when comparing the performance drop between VPQ6 and VPQ1. One thing to note that we only used single scale inference to achieve the result without using any test time augmentation or model ensemble.

In the VSS track of PVUW Challenge 2024, we ranked 3rd in the test phase, ranked by the mIoU metric and first ranked by the VC16 (16-frame video-consistency) metric. The leaderboard for the test phase is displayed in Table 6. Our method achieved a mIoU of 55.69 i and VC16 of 0.9325, in the PVUW VSS test phase.

In particular, our model tends to have better temporal consistency in terms of metrics VC8 and VC16. We ranked 2nd in VC8 and ranked 1st in VC16, which means our model tends to have better tracking reliability since it is finetuned from a VIPSeg pretrained model. We believe that better mIoU can be achieved if we follow the same training pipeline as VIPSeg dataset, rather than simply finetuning a VIPSeg pretrained model on VSPW dataset.

We also carefully tuned the hyper-parameters for several aspects using the development stage dataset. As shown in Table 7, a finetuned model with crop size (840, 840) achieved slightly better performance than baseline (604, 604) and also (630, 1120) with a different aspect ratio 16:9. We also compared different query sizes and found that 300 performs similar as 200, so we still keep 200 query size as our choice, as shown in Table 8.