Video Frame Interpolation with Region-Distinguishable Priors from SAM

Task setting. Given two frames $I_{0},I_{1}\in\mathbb{R}^{H\times W\times 3}$, the target of VFI is to synthesize an intermediate frame ${\hat{I}_{t}\in{\mathbb{R}}^{H\times W\times 3}}$ at arbitrary time step $t\in(0,1)$, as

$${\hat{I}_{t}=\mathcal{O}(I_{0},I_{1},t),}$$

(1)

where $\mathcal{O}$ denotes the VFI method that shares a common framework as illustrated in Fig. 2. Motion-based VFI typically comprises three key stages. These stages involve feature extraction for $I_{0}$ and $I_{1}$, with the extracted features labeled as $f_{0,l}$ and $f_{1,l}$, where $l\in[1,L]$ signifies the $l$-th layer in the encoder. Additionally, it includes motion estimation between the extracted features and warping these features to synthesize the final results. The accuracy of the motion estimation stage holds pivotal importance within VFI, as it directly influences the ultimate performance.

Challenge. While numerous motion estimation strategies have been introduced in recent years, their effectiveness is predominantly evident in scenarios involving continuous motions. However, in the context of VFI tasks, there exists a substantial temporal gap and limited continuity between adjacent frames. This presents a significant challenge for accurate motion estimation. The primary obstacle in this motion estimation process arises from the inherent ambiguity associated with identifying corresponding areas in neighboring frames for interpolation. Consequently, achieving precise estimation accuracy in current VFI frameworks remains a formidable challenge

Motivation. To address the aforementioned challenge, we propose a method to enhance the extracted features for interpolation by introducing specific priors capable of distinguishing different objects within frames. This serves to reduce ambiguity in the identification of matching areas in adjacent frames. These priors are obtained through the utilization of the current open-world segmentation module, such as SAM, resulting in $M_{0}$ and $M_{1}$ for $I_{0}$ and $I_{1}$. Furthermore, these priors are integrated hierarchically into the feature extraction stage of VFI models, given that VFI models typically employ pyramidal structures in their encoders. The primary objective is to provide distinct feature representations for different areas within $I_{0}$ and $I_{1}$. This, in turn, enables more accurate motion estimation by distinguishing between various objects and being aware of boundaries.

Implementation. Given $I_{0}$ and $I_{1}$, we first obtain their SAM outputs as $M_{0}$ and $M_{1}$. Then, $M_{0}$ and $M_{1}$ (${M_{0},M_{1}}\in\mathbb{R}^{H\times W\times 1}$) are transformed into the desired Region-Distinguishable Priors (RDPs) that can distinguish different regions in frames with a unified representation dimension. Thus, Eq. 1 can be written as

$${\hat{I}_{t}=\mathcal{O}(I_{0},I_{1},\mathcal{G}(M_{0}),\mathcal{G}(M_{1}),t),}$$

(2)

where $\mathcal{G}$ is the transformation function to produce RDPs, and we denote $\mathcal{S}_{0}=\mathcal{G}(M_{0})$ and $\mathcal{S}_{1}=\mathcal{G}(M_{1})$. The extracted features $f_{0,l}$ and $f_{1,l}$ are enhanced with our proposed Hierarchical Region-aware Feature Fusion Module (HRFFM) (as displayed in Fig. 2), as

$$f^{\prime}_{0,l}=\mathcal{H}(f_{0,l},\mathcal{S}_{0}),\;f^{\prime}_{1,l}=\mathcal{H}(f_{1,l},\mathcal{S}_{1}),$$

(3)

where $\mathcal{H}$ is the designed HRFFM. The enhanced $f^{\prime}_{0,l}$ and $f^{\prime}_{1,l}$ are then sent to the following original motion estimation and frame synthesis stages to obtain the final result.

The drawback of SAM outputs for VFI. The original SAM model provides segmentation outputs for all instances within an image. SAM generates masks for frames, with each pixel value representing an object. Its remarkable segmentation capabilities make it a valuable choice as a region-distinguishable prior. Over time, several variants of SAM have been introduced, enhancing its capabilities, including semantic and panoptic segmentation when combined with other models. However, SAM’s output has limitations when it comes to representing objects with arbitrary numbers, a requirement for RDP. The semantic one-hot embedding is constrained by semantic categories, and the instance one-hot embedding assumes a maximum instance number, making it unable to accommodate new instances during real-world evaluation. Consequently, there is a need to transform SAM’s output to make it more suitable for RDPs.

Mixture Gaussian embedding strategy. We posit that the representations of segmentation priors can be conceptualized as distributed sampling results with distinct parameters across different regions of an image. These parameters enable the discrimination of regions and the alignment of the same region across multiple frames. In particular, each segmented area can be interpreted as a sampling result from a Gaussian distribution characterized by individual parameters. To facilitate this corresponding sampling process, we begin by establishing a codebook $\mathcal{C}$ that comprises a range of Gaussian parameters, encompassing both mean and variance. Subsequently, each object identified by the SAM output can retrieve its specific Gaussian parameters via a hashing mechanism. Therefore, the transformation procedure can be written as:

$${{S}_{i}=\mathcal{G}(M_{i})=\mathcal{N}(\mathcal{C}_{m}(M_{i}),\mathcal{C}_{v}(M_{i})),\;i=0,1,}$$

(4)

where ${S_{i}}\in\mathbb{R}^{H\times W\times c}$, $\mathcal{N}$ is the Gaussian distribution sampler, $\mathcal{C}_{m}$ is the codebook for Gaussian mean values, and $\mathcal{C}_{v}$ is the codebook for Gaussian variance scores. This Gaussian mixture is independent of the number of object types (adding a new object in the frame equals sampling a new Gaussian parameter), and distinguishes an arbitrary number of areas with a unified modality.

Training dataset. The Vimeo90K dataset [49] contains 51,312 triplets with a resolution of 448$\times$256 for training. We augment the training images by randomly cropping 256$\times$256 patches. We also apply random flipping, rotating, and reversing the order of the triplets for augmentation.

Evaluation datasets. While these models are exclusively trained on Vimeo90K, we assess their performance across a diverse range of benchmarks featuring various scenes.

• •

  UCF101 [46]: The test set of UCF101 contains 379 triplets with a resolution of 256$\times$256. UCF101 contains a large variety of human actions.

• •

  Vimeo90K [49]: The test set of Vimeo90K contains 3,782 triplets with a resolution of 448$\times$256.

• •

  SNU-FILM [8]: This dataset contains 1,240 triplets, and most of them are of a resolution of around 1280$\times$720. It contains four subsets with increasing motion scales – easy, medium, hard, and extreme.

Quantitative comparison. The comparison results are presented in Tab. 1, where we integrate our proposed approach with VFI baselines to assess performance improvements. It is observed that almost all baselines exhibit enhanced results across all testing sets when our strategy is applied, with only a minimal increase in parameters and computation costs. Notably, our method demonstrates a substantial improvement of 0.31dB on Vimeo90K for the robust baseline, VFIformer. Additionally, for other methods, there is an improvement of more than 0.1dB, which is significant in the context of VFI tasks where performance has almost approached the upper limit.

Qualitative comparison. We present a visual comparison between the baselines and their counterparts combined with our approach, illustrated in Fig. 5. Evidently, our strategy yields perceptual improvements by reducing undesirable artifacts and enhancing the accuracy of details.

Evaluation with downstream tasks. The VFI capability can be leveraged for various downstream tasks, including video segmentation. Large temporal gaps in videos can disrupt the effective propagation of semantic information. To assess the performance of our framework in terms of its impact on downstream video segmentation tasks, we employ the SOTA video segmentation approach SAM-Track[5]. The results, presented in Fig. 6, showcase three consecutive frames in the first row, with segmentation results of synthesized intermediate frames generated by VFIformer and VFIformer_ours in the second and third row, respectively. It is evident that the intermediate frames produced by our model exhibit more accurate segmentation. Our method’s results enhance better temporal propagation among frames and can even rectify incorrect segmentation results in the first frame. For instance, the dog in the second row is not clearly separated from the shadow on the ground, whereas in the third row, the separation is more distinct.

Effect of Mixture Gaussian embedding. Mixture Gaussian embedding serves as a crucial representation for distinguishing objects between two frames, playing a pivotal role in adapting SAM outputs for an arbitrary number of instances. To investigate the impact of Mixture Gaussian embedding, we replaced it with alternative methods, including naive one-hot encoding or learnable embeddings. Both alternatives require assuming a maximum instance number, denoted as “Ours with O.H.” and “Ours with L.E.”, respectively. The results, presented in Tab. 2, indicate that their performance is lower than the results achieved with Mixture Gaussian embedding, highlighting the effect of the proposed approach outlined in Sec. 3.2.

Effect of softmax operation and residual learning in HRFFM. After the feature extraction for RDP in each layer, the softmax operation ensures the consistency of feature representations at different scales. Additionally, to mitigate the impact of SAM errors on subsequent feature fusion, a residual learning component is incorporated after RDPFN. To assess their effectiveness, we trained two models without the softmax operation and residual learning, labeled as “Ours w/o S.O.” and “Ours w/o R.L.”, respectively. As depicted in Tab. 2, the performance of both models is lower than the original full setting, underscoring the rationality of the softmax operation and residual learning in HRFFM.

Effect of parallel CNN and transformer blocks in RDPFN. RDPFN is designed to leverage both long- and short-range dependencies, formulating normalization parameters for regions with varying shapes and areas. To demonstrate the effectiveness of this parallel setting, we trained two models with only a convolutional layer and a Transformer layer in RDPFN, labeled as “Ours with CNN” and “Ours with Trans.”, respectively. The results in Tab. 2 indicate that removing either component leads to an overall performance degradation, underscoring the necessity of the parallel CNN and Transformer strategy in formulating suitable region-aware normalization parameters.