FloLPIPS: A Bespoke Video Quality Metric for Frame Interpolation
^††thanks: This work was funded by the China Scholarship Council, University of Bristol, and the UKRI MyWorld Strength in Places Programme (SIPF00006/1).

Video frame interpolation (VFI) has recently attracted significant interest in the video compression research community as a means of generating intermediate frames between every two consecutive frames in a sequence. It has been employed in video coding to replace motion prediction in conventional codecs [1], to perform error concealment [2], and as the basis of end-to-end deep video compression systems [3].

Existing works on VFI largely focus on developing new algorithms to improve interpolation performance under various challenging scenarios. These techniques include the use of deformable convolution [4, 5], transformer networks [6, 7], coarse-to-fine architectures [8, 9, 10], and the design of more flexible optical flow estimation mechanisms [11, 12]. While a plethora of VFI algorithms have been reported, the perceptual evaluation of frame interpolated content has attracted less attention. The most common quality assessment methods used for VFI are PSNR, SSIM [13], and LPIPS [14]. However, these metrics have recently been shown exhibit poor correlation with subjective opinion scores for the case of frame interpolated videos [15]. Other popular image/video quality assessment models developed for more generic applications, such as VMAF [16], VIF [17], as well as the ones that specifically consider frame rate-related artefacts including ST-GREED [18] and FRQM [19], have also failed to provide satisfactory performance in the context of VFI [15]. Due to the lack of a high performance quality metric, many VFI works [20, 21, 5] resort to performing costly subjective experiments to evaluate the visual quality of their interpolation results. Hence, there is an urgent need to develop a quality assessment model that can more accurately capture the perceptual degradation in frame interpolated videos.

In this context, we propose a full reference video quality assessment model, Flow difference-weighted LPIPS, FloLPIPS, which, to the best of our knowledge, is the first bespoke metric specifically designed for video frame interpolation. FloLPIPS combines the spatial distortion captured by LPIPS with the temporal degradation estimated using the discrepancy between the optical flow maps of the reference and distorted videos. Our experiments show that FloLPIPS achieves state-of-the-art performance on a publicly available subjective database for VFI, significantly outperforming 12 other tested quality metrics.

The rest of the paper is organised as follows. We first describe the FloLPIPS algorithm in Section II. The quantitative evaluation results and analysis of the proposed method are then presented in Section III. Finally, we draw conclusions in Section IV.

The workflow to calculate the proposed Flow difference-weighted LPIPS, FloLPIPS, is illustrated in Fig.1. This comprises two primary stages: (i) the calculation of LPIPS features and (ii) flow difference based feature aggregation.

The Learned Perceptual Image Patch Similarity (LPIPS) method measures the perceptual distortion between two images based on the feature maps extracted by several early layers of a CNN (e.g. five for VGG [22] and AlexNet [23], and seven for SqueezeNet [24]), which is pre-trained for image classification. Specifically, given a reference and a distorted frame, $I^{ref},I^{dis}$, a feature extractor $\phi(\cdot)$ extracts feature maps $\phi_{l}^{ref}$ and $\phi_{l}^{dis}$ from these two frames at multiple layers, where $l=1,2,...,L$ and $L$ is the total number of layers employed. The reference and distorted feature maps are then normalised, and their differences are calculated. Next, the difference maps are weighted linearly at each layer by learned vectors $\omega_{l}$, and the $\ell 2$ norm is obtained across the channel dimension.

In the original LPIPS, the resulting feature difference of each layer is averaged spatially and summed to obtain the final LPIPS score, shown in (4).

where $H_{l},W_{l}$ are the height and width of the feature maps at layer $l$, and where $h,w$ index over all the spatial locations. The symbol $\odot$ represents a Hadamard product. More details on LPIPS training can be found in the original paper [14].

It is noted that, although LPIPS offers promising performance for measuring the perceptual quality of images, we previously reported that it exhibits unsatisfactory correlation with human judgement when used to evaluate the quality of interpolated videos [15]. We observed two major reasons for this. Firstly, LPIPS only captures distortions in the spatial domain, without any measure of temporal consistency. Secondly, when aggregating the spatial information, an arithmetic mean is taken over the feature difference maps where each pixel is weighted equally. This conflicts with VFI algorithms which tend to introduce salient artefacts in regions with movement which have a non-uniform spatial distribution.

Based on these observations, instead of considering single reference and distorted frames, we additionally involve the previous frames (both for reference and distorted sequences) to better capture the temporal characteristics of the video. Specifically, we compute the optical flow $F^{ref}$ between the reference frames $I^{ref}_{t-1}$ and $I^{ref}_{t}$, as well as the flow $F^{dis}$ between the corresponding distorted frames $I^{dis}_{t-1}$ and $I^{dis}_{t}$. To measure the distortion in the temporal domain, we compute the magnitudes of the differences between the reference and distorted flow maps. This difference, after normalisation, is used to weight the feature difference maps obtained during the calculation of LPIPS, so that a weighted spatial average is computed instead of the original arithmetic mean. Such a spatial pooling strategy places more emphasis on those pixels where there is more motion discrepancy. The underlying assumption is that the regions with distorted motions correspond to more salient parts of the video. This process is summarised in (2)-(6) below.

The optical flow estimator used is a pre-trained PWC-Net [25], and a pre-trained AlexNet (the first five layers) is used for LPIPS feature extraction. It is noted that all the model and hyper-parameters in the proposed FloLPIPS are identical to those in the original LPIPS or in the pre-trained PWC-Net and AlexNet. No other parameters need to be further optimised for this approach.

In order to obtain the quality score for the whole video, FloLPIPS is calculated for every two frames in a sliding window with a stride of 1, and all the frame level scores are averaged.

In this section, we first describe the experimental setup, then quantitatively evaluate the performance of FloLPIPS by comparing it with 12 commonly used image/video quality metrics. Ablation study results are also presented to demonstrate the efficacy of the proposed flow difference-based weighting.

In this paper, we introduced FloLPIPS, a full reference video quality assessment method specifically designed for video frame interpolation quality assessment. The proposed method builds upon the popular perceptual image quality metric, LPIPS, and improves its performance by incorporating distortions in the temporal domain represented by discrepancy between the reference and distorted optical flow fields. Such degradation in optical flow maps is used to weight the distortion maps in LPIPS during spatial pooling. FloLPIPS has been quantitatively evaluated and benchmarked against 12 commonly used (or recently reported) quality assessment models on the BVI-VFI database. The results demonstrate that FloLPIPS offers superior performance compared to all tested metrics with statistical significance, while requiring a moderate runtime. The proposed quality metric serves as a better quality assessment tool for VFI applications, and can also be used as a perceptual loss function for training learning-based VFI methods.