Full Reference Video Quality Assessment for Machine Learning-Based Video Codecs

The CVPR 2022 CLIC video compression challenge²²2http://compression.cc resulted in 12 teams compressing 30 videos in the test set targeting 1 Mbps and 0.1 Mbps average bitrate. This included 27 combinations of codecs and bitrates, including the original sequences and two sets encoded by H.264 and AV1 [14]. Overall, the resulting dataset contains 810 videos which are either processed by 27 encoder-bitrates combinations or are uncompressed videos.

The Microsoft P.910 Crowdsourcing Toolkit [32] was used to assess the quality of the videos using three different subjective test methodologies. The toolkit provides an open-source implementation of ITU-T Rec. P.910 [34] for crowdsourcing. The CLIC 2022 dataset is available at http://compression.cc.

The Absolute Category Rating (ACR) is a single-stimulus subjective test method, in which participants watch a sequence and rate the quality on a five-point discrete scale. The Degradation Category Rating (DCR) and Comparison Category Rating (CCR) are double-stimulus test methods, in which participants watch the source and the processed clips before casting their votes. In DCR, participants are informed which sequence is the reference clip and rate the level of degradation in the processed sequence compared to the reference. However, in CCR participants are not aware of which sequence is processed and rate the quality of the second clip compared to the first clip they watched. As the source clips were included in the dataset, ACR with Hidden Reference (ACR-HR) was computed as well (see [34] for further description of the test methods). On average, 21.6 valid votes per sequence in the ACR test, 12.1 in the DCR test (on a 9-point discrete scale), and 15.2 in the CCR test were collected. Table 1 reports the correlation between Mean Opinion Scores (MOS) of different test methods aggregated in clip or model level. The highest correlations are observed within double-stimulus methods.

According to the P.910 recommendation, the double-stimulus test methods with an explicit reference should be employed when high-quality systems are under test, or the transparency or fidelity of the underlying process or transmission should be evaluated. These test methods lead to longer test sessions and consequently higher costs compared to the single-stimulus approach (about $\times 2$). We also observed a large difference in the quality rating of clips according to single- or double-stimulus tests when they contain (1) imperfect source sequence (e.g., low image quality, or slow motioned), (2) specific impairment due to encoding process (e.g., color changes or blur effect on non-saliency area), (3) source sequence with bokeh.

As FRVQA models use both reference and processed videos for prediction, we decided to use scores from a double-stimulus subjective test method. Consequently, participants are also exposed to both reference and processed videos before casting their votes. We chose DCR scores (known as DMOS) since the test set includes only high-quality video clips and the current codecs do not improve the original video quality. We repeated the DCR subjective test (9-point scale) again to increase the number of valid votes. The results of both subjective tests highly correlated (see Table 2) leading to 26.6 valid votes per clip on average. Reported statistical metrics also indicate the presence of uncertainty in subjective ratings, even in test-retest experiments, and the importance of considering this uncertainty when evaluating objective models.

Two issues with the CVPR 2022 CLIC dataset are (1) It does not have broad coverage of spatial and temporal information (see Figure 3), and (2) It only includes high-quality videos, which is not the case in user-generated content and video conferencing scenarios. To address these issues, we created a more comprehensive dataset based on several datasets described in Table 3. Each clip in the MLVC-FRVQA is resized to the resolutions given in Table 4 and processed by ML video codecs at the target bitrates given in Table 4. The MLVC-FRVQA dataset uses 7 open source ML video codec implementations [22, 23, 18, 4, 30, 27, 51], 4 traditional DSP codecs (H.264/AVC, H.265/HEVC, H.266/VVC, AV1), and 2 from CVPR CLIC 2022 challengers, for a total of 13 codecs. The total number of rated clips in the MLVC-FRVQA dataset is $147\times 13\times 12=22,932$.

The video clips were subjectively labeled using P.910 CCR with an average number of raters of $N=25$. We used the CCR test method as a portion of videos in the MLVC-FRVQA dataset have less than perfect quality, and some of the codecs enhance the perceived quality. The MLVC-FRVQA dataset is available at https://github.com/microsoft/MLCVQA. MLVC-FRVQA is not used in the MLCVQA model in this paper due to the timing of the dataset completion.

Raw measurements of subjective quality scores are often very noisy [3]. To get more accurate subjective quality scores and help machine learning-based FRVQA models learn easier we applied SUREAL [26]. We first checked the effectiveness of SUREAL by performing bootstrapping simulations with the VQEG HDTV datasets and the subjective quality scores collected in [32]. This dataset contains 81 votes per clip on average and we performed similar bootstrapping simulations as in Section 4.6 in the paper. N votes were randomly selected per clip with a replacement while making sure a minimum of N or 10 votes are selected from each voter. The selected votes were used to calculate both raw DMOS and SUREAL corrected DMOS and then resulting DMOS values were compared against the ground truth DMOS values - the average of all votes collected per clip - in terms of correlations and RMSE. We repeated this process 250 times per number of selected votes (N). Figure 2 shows the results. Raw DMOS aligns better with the ground truth DMOS than SUREAL corrected DMOS until the number of votes reaches around 20. The pattern is reversed after 20 votes. This may be because SUREAL is a Maximum Likelihood Estimator and it requires a certain number of samples to perform reliably per the Central Limit Theorem. Our dataset has 26 votes per clip on average and it lies on the borderline of getting benefits by applying SUREAL, so we compared our results with and without SUREAL-corrected DMOS. We trained and tested our models without any augmentation using 5-fold cross-validation. Table 5 summarizes the results. We observe that SUREAL does not improve our results significantly. We believe that this is mainly because the P.910 Crowdsourcing Toolkit [32] we used for data collection already applies intensive rater, environment, and device qualification tests, and includes rating quality tests, which altogether leads to less bias and noise in the aggregated data.

We used the SI-TI Tool³³3https://github.com/VQEG/siti-tools to calculate the Spatial perceptual Information (SI) and Temporal perceptual Information (TI) of all the source videos in the MLVC-FRVQA dataset. The SI indicates the amount of spatial detail in a frame, and TI refers to the amount of temporal changes in a video sequence [34]. Figure 3 illustrates the average SI and TI distribution and 95% percentile range per video. Besides the MS-Webcam set, the average SI and TI for all videos cover a wide range of spatial and temporal information. The new MS-Webcam dataset focuses on head and shoulder scenarios, mainly representing the video conferencing use case, showing a lower range of TI values.

We choose the SlowFast [10] model pre-trained using the action recognition task on the Kinetics-400 [8] for feature extraction. This choice was motivated by: (1) the fully convolutional nature of the model, allowing input videos of arbitrary resolution, and (2) the design of SlowFast being inspired by the Human Visual system.

We extract the features using the SlowFast model pre-trained on Kinetics [8]. Specifically, we use the SlowFast 8x8 with a ResNet-101 [15] backbone available on PyTorch Hub.

For each video, we fed a sliding window of 32 frames (sampled with a step size of 2 from a chunk of 64 consecutive frames), with a stride of 16, extracting 2304-D features before the last fully connected layer. Each window thus covers $2.133$ seconds (64 frames / 30 fps) of the video.

As mentioned above, SlowFast is a fully convolutional model allowing us to forward each clip (sequence of frames), in full resolution ($1920\times 1080$) without resizing or cropping. Avoiding resizing is critical because many distortions introduced by the ML video codecs are not noticeable if the videos are resized to a smaller resolution. This is especially true for the top-performing ML video codecs from the CLIC 2022 Challenge.

For each pair of encoded video ($X_{enc}$) and reference video ($X_{ref}$) features, we calculate the difference ($X_{diff}$) as:

We further concatenate this difference to the encoded video features resulting in the concatenated feature vector ($X_{SF}$):

Theoretically, the model should learn to model the difference on its own, but, in our experiments, we found providing the difference explicitly helps the model achieve better results as well as converge faster.

In our experiments, we noticed that in the few cases where a video does not have a lot of motion (i.e., it is more or less like a still image), then as one would expect, SlowFast features as well as learning-based baselines [50, 42] do not perform as well as frame-metric based methods such as VMAF (see Figure 10). Thus, in order to make our model more robust to these outlier videos, we also input frame-level metrics. We pool frame-level metrics such as Visual Information Fidelity (VIF) [45] and Detail Loss Metric (DLM) [24], in a similar manner as the sliding window approach described above. For each window of 32 frames, we end up with $32\times 11$ frame level metrics which are flattened to a vector of size 352. This vector is concatenated with the SlowFast features resulting in the final feature representation ($X$) of the video pair which is fed to the model:

In order to increase the data size, we propose augmentation strategies that do not affect a clip’s perceptual quality relative to a reference clip. For spatial augmentation, we extract features using different transforms such as horizontal and vertical flips, and rotation by small angles ($|\theta|\in[0^{\circ},5^{\circ},10^{\circ}]$). We also add a center crop ($500\times 500$) augmentation to implicitly give more weight to the center region of each frame, since the human visual system often pays more attention to the center of its view[53], thus having more impact on the perceived quality of the video.

Before settling on the final set of augmentations, we explored different strategies, including modifying the distorted videos at the just noticeable difference (JND) level, and 3x the JND level. For these experiments we included changes in brightness, contrast, hue, saturation, gamma, Gaussian noise, rotation, and horizontal flipping, parameter levels were defined through crowd-sourced P.910 DCR subjective tests. These experiments are discussed in Section 5.2 in more detail and did not improve the model’s performance.

We also augment (temporal sample) the data along the temporal axis. Specifically, given a $T\times 2304$ feature representation of a video, we sample feature vectors with a step size of 2, resulting in two features of size $[T/2]\times 2304$.

We follow the standard practices of training augmentation [50, 10, 8, 39]. During training, random augmentations are applied to both the encoded video as well as the reference video. The resulting features are then sampled temporally as mentioned above. It should be noted that the same augmentation is applied to both the encoded and reference video to ensure the relative perceptual quality stays the same. We also experimented with test-time augmentation, but it showed no improvement in performance.

An overview of the model is shown in Fig. 4. Our model consists of a projection layer followed by two 1-D convolutional layers and a quality prediction head.

The projection layer consists of a 1-D convolutional layer with a kernel size of 3, a stride of size 1, and padding of size 1. This layer projects the input 4608-D features to a size of 128-D. The two subsequent convolutional layers have the same configuration with the only difference being that of keeping the number of input and output channels the same, viz. 256-D. ReLU activation functions follow each convolutional layer.

We chose the kernel size of $3$ to help incorporate information from neighboring features. Setting stride and padding to $1$ ensures the temporal dimension does not change during convolutions.

After the 1-D convolutions is a 2-layer multi-layer perceptron to predict the score at each time step. This allows a fine-grained quality score prediction since the quality of a video clip can be dynamic over time. Finally, we average (arithmetic mean) the scores across the time dimension to get the final predicted score for the video.

We use Smooth L1 loss to model the error between the predicted quality score ($\hat{y}$) and the ground truth MOS ($y$):

We optimize the model using the Adam optimizer with a batch size of $32$, weight decay set to $10^{-4}$, and momentum of $0.9$. The maximum learning rate is set to $4\times 10^{-4}$ and a linear warm-up is applied in the first $10$ epochs. After $10$ epochs the learning rate is decayed using the cosine decay schedule. We train for $200$ epochs after warm-up, which takes about $75$ minutes on 8 NVidia V100 GPUs ⁴⁴4We pre-extract the features to speed up the training process. Without pre-extraction, we expect the training to take longer..

A key performance indicator for an FRVQA model is the validity of the rank order of ML video codecs produced by it and compared to the subjective test rank order. Typically, SRCC and Kendall’s Tau ($\tau$) [19] are used for measuring the ordinal association between two sets of scores (i.e., DMOS values from a subjective test and predicted scores from an FRVQA model). The SRCC represents the linear relationship between two rankings, for which a high coefficient can be achieved when the two ranked sets are monotonically related. Kendall’s Tau measures how much two ranking procedures agree for all combinations of item pairs in the set. It is well known that Kendall’s Tau is a better estimate of correlation in population and is the preferred method when there is a small number of items, and many tied ranks [17, 11]. It also provides a better interpretation compared to the SRCC. $\tau=t$ shows that for a randomly selected pair of items (e.g., ML video codecs), the probability that the two items will be ranked in the same order as the subjective test is $t$ higher than the probability that they will be ranked in the reverse order. In other words, it shows how far the conclusions made from the prediction of an objective model correspond to the conclusions made from the subjective test. Fig. 5 illustrates the prediction of a poor-performing FRVQA model compared to the subjective scores resulting in SRCC $=0.92$ and $\tau=0.75$. Consequently, a high SRCC score, in this case, can be a misleading metric.

Furthermore, none of these methods consider the uncertainty in subjective scores. Both methods consider two items to construct a tied rank only when they have equal numeric values. However, the MOS from a subjective test represents the sample mean and is subject to uncertainty, i.e., the numeric MOS value can change by a minor change in the rating of a single participant. Therefore, it is recommended to consider the distribution of ratings (e.g., through 95% CI ranges) when comparing MOS values of two items [33, 38]. For our evaluation, first, we create a rank order of the items given the DMOS and the 95% CI values from the subjective test according to [33], i.e., two items considered a tied rank when the DMOS value of one item is in the 95% CI range of the other item. Afterward, we calculate Kendall’s Tau-b on the provided ranked-order list (hereafter referred to as Tau-b 95). Given the higher number of tied ranks, a small number of ML video codecs, and better interpretability of Kendall’s Tau, we consider Tau-b 95 the primary metric in our evaluation.

Besides ranked-based metrics, we also calculate Pearson Correlation Coefficient (PCC) and Root Mean Square Error (RMSE) to evaluate the linearity and accuracy of predictions, respectively.

We used a bootstrapping simulation to estimate the accuracy of DMOS scores as a function of the number of votes used per clip and model. In the simulation, N votes were randomly selected per clip (model) with replacement and used to calculate the DMOS values of that subset. Consequently, we calculated correlations and RMSE between DMOS from the subset and when the entire ratings are used. We repeated this process 200 times per the number of selected votes (N) and calculated the mean and 95% CI for all metrics⁵⁵5The average 95% CI for all metrics in model level was 0.001.. Fig. 6 illustrates how each metric changes as a function of the number of votes used in the calculation of DMOS. At the model level, PCC is saturated by $\sim$100 votes, whereas the SRCC shows a slight increase up to $\sim$200 votes. Kendall’s Tau-b and Tau-b 95 partially saturated by 500 votes however with only a change of $\sim$0.01 compared to 200 votes. RMSE is also saturated by $\sim$600 votes although with only a 0.05 difference compared to 200 votes (see Figure 2(b)). At the clip level, although PCC and SRCC are saturated by $\sim$25 votes, Tau-b has a steady slope within the range of simulation. Similarly, RMSE tends to decrease further when more votes are used. A similar comparison for ACR ratings showed that $\sim$40 votes per clip are recommended [32].

Subjective scores have an uncertainty that decreases by increasing the number of votes collected in a dataset. One can conclude that the prediction accuracy of a hypothetical objective model, when compared to subjective scores, may only reach the limit observed during simulation, given the number of subjective votes collected for the test set. In this paper, we use five-fold cross-validation, in which 6 videos (encoded by 27 codecs) belong to the test set in each fold (details in section 5). Consequently, the predictions of the FRVQA models were compared to the subjective DMOS values calculated from $\sim$26 votes per clip and $\sim$160 votes per model. Given the simulation result, the expected upper limit for each statistical metric is given in Table 6.

Figure 8 illustrates how the RMSE is changing as a function of the number of selected votes in the bootstrapping simulation.

In this section, we show how different components of MLCVQA contribute to its performance. Table 8 shows the results of our ablation study. Specifically, we test the contribution of (1) SlowFast features, (2) Augmentation, and (3) Adding image level metrics. We observe how using SlowFast features alone results in very high performance at the model level. This performance is boosted by simple geometric augmentations and further improved by adding image-level metrics.

To utilize augmentation, we performed multiple experiments with different types, levels, and augmentation strategies. First, within an expert viewing session, we established the highest level of augmentation without a noticeable difference in quality for all augmentations. Table 9 reports the type and level of augmentations. We also examined these levels in a follow-up DCR crowdsourcing test, where the quality of the augmented clips was rated compared to the reference clips. We observed less than a 1 DMOS change in quality for Rotation and Additive Gaussian Noise and less than a 0.5 DMOS change for the rest of the augmentations. Therefore, we considered these as minimum augmentation levels.

We examined different augmentation strategies with the following configurations: 1) only processed clips augmented with minimum levels (AugDistortedOnly), 2) Both reference and processed clips are augmented, with 3$\times$ of the minimum levels (ExtremeAug), 3) geometric augmentations as described in Section 4.2 of the paper and the subset of geometric augmentations from table 9 (GeometryOnly), 4) ensemble testing [40] where many augmentations are averaged for a final prediction value (TestAugmentation), 5) a subset of GeometryOnly where all geometric augmentations coming from Table 9 were removed (SubsetGeometryOnly). We evaluated these strategies on the performance of the MLCVQA model. Results are reported in Table 10 and show that the SubsetGeometryOnly and TestAugmentation strategies beat the rest. We use SubsetGeometryOnly in the next steps as it is the simpler approach.

We also evaluated the effect of augmentation on other ML-based FRVQA models. Table 10 also shows the results of experiments with CompressedVQA and C3VQA, both retrained using SubsetGeometryOnly, the best set of performing augmentations for MLCVQA. The MLCVQA (ours) result adds frame-level features to the best-performing features which are described in Section 4.1 of the paper. 5-fold cross-validation results are used to compare all experiments with MLCVQA.

We observed that MLCVQA is performing well in predicting the quality of videos on both extremes of the DMOS scale (1 and 9) and shows lower accuracy for videos in the middle-quality range. We trained a random forest regressor to predict the absolute error of MLCVQA prediction (per clip) given some features from source videos like SI and TI, and the uncertainty of subjective ratings. We randomly selected 80% of the data for training and repeated the process 20 times with newly selected training and test sets. On average, the regressor’s prediction on the test set has a PCC = 0.63 with the absolute error. The essential features and their average importance over all repetitions are reported in Table 11. Results show that the uncertainty of subjective ratings is moderately correlated with the absolute error. However, it should be interpreted with caution as it is known that items located in the middle-quality range mostly have a larger 95% CI compared to items located in both poles of the scale. There could be a confounding factor that both increases uncertainty in subjective rating and prediction error of MLCVQA.

Figure 10 shows the TI and SI for each split. In addition, we show if VMAF is performing better than MLCVQA based on which model has a higher SRCC for that split. Note that VMAF tends to do better for lower TI and lower SI values.