DCVQE: A Hierarchical Transformer for Video Quality Assessment

The left side of Fig. 2 represents the architecture of a key video quality analysis layer, Divide and Conquer Transformer (DCTr) in our proposal. In order to simulate the second action of human annotation mentioned above, we split the input sequence into a number of clips, and introduce a Transformer module $TransformerD$ to learn quality representations for each clip. As shown in Fig. 2, for the $k^{th}$ DCTr layer, we split the whole sequence into $I$ clips, and each clip covers $J$ frame-level quality representations generated by the previous layer. For the $i^{th}$ ($1\!\leq\!i\!\leq\!I$) clip $C^{k}_{i}$, a module $TransformerD_{i}$ is applied to combine the two levels of quality embeddings (QE) generated by the previous layer, that is, all the $J$ frame-level QE $F^{k-1}_{i,j}(1\!\leq\!j\!\leq\!J)$ and the video-level QE $Q^{k-1}$, to simultaneously learn the clip-level QE $C^{k}_{i}$ and update the frame-level QE $F^{k}_{i,j}$ for the current layer. We further simulate the first action of human annotation by integrating the learned clip-level QE to generate a video-level QE. The other Transformer module $TransformerC$ with a topped average pooling layer are proposed to merge all clip-level quality representations $C^{k}_{i}$ to predict the video-level QE $Q^{k}$ for the current layer. In general, our proposed video quality analysis module is constructed in a divide and conquer format, so we coin it as a Divide and Conquer Transformer (DCTr) layer.

The overall architecture of our DCVQE model is shown in the right side of Fig. 2. As seen, we stack several DCTr layers (3 layers for our proposal, refer to supplementary material for details about this setting) to extract the final video-level QE $Q^{k}$ for the input video. The final quality score is predicted by topping this embedding $Q^{k}$ with a regressor. In practice, to improve the temporal sensitivity of our model, we progressively expand the coverage of video clips in the DCTr as the layers deepen. For example, supposing that each clip in the $k^{th}$ DCTr layer covers $J$ frame-level QE from the previous layer, the coverage of each clip in the $(k\!+\!1)^{th}$ DCTr layer will be $2J$, which means two neighbor clips of the $k^{th}$ DCTr layer are combined to extract the clip-level QE in the $(k\!+\!1)^{th}$ DCTr layer. For the first DCTr layer, the input frame-level QE $F^{1}_{*,*}$ is generated by our feature extractor described in subsection 3.2, and the input video-level QE $Q^{1}$ is initialized randomly. Both of them are integrated with positional embeddings $P_{*,*}$ [9] for the following processes.

$TransformerD$ and $TransformerC$ are the two most important modules in our DCVQE. As shown in Fig. 3, they have similar architecture to the classic Transformer [55]. Three fully connected layers are used to convert the input features into Query, Key, and Value tensors. Split operations are then adopted to group these three tensors into $H$ heads, respectively. For the input frame-level feature with the shape of $B\!\times\!S\!\times\!D$, where $B$, $S$ and $D$ denote the batch size, sequence length and dimension of the frame embedding respectively, the shapes of its corresponding grouped Query, Key and Value tensors will be $H\!\times\!B\!\times\!S/H\!\times\!D$. The attention weights can be computed by the dot-product operations between the grouped Query and grouped Key tensors. To make $TransformerD$ module more concerned with the clip-level quality information, except the first position is reserved for the input video-level QE, we mask out the attention weights outside a predefined temporal range by using sequence masks (the dashed square plotted in Fig. 3) and adopt a Softmax layer to normalize the rests. The normalized weights are then applied to the grouped Value tensor to update the quality features. A stack operation is conducted on the head axis to recover the shape of the tensor. Here we also introduce the residual technique [17] to improve the robustness of $TransformerD$ module, so the final output of this module is the addition of the recovered features and residual. $TransformerC$ module is designed to update the video-level QE for each DCTr layer. The difference between $TransformerC$ and $TransformerD$ modules is that $TransformerC$ module does not contain sequence masks and residual block, as shown in Fig. 3.

To avoid the additional distortions that may be caused by applying some preprocessing steps to the input data, such as image resizing and filtering, we take the original full-size image frames as the input of our model. The pretrained Resnet-50 [17] is adopted as the CNN backbone of our feature extractor. To increase the sensitivity of the feature extractor to capture frame-level distortions, we first conduct a typical IQA task to fine-tune the CNN backbone. As shown in the top figure of Fig. 4, all CNN layers except the last group of CNN blocks are frozen for fine-tuning. We find this process can help to transfer the learned knowledge from ImageNet [8] to our application. It will be verified by our experiments.

Based on the fine-tuned backbone, as the bottom figure of Fig. 4 shows, the quality feature of each input frame is then extracted by adopting a concatenation operator to the outputs of global average (AVG) pooling and global standard deviation (STD) pooling processes [23, 24]. Since both the dimensions of the outputs of AVG pooling and STD pooling are 2048, the frame-level QE for each frame will be a vector with a dimension of 4096. In practice, refer to [23], we further add a fully connected (FC) layer to reduce the dimension of the frame-level QE to 128. It is an efficient way to balance the accuracy of frame-level feature representation and overall computational cost.

Prior works usually use the following L1 loss to optimize the model:

$$L_{1}=\frac{1}{N}\sum_{n=1}^{N}\left|p_{n}-g_{n}\right|$$

(1)

where $p_{n},n\in[1,N]$ represents the $n^{th}$ predicted MOS, $g_{n}$ denotes its corresponding ground truth, $N$ is the total number of videos. Mathematically, this criterion to some extent ignores the relative order relationship of quality scores of the training samples in a batch, it sometimes may lead to limited ability to quantify the perceptual differences between the videos with similar quality scores. For example, considering the training processes of two NR-VQA models based on L1 loss criterion, for two video samples A and B with the MOS ground truths of 7.0 and 7.1, if their quality scores are predicted as $P_{A_{1}}=7.1$, $P_{B_{1}}=7.0$ and $P_{A_{2}}=6.9$, $P_{B_{2}}=7.0$ by the two models respectively (where $P_{A_{1}}$ denotes the quality score of video A predicted by model 1, and so forth), the same strengths coming from the L1 losses (0.1) of these two training samples will be contributed to optimize the models correspondingly. Though it is hard to say which model is better than the other, in our opinion, it may be easier to optimize model 2 because of a positive correlation between the ground truths and its loss. Based on this idea, a new correlation loss, aiming to bring an additional order constraint of video quality to guide the training procedure, is proposed as follows:

$$L_{c}=\frac{1}{N}\sum_{n=1}^{N}\max\left(0,-\left(\sum_{m=1}^{N}\left(p_{n}-p_{m}\right)\right)\left(\sum_{m=1}^{N}\left(g_{n}-g_{m}\right)\right)\right)$$

(2)

where $p_{m}$ represents the prediction of a training sample in a batch with size $N$, and $p_{n}$ denotes the $n^{th}$ prediction which is picked to compare the ranking order with each element in the prediction set. $g_{m}$ and $g_{n}$ stand for the corresponding ground truth scores of $p_{m}$ and $p_{n}$, respectively. This loss equation can be considered as that, we take the $n^{th}$ video in the batch as an anchor and compute the positive or negative correlations between predictions and ground truths for this anchor and any other one in the batch. Eq. 2 can be further simplified as:

$$L_{c}=\frac{1}{N}\sum_{n=1}^{N}\max\left(0,-N^{2}\left(p_{n}-\sum_{m=1}^{N}\frac{p_{m}}{N}\right)\left(g_{n}-\sum_{m=1}^{N}\frac{g_{m}}{N}\right)\right)$$

(3)

$$L_{c}=N\sum_{n=1}^{N}\max\left(0,-\left(p_{n}-\bar{p}\right)\left(g_{n}-\bar{g}\right)\right)$$

(4)

where $\bar{p}$ and $\bar{g}$ represent the mean values of predictions and ground truths, respectively. We find that this simplified $L_{c}$ term is actually equivalent to the Spearman correlation coefficient without normalization. A total loss $L$, which combines the above two loss equations (1) and (4), is finally constructed to optimize our DCVQE model as

$$L=\alpha\times L_{1}+\beta\times L_{c}$$

(5)

where $\alpha$ and $\beta$ represent the weights of $L_{1}$ loss and $L_{c}$ loss. We will discuss the optimal settings for these two weights in subsection 4.4.

We conduct experiments on four in-the-wild VQA datasets: KoNViD-1k [18], LIVE-VQC [48], YouTube-UGC [58], and LSVQ [63]. In these datasets, KoNViD-1k contains 1,200 unique contents with a duration of 8 seconds; LIVE-VQE, the smallest among them, consists of 585 video contents; YouTube-UGC contains 1,500 UGC video clips sampled from millions of YouTube videos; A recently released dataset LSVQ, containing 39,075 video samples with diverse durations and resolutions, is the largest publicly available UGC datasets for research right now. We also follow the suggestion given in [53, 40] to select YouTube-UGC dataset as the anchor and map MOS values of KoNViD-1k and LIVE-VQC data onto a common scale to generate an All-Combined dataset for performance evaluation. The typical random 60-20-20 strategy [23] is applied to split data into three sets, i.e. 60% for training, 20% for evaluation, and the remaining 20% for testing. Especially, for LSVQ we follow the setting in [63] to first generate a Test-1080p set and then conduct the random 80-20 splitting on the rests to generate training and testing sets for experiments.

Here we adopt four commonly used criteria, Spearman Rank-Order Correlation Coefficient (SRCC), Kendall Rank-Order Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC) and Root Mean Square Error (RMSE) to measure prediction monotonicity and prediction accuracy. For fair comparisons, we conduct each evaluation 100 times individually and report the median values of these four metrics as their final results.

We first fine-tune ImageNet-pretrained Resnet-50 backbone on KonIQ-10k dataset [19] with random 80-20 splitting and maximum 20 epochs. The CNN backbone with the best performance of SRCC on the test set is selected to construct the feature extractor for our VQA task. The frame-level features are extracted using this feature extractor, where all frames of each video are considered without sampling. The temporal range of the sequence mask in the proposed $TransformerD$ (see Fig. 3) is empirically set to 15. For a tradeoff between GPU memory, running speed, and performance, we set the maximum length of each input video to 600 (i.e. only the first 600 frames of a video are considered if the video is longer than this maximum length) and the maximum training epoch to 75. An evaluation process will be conducted after each epoch, and the model with the lowest loss on the validation set will be stored.

In this subsection, we perform ablation studies to better understand the different components of the proposed DCVQE model.

CNN Backbone: Training of the CNN feature extraction backbone can also be regarded as a typical IQA task. Two series of architectures based on ResNet and ViT backbones are fine-tuned fully or partially in our study. Here we note the partial fine-tuning means that only the last blocks of these models, topped with average pooling and fully connected layers (see Fig. 4) are trainable. We determine that fully fine-tuned backbones act negatively with regard to the complexities of the networks. i.e. deeper structure results in worse performance, and partial fine-tuned backbones usually work better than fully fine-tuned ones for both two series of architectures. We also incorporate a new attention-based IQA architecture PHIQNet [65] to our study, and similar performance as partial fine-tuning on ResNet is observed (refer to supplementary material for details). In our application, for a tradeoff between model complexity and performance, also a fair comparison with previous work [23], we select the partially fine-tuned ResNet-50 as the feature extraction backbone to construct our DCVQE model.

Temporal Range of Sequence Mask: We introduce a group of experiments on KoNViD-1k dataset to study how the temporal range affects the DCVQE model. By directly replacing the proposed DCTr layers with the traditional Transformer layers [55], a baseline Transformer model is constructed for performance comparisons. Test results are plotted in Fig. 5. From these figures, we can see that the best temporal range for the baseline Transformer model is about 9, and the performance of this baseline model slightly decreases as the temporal range increases. Especially if we set the range to “all”, which means that all the frame-level representations are involved in the self-attention processes [55] of the Transformer, the performance of this baseline model drops dramatically. Switching to our DCVQE, we can see that the overall performance stays at a high level. The best performance can be observed when the temporal range reaches 15. The results of the “all” setting for DCVQE, for which one frame only conducts self-attention with all frames in its own clip, also demonstrate the robust performance of our DCVQE model.

Correlation Loss: To show how the proposed Correlation Loss (CL) contributes to our VQA task, the performances of DCVQE with different weight settings of CL and L1 loss terms are reported in Table 1. The best performance can be identified when L1 Loss weight $\alpha$ reaches 0.7 and CL weight $\beta$ reaches 0.3. Here we note that our CL does not intend to play a critical role in model optimization since it tries to describe the relative order relationships between videos, particularly for those videos with similar quality scores, so the overall improvement by adding this CL term may not look very significant. Overall, we confirm that our model can achieve better performance with the introduction of CL (Eq. 4). The rest experiments are conducted with the setting of $\alpha=0.7$ and $\beta=0.3$ if not specified.

Component by Component Ablation: We further stack the key parts in our DCVQE component-by-component to analyze their impacts individually and gradually. To additionally study the effectiveness of our new CL term, a well-known pairwise ranking loss (PW-RL) [4], which is similar to our proposal, is also involved in our experiments. As shown in Table 2, compared with Vanilla-Transformer (row 1), only introduces the Divide operation (row 2) does not impact the system’s performance. In row 3 we employ the AVG pooling operator to integrate clip-level embeddings to generate video-level embeddings, and the results show that this simple “conquer” operation can slightly improve the performance. In row 4 the conquer operation is switched to the proposed $Transformer_{C}$, where similar correlation scores with a lower RMSE are presented (compared with row 3). The temporal range of sequence mask 15 is introduced to tests in rows 5 and 6, where we can see that this change apparently benefits the system. Furthermore, the losses CL and PW-RL are compared directly in rows 7 and 8. The results confirm that the proposal CL provides the best performance for our VQA task.

We compare our method with 13 state-of-the-art VQA methods, which can be roughly divided into NR-IQA based methods: BRISQUE [33], GM-LOG [61], HIGRADE [22], FRIQUEE [14], CORNIA [62], HOSA [60], Koncept-512 [19], PaQ-2-PiQ [64], and NR-VQA based methods: V-BLIINDS [46], TLVQM [21], VSFA [23], VIDEVAL [53] and RAPIQUE [54]. For NR-IQA based methods, we abstract one frame feature per second and regard the average value of the features as the video representation to train a support vector regressor (SVR) for prediction [54, 53, 63].

Test results are reported in Table 3 and the best solution for each evaluation metric is marked in bold. Since it is hard to reproduce VIDEVAL and RAPIQUE methods, in Table 3 we directly cite the results given in [53, 54] (marked as “*”). Different from our default 60-20-20 data splitting setting, these two methods used the random 80-20 strategy to split data for training and testing. For a fair comparison, we train an additional version of the DCVQE model under the same 80-20 data splitting setting. Test results of this additional DCVQE model are marked as “†”. As seen from this table, our model (DCVQE†) significantly outperforms the current top method RAPIQUE on KoNViD-1K dataset by 4.37% SRCC and 2.45% PLCC, and greatly exceeds the second place method VIDEVAL on YouTube-UGC dataset by 5.62% SRCC and 5.67% PLCC. TLVQM method performs best on LIVE-VQC dataset. Looking into this dataset, we can see that most videos are captured using mobile devices, thus camera motion blurring is one of the common distortions affecting the video quality. TLVQM introduces several hand-crafted motion-related features to handle this issue, so it achieves the best performance, especially compared with all the deep learning based solutions. Nevertheless, besides TLVQM, our proposal performs the best among the remaining. The evaluation on the All-Combined dataset of KoNViD-1k, LIVE-VQC, YouTube-UGC also confirms the robustness of our model. As shown in the column “All Combined”, our method (DCVQE†) surpasses the second place method (RAPIQUE) by 2.84% SRCC and 1.12% PLCC.

To demonstrate how well our model is learned from a given dataset, in Fig. 6 we also visualize the correlation between the video quality predictions and their ground truths (top row), as well as the video-level QE learned by the “conquer” operation of the last DCTr layer (bottom row). From the top row, we can see a strong correlation between our predictions and their ground truths. The figures in the bottom row plot the dimension-reduced video-level QE, where the gray value of each point is assigned with the normalized ground truth MOS score of the corresponding video sample, that is, the darker a point, the greater its corresponding MOS score. From these figures, we can observe that the points with higher MOS scores scatter on one side while the points with lower MOS on the other. It indicates that the separability of quality for those video samples has been significantly enhanced by using our DCVQE model.

Additionally, we compare our model with several others on the recently released large-scale LSVQ dataset. The PVQ model [63], which was proposed along with this dataset, is also involved in our evaluation. Test results listed in Table 4 confirm the excellent performance of our model again.