YouTube SFV+HDR Quality Dataset

Our raw video pool ideally includes all YouTube SFV with the Creative Commons license. However, it doesn’t mean we can cover all video characteristics with sufficient samples (e.g. 10 sampling points per dimension) in one dataset. Common video characteristics include content, original quality, video (spatial/temporal) complexity, color space, resolution, frame rate, video length, freshness, etc. It is preferable to identify key characteristics of the study, and remove less important dimensions. Regarding a SFV quality dataset, we think resolution, frame rate, and video length are less important for exploring distinct characteristics of SFV. Thus we chose the most popular settings for resolution and frame rate, which are $1080\times{1920}$ and 30FPS respectively. For video length, we cropped the first 5 seconds of the entire video, which fairly represented the quality for most SFV. To sufficiently represent the current trend of SFV, we selected 80,000 recently uploaded SDR videos with Creative Commons license. Video content is another important dimension to study human opinions on SFV quality. We selected 10 popular SFV categories annotated by Knowledge Graph [10]: Animal, Cooking, Dance, Gameplay, Health, Hobby, Music, Society, Speech, and Sports. In this way, our initial SDR pool has 10 subsets, corresponding to 10 content categories, and each subset contains 1000 to 7000 samples.

HDR is another focus of our dataset, and has additional constraint. Compared to SDR SFV, the total number of HDR SFV with Creative Common license is smaller, so we included all available 1080P HDR SFV in the initial HDR pool, still cropped the first 5s. Only part of HDR SFV were associated with above mentioned content labels, and others’ content categories are labeled as “unknown”.

The initial SDR and HDR sampling pools were created as outlined above. Although they are in different situations in terms of the pool size, we still followed the same creation rationale, i.e. maintaining key characteristics of the interest and removing less important dimensions to reduce noise.

The identified sampling pool usually contains orders of magnitudes more videos than the target size. In our case, the target size for SDR SFV dataset is 2000, while the size of the SDR pool is 80000. How to fairly sample videos is another important topic. Purely random sampling has a risk to be biased with the current data distribution and poorly represent minor categories. To better represent the entire set, we divided the sampling space by three basic video features: spatial information (SI), temporal information (TI), and perceptual quality. Here we followed ITU-T Rec. P910 [11] to calculate SI and TI. For perceptual quality, the precise value requires subjective tests, which is not available at this stage. Alternatively we used UVQ (old name CoINVQ) [12] to approximate perceptual quality, which should be sufficient for the sampling purpose.

Figure 1 shows the scatter plots for pairs of SI, TI, and UVQ for three content categories: Cooking, Health, and Gameplay. We can see most Cooking videos have low SI and high TI, while most Health videos have relatively low TI. Both Cooking and Health videos have relatively high quality (UVQ $>4.0$), while Gameplay videos have wider distribution in all three dimensions. This analysis demonstrated that the content categories had significant discrepancy, and needed to be investigated separately if possible. We used the mean values of SI, TI, and UVQ to divide the entire pool into 8 ($=2\times{2}\times{2}$) subregions, and randomly selected equal number of samples from each subregion for each content category. This sampling strategy includes more samples from low density regions, which effectively suppressed the bias of original data distribution. In practice, we selected 50 samples per subregion per category for SDR SFV. We keep the entire HDR set for the final review since its total size is already close to the target size (4000 v.s. 2000). After this step, both SDR and HDR pools remain about 4000 samples. Fig. 3 shows samples in high and low quality (approximated by UVQ) for each content category, where we can see a distinct gap between high and low quality samples. Similar distinct gaps can be found in SI and TI, which demonstrated good diversity.

Step 1 and 2 in our sampling framework mainly rely on objective features (e.g. content category) and metrics (SI, TI, and UVQ), which still has some problems. For example, videos with inappropriate contents cannot be filtered out by above objective metrics. Also to maximize the diversity of the dataset, it is preferable to remove duplicates and limit the number of similar samples. This duplicates issue is more severe in our HDR pool than the SDR pool, due to limited candidates. Several creators contributed hundreds of samples with similar contents, which could lead to a significant bias of the final dataset. Thus a careful manual review is a necessary step to finalize the dataset. For this SFV+HDR dataset, we cleaned up the content with multiple manual reviews to maximize the content diversity, as shown in Fig. 2.

After this three step sampling, we finally selected 2030 SDR and 2000 HDR samples. For SDR, most content categories have 210 samples except Dance (140 samples). For HDR, we selected 30 samples per content category, and the other 1700 samples with unknown category.

With the finalized video set, the next challenge is to collect subjective quality scores. Original YouTube uploads are in various formats (e.g., different codecs, color spaces, and frame rates). In order to be playable on all clients’ browsers, we transcoded all original videos using H.264 [13] with YUV420P and Constant Rate Factor (CRF) value of 10 to preserve the quality as close as possible to the original version. For HDR videos, two major types are Perceptual Quantization (PQ) and Hybrid Log-Gamma (HLG). We converted these HDR videos to SDR using corresponding tone mappings. The subjective tests were run on mobile phone, since it is the main device interface for SFV.

We conducted two subjective tests for SDR and HDR respectively. The SDR test contained 4030 (2030 native SDR and 2000 HDR2SDR) videos. 66 subjects (from an internal data labeling team) participated in this SDR test using their own mobile phones. Each subject participated in 7 sessions, where each session contained 300 randomly selected SDR videos. The HDR test included 300 native HDR videos and 300 corresponding HDR2SDR versions. After preliminary tests, we found HDR experiences were highly different on different devices. For example, the same HDR videos look much brighter on Pixel 7 pro than on Pixel 5, due to different peak brightness. To get more reliable subjective data, 10 subjects who used the same phone (Pixel 7 pro) participated in an additional HDR test (2 sessions, 300 videos per session). Subjects were first shown three training videos to get them familiar with the testing process. These three videos were chosen to exemplify bad, okay, and good qualities. After the training, subjects were presented testing videos that were randomly sampled from the entire dataset. For each video, subjects had to watch the entire duration of the clip, and they were allowed to replay the video if necessary. Then they were asked the quality assessment question of How was the overall video quality? The rating was given on a 1 to 5 scale slider, adjustable in 0.1 increments, where each integer is marked as Bad (1), Poor (2), Fair (3), Good (4), and Excellent (5). The test was designed to be finished within 30 minutes. Each SDR video clip was finally rated by 25 to 40 subjects, and HDR videos had 10 ratings. Since our raters were from a professional data labeling team, whose ratings were reliable, we used all their ratings to compute Mean Opinion Score (MOS).

The left histogram in Fig. 4 showed the overall MOS distribution for the entire SDR set (4030 samples, including native SDR and HDR2SDR). We found that the MOS distribution is relatively narrow, where $80\%$ MOS values are within [3.8, 4.6]. It suggested that many SFV have equally good quality, and meet viewers’ quality expectation. We further broke down the set into native SDR and HDR2SDR (middle and right histograms in Fig. 4). We can clearly see that MOS of most HDR2SDR ($90\%$) are higher than 4.0, which has two potential reasons: 1) HDR videos are usually captured by high end devices which natively provides high picture quality; 2) The color plays an important role in quality assessment. The first reason is intuitively major, but we also found some examples supporting the second reason. As shown in Fig. 5, the two HDR2SDR samples have saturated color and keep more local color details, which makes them look professional and high quality. It implies that good attributes of HDR are still maintained in their HDR2SDR versions.

Content dependency is also an important feature of UGC quality assessment. Fig. 7 shows the MOS distributions of native SDR for individual categories. We can see that Society and Speech have relatively uniform distributions (and lower average quality) than other categories. One potential reason is that many content were recorded in public spaces with restricted lighting and device control. Another potential indication is that viewers are not very interested in such contents and intuitively avoid giving very high scores. In contrast, Cooking and Hobby have the highest average quality, since the contents are widely interesting, and creators fully control the recording environments and are able to do sophisticated post-enhancements. Above are just preliminary observations. Understanding the content impact on perceptual quality is an important topic with practical impacts (e.g. calibrating quality score among different contents). We hope our dataset encourage more thorough research on this topic.

Fig. 6 compared MOS for native HDR and corresponding HDR2SDR versions. We can see most HDR MOS are higher than corresponding HDR2SDR version, and the average MOS difference is about 0.18, which means a significant gap. Based on raters’ feedback, the brightness played an important role in the quality assessment. HDR videos are significantly brighter with more clear details than SDR versions, which makes HDR look better.