Wide Color Gamut Image Content Characterization: Method, Evaluation, and Applications

In order to reproduce the original color of contents in devices having smaller color gamuts, several GMAs have been proposed. They can be categorized into global and local strategies. The former changes all colors of out-of-gamut pixels towards the inside of the target gamut by gamut compression or clipping [14, 15, 16, 17, 18]. QoE of the gamut-reduced image often decreases since the color may become blurred around the pixels where the color changes. The latter considers spatial relationship between pixels at the expense of increased computational complexity in order to enhance perceived quality of the gamut-reduced images [7, 19, 20, 21, 22, 23, 24, 25].

QoE of gamut-mapped contents is usually assessed by conducting subjective or objective studies. In [26], a psychophysical experiment is conducted to evaluate four GMAs, where the subjective quality of the gamut-reduced images is assessed. In [27, 28, 29, 30, 31], various color image difference metrics are proposed to measure objective quality of gamut-mapped images. In [32], subjective scores of gamut-reduced images using different GMAs are obtained by a psychophysical experiment, and are used to evaluate four objective metrics.

However, in [33], it is concluded that the color difference measured by objective metrics and the perceived image difference between original and gamut-reduced images do not correlated well. There are attempts to improve objective metrics by employing spatial filtering that simulates the human visual system [34] and by extracting features based on perceptually important distortion [35]. On the other hand, studies that consider measuring QoE of WCG contents are rare. In [36], a physiological experiment is conducted to measure electroencephalography during watching WCG video contents.

Winkler [37] quantifies the characteristics of the contents in existing image and video datasets, including spatial information and colorfulness for color images, and motion vectors for video contents, based on which the representativeness of a set of contents can be evaluated [38, 39]. In [40], it is suggested to consider attributes of the test material such as brightness, colorfulness, amount of motion, scene cuts, types of the content, etc. for subjective video quality assessment. In [41], contrast, colorfulness, and naturalness are considered to characterize tone-mapped images for HDR contents. In [42], a content selection procedure for light field images is proposed using high-level features consisting of depth properties, disparity range of pixels, refocusing features, etc. as well as general image quality features.

In [43], however, it is argued that those simple characteristics do not sufficiently cover the perceptual aspects of visual contents when processing steps (i.e. tone-mapping) are involved. Therefore, an approach is proposed to characterize HDR contents in the viewpoint of whether an HDR content is challenging for tone mapping operators. It focuses on the perceptual change due to the dynamic range reduction that is frequently applied to HDR contents. Using this characterization method, a framework to build a representative HDR dataset is proposed in [44]. In a similar spirit, we propose a novel characterization framework for WCG contents.

We propose a framework for WCG content characterization based on the perceptual change caused by gamut mapping. We define WCG content characteristics as degrees of the perceptual differences due to successive gamut reduction. The overall procedure of the proposed method is summarized in Algorithm 1.

The framework in Algorithm 1 produces an $N$-dimensional feature vector of perceptual difference for each WCG source content. First, we obtain $N$ gamut-reduced images by applying a gamut reduction operator that converts the color gamut of the reference image $G_{0}$ into a target gamut $G_{n}$ ($n=1,\cdots,N$). For each gamut-reduced image $I_{n}$, we apply an objective metric that measures the perceptual difference from the reference image $I_{0}$. Finally, we obtain a feature vector $D$ describing the behavior of the WCG content in terms of perceptual difference due to gamut reduction. We can utilize this feature in various applications such as WCG dataset analysis, content clustering, and selection, which will be presented in Sections IV and V.

Hereafter, we provide implementation details of the proposed framework. In Algorithm 1, we use an objective metric $PD$ to measure the perceptual difference due to gamut reduction. Although various image quality metrics have been proposed in literature, metrics specifically designed to measure perceptual difference of images exposed to color gamut change do not exist. Therefore, we conduct a subjective test in which the mean opinion score (MOS) of the perceptual difference between gamut-mapped images is measured. MOS is then used to benchmark existing color metrics and optimize the best one via nonlinear transformation.

In order to approximate the subjective score of the color difference due to gamut reduction in an objective manner, we employ the color extension of the structural similarity index ($cssim$) [49, 50], which can effectively measure perceptually significant structural differences due to gamut reduction between two color images. The preliminary study [1] shows that it performs best with the highest accuracy among eight commonly-used objective color difference metrics [51, 52, 53, 54, 55, 56, 57].

For each pair of the reference and gamut-reduced image, we measure the $cssim$ score. The score is further fitted to the MOS by a monotonic nonlinear function as described in [58]:

where the fitted values of the parameters are $\alpha=2$, $\beta=-3.5$, and $\gamma=1.9$. The result of fitting for the training dataset is shown in Figure 2. In order to evaluate the prediction performance, we obtain MOS for the validation dataset from 20 subjects by following the same procedure described in Section III-B2. The Pearson correlation coefficients (PCCs) between the ground truth MOS and predicted MOS using the fitting function are 0.92 and 0.80 for the training and validation sets, respectively. Therefore, we calculate the perceptual difference in Algorithm 1 as

We validate the framework by applying it to a simple content selection task. As mentioned in Section I, using representative contents is crucial to draw reliable conclusion in studies on QoE of WCG images. In the task, the main objective is to select representative images that have diverse behaviors in terms of the perceptual difference due to successive gamut reduction. We use the framework to obtain predicted perceptual differences due to gamut reduction to the two target gamuts (Rec.709 and Toy) as two-dimensional features characterizing the 24 candidate images in the validation dataset. Then, the $k$-means clustering algorithm is applied to the predicted perceptual differences. The value of $k$ determines the number of representative clusters for content selection, which should be chosen by the user according to the purpose of content selection. In this experiment, we set the value of $k$ to five based on the distribution of the images in terms of the predicted perceptual differences. One image for each cluster is randomly selected to construct a representative image set, which maximizes the coverage of the feature space. For comparison, we also apply a random selection method where five images are selected randomly in the same dataset.

The result for each selection method is shown in Figure 3. It can be seen that the selected images by our framework in Figure 3a are more spread than the randomly selected images. In Figure 3b, however, the selected images are biased to the upper-side of the feature space. In this case, images having small perceptual difference by severe gamut reduction are not considered, and the obtained image set cannot be said to be representative. Figure 4 shows two example images (marked in Figure 3a) in different gamuts. In Figure 4a, as predicted, large perceptual differences are observed for both gamut-reduced images compared to the reference P3 image, i.e., the overall color of the scene and the green laser lights at the top area. On the contrary, Figure 4b hardly shows any difference between gamut-reduced ones, which is also predicted in Figure 3a. By selecting images with diverse characteristics, a representative dataset can be constructed by our framework.

We also evaluate robustness of content selection with our framework. For each of the two methods (random selection and our framework), the selection task is repeated two times to obtain two sets of selected images, and the PCC between the MOSs of the two sets is measured. We consider that a high value of PCC by a selection method represents a high level of robustness of the method, because it means that the characteristics of the selected images are consistent regardless of repetition or random effects. We repeat the procedure 100 times. Much higher PCC values are obtained by our framework than random selection (0.83 vs. 0.15 on average), which is found to be statistically significant via a t-test, $t(137.1)=21.1$, $p<0.001$³³3The statistical significance of higher PCC values by our framework is obtained in all cases with $k$ from 2 to 10..

By extending the dataset characterization criteria presented in [37], we propose to measure four statistics of perceptual difference measured by the framework as follows. In [37], three statistics of various characteristics extracted from images or videos in the dataset are proposed. They are two criteria measuring the coverage and uniformity in each dimension, and a multidimensional coverage criterion. In addition to these, we also consider the multidimensional uniformity. Note that we normalize the perceptual differences in each dimension to scale the span of the criteria within [0, 1], i.e., $\tilde{d_{i}}=d_{i}/s_{i}$, where $s_{i}$ is a normalization factor that is equal to the maximum possible value of MOS ($s_{i}=2$ in our case) because the minimum value of MOS is zero.

We analyze the two existing WCG datasets, HdM and Arri, in terms of the four criteria described above⁴⁴4These are the only publicly available datasets that support Rec.2020.. In this experiment, we collect 38 and 11 images from each dataset, respectively. We use the perceptual difference for the 49 images due to successive gamut reduction from the reference P3 gamut to the Rec.709 and Toy gamuts as in Section III-C. We then measure the four criteria of the two WCG datasets. For (total) uniformity, we use 10 bins for each dimension of the histograms (i.e., $B=10$). The measured criteria are summarized in TABLE II. In addition, the distributions of the perceptual difference for the two datasets are shown in Figure 5.

First, the coverages of the two datasets have different behaviors depending on the target gamuts. The perceptual differences of the images in the HdM dataset cover over about a half of the scale for both target gamuts as shown in Figure 5a. For the case of gamut reduction to Toy, the perceptual difference is biased to large values because most images of HdM contain many pixels with highly saturated colors, which produces large perceptual difference when the gamut is reduced. On the contrary, pixels with highly saturated color are few in the images of the Arri dataset, so the coverage criterion for Rec.709 is low while that for Toy is high as shown in Figure 5b.

Similarly to the results of the dimension-wise coverage criterion, the HdM dataset has a medium level of total coverage of perceptual differences, showing the convex hull covering almost the upper-half area in Figure 5a. On the other hand, although the coverage value for the Toy gamut is large as shown in Figure 5b, the total coverage of the Arri dataset is small due to the extremely low coverage for Rec.709. Note that $z_{Toy}$ would be always higher than $z_{709}$ for the same image because the details of color are more distorted in Toy, so the practical maximum possible value of total coverage is 0.707 ($=\sqrt{0.5}$).

In terms of uniformity, the perceptual differences caused by the large gamut difference (i.e., the case of Toy) are quite uniformly distributed for both datasets. For the small gamut reduction (to Rec.709), the perceptual differences of the HdM dataset are slightly biased to low values. The perceptual differences of the Arri dataset are extremely biased, so all data points are allocated in a single bin and the uniformity is zero.

In the case of total uniformity, there exist differences between the two datasets. The perceptual differences of the HdM dataset are quite uniformly distributed on the two-dimensional space in Figure 5a, although the data points are slightly biased to the upper region (where large perceptual differences occur due to large gamut reduction). For the Arri dataset, the perceptual differences are biased to the left-side in Figure 5b, so the total uniformity becomes low.

Overall, each of the two datasets has its own strengths and limitations in a complementary manner. HdM has a relatively small coverage of $z_{Toy}$, while Arri has limited characteristics in the Rec.709 gamut. For example, if the Arri dataset is used for an experiment involving gamut changes, the experiment would draw biased conclusion for small gamut difference. Based on this understanding, one can choose either of the two datasets for particular research problems; for instance, the Arri dataset could be more effective for the experiments that focus on large gamut difference. Furthermore, one can obtain an enhanced dataset by supplementing one of the two datasets with particular contents having characteristics desired for the given objective.

The main goal of the scenario is to benchmark performance of GMAs. Each GMA is applied to a set of source image contents having wide gamuts, and its performance is measured by an objective quality metric in terms of perceptual color information loss in the gamut-reduced images in comparison to the original ones. Here, which image dataset is used is an important issue. For instance, if images that do not have color profiles challenging enough to reveal distinguished gamut mapping performance, the GMAs may be evaluated to perform similarly, which may not be the case if challenging images are included. Therefore, careful selection of the images is required to obtain unbiased benchmarking results, for which the proposed framework can be used. Therefore, our objective is to evaluate the reliability of the benchmarking results between different source content selection methods.

We limit the number of GMAs for comparison to two in order to validate the effectiveness of the proposed framework clearly rather than to present extensive benchmarking of many GMAs. One is the state-of-the-art gamut reduction algorithm [7] that adaptively modifies local contrast of pixels residing outside of the target gamut based on the Retinex theory [8]. For the other one, we use the gamut compression algorithm [6] that maps the entire color of the source image inside the target gamut in the CIE 1931 space.

To evaluate the performance of gamut mapping, we use the color image difference (CID) [35] that predicts perceptual color difference between the reference and gamut-reduced image, which is used to evaluate performance of the gamut reduction algorithm in [7]. As the main objective of conducting the scenario, we focus on the reliability and robustness of test results with representative contents selected by our framework. First, the selected dataset should sufficiently cover diverse gamut characteristics so that it is representative. Second, in terms of robustness, experiments with content selection followed by the same procedure should produce consistent results and conclusions regardless of repetition.

The pool of candidate source images consists of half-HD ($960\times 1080$ pixels) WCG images from both the HdM and Arri datasets. After excluding images containing no or too few pixels in WCG (outside the Rec.709 gamut) from the data used in Section IV-B, 35 candidate images are used. The reference gamut is Rec.2020, and we use three target gamuts for gamut mapping: P3, Rec.709, and Toy.

The proposed framework is applied to select representative images from the pool. As described in Section III-C, each candidate image is represented by a two-dimensional perceptual feature vector. Then, the $k$-means clustering algorithm with $k=3$ is used to group them into three clusters, from each of which three images are randomly selected. For comparison, content selection using an existing content feature, colorfulness [53], is also conducted. It measures the variety and intensity of colors in an image. The colorfulness features computed for the candidate images are also clustered into three groups and three images are randomly chosen from each group. These content selection procedures are repeated 100 times with different random seeds.

In order to compare the two GMAs, we define CID gain $g_{t}$ for target gamut $t$ and for a source image as

where $I_{0}$ is the reference image, and $\textrm{GC}(I_{0},t)$ and $\textrm{GR}(I_{0},t)$ are the gamut-compressed and gamut-reduced versions of $I_{0}$, respectively. $g_{t}$ becomes positive when the gamut reduction algorithm performs better than the gamut compression algorithm, and its absolute value indicates the degree of the performance difference.

Using the CID gains for 100 repetitions, the two content selection methods are compared with respect to two aspects: robustness and representativeness. First, a content selection method is considered to be robust when the CID gains remain consistent, i.e., the averages and standard deviations of the CID gains over the selected images are similar across the repetitions. Second, a dataset of images chosen by a content selection method is regarded as being representative if the images have diverse color characteristics. Thus, the CID gains lie in a wide range, resulting in a large average and standard deviation over the images.

Figure 6 shows the average and standard deviation of CID gains for the selected images with respect to the target gamut and selection method. In all cases, the average CID gains are positive, which indicates that the gamut reduction algorithm produces gamut-reduced images with smaller difference from the reference ones compared to the gamut compression algorithm. When the three target gamuts are compared, a smaller gamut yields larger CID gains because more color distortion is introduced by the gamut compression algorithm than the gamut reduction algorithm as the gamut difference becomes larger.

The two selection methods show clearly distinct results. First, the average and standard deviation of the CID gains appear more similar across 100 trials when the proposed framework is used, particularly when the target gamut is small. In order to statistically assess this, we conduct one-sided F-tests under the null hypothesis that the two populations (one for the proposed framework and the other for the method using colorfulness) of the average (or standard deviation) values of the CID gains have the same variance. The results are shown in TABLE III, which confirms that the cases involving large gamut changes show statistically significant difference (i.e., Rec.709 and Toy for the average and Toy for the standard deviation). Note that for P3, the gamut difference from Rec.2020 is small, so the average and standard deviation of the CID gains are also small. These results demonstrate that the selection method has an impact on the results of GMA comparison, where content selection using the proposed framework provides improved robustness.

Second, on average, the average and standard deviation values are larger for the case using the proposed framework than for the case using colorfulness. Since many images in the pool are not challenging for GMAs as shown in Section IV-B, for which the CID gain is small, a larger average or standard deviation value indicates a more representative dataset. We perform one-sided t-tests under the null hypothesis that the two populations of the average (or standard deviation) values of the CID gains have the same mean. As shown in TABLE III, the null hypothesis is rejected in all cases, indicating that the average and standard deviation values are significantly larger for our method. This confirms representativeness of the dataset obtained using our method and, consequently, reliability of the results of the benchmarking.

For comparison, we provide further results using selection features other than colorfulness. We use two no-reference color quality metrics: contrast enhancement based contrast-changed image quality measure (CEIQ) [59] and accelerated screen image quality evaluator (ASIQE) [60]. The former is a metric based on a learned support vector machine using multiple features estimating contrast distortion, while the latter assesses image quality considering four types of quality features consisting of picture complexity, screen content statistics, global brightness quality, and sharpness of details. We conduct statistical tests comparing the CID gains obtained by our framework and the method using either CEIQ or ASIQE. The results are shown in TABLE IV. Similar to the results in TABLE III using colorfulness, statistical significance is observed for F-tests in the cases of the large gamut change (i.e. between Rec.709 and Toy) and for t-tests in all cases. Thus, our framework can effectively select representative contents reliably compared to the methods using these image quality metrics.