THE FIRST COMPREHENSIVE DATASET WITH MULTIPLE DISTORTION TYPES FOR VISUAL JUST-NOTICEABLE DIFFERENCES

To select JND images from these datasets, we use the MOS of distorted images as the preliminary selection criterion, since the MOS is a subjective viewing score, reflecting the perceptual quality of distorted images among all the subjects. Commonly, the higher the distortion image quality, the smaller the difference between it and its associated source image, and vice versa. Therefore, we can use MOS to select distorted images with high perceptual quality as the JND maps, so as to reduce the amount of data in subsequent subjective tests. The involved IQA datasets in this work include TID2013 [13], TID2008 [12], and KADID-10k [10], which have a total of 14,825 distortion images from 106 source images, and 39 distortion types, each with 4 or 5 distortion levels.

Since different IQA datasets use different algorithms to subjectively evaluate distortion images, the range of the obtained MOS values is different. In view of this, we first normalize the MOS values. This process can be formulated as

$$Q=\frac{P-Min}{Max-Min},$$

(1)

where $P$ represents the MOS value of the distortion image in a certain IQA dataset, $Min$ and $Max$ are the minimum and maximum values of the MOS in the current IQA dataset. $Q$ represents the normalized MOS. The range of the normalized MOS is [0,1]. Fig. 2 shows the MOS distribution of each IQA dataset after normalization, and it can be seen that the MOS distribution in different datasets is different. Among them, the MOS distribution of TID2013 and TID2008 conforms to the normal distribution, while the KADID-10k dataset is relatively evenly distributed in each interval. To determine a threshold that can distinguish between JND and non-JND maps, we randomly sample 10%-20% of the source images and all their corresponding distorted images from each IQA dataset and then pick out JND maps from them through a subjective viewing test by well-trained and experienced subjects. We only sample 10%-20% of the whole images because the number of images is huge and we don’t need to ensure the high accuracy of the threshold due to the fine JND map selection after. As a result, 112 JND maps are confirmed out of 2,190 distorted images with MOS above 0.7. To determine the MOS threshold, we calculate the average MOS of the confirmed JND images for each image dataset and then reduce the calculated average MOS by about 5% so that more JND-eligible maps can be involved. Then, we obtain the MOS thresholds 0.855 for the KADID-10k dataset, and 0.795 for the TID2013 and TID2008 datasets, respectively. Afterward, we use the MOS thresholds above to pick out images from their corresponding datasets as the JND candidates. In this way, we coarsely obtain 1,962 JND candidates.

After coarse JND selection, there are still some JND candidates that can be noticed the differences compared with their associated source images. In view of this, we conduct a crowdsourced subjective assessment of the JND candidates to achieve fine JND selection by abandoning non-JND maps. The crowdsourced subjective assessment was conducted on Amazon Mechanical Turk (AMT). On the AMT platform, requesters—companies, organizations, or individuals—create and submit tasks requiring human intelligence for workers, who may be paid for each task successfully completed.

In this work, we adopt the flicker test as used in [14], where the JND candidate and its associated source image are toggled back and forth successively at a frequency of 8 Hz. Here, we set three options for subjects: obvious flicker, slight flicker, and no flicker. The definitions of the three options are as follows. The obvious flicker indicates that the flicker is intense, affecting the entire image. The slight flicker means that slight changes can be seen in the image, such as a subtle variation in the image brightness. These changes can resemble a mild film grain in a static scene of a movie and are often localized to small regions of the image. No flicker represents the image is static, there is no visible change in the image, no matter how small it is.

According to the definition of JND as aforementioned in Sec. 1, no flicker and slight flicker JND candidates are selected as the final JND maps. Here, we randomly divide all the 1962 JND candidates into 103 groups. Each group contains 19 or 20 JND candidates, which are assessed by 30 workers (subjects). A total of $1962\times 30$ results are obtained.

Considering that the screens used by the workers may have different sizes and resolutions, resulting in different physical sizes of pictures displayed on the screen, we carry out the calibration process in [9] to make the image appear the same physical size on all workers’ computers. That is, workers must prepare a card the size of a credit card ($85mm\times 53.98mm$) and change the size of a frame on the screen to fit the card. By computing the Logical Pixel Density (LPD) of the display in Pixels Per Inch (PPI), we can estimate the physical size of the display. For more details refer to [9]. Following the calibration, we instruct the workers to change their viewing distance to $30cm$ in accordance with the trigonometric calculation [15], [16] and ISO standard [17].

After passing a training phase, the workers will enter the assessment phase. In the assessment phase, 19 or 20 images are set for each task, and the workers choose the most suitable option from the three options of no flicker, slight flicker, and obvious flicker, as aforementioned.

To get effective results, we need to remove outliers in the collected results. Since we can not guarantee that all workers complete the test with seriousness, for example, some workers may randomly choose an option during the test, or workers may submit unreliable answers because they have been tested for a long time. In this paper, we adopt the result processing method mentioned in the ITU-R Recommendation BT.500 [18] to remove unreliable results. We use the scores 1, 2, and 3 to represent no flicker, slight flicker, and obvious flicker respectively. Then, we calculate the mean and standard deviation of the scores rated by all the subjects for each image as follows

$$\bar{u}_{j}=\frac{1}{N}\sum_{i=1}^{N}{u_{ij}},$$

(2)

$$S_{j}=\sqrt{\sum_{i=1}^{N}{\frac{\left(\bar{u}_{j}-u_{ij}\right)^{2}}{\left(N-1\right)}}}.$$

(3)

$u_{ij}$ denotes the score rated by the $i$-th subject on the $j$-th image, where $i=1,2,...,N$ and $j=1,2,...,M$. $\bar{u}_{j}$ and $S_{j}$ are the mean and standard deviation of scores rated by all the subjects for the $j$-th image. Subsequently, we adopt the confidence interval in [18] for outlier removal, that is

$$\left[\bar{u}_{j}-\delta_{j},\bar{u}_{j}+\delta_{j}\right],\text{where}\ \delta_{j}=1.96\frac{S_{j}}{\sqrt{N}}.$$

(4)

$[\cdot]$ is rounding operation. $u_{ij}$ outside such an interval is considered as an outlier. After removing the outliers, we re-calculate the average score of each image. We regard the image with an average score of less than 2.5 as a JND map.

As a result, we select a total of 1,642 distortion images (corresponding to 106 source images) as the JND maps, which contain 25 distortion types. Table 1 shows the 25 distortion types, their number of JND maps, and the ratio of the number of JND maps to the total number of samples. From the table, we can see that the number of JND maps with different distortion types varies greatly. There are two reasons to explain this. First, since we select JND maps from three IQA datasets KADID-10k, TID2008 and TID2013, some distortion types are not contained by all three datasets such as Comfort noise and Sparse sampling and reconstruction, which causes different numbers of samples for different distortion types. The distortion type with a small number of samples probably has fewer JND maps and vice versa. Second, the HVS has different sensibilities to different types of distortion [6]. For the distortion type with a high sensibility of the HVS, even low-level distortion may be perceived by the HVS and this will make fewer JND maps can be obtained from the IQA datasets, and vice versa. Therefore, the distortion with a higher sensibility to the HVS, the fewer JND maps we obtained from the samples with such type of distortion. Besides, for each distortion type, we also calculate the ratio of the number of JND maps and total samples. The difference in such ratios reflects a certain extent to which types of distortion are more easily sensitive to the HVS.

A good JND dataset tends to contain a wide variety of visual content. To demonstrate the diversity of the JND maps in our dataset, we calculated the spatial information (SI) [19] and the colorfulness (CF) [20] of 106 source images, as shown in Fig. 3. The spatial information reflects the spatial complexity of the image, and the colorfulness reflects the richness of the color of the image. In Fig. 3, we can see that the selected source images do have wide coverage in this plot. Furthermore, we divide the 106 source images into 8 semantic categories, as shown in Table 2.