Towards Top-Down Just Noticeable Difference Estimation of Natural Images

KLT is a signal dependent linear transform, the kernels of which are derived by computing the principal components along eigen-directions of the autocorrelation matrix of the input data. Given an image $\mathbf{X}$ with size $M\times N$, a set of non-overlapping patches with size $\sqrt{K}\times\sqrt{K}$ are extracted. These image patches are vectorized and combined together to form a new matrix $\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{S}]\in\mathbb{R}^{K\times S}$, where $\mathbf{x}_{s}\in\mathbb{R}^{K\times 1},s=1,2,\cdots,S$ represents the $s$-th vectorized patch and $S$ is the total number of image patches in $\mathbf{X}$. The covariance matrix of $\mathbf{X}$ is defined as follows

where $\mathbf{\bar{x}}=\frac{1}{S}\sum_{s=1}^{S}\mathbf{x}_{s}$ denotes the mean vector obtained by averaging each row of $\mathbf{X}$ and $\mathbf{C}\in\mathbb{R}^{K\times K}$. Then, the eigenvalues and eigenvectors of $\mathbf{C}$ are calculated via eigenvalue decomposition. The eigenvectors are arranged according to their corresponding eigenvalues in the descending order to form the KLT kernel $\mathbf{P}=[\mathbf{p}_{1},\mathbf{p}_{2},\cdots,\mathbf{p}_{K}]\in\mathbb{R}^{K\times K}$ where $\mathbf{p}_{k}\in\mathbb{R}^{K\times 1},\;k=1,2,\cdots,K$ represents the $k$-th eigenvector. Using the KLT kernel $\mathbf{P}$, the KLT of $\mathbf{X}$ is expressed as follows:

where $\mathbf{Y}=[\mathbf{y}_{1},\mathbf{y}_{2},\cdots,\mathbf{y}_{K}]^{\mathrm{T}}\in\mathbb{R}^{K\times S}$ is the KLT coefficient matrix and $\mathbf{y}_{k}\in\mathbb{R}^{S\times 1},\;k=1,2,\cdots,K$ refers to the $k$-th spectral component obtained by $\mathbf{y}_{k}=\left({\mathbf{p}_{k}}\right)^{\mathrm{T}}\mathbf{X}$.

After obtaining the KLT coefficient matrix $\mathbf{Y}$, we calculate the KLT coefficient energy $E_{k}$ for spectral component $y_{k}$ as follows:

In order to remove the influence of image content, we further calculate the normalized KLT coefficient energy $p_{k}$ as follows:

Fig. 3(a) shows the normalized KLT coefficient energy distribution curve for the image in Fig. 2(a). Since $p_{1}$ is particularly large than others, we further plot another curve in Fig. 3(b) by excluding $p_{1}$. As shown in Fig. 3(b), as the spectral component index $k$ increases, the KLT coefficient energy $p_{k}$ first drops dramatically and later converges to be stable. When $k$ is larger than 20, $p_{k}$ becomes extremely small and gradually converges to zero. This phenomenon is consistent with the inverse KLT-based image reconstruction process illustrated in Fig. 2. The former spectral components occupy much larger energies than the later ones so that they take charge of the macro-structure image information. The latter spectral components own relatively small energies such that they are in charge of the micro-structure image information.

Now, let us take a look at the normalized KLT coefficient energy curve from an cumulative perspective. The cumulative normalized KLT coefficient energy $P_{k}$ is obtained as follows:

The cumulative distribution curve of the normalized KLT coefficient energy for the image in Fig. 2(a) has been shown in Fig. 3(c). Note that $P_{1}=p_{1}$ and has been already close to 1 due to the particularly large energy of the first spectral component. As $k$ increases, $P_{k}$ monotonously increases. When $k>20$, $P_{k}$ will gradually converges to 1, indicating the recovered visual information gradually become saturated.

For images with different contents, although their cumulative distribution curves have the same convergence tendency, their critical points that lead to sufficient visual information may be different, as demonstrated in Fig. 4. Therefore, it is required to design an adaptive scheme to automatically determine the critical point for different images. In the following, we will illustrate how this issue is addressed with subjective studies and statistical analyses.

We select a total number of 500 high visual quality images from the DIV2K [41, 42] dataset. For each image, we apply the KLT-based image transform and inverse KLT-based image reconstruction with the first $k$ spectral components, $k=1,2,\cdots,K$. Thus, we can obtain $K$ reconstructed images for each original image. Then, we conduct a user study to subjectively determine the critical point $L\in\{1,2,\cdots,K\}$ for each original image. Details of the user study are illustrated as follows. There are 60 participants (38 males and 22 females) in our subjective experiment. Each participant $s^{m}$ is asked to compare each original image $I_{o}$ and its corresponding $K$ reconstructed versions $\{I_{r}^{k}\}$ one-by-one. During the subjective experiments, the image pairs are presented in a pre-defined order: $\{I_{o},I_{r}^{1}\}$,$\{I_{o},I_{r}^{2}\}$,$\cdots$,$\{I_{o},I_{r}^{K}\}$. The image pairs are displayed on a DELL U2419HS monitor without resizing. The display is set to the sRGB color profile, the size of display is 23.8 inches (593.5mm$\times$353.5mm), the resolution is 1920$\times$1080, the peak luminance is set to 220$\rm cd/m^{2}$, the contrast is 1000:1, the viewing distance is 0.45m, and the pixels per visual degree (ppd) is 29.6266, approximately 30. All the subjective experiments are conducted in a dark room with dimmed lights. Each participant observes the presented image pairs carefully and conduct a binary judgment to answer the question whether the presented image pair has visible difference or not. Typically, the participants will observe obvious difference for the first several pairs and then gradually have difficulties in differentiating the later ones. For each participant, the user study will stop once he/she cannot observe any visible difference for a certain image pair $\{I_{o},I_{r}^{k}\}$ and we cosider this spectral component index $k$ as the critical point given by the $m$-th participant $s^{m}$ for the $n$-th original image $I_{o}^{n}$: $L_{n}^{m}=k$. The experiment then continues untill each participant have finished determining the critical points for all 500 images. To understand the distribution of subjective votes, we also take the four images shown in Fig. 4 as examples and plot their vote distributions from all participants in Fig. 5. We can see that the votes of critical points from all participants for each individual image approximate a Gaussian distribution.

Let us denote the critical points given by all participants for the $n$-th original image as ${\mathbf{L}}_{n}=[L_{n}^{1},L_{n}^{2},\cdots,L_{n}^{60}]$. For Gaussian-like distribution, we remove the outlier data points from ${\mathbf{L}}_{n}$ according to the well-known 3-$\sigma$ criterion which assumes that a set of test data only contains random errors, calculate it to obtain standard deviation, and determine a range according to a certain probability of $99.7\%$ [43]. It is considered that the error exceeds this interval is not a random error. That is, a specific element $L_{n}^{m}$ will be excluded if it does not satisfy ${\mu}_{{\mathbf{L}}_{n}}-3{\sigma}_{{\mathbf{L}}_{n}}\leq L_{n}^{m}\leq{\mu}_{{\mathbf{L}}_{n}}+3{\sigma}_{{\mathbf{L}}_{n}}$ where ${\mu}_{{\mathbf{L}}_{n}}$ and ${\sigma}_{{\mathbf{L}}_{n}}$ denote the mean value and standard deviation of ${\mathbf{L}}_{n}$, respectively. The identified outliers have been marked in orange. After outlier removal, ${\mathbf{L}}_{n}$ will be updated to be ${\mathbf{L}}_{n}^{{}^{\prime}}$. We further calculate the mean value ${\mu}_{{\mathbf{L}}_{n}^{{}^{\prime}}}$ and set the final critical point for the $n$-th original image as follows:

where $\lceil\cdot\rceil$ represents the round up operation. A detailed data analysis on vote distribution of different images is summarized in Table I.

In order to understand whether the vote distributions for all 500 images follow the normal distribution or not, we further conduct a normality test. Here, the Shapiro-Wilk test (i.e., SW test) [44] is adopted. The significance level is set to $\alpha$=0.05 [44]. If the P-value is less then $\alpha$, it means the sample data is significantly different from the normal distribution. Fig. 6 shows the histgram of all 500 P-values. It is observed that only 13 out of all P-values were less than 0.05, only accounting for 2.6$\%$. This demonstrates that the vote distributions of most images (97.4$\%$) conform to normal distribution. Thus, our strategy using the 3-$\sigma$ criterion to filter the outliers is rational and appropriate.

Next, we calculate the corresponding cumulative KLT coefficient energy $P_{L_{n}}$ for each original image with the obtained critical points $L_{n}\in\{1,2,\cdots,K\}$. As a result, we can obtain a vector ${\mathbf{P}}_{L}=[P_{L_{1}},P_{L_{2}},\cdots,P_{L_{500}}]$ with each element representing the critical cumulative KLT coefficient energy of a specific original image. The calculated cumulative KLT coefficient energy values of the same four images are given in the last column in Table I. The distribution of cumulative normalized KLT coefficient energy over all 500 original images is also presented in Fig. 7. It is found that, although the critical points vary with different image contents, their corresponding cumulative KLT coefficient energy values tend to highly concentrate on the range of [0.99,1]. We can see from the figure that the distribution is right-skewed and can be well approximated by a Weibull distribution [45]. The Weibull distribution is a continuous probability distribution typically used for fitting both left- and right-skewed data. The widely used two-parameter Weibull distribution is expressed as follows:

where $\beta$ and $\eta$ are the shape parameter and scale parameter, respectively. The above distribution can be well fitted by the Weibull distribution with $\beta=894.16$ and $\eta=0.998$. The fitted Weibull distribution curve (the red curve in Fig. 7) can be considered as a statistical prior which directly shows the probability of each $k$ (the index of spectral component) to be determined as the critical point.

Given a test image, the expected value of $k$ is calculated to derive its corresponding critical point where the probabilities are determined by the fitted Weibull distribution function:

where $\lceil\cdot\rceil$ represents the round up operation.

When we take the first $L$ spectral components for image reconstruction via inverse KLT, it is expected to generate the CPL counterpart of the original image. How the CPL image can be reconstructed with the first $L$ spectral components will be detailed in the following subsection.

To utilize the first $L$ spectral components for image reconstruction via inverse KLT, we first define the corresponding reconstruction KLT coefficient matrix $\mathbf{Y}^{(L)}$ as follows:

where $\mathbf{Y}^{(L)}\in\mathbb{R}^{K\times S},\;k=1,2,\cdots,K$. Then, the image is reconstructed as follows:

where $\mathbf{X}^{(L)}$ represents the reconstructed image by only considering the first $k$ spectral components. Since $L$ is the estimated critical point, $\mathbf{X}^{(L)}$ can be considered as the CPL image. Fig. 8 illustrates the progressive image reconstruction process. In our experiment, we set $K=64$. The images shown in the top row are the reconstructed results using only each individual single spectral component while the images shown in the bottom row are the reconstructed results using all previous spectral components. It is obvious that only using the first spectral component for reconstruction could recover most macro-structures of the original image. When more spectral components are involved, the image is progressively reconstructed with richer and finer details. Suppose $\tilde{L}$ is the boundary between macro- and micro-structures. Using the former $\tilde{L}$ spectral components will successfully reconstruct all the macro-structures and the latter $(64-\tilde{L})$ spectral components will mainly responsible for the reconstruction of micro-structures. For the critical point $L$ we have estimated, it means that we at least need to use the former $L$ spectral components to reconstruct all the macro-structures and sufficient micro-strucutres required for a perceptual lossless image. Finally, if all the spectral components are involved, the original image can be completely reconstructed.

In Fig. 9, we present our predicted CPL images of the four original images. As we can see, the CPL images (second row) are quite similar to the original ones, without obvious visible distortions. We also use the HDR-VDP2.2 metric [46] to calculate the distortion visibility maps, as shown in the third row. It is observed that most regions in the visibility maps are blue, implying that it is hard to perceive distortions from these CPL images. These results show that our determined CPL images are close to their original counterparts from the perspective of human perception.

Once the CPL image ${\mathbf{X}}^{(L)}$ is obtained, we compute the difference map between the original image $\mathbf{X}$ and ${\mathbf{X}}^{(L)}$ as the final JND map $\mathbf{M}$:

where $\lvert\cdot\rvert$ denotes the absolute value operator, $(i,j)$ are the pixel coordinates in spatial domain.

Apparently, with the guidance of a more accurate JND map, the JND-contaminated image (obtained by Eq. (13)) with same noise level should have better visual quality. As stated, for different JND models, we can adjust the value of $\theta$ to ensure that almost the same amount of noise is injected. Then, the performances of differenta JND model can be compared objectively and subjectively. Specifically, objective evaluation is performed based on the HDR-VDP2.2 metric [46] which is known as a popular distortion visibility detection metric for both high-dynamic range and standard images. A higher HDR-VDP score indicates less visible distortions. Subjective evaluation is performed by humans to obtain MOS score. Another 30 participants are invited to participate the subjective experiments. Each participant is asked to assign an opinion score in the range [-1,0] to a specific image pair including one original image and one JND-contaminated image. A score equals to 0 means the visual quality of the JND-contaminated image is equal to that of the original image while a score equal to -1 means the visual quality of the JND contaminated image is dramatically worse than that of the original image. As a result, for each contaminated image, 30 scores are collected from all participants. After removing the outlier, the mean value of the retained opinion scores is calculated as the MOS. Overall, a more accurate JND model will result in higher HDR-VDP score and MOS than the competitors.

We select another 20 high-quality images from the DIV2K [41, 42] dataset for testing. These images are shown in Fig. 11. The proposed JND model is compared with seven existing JND models including Yang2005 [21], Zhang2005 [47], Wu2013 [23], Wu2017 [24], Jakhetiya2018 [25], Chen2020 [26], and Shen2021 [27]. Table III provides the performance results of different JND models in terms of HDR-VDP score and MOS at the noise level of PSNR=26dB (comparisons under other different PSNR settings are also conducted and will be reported later in this section). It can be seen that our proposed JND model delivers higher HDR-VDP scores on 11 out of 20 test images and the highest HDR-VDP score in average when considering all 20 images. Followed by our proposed JND model, Wu2017 and Shen2021 take the first place for 5 and 4 times, respectively. In terms of MOS, we achieve the highest MOS values for almost all the images except the image ‘I12’ on which Chen2020 is the best. These results demonstrate the superiority of our proposed JND model against others in allocating noise at the noise level of PSNR=26dB (comparisons under other different PSNR settings are also conducted and will be reported later in this section). Note that we inject noise to the original images at the noise level of PSNR=26dB to obtain the JND-guided noise contaminated images. Generally, 26dB is a relatively large noise intensity which will probably break the transparency, and thus it is inevitable to induce visible distortions on the JND-guided noise contaminated image to some degree.

In order to make a more clear comparison among these JND models, Fig. 12 gives a visual example of the JND maps and the noise contaminated images. Images in the first row are the estimated JND maps. Images in the second row are the JND-guided noise-injected images generated according to Eq. (12). Images in the third row are the enlarged versions of sub-regions cropped from the noise-injected images in the second row. We can observe that the noise-contaminated image guided by our proposed JND model is the ‘cleanest’ one among all the compared ones. It seems that Fig. 12(h) is perceptually the same with the original image as it seems to have no visible noise. Among the competitors, Fig. 12(d) also has achieved satisfactory visual quality while it is still worse than Fig. 12(h). With careful observations, one can still perceive visible noise in the sky area of Fig. 12(d).

By taking a closer look at Fig. 12(h), we find that the injected noises are hidden in those textured areas such as the walls and leaves. However, it is hard to be perceived by the HVS at the first glance. As has been reported in [24], the HVS is highly adapted to extracting the repeated patterns for visual content representation and it is hard to perceive the noise which is injected into the textured areas with high pattern complexity. By contrast, it is much easier for the HVS to perceive the noise in those relatively smooth areas. Compared with the smooth areas, textured areas tend to contain much more details (e.g., micro-structures) and are likely to be redundant. As we have demonstrated before, the difference map between the original image and the derived CPL image can well resemble the redundant micro-structure image information to the HVS. Thus, our proposed JND model using such difference map as the JND map will encourage to hide the noise in those highly redundant areas. Therefore, the injected noises will not be easily visible by the HVS. What’s more, instead of modeling and aggregating the masking effects of diverse contributing factors in isolation, our proposed JND model dedicates to deriving a CPL image to best exploit the potential perceptual redundancies exist in the original image from a top-down perspective. Thus, our proposed JND model would be able to implicitly characterize the influence of more potential factors beyond those have been considered previously. As a result, our proposed JND model can finally generate a perceptually better noise-contaminated image.

Note that comparing the performance under PSNR=26dB only may be unfair to certain JND models. Therefore, we also conduct experiments under more PSNR settings. Specifically, we set PSNR=$\{22,23,24,25,26,27,28,29,30\}$ and for each PSNR setting we generate JND-contaminated images of I01-I20 by using different JND models as guidance. We also apply the HDR-VDP metric to compute the objective score of each JND-contaminated image, and we take the averaged HDR-VDP score as the indicator of JND model performance. Finally, we plot the curve of average HDR-VDP score versus PSNR value, as shown in Fig. 13. We can see that the curve of our proposed JND model is among the top, which indicates our JND model always achieves the highest HDR-VDP score at each PSNR value.

It should be emphasized that, although we have compared different JND models on noise injection with different noise levels, such experiments do not directly reflect the capability of predicting JND. The experimental setups for directly comparing the capability of predicting JND still deserve more rigorous treatments.

The above experiments are conducted to compare the visual quality of different JND-guided noise contaminated images under the same PSNR level. While the results have demonstrated promising performance of our proposed JND model, it is still necessary to understand the maximum tolerable noise level by different JND models. A better JND model should tolerate more noise while keeping the quality of the noise-contaminated image perceptually unchanged. In our experiment, we use PSNR as a measure of the noise level and a lower PSNR value actually indicates a better performance of a JND model. The results are listed in Table IV. As we can see, the proposed JND model can tolerable more noise on 11 out of 20 images and also has the lowest PSNR in average when considering all 20 images, which again validates the superiority of our proposed JND model.