Blind Omnidirectional Image Quality Assessment: Integrating Local Statistics and Global Semantics

The rapid recent advancement in virtual reality (VR) technologies makes it possible to create immersive multimedia quality-of-experience (QoE) for end-users. As a representative form of VR, omnidirectional content has increasingly emerged in our daily life. To evaluate and optimize the perceptual QoE of omnidirectional content, objective omnidirectional image quality assessment (OIQA) models play a critical roles in the development of modern VR systems.

In the literature, objective OIQA models have emerged that follow both full-reference (FR) and no-reference (NR) frameworks. FR-OIQA models assume full access to information of the reference image and are usually direct extensions of traditional FR methods developed for regular rectangular 2D image quality assessment (IQA). For example, based upon the peak signal-to-noise ratio (PSNR), Yu et al. [1] propose the spherical PSNR (S-PSNR) algorithm, where PSNR is calculated for uniformly distributed points on a sphere instead of projected rectangular image. In [2], the weighted-to-spherically uniform PSNR (WS-PSNR) method is presented, where a weighting map is created by considering the stretched degree. Zakharchenko et al. [3] propose the Craster parabolic projection PSNR (CPP-PSNR) approach, which maps the reference and distorted omnidirectional images on the Craster parabolic projection followed by PSNR computation.

NR-OIQA methods do not require access to the reference image and are more desirable in many application scenarios. Existing NR-OIQA approaches can generally be classified into two categories, depending on whether the conventional hand-crafted or learned deep features are employed for quality prediction. Multi-frequency information and local-global naturalness are applied to develop the MFILGN model [4]. More recent models employ deep convolutional neural networks (CNNs) or graph convolution networks (GCNs). These models demonstrate promising performance, including the multi-channel CNN for blind 360-degree image quality assessment (MC360IQA) [5], the viewport oriented graph convolution network (VGCN) [6], and its variant named adaptive hypergraph convolutional network (AHGCN) [7].

In a 360-degree viewing environment, e.g. using a head-mounted device, the observer is not able to visualize the whole omnidirectional content simultaneously, and thus an important step in the human subjective viewing experience is to establish or reconstruct a sense of the global semantics by browsing and integrating information from many viewports. During the course of image quality assessment, such global semantics are integrated with local observations on image fidelity, naturalness, and/or artifacts to produce an overall quality evaluation. Motivated by this observation, we propose a statistic and semantic oriented quality prediction framework named S² for blind OIQA as illustrated in Fig. 1, by integrating features extracted from both low-level image statistics of multiple local viewports and high-level semantics of the hallucinated global omnidirectional image. A quality regression module is then leveraged to map the collection of the quality-sensitive features extracted from the two separate paths to an overall prediction of the subjective quality rating. Extensive experimental results demonstrate that the proposed method is superior to many state-of-the-art quality assessment models. In addition, we make some interesting observations on the relationship between semantic confidence and image distortions, as well as how the individual components affect the ultimate quality prediction performance in ablation studies.

The overall framework of the proposed S² method is shown in Fig. 2, which consists of a statistic path, a semantic path, and a final quality regression step.

Since a variety of viewports are browsed by the viewers, we first convert the distorted omnidirectional image (OI) to multiple viewports. Given each input distorted OI denoted by $D$, we exploit the non-uniform viewport sampling strategy [8, 9] and obtain $N$ viewports $V_{n},n=1,2,...,N$.

To capture the multi-scale characteristics of the human visual system [10], we construct pyramid representations [11, 12] of multiple local viewports. Specifically, multi-level Laplacian pyramids [13] are created by iterative Gaussian filtering, down-sampling, and subtracting, resulting in Gaussian and Laplacian pyramids in the same process. For a specific viewport $V_{n}$, layers of the Gaussian pyramid are calculated as follows:

$$\small G_{n}^{i}(x,y)=\left\{\begin{array}[]{l}V_{n},i=1\\ \sum\limits_{u=-2}^{2}{\sum\limits_{v=-2}^{2}k(u,v)G_{n}^{i-1}(2x+u,2y+v)},i>1\end{array}\right.,$$

(1)

where $i$ is the layer index of the Gaussian pyramid, $x\in[0,X)$ and $y\in[0,Y)$ are the pixel position indices in which $X$ and $Y$ are the image dimensions, and $k(u,v)$ denotes the generating kernel that is typically defined by the coefficients of a low pass filter such as a 2D Gaussian filter.

We then interpolate each layer of the Gaussian pyramid by:

$$\small\hat{G}_{n}^{i}(x,y)=4\sum_{u=-2}^{2}\sum_{v=-2}^{2}k(u,v)G_{n}^{i}\left(\frac{u+x}{2},\frac{v+y}{2}\right).$$

(2)

The residual between the current layer of the Gaussian pyramid and the interpolation result from the next layer defines the current layer of the Laplacian pyramid:

$$\small L_{n}^{i}=G_{n}^{i}-\hat{G}_{n}^{i+1}.$$

(3)

Since the computation of the $i$-th layer in the Laplacian pyramid requires the $(i+1)$-th layer of the Gaussian pyramid, the number of layers in the Laplacian pyramid is one less than that in the Gaussian pyramid.

To extract features from the Gaussian pyramid, we compute the default uniform local binary pattern (LBP) descriptors, resulting in 59 statistics for each Gaussian layer. When a 3-layer Gaussian pyramid is employed, this leads to 177 Gaussian pyramid features denoted by $f_{GP}$. For a Laplacian pyramid, motivated by the success of natural scene statistics (NSS) in IQA research [14, 15, 16], we extract mean subtracted and contrast normalized coefficients, leading to 36 features for each layer. When a 2-layer Laplacian pyramid is employed, this results in 72 Laplacian pyramid features denoted by $f_{LP}$. The full statistic feature set $f_{st}$, one for each viewport, is obtained by concatenating the statistical features extracted from the Gaussian and Laplacian pyramids as:

$$\small f_{st}=\left[f_{GP},f_{LP}\right].$$

(4)

We employ the VGGNet trained on the large ImageNet dataset [17] as the semantic feature extraction backbone, mainly for its simplicity and ability to capture image distortion-related representations [18]. In [19], three different structures of VGGNet have been proposed to balance between complexity and accuracy, namely fast VGG (VGG-F), medium VGG (VGG-M), and slow VGG (VGG-S). Each of them contains 5 convolutional (Conv) layers and 3 fully connected (FC) layers. The first two FC layers have 4,096 neurons, while the last one has 1,000 nodes indicating the 1,000 classes for image recognition. In our current implementation, we select the deep features from the first FC layer of VGG-M as our semantic feature set $f_{se}$:

$$\small f_{se}=FC_{1}(D).$$

(5)

To learn the mapping from features to quality labels, we feed the statistic features and semantic features separately to support vector regression (SVR) models [20], and denote the regressed statistic and semantic quality scores as $Q_{st}$ and $Q_{se}$, respectively. The overall quality score is calculated by a weighted average:

$$\small Q_{\text{overall }}=wQ_{st}+(1-w)Q_{\text{se }},$$

(6)

where $w$ is a weighting factor that determines the relative importance of the statistic and semantic feature predictors.

We propose a novel S² framework for blind omnidirectional image quality assessment that integrates both local low-level statistic and global high-level semantic features. Extensive experiments show that the proposed method achieves state-of-the-art performance. Observations on the relationship between semantic confidence and image distortion, and the ablation/sensitivity tests offer additonal useful insights. Under the same framework, more advanced models for statistic and semantic analysis may be employed in the future, aiming for more accurate QoE assessment models that may help drive the advancement of immersive multimedia systems.