Multi-Scale Estimation for Omni-Directional Saliency Maps Using Learnable Equator Bias

Many saliency models have been proposed to estimate the locations in 2D images which attract attentions of people. The saliency map computed from image features with these models represents the probability density function of fixations when people look at an image. In the earlier studies, the saliency map has been estimated by extracting low-level features. One of these saliency-map estimation models is that proposed by Itti et al. [10], which estimates saliency maps by extracting early visual features such as intensities, colors, and orientations. AWS (Adaptive Whitening Saliency) [11] is another conventional model based on the whitening of low level features.

The recent progress of deep learning has also contributed to the improvement of saliency-map estimation models. Although the conventional models described above have used the hand-crafted features to estimate the saliency maps, the image feature vectors extracted from convolutional neural networks (CNN) have been used instead [12, 13, 14, 15, 16, 17]. The ensembles of deep networks (eDN) [14] have trained small-size CNN (up to 3 layers) to extract multiple-layer features for estimating the saliency maps by combining the features with support vector machines. DeepGaze I [15] has shown that the image features extracted from the multiple layers of CNN trained for an image-classification task on ImageNet [18] are useful to estimate the saliency maps. This implies that the transfer learning from the image classification to the saliency map estimation is effective because people tend to look at the centers of objects [19], which are learned to recognize in the pretrained model for the ImageNet classification task.

In contrast to DeepGaze I which uses AlexNet as the backbone network, SaliconNet [12] and DeepFix [16] uses fully convolutional neural networks based on the VGG architecture which has shown better performance than AlexNet on the image classification task. SaliconNet [12] estimates saliency maps with two image scales to allow the model to be robust against the sizes of objects in images. DeepFix [16] uses dilated convolution filters [20] to enlarge the receptive field and inception modules to capture multi-scale information. The deep-type network of SalNet [13] has also adopted the VGG architecture with simplification composed of 10 layers. DeepGaze II [17] is also based on the VGG architecture, which uses a center-bias layer to incorporate the prior distribution and log-likelihood as the loss function.

In addition to the VGG-based CNN models, several CNN architectures have been used for the saliency-map estimation. For example, SalGAN [21] is a model that estimates saliency maps using generative adversarial networks. DenseSal and DPNSal [8] have shown better performance based on densely connected neural networks (DenseNet) [22] and dual path networks (DPN) [23], respectively. EML-NET [24] combines the feature vectors extracted from DenseNet and NasNet [25]. DeepGazeIIE [26] also combines saliency predictions from multiple saliency models. These methods have achieved the state-of-the-art results on the MIT Saliency Benchmark [6] and MIT/Tuebingen Saliency Benchmark [7].

In addition to the saliency-map estimation for 2D images, the saliency models for ODI have been developed since the ICME2017 Competition [27]. ODI is represented in various projections, among which ERP is commonly used to represent ODI in 2 dimensions as shown in Fig. 1, though it has distortions at poles (the top and bottom regions in ERP). The model by Abreu et al. [2] has used SaliconNet [12] to estimate the ODI saliency map by directly inputting ODI in ERP. The ERP saliency maps for horizontally different viewing directions are fused into an ERP saliency map to suppress the center-bias effect induced by the 2D saliency-map model (SaliconNet). However, the ERP images have distortions at poles, so that saliency at the poles cannot be correctly estimated. Moreover, the center-bias effect cannot be completely suppressed at the edges of the ERP image.

Since the database for ODI saliency maps are relatively small, it would be important to first pre-train the CNN model for 2D images with large databases, and then fine-tune it with the ODI database. In SalNet360 [3], this has been realized by subdividing ODI into 6 undistorted plane patches in cubic projection, estimating the 2D saliency maps with SaliconNet (2D saliency model) [12] and a refinement network, and then integrating the 2D saliency maps into an ODI saliency map in ERP. In order to incorporate the dependence of saliency on the locations in ODI, the ODI spherical coordinates of each pixel in each 2D patch were input to the refinement neural network. However, since SaliconNet outputs a saliency map biased in the center of an image, the saliency at the 2D image edges is estimated lower. SalGAN360 [28] based on SalGAN [21] has estimated the ODI saliency maps by fusing the saliency map in ERP and saliency maps for undistorted plane patches in the cubic projection to integrate the global and local information. This method employs multiple cubic projection, where overlapping cubic faces are extracted as in our proposed method. However, the angles of view are limited to 90^∘ in the multiple cubic projection, and biases to represent prior distributions are not considered. Qing et al. [4] also have proposed a method to train a model with multiple cubic projection and to estimate saliency maps using ERP with multiple sphere rotation to reduce the influence of distortion at poles. Although global information can be utilized from ERP in the inference, the angle of view in the training stage is limited to 90^∘ and the equator bias is not explicitly considered.

As shown in Fig. 1, the center bias represents the property of tendency for fixations to concentrate on the center of a 2D image while the equator bias represents that to concentrate on the equator of ODI. The average of saliency maps over 2D images and ODI in databases are shown in Fig. 4 (a) and (b), respectively. They represent the prior distributions of fixations, independent on the local image features. Thus, the prior distributions are different between 2D images and ODI, so that these differences have to be compensated in the ODI saliency-map estimation.

The saliency is defined as a probability of observing a fixation ($o$: binary random variable) at a location $x$ for a local image feature $f$: $p(o|f,x)$. If $x$ and $f$ are assumed to be independent,

where $p(o|f)$ represents the saliency map depending only on the local image feature $f$ without the prior probability $p(o|x)$, and $p(o)$ is a normalization constant. Thus, the saliency map can be modeled by the multiplication of $p(o|f)$ and $p(o|x)$ with normalization. In the proposed ODI saliency model, a 2D saliency-map estimation model, DenseSal [8], was used with an additional layer (bias layer) for learning the prior probability $p(o|x)$ in 2D saliency maps, as shown in Fig. 5. This layer consists of pixel-wise scaling weights for a feature map from the 2D saliency model. By learning the weights with 2D saliency datasets, the saliency depending only on a location $x$ in an image can be estimated in the center-bias layer. When estimating ODI saliency maps, this center-bias layer is replaced by an equator-bias layer, which consists of pixel-wise scaling weights in multiple channels corresponding to elevation angles of viewing directions for 2D patches.

In the 2D saliency model, the saliency at image boundaries are underestimated by the center-bias effect even if there are some objects at the edges of an image [29, 30]. In the extracted 2D images from ODI, objects can appear at the edge of the extracted images, which may be focused by people with HMD. Therefore, the center bias can give the negative effect to the saliency estimation for ODI. In our approach, first the 2D saliency model was trained with center-bias layer to model the center-bias effect existing in the 2D saliency dataset similar to [29], so that the MainNet (excluding the center-bias layer) represents only the saliency for contents (such as objects) not depending on the position of the image. By replacing the center-bias layer to the equator-bias layer, and by learning the bias layer using the training data extracted from ODI, the bias layer expresses the equator-bias effect (saliency depending on the elevation angle in the spherical coordinates, and not depending on the horizontal angle), so that the negative effect of the center bias can be eliminated.

Since ODI in ERP has distortion at poles as shown in Fig. 1, 2D images are extracted from ODI to estimate an ODI saliency map by removing the distortion. The correspondence between ERP and the spherical coordinate system is shown in Fig. 6. The unit vectors of a 2D image extracted at a viewing direction ($\theta_{c},\phi_{c}$) in spherical coordinates are represented in the following equations in the 3D Euclidean coordinate system:

The ERP coordinates of the 2D image in a tangent plane can be obtained by transforming the 3D Euclidean coordinates of points in the 2D image to spherical coordinates. Thus, 2D images can be extracted from ODI in ERP using the coordinates. When 2D saliency maps are integrated into an ODI saliency map in ERP, each point in ERP is assigned to the nearest point in 2D saliency maps of multiple viewing directions.

When the 2D images are extracted from ODI without overlapping (for example, cube mapping), the objects placed at the edge of the 2D patch are cut off as shown in Fig. 7 (a), so that it would be difficult to be recognized for accurate saliency estimation. However, if the 2D images are extracted from ODI with overlapping as in Fig. 7 (b), the objects can be detected, leading to more accurate estimation. In the integration of 2D saliency maps for multiple viewing directions, the saliency values in ERP are calculated by averaging the overlapping saliency values. Note that the 2D saliency maps with multiple angles of view are integrated using an integration layer by weighting the optimal scales, as explained in the following subsection and shown in Fig. 3. Although the multiple viewing directions can also be integrated using importance weights depending on the position of an extracted image (saliency at the center of an extracted image may be more accurately predicted than at the edge of an extracted image), it would be less effective than weighting optimal scales, so that a simple averaging integration was selected in the proposed method for the simplicity.

In the proposed method, 2D images are extracted in multiple angles of view, as shown in Fig. 3. The relationship between the angle of view ($\theta_{a},\phi_{a}$) and the extracted 2D image size ($W,H$) can be represented by the following equation, where $L$ is the distance from the camera to the image plane.

The 2D image size ($W,H$) are set to constant values for multiple angles of view.

The structures of the integration layer for the 2D saliency maps estimated from the extracted patches with multiple angles of view at a viewing direction are shown in Fig. 8. Since the receptive fields are different among the angles of view, the 2D saliency maps with larger angles of view are cropped and resized to cover the same range with the smallest angle of view, using the relationship in Eq. 3. Then, they are integrated into a 2D saliency map using pixel-wise channel weighting, whose weights are calculated by an attention module. In the experiments, two integration structures were examined, as in Fig. 8 (a) and (b). The attention weights are calculated from the concatenated saliency maps in (a), whereas the weights are obtained from image features in MainNet for the smallest angle of view in (b) since more information for importance of scales for each objects can be included in the image features. Since two structures of attention modules were also examined, the 4 types of architectures shown in Table 1 were tested in the experiments.

The output of the 2D saliency model is usually normalized by its norm, such as the L1 norm. If the 2D saliency model was used with the normalization to estimate the saliency for 2D patches extracted from ODI, the information of the dependence on the viewing direction for the extraction would be lost by the normalization at the integration to an ODI saliency map. Therefore, the 2D saliency model is used without the normalization at the output of the 2D saliency model. After the integration to the ODI saliency map, it is normalized by the L1 norm.

First, the effect of fine-tuning on the ODI dataset and the learnable equator-bias layer was examined in the condition of 2D-image extraction with the single angle of view 100^∘. The results are shown in Table 2. The experimental conditions were separated into two groups, MainNet without the fine-tuning to the ODI dataset (w/o FT in MainNet) and MainNet with the fine-tuning (w/ FT in MainNet). For each condition in MainNet, several biases were compared. The ”Learned (single bias)” means that the 2D saliency model with a single-channel bias layer (same as the center-bias layer) was fine-tuned on the ODI dataset. The ”Learned (multi-bias)” is the proposed method but using only a single angle of view for 2D-image extraction. The ”Constant (Avg)” bias means that the average equator bias over images in the ODI dataset was used for weighting the output of 2D saliency map. From the results, the fine-tuning of MainNet was highly effective to improve the performance. The learnable bias layers were also effective in both the conditions in ’MainNet w/o FT’ and ’MainNet w/ FT’, especially in the multiple-channel equator bias which achieved the best performance in the condition of MainNet with the fine-tuning in all the metrics. This means that leaning bias layer depending on the elevation angle for the 2D image extraction was useful to estimate the ODI saliency maps. To see this more clearly, NSS was calculated against the elevation angle of ODI in ERP, as shown in Fig. 9. It can be seen from the figure that the fine-tuning with the multiple-channel bias was better than other conditions, especially around at $-45^{\circ}$.

Next, the effect of extracting 2D images in multiple angles of view (MAV) was studied for the integration layers in Table 1. The results are shown in Table 3. The 4 architectures of the integration layer with MAV ($100^{\circ},110^{\circ},120^{\circ}$) were compared with the results with single angle of view (SAV) $100^{\circ}$. It can be seen from the table that the performance on all the architectures with MAV was better than that with SAV. Among the 4 architectures, ”Arch. 4” achieved the best performance. This means that the image features from MainNet (DenseSal) with deeper attention modules were effective to calculate the attention weights on multiple scales for each pixel.

To examine the effective combination of angles of view, several combinations were tested, as shown in Table 4. It can be seen from the results that smaller angles of view than the field of view in HMD ($100^{\circ}$) were not effective, as in ($80^{\circ},90^{\circ},100^{\circ}$). Since the performance with more than 3 angles of view decreased, the combination of ($100^{\circ},110^{\circ},120^{\circ}$) was the best on performance among the conditions tested. The reason for this might be that ($100^{\circ},110^{\circ},120^{\circ}$) was enough to capture the saliency for people, or that the larger angles of view detected the saliency which was not focused by people.

To compare the performance with conventional methods, the proposed model was evaluated in the same experimental setup as the conventional methods following the description in [3] and [27]. In this experiment, two additional state-of-the-art (SOTA) 2D saliency models (DPNSal [8] and DeepGaze IIE [26]) in addition to DenseSal were tested as MainNet in the proposed method. The results are shown in Table 5. The values for the conventional methods were taken from the reference [3]. As can be seen from the table, the proposed method with DenseSal achieved the best performance in all the metrics. Although DeepGaze IIE [26] has shown the best performance in the 2D saliency benchmark [7], the performance on the ODI saliency estimation was worse than DenseSal and DPNSal. Since DeepGaze IIE has frozen the base networks (ShapeNetC, EfficientNetB5, DenseNet201, and ResNext50) and has trained only a readout network, it would be difficult to be adapted to new datasets such as the ODI saliency dataset. The effectiveness of integrating multi-view angles (MAV) was confirmed for all the 2D saliency models tested. The proposed method with DenseSal outperformed the conventional methods by a large margin.

The examples of saliency maps estimated by the proposed methods (SAV and MAV with Arch. 4 in Table 3) are shown in Fig. 10. As can be seen from the sample images, most of the salient regions were detected both in SAV and MAV. To see more details, 2D images were extracted from the estimated ODI saliency maps, as shown in Fig. 11. In the left column of the figure, a small painting was fixated in the ground truth (GT), whereas SAV estimated lower saliency than GT. By using the multi-scale method with MAV, high saliency was estimated to the small painting. Similarly, a left hand of a human in the middle column example and a small head of a bird in the right column example were provided higher saliency by MAV than SAV as in GT.