Scanpath Prediction in Panoramic Videos via Expected Code Length Minimization

Scanpath prediction has been first investigated in planar images as a generalization of non-ordered prediction of eye fixations in the form of a 2D saliency map. Ngo and Manjunath [23] used a long short-term memory (LSTM) to process the features extracted from a DNN for saccade sequence prediction. Wloka et al. [24] extracted and combined saliency information in a biologically plausible paradigm for the next fixation prediction together with a history map of previous fixations. In contrast, Xia et al. [25] constrained the input of the DNN to be localized to the current predicted fixation with no historical fixations as input. Sun et al. [17] explicitly modeled the inhibition of return¹¹1Inhibition of return is defined as the relative suppression of processing of (detection of, orienting toward, responding to) stimuli (object and events) that had recently been the focus of attention [26]. (IOR) mechanism when predicting the fixation location and duration. The GMM was adopted for probabilistic modeling of the next fixation. A similar work was presented in [27], where the IOR mechanism was inspired by the Guided Search 6 (GS6) [28], a theoretical model of visual search in cognitive neuroscience.

For scanpath prediction in 360° images, Assens et al. [32] advocated the concept of the “saliency volume” as a sequence of time-indexed saliency maps in the ERP format as the prediction target. Scanpaths can be sampled from the predicted saliency volume based on maximum likelihood with IOR. Zhu et al. [33] clustered and organized the most salient areas into a graph. Scanpaths were generated by maximizing the graph weights. Assens et al. [18] combined the mean squared error (MSE) with an adversarial loss to encourage realistic scanpath generation. Similarly, Martin et al. [19] trained a generative adversarial network (GAN) with the MSE replaced by a loss term based on dynamic time warping [46]. Kerkouri et al. [35] adopted a differentiable version of the argmax operation to sample fixations memorylessly, and leveraged saliency prediction as an auxiliary task.

For scanpath prediction in 360° videos, Fan et al. [12] combined the saliency map, the optical flow map, and historical viewing data (in the form of scanpaths or tiles²²2Typically, an ERP image can be divided into a set of nonoverlapping rectangular patches, i.e., tiles. Any FoV can be covered by a subset of consecutive tiles.) to calculate the probability of tiles in future frames. Built upon [12], Li et al. [16] added a correction module to check and correct outlier tiles. Nguyen et al. [14] improved panoramic saliency detection performance for scanpath prediction with the creation of a new 360° video saliency dataset. Xu et al. [13] improved saliency detection performance from a multi-scale perspective, and advocated relative viewport displacement prediction but applying Euclidean geometry to spherical coordinates. Xu et al. [42] used deep reinforcement learning to imitate human scanpaths, but the prediction horizon is limited to $30$ ms (i.e., one frame). Li et al. [43] made use of not only the historical scanpath of the current user but also the full scanpaths of other users who had previously explored the same 360° video (also known as cross-user behavior analysis). Rondón et al. [11] performed a thorough root-cause analysis of existing scanpath prediction methods. They identified that visual features only start contributing to scanpath prediction for horizons longer than two to three seconds, and an RNN to process the visual features is crucial before concatenating them with positional features. To respect the spherical nature of 360° videos, spherical convolution [47, 48, 49] has been adopted to process visual features [50, 15] in combination with spherical MSE as the loss function. Additionally, Chao et al. [45] explored a transformer [51, 52] to predict the future scanpath using its history as the sole input.

In Table I, we contrast our scanpath prediction method with existing representative ones in terms of the input format and modality, whether to rely on external algorithms, the sampling method for the next viewpoint, the prediction horizon, whether to specify ground-truth scanpaths, and the loss function. From the table, we see that most existing panoramic scanpath predictors work directly with the ERP format for computational simplicity. Like [42], we choose to sample along the scanpath a sequence of 2D viewports as the visual input, and further project the scanpath onto each of the viewports for relative scanpath prediction, both of which are beneficial for mitigating geometric deformations induced by ERP. Moreover, nearly all panoramic scanpath predictors take a supervised learning approach: first specify ground-truth scanpaths, and then adopt the MSE to quantify the prediction error, which essentially corresponds to sampling the next viewpoint by maximizing unimodal Gaussian likelihood. Such a supervised learning formulation is limited to capturing the uncertainty and diversity of scanpaths. Interestingly, early work on planar scanpath prediction suggests taking an unsupervised learning approach: first specify a parametric probability model of scanpaths, and then estimate the parameters by minimizing the negative log likelihood (NLL). In a similar spirit, we optimize a probability model of panomaric visual scanpaths, as specified by a product of discretized GMMs, by minimizing the expected code length. The proposed sampling strategy is also different and physics-driven. Additionally, our method is end-to-end trainable, and does not rely on any external algorithms for visual feature analysis.

Panoramic scanpath prediction aims to learn a seq2seq mapping $f:\{\mathcal{X},\bm{s}\}\mapsto{\bm{r}}$, in which a sequence of seen 360° video frames $\mathcal{X}=\{\bm{x}_{0},\ldots,\bm{x}_{t},\ldots,\bm{x}_{T-1}\}$ and a sequence of past viewpoints (i.e., the historical scanpath) $\bm{s}=\{(\phi_{0},\theta_{0}),\ldots,(\phi_{t},\theta_{t}),\ldots,(\phi_{T-1},\theta_{T-1})\}$ are used to predict a sequence of future viewpoints (i.e., the future scanpath) $\bm{r}=\{(\phi_{T},\theta_{T}),\ldots,(\phi_{T+S-1},\theta_{T+S-1})\}$. Here, $S$ indexes the discrete prediction horizon; $(\phi_{t},\theta_{t})$ specifies the $t$-th viewpoint in the format of (latitude, longitude), and can be transformed to other coordinate systems as well (see Fig 1); $\bm{x}_{t}$ denotes the $t$-th 360° video frame in any format, and in this paper, we adopt the viewport representation by first inversing the plane-to-sphere mapping followed by rectilinear projection centered at $(\phi_{t},\theta_{t})$.

A supervised learning formulation of panoramic scanpath prediction relies on the specification of the ground-truth scanpath $\bm{r}$, and aims to optimize the predictor $f$ by

where $D(\cdot,\cdot)$ is a distance measure between the predicted and ground-truth scanpaths. It is clear that Problem (1) encourages deterministic prediction, which may not adequately model the scanpath uncertainty and diversity.

Inspired by early work on planar scanpath prediction [23, 17] and optical flow estimation [53], we argue that it is preferred to formulate panoramic scanpath prediction as an unsupervised density estimation problem:

Generally, estimating the probability distribution in high-dimensional space can be challenging due to the curse of dimensionality. Nevertheless, we can decompose $p(\bm{r}|\mathcal{X},\bm{s})$ into the product of conditional probabilities of each viewpoint using the chain rule in probability theory:

where $\bm{c}_{t}=\{({\phi}_{T},{\theta}_{T}),({\phi}_{T+1},{\theta}_{T+1}),\ldots,({\phi}_{T+t-1},{\theta}_{T+t-1})\}$, for $t=1,\ldots,S-1$, is the set of all preceding viewpoints, and $\bm{c}_{0}=\emptyset$. The set of $\{\mathcal{X},\bm{s},\bm{c}_{t}\}$ constitutes the contexts of $({\phi}_{T+t},{\theta}_{T+t})$, among which $\mathcal{X}$ is the historical visual context, $\bm{s}$ is the historical path context, and $\bm{c}_{t}$ is the causal path context. For computational reasons, we may as well keep track of only the most recent visual and path contexts by placing a context window of size $R$. As a result, $\mathcal{X}$ and $\bm{s}$ become $\{\bm{x}_{T-R},\ldots,\bm{x}_{T-1}\}$ and $\{(\phi_{T-R},\theta_{T-R}),\ldots,(\phi_{T-1},\theta_{T-1})\}$, respectively. As for the causal path context, we adopt human scanpaths during training, and sample them from the learned probability model during testing.

Often, viewpoints in a visual scanpath are represented by continuous values, which are amenable to lossy compression. A typical lossy data compression system consists of three major steps: transformation, quantization, and entropy coding. The transformation step maps spherical coordinates in the form of $(\phi,\theta)$ to other corrdinate systems such as 3D Eculidean coordinates used in [19] and the relaitve $uv$ coordinates advocated in the paper. The quantization step truncates input values from a larger set (e.g., a continuous set) to output values in a smaller countable set with a finite number of elements. The uniform quantizer is the most widely used:

where $\xi\in\{\phi,\theta\}$ indicates viewpoint coordinates, $\Delta$ is the quantization step size, and $\lfloor\cdot\rfloor$ denotes the floor function.

After quantization, we compute the discrete probability mass of $(\bar{\phi}_{T+t},\bar{\theta}_{T+t})=(Q(\phi_{T+t}),Q(\theta_{T+t}))$ by accumulating the probability density defined in the righthand side of Eq. (3) over the area $\Omega=[\bar{\phi}_{T+t}-1/2\Delta,\bar{\phi}_{T+t}+1/2\Delta]\times[\bar{\theta}_{T+t}-1/2\Delta,\bar{\theta}_{T+t}+1/2\Delta]$:

Finally, given a minibatch of human scanpaths $\mathcal{B}=\{\mathcal{X}^{(i)},{\bm{s}}^{(i)}\}_{i=1}^{B}$, where $\mathcal{X}^{(i)}=\{\bm{x}_{0}^{(i)},\ldots,\bm{x}_{T-1}^{(i)}\}$ and $\bm{s}^{(i)}=\{(\phi^{(i)}_{0},\theta^{(i)}_{0}),\ldots,(\phi^{(i)}_{T-1},\theta^{(i)}_{T-1})\}$, we may use stochastic optimizers [54] to minimize the negative log-likelihood of the parameters in the discretized probability model:

It can be shown that this optimization is equivalent to minimizing the expected code length of training scanpaths, where $-\log_{2}(P(\bar{\phi}^{(i)}_{T+t},\bar{\theta}^{(i)}_{T+t}\Big{|}\mathcal{X}^{(i)},\bm{s}^{(i)},\bm{c}^{(i)}_{t}))$ provides a good approximation to the code length (i.e., the number of bits) used to encode $(\bar{\phi}^{(i)}_{T+t},\bar{\theta}^{(i)}_{T+t})$.

We conclude this subsection by pointing out the advantages of optimizing the discretized probability model defined in Eq. (5) over its continuous counterpart in Eq. (3). From the probabilistic perspective, estimating a continuous probability density function in a high-dimensional space with a small finte sample set (as in the case of panoramic scanpath prediction) can easily lead to overfitting [55]. Discretization (Eqs. (4) and (5)) introduces a regularization effect that encourages the esimated probability to be less spiky. From the computational perspective, we introduce an important hyperparameter—the quantization step size $\Delta$—that includes the continuous probability density esimation as a special case (i.e., $\Delta\rightarrow 0$). Thus, with a proper setting of $\Delta$, a better probability model for scanpath prediction can be obtained (see the ablation experiment in Sec. 5.4). From the conceptual perspective, optimizing a discretized probality model is deeply rooted in the well-established theory of lossy data compression, which gives us a great opportunity to transfer the recent advances in learned-based image compression [56, 57, 58] to scanpath prediction.

Inspired by the entropy modeling in the field of learned image compression [56, 57], we construct the probability model of $\bar{\bm{\eta}}_{T-1,t}=(\bar{u}_{T-1,t},\bar{v}_{T-1,t})^{\intercal}$, the quantized version of ${\bm{\eta}}_{T-1,t}=({u}_{T-1,t},{v}_{T-1,t})^{\intercal}$, using a GMM with $K$ components. Our GMM is conditioned on the historical visual context $\mathcal{X}$, the historical path context $\bm{s}$, and the causal path context $\bm{c}_{t}$:

where we omit the subscript $T-1$ in $\bar{\bm{\eta}}_{t}$ to make the notations uncluttered. Due to the fact the gradients of the quantizer in Eq. (4) are zeros almost everywhere, we follow the method in [56], and approximate the quantizer during training by adding a random noise $\epsilon$ uniformly sampled from $[-\Delta,\Delta]$ to the continuous value:

$\{\alpha_{i},\bm{\mu}_{i},\bm{\Sigma}_{i}\}$ in Eq. (3.3) represent the mixture weight, the mean vector, and the covariance of the $i$-th Gaussian component, respectively, to be estimated. Such estimation can be made by concatenating the visual and path features (with the size of $B\times S\times(C_{v}+C_{h}+C_{c})$) followed by three prediction heads to produce the mixture weight vector, $K$ mean vectors, and $K$ covariance matrices, respectively. We assume the horizontal direction $u$ and the vertical direction $v$ to be independent, resulting in diagonal covariance matrices. Each prediction head consists of a front-end FC layer, two FC residual blocks, and a back-end FC layer. We append a softmax layer at the end of the weight prediction head to ensure that the output is a probability vector. Similarly, we add the ReLU nonlinearity at the end of the covariance prediction head to ensure nonnegative outputs on the diagonals.

We then discretize the GMM model by integrating the probability density over the area $\Omega=[\bar{u}_{t}-1/2\Delta,\bar{u}_{t}+1/2\Delta]\times[\bar{v}_{t}-1/2\Delta,\bar{v}_{t}+1/2\Delta]$:

Finally, we end-to-end optimize the entire model by minimizing the expected code length of the scanpaths in a minibatch:

Panoramic video datasets typically contain eye-tracking data, in the form of eye movements and head orientations, collected from human participants. We list some basic information of commonly used 360° video datasets in Table II.

Based on the dataset scale, we have selected the CVPR18 dataset [13] and the VRW23 dataset [66] for the main experiments, and leave some of the remaining for cross-dataset generalization testing. To illustrate the diversity of the scanpaths in the two datasets, we evaluate the consistency of two scanpaths of the same length using the temporal correlation:

where $\mathrm{PCC}(\cdot)$ is the function to compute the Pearson correlation coefficient. The mean temporal correlation over $N$ scanpaths for the same 360° video can be computed by

meanTC ranges from $[-1,1]$, with a larger value indicating higher temporal consistency.

We visualize the meanTC histograms of the CVPR18 and VRW23 datasets in Fig. 4, from which we observe that scanpaths in both datasets exhibit considerable diversity, which shall be computationally modeled. Moreover, the scanpaths with longer horizons in the CVPR18 dataset (e.g., more than $30$ seconds) are even less consistent, showing the difficulty of the long-term scanpath prediction.

In the main experiments, we inherit the same sampling rate in previous studies [13, 11] to downsample both the video (and the corresponding) scanpaths to five frames (and viewpoints) per second. We use one second as the context window size to create the visual and path history (i.e., $R=5$), and produce one-second future scanpath (i.e., the prediction horizon $S=5$). As for predicting scanpaths that are longer than one second, we just apply our PID controller-based sampling strategy multiple rounds, as described in Sec. 4.

We set the quantization step size $\Delta$ in Eq. (4) to $0.2$ (with a quantization error $<0.034\degree$). The resolution of the extracted viewport is set to $H_{v}\times W_{v}=252\times 484$, covering an FoV of $\phi_{v}\times\theta_{v}={63}\degree\times{112}\degree$. As shown in Fig. 3, for the historical visual context, we set $H=8$, $W=14$, $C=16$, and $C_{v}=128$; for the historical path context, we set $C_{h}=128$; for the causal patch context, we set $C_{c}=32$. The number of Gaussian components $K$ in GMM in Eq. (3.3) is set to $3$. For the PID controller, we set the sampling interval $\Delta\tau$ in Eq. (28) as the inverse of the sampling rate, i.e., $\Delta\tau=0.2$ second. The set of parameters in the PID controller to adjust the accelaration in Eq. (31) are set using the Ziegler–Nichols method [67] to $K_{p}=0.6K_{u}$, $K_{i}=2K_{u}/P_{u}$, and $K_{d}=K_{u}P_{u}/8$, respectively. For the CVPR18 dataset, we set $K_{u}=60$ and $P_{u}=0.29$, while for the VRW23 dataset, we set $K_{u}=90$ and $P_{u}=0.29$.

During model training, we first initialize the convolution layers in ResNet-50 for visual feature extraction with the pre-trained weights on ImageNet, and initialize the remaining parameters by He’s method [68]. We then optimize the entire method by minimizing Eq. (27) using Adam [54] with an initial learning rate of $10^{-4}$ and a minibatch size of $B=48$ (where we parallelize data on $4$ NVIDIA A100 cards). We decay the learning rate by a factor of $10$ whenever the training plateaus. We train two separate models, one for the CVPR18 dataset by following the suggestion of the training/test set splitting in [13] and the other for the VRW23 dataset, in which we use the first $400$ videos for training and the rest $102$ videos for testing.

We evaluate panoramic video scanpath predictors from three perspectives: 1) prediction accuracy, 2) perceptual realism, and 3) generalization to unseen datasets. For prediction accuracy evaluation, we introduce two quantitative metrics: minimum orthodromic distance⁷⁷7The orthodromic distance is also known as the great-circle or the spherical distance. and maximum temporal correlation. Specifically, given a panoramic video, we define the set of scanpaths, $\mathcal{S}=\{\bm{s}^{(i)}\}_{i=1}^{|\mathcal{S}|}$, corresponding to $|\mathcal{S}|$ different viewers as the ground-truths. The minimum orthodromic distance between $\mathcal{S}$ and the set of predicted $\hat{\mathcal{S}}=\{\hat{\bm{s}}^{(i)}\}_{i=1}^{|\hat{\mathcal{S}}|}$ can be computed by

where the $\mathrm{OD}(\cdot,\cdot)$ between two scanpaths of the same length is defined as

Similarly, the maximum temporal correlation between $\mathcal{S}$ and $\hat{\mathcal{S}}$ is calculated by

It is noteworthy that we intentionally opt for best-case set-to-set distance metrics to avoid specifying, for each predicted scanpath from $\hat{\mathcal{S}}$, one ground-truth from $\mathcal{S}$. Moreover, such distances have the advantage over path-to-path distances in terms of quantifying the prediction accuracy without penalizing the generation diversity.

Additionally, inspired by the time-delay embedding technique in dynamical systems [69, 70], we introduce the sliced versions of the minimum orthodromic distance and the maximum temporal correlation, respectively. We first slice each ground-truth scanpath $\bm{s}^{(i)}\in\mathcal{S}$, for $i\in{1,\ldots,|\mathcal{S}|}$, into $N_{s}$ overlapping sub-paths of length $T_{s}$, $\{\bm{s}^{(i)}_{t}\}_{t=1}^{N_{s}}$, in which the overlap between two consecutive sub-paths is set to $\lfloor T_{s}/2\rfloor$. This gives rise to $N_{s}$ sets of sliced scanpaths $\{\mathcal{S}_{t}\}_{t=1}^{N_{s}}$, where $\mathcal{S}_{t}=\{\bm{s}^{(i)}_{t}\}_{i=1}^{|\mathcal{S}|}$. Similarly, for the predicted scanpath set $\hat{\mathcal{S}}$, we create $N_{s}$ sets of sliced scanpaths $\{\hat{\mathcal{S}}_{t}\}_{t=1}^{N_{s}}$, where $\hat{\mathcal{S}}_{t}=\{\hat{\bm{s}}^{(j)}_{t}\}_{j=1}^{|\hat{\mathcal{S}}|}$, and compute the sliced minimum orthodromic distance and the sliced maximum temporal correlation by

and

respectively. In the experiments, $T_{s}$ is set to $\{5,10,15\}$, corresponding to one-second, two-second, and three-second sliced scanpaths, respectively. We will append the corresponding number to the evaluate metric (e.g., $\mathrm{SminOD}$-$5$) to differentiate the three different settings. After determining $T_{s}$, $N_{s}$ can be set accordingly. Generally, the OD metric family focuses more on the pointwise local comparison, while the TC metric family emphasizes more on global covariance measurement.

For perceptual realism evaluation, we first train a separate classifier for each scanpath predictor to discriminate its predicted scanpaths from those generated by humans. The underlying idea is conceptually similar to that in GANs [71], except that we perform post hoc training of the classifier as the discriminator. A higher classification accuracy indicates poorer perceptual realism. Rather than solely relying on machine discrimination, we also perform a formal psychophysical experiment to quantify the perceptual realism of scanpaths. We reserve the details on how to train the classifiers and how to perform the psychophysical experiment in later subsections.

For generalization evaluation, we resort to the MMSys18 [65] and the PAMI19 [42] datasets, which consist of $19$ and $76$ distinct paromanic scenes, respectively (see Table II).