Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors

We developed a web application that presents images to participants and prompts them to provide textual descriptions while performing click-contingent image explorations. Users were asked to visually explore and provide a caption for up to fifty distinct images from the MIT1003 dataset [11]. Images were presented sequentially, one after another. We used the protocol defined in Jiang et al., [10] for which crowdsourcers are presented with a completely blurred version of the image, and can unveil details on specific locations by clicking. A click will reveal information in an area corresponding to two degrees of visual angle, calculated with respect to the original dataset experimental setup described in [11]. This process is aimed at simulating a human’s foveated vision. Subjects can click up to ten times, before providing a caption. We instructed users to describe the content of the image with "one or two sentences", in English. All clicks and captions are stored in a database, while no personal information has been collected.

In total, we collected 27865 clicks on 4573 observations over 153 sessions. We excluded 654 observations that were skipped, 33 with no recorded clicks, and 38 captions with less than 3 characters. After these exclusions, we analyzed the data for 1002 of the 1003 images in the pool. The average number of characters per caption was 50.89, with a standard deviation of 33.88. The average number of words per caption was 9.78, with a standard deviation of 6.47. The maximum caption lengths were 259 characters and 51 words, respectively. Figure 2 shows the click density based on all collected clicks. The figure was generated by scaling the position of each click with the size of the corresponding image. Afterward, we used a Gaussian kernel density estimation to calculate the click density. Figure 3 shows the number of observations with each click count from 1 to 10. Using all 10 available clicks was the most favored option.

Contrastive Language-Image Pretraining (CLIP) [21], is a method of learning semantic concepts efficiently from natural language supervision, which is trained on a large dataset of $N$ (image, text) pairs. CLIP uses a shared embedding space for both modalities. Let $f_{\theta}^{I}$ be the image encoder and $f_{\theta}^{T}$ be the text encoder, where $\theta$ represents the trainable parameters of the models. Given an image $I_{i}$ and a corresponding text $T_{i}$, the image embedding is defined as $e_{i}^{I}=f_{\theta}^{I}(I_{i})$, and the text embedding is defined as $e_{i}^{T}=f_{\theta}^{T}(T_{i})$.

CLIP models are trained by maximizing the cosine similarity of the embeddings of the $N$ correct pairs in the batch while minimizing the cosine similarity of the embeddings of the $N^{2}-N$ incorrect pairings. The cosine similarity between two embeddings $e_{i}$ and $e_{j}$ is computed as $S_{c}(A,B)=(A\cdot B)/(\lVert A\rVert\lVert B\rVert)$.

Let $\xi=\{\xi_{t}\}_{t\in\{1,...,N\}}$ be a sequence of fixation points, also called scanpath in what follows. Given a visual stimulus $S$ and its coarse version $\tilde{S}$, Schwinn et al., [22] define a foveated stimulus as

where $\xi_{t}$ is the current fixation point at time $t$, and $G_{\sigma_{\xi}}$ is a Gaussian blob centered on $\xi_{t}$ with standard deviation $\sigma_{\xi}$. A scanpath is generated iteratively, by an optimization procedure. Let $\tau$ be any task model, and $\mathcal{L}_{\tau}$ its associated loss. The next fixation location $\xi_{t+1}$ is determined by a local search in order to minimize the loss

where $y$ is a target output for the specific downstream task. This approach can be applied to any pretrained vision model.

To use NeVA in combination with CLIP, we need to modify the optimization procedure to incorporate the cosine similarity between (foveated) stimuli and caption embeddings in the loss function.

Let $S$ be a visual stimulus and $C$ be its associated caption. We use CLIP’s visual encoder $f_{\theta}^{I}$ as the task model, where the downstream task is, at each step, to maximize the cosine similarity between the foveated image embedding and caption embeddings, i.e.,

where $e^{\pi}=f_{\theta}^{I}\left(\pi\left(S,\xi_{t}\right)\right)$, and $e^{C}=f_{\theta}^{T}\left(C\right)$. Then, the NeVA optimization procedure described in the previous subsection can be extended to generate a complete visual scanpath for a given stimulus and its associated caption embedding $e^{C}$. To simplify the notation, we denote this procedure as

where $\xi^{S,C}$ is the sequence of fixation points that constitute the scanpath. This approach provides an efficient method for generating scanpaths that are optimized to minimize the distance between the foveated versions of the stimulus, highlighting specific portions of the image, and the semantic concepts included in the captions. The pseudo-code in algorithm 1 illustrates the overall procedure.

We observe that, as a consequence of its training, the CLIP visual embedding of the clean image $e^{S}=f_{\theta}^{I}\left(S\right)$ (i.e., not foveated), can be considered as the ideal caption’s embedding. By employing the CLIP visual embedding of the clean image as a target in the NeVA optimization procedure, we can generate a scanpath without an explicit caption, i.e.,

We refer to this approach as the NevaClip (visually-guided). Although it violates the biological constraint of partial information by having access to the whole stimulus (we in fact need the entire clean image in advance to compute $e^{S}$), this serves us as a gold standard as it potentially minimizes the distribution shift between captions provided by our users in our study and captions that CLIP would assign the same image. Furthermore, this configuration of the NevaClip model is relevant for practical applications in which processing the entire stimulus is feasible, while we do not have real-time access to the subject’s caption.

We adhere to the original implementation of the Neva algorithm with 20 optimization steps¹¹1https://github.com/SchwinnL/NeVA. Random restarts during the local search were not allowed, as we found out this induces a favorable proximity preference [13] to the next fixation. In [22] past explorations are incorporated into the agent’s state, modulated by a forgetting factor. In our experiments, we found that optimal results were achieved for NevaClip with a forgetting factor of $0$. This indicates that the system operates under a Markovian assumption, where only information from the last fixation is used in order to determine the next location to be attended. We use the implementation of the CLIP model provided by the original authors²²2https://github.com/openai/CLIP. When not stated otherwise, all NevaClip results are presented for the Resnet50 backbone. For all competitor models, the original code provided by the authors is used.

In order to investigate the relationship between captions and their corresponding visual explorations, we evaluate four distinct variations of our NevaClip algorithm. These versions differ only on the target embedding used as a guide for visual exploration, as described in equations 1, 2, and 3.

NevaClip (correct caption). Scanpaths are generated maximizing the alignment between visual embeddings of the foveated stimuli, and text embeddings of the corresponding captions, as described in equation 2.
NevaClip (different caption, same image). Scanpath are generated to maximize the alignment between visual embeddings of the foveated stimuli, and text embeddings of the captions provided by a different subject on the same image.
NevaClip (different caption, different image). Scanpath are generated to maximize the alignment between visual embeddings of the foveated stimuli, and text embeddings of the random captions provided by a different subject on a different image.
NevaClip (visually-guided). Scanpath are generated to maximize the alignment between visual embeddings of the foveated stimuli, and the visual embedding of the clean version of the same image, as described in equation 3.

We define three baselines, to better position the quality of the results.

Random. Scanpaths are generated by subsequently sampling fixation points from a uniform probability distribution defined over the entire stimulus area.
Center. Scanpaths are randomly sampled from a $2$-dimensional Gaussian distribution centered in the image. We used the center matrix provided in Judd et al., [11], associated with the MIT1003 dataset.
Clicks Density. Scanpaths are randomly sampled from a $2$-dimensional density distribution obtained by cumulating all clicks in the dataset.

We compare our approach to four state-of-the-art unsupervised models of human visual attention. It is worth mentioning that none of these models is designed to exploit information from the caption.

NeVA (original) [22]. We employ the guidance of a classification downstream task, which was demonstrated to produce the highest similarity to humans in the original paper. We use the optimized version of the algorithm, provided by the authors.
Constrained Levy Exploration (CLE) [3]. Scanpath is described as a realization of a stochastic process. We used saliency maps by [9] to define non-local transition probabilities. Python implementation provided by the authors in [2].
Gravitational Eye Movements Laws (G-EYMOL) [28]. Scanpath is obtained as the motion of a unitary mass within gravitational fields generated by salient visual features.
Winner-take-all (WTA) [13]. Based on the saliency maps by Itti et al., [9], the next fixation is selected at the location of maximum saliency. Inhibition-of-return allows switching to new locations.

To measure the similarity between simulated and human scanpaths, we compute ScanPath Plausibility (SPP) [8], using the string-based time-delay embeddings (SBTDE) [22] as a basic metric. The SBTDE accounts for stochastic components which are typical of the process of human visual attention [20], by searching for similar subsequences at different times. This metric can be computed for each subsequence length, expressively indicating the predictive power of models at each stage of the exploration. The SBTDE at subsequence length $k$ is computed as

where $\xi^{A},\xi^{B}$ are the two scanpaths to be compared, and $\Xi^{A}_{k},\Xi^{B}_{k}$ are the sets of all possible contiguous sublengths of size $k$ in $\xi^{A}$ and $\xi^{B}$, respectively. We use the string-edit distance to compute the distance $d(x,\Xi^{B}_{k})$ between subsequences. The SPP accounts for large variability between human subjects, by considering only the human scanpath with the minimum distance for each stimulus. To obtain a similarity measure, ranging from $0$ (minimum similarity) and $1$ (maximum similarity), we consider the SBTDE complement to one. Let $\xi^{S}$ be a simulated scanpath, and $\Xi^{H}$ a set of human reference scanpaths, for each sublength $k$, scanpath plausibility is defined as

Since human explorations, both for clicks and eye-tracking, can be of different lengths, for each subject we compute the metric up to $min(L,10)$, where $L$ is the length of the human scanpath.

Table 1 summarises results on the CapMIT1003 dataset. The NevaClip (visually-guided), which exploits the guidance of the clean image embedding, significantly outperforms all other approaches by a substantial margin. As expected, NevaClip (different subject, different image), which generates a scanpath using a label from a different image and a different subject, performs similarly to the Random Baseline. We observed that this configuration systematically explores non-relevant objects that align with the misleading caption or, in their absence, emphasizes the background, which typically possesses an average feature representation (see Figure 4). The NevaClip (correct caption) performs slightly better than the NevaClip (different subject, same image), demonstrating that the caption provided by the subject brings useful information about their own visual exploration behavior. Among competitors, G-Eymol and Neva (original) compete well, despite not incorporating any caption information. It is worth noting that all approaches are unsupervised, and the scanpath prediction is performed zero-shot. Supervised approaches could benefit from exploiting more information contained in the caption for personalized scanpath prediction.

The NevaClip (visually-guided) approach again significantly outperforms all competitors by a substantial margin. NevaClip (correct caption) and NevaClip (different subject, same image) also performs better than state-of-the-art models NeVA (original) and G-Eymol. These results demonstrate a substantial overlap between the two tasks, as discussed later in section 4.7.

We compare the performance of NevaClip (correct caption) for different backbones. Results are presented in table 2. ResNet50 obtains a SPP SBTDE of 0.35 ($\pm$ 0.18), while ResNet101 and ViT-B/32 obtain 0.33 ($\pm$ 0.17) and 0.33 ($\pm$ 0.18), respectively. While the performance of different backbones is comparable, simpler models such as ResNet50 might benefit from simpler representations, which lead to less over-specific visual explorations. A similar observation was made by Schwinn et al., [22], where a combination of a simpler model and task exhibited the best performance in terms of similarity to humans.

As expected, capturing attention in captioning tasks proves to be more challenging compared to free-viewing, as it involves the simultaneous consideration of multiple factors such as image content, language, and semantic coherence. This is reflected in the results, where all approaches achieve lower performance in terms of scanpath plausibility (see table 1) for the captioning task. The ranking of models remains consistent across both tasks, with free-viewing models performing reasonably well on both datasets. This observation suggests a substantial overlap between free viewing and captioning, possibly due to the initial phases of exploration being predominantly driven by bottom-up factors rather than task-specific demands.

Simulating realistic scanpaths becomes increasingly challenging after just a few fixations, and this difficulty is particularly pronounced in the context of captioning, as illustrated in Figure 5. This result is in line with existing literature [24] demonstrating how human visual attention diverges quickly due to individual goals.

Upon analyzing the click density map depicted in Figure 2, we observe that captioning elicits significantly more diverse exploratory patterns compared to free viewing, which exhibits a more pronounced center bias [11]. The same figure also reveals that a portion of the clicks is distributed in a grid-like fashion, indicating a tendency of users to perform a systematic strategy for spread out visual exploration. This finding makes sense considering the nature of captioning, which assumes a holistic understanding of the scene, unlike free viewing which does not impose such requirements.