Evaluating Atypical Gaze Patterns through Vision Models: The Case of Cortical Visual Impairment

Human behavior presents a complex interplay influenced by social surroundings, the environment, and an individual’s cognitive or biomedical conditions. There is hence an increasing interest in studying behavioral markers that could indicate neurological diseases (e.g., Alzheimer’s disease [1]), mental illness (e.g., depression [2]), developmental disorders (e.g., Autism Spectrum Disorder [3]) or affective patterns (e.g., mood [4], stress [5]). Aside from the profound implications to the quality and length of human life, such conditions incur a heavy financial burden for testing and treatment. The investigation of behavioral markers could thus assist specialists in both interpreting those conditions and diagnosing them with reduced costs. Behavioral Signal Processing (BSP) [6] and Artificial Intelligence (AI) technologies seek to quantify subtle nuances of behavioral signals that may serve as indicators of underlying cognitive or biomedical conditions. Various modalities have been researched to that end, with the most prominent being speech and paralinguistic patterns [7], facial expressions [3], movement and physiology, as quantified through specialized biosensors [5, 8].

In this paper, we apply this premise and use eye tracking technology to study Cortical Visual Impairment (CVI), one of the leading causes of pediatric visual impairment worldwide [9]. Yet, CVI lacks standardized, evidence-based methods that interpret its impacted visual characteristics. Eye tracking is a favorable behavioral modality, as it has been previously utilized to quantify atypical gaze patterns to predefined stimuli [10]. By scrutinizing gaze patterns, experts can gain a deeper understanding of how clinical populations perceive their environment [11]. Notably, people in the ASD spectrum report reduced attention to social stimuli such as the human face, voice and hand gestures [12, 13], whereas people with diagnosed depression show attentional biases to emotional faces [14]. In our case, eye tracking could quantify the visual function of CVI patients on images of higher-order attributes such as social interaction and texture [15, 16]. However, assessing human visual saliency requires a model for efficient extraction of saliency for the stimuli used for diagnosis. This is typically done through expert annotations, however these are time-consuming and prone to individual biases. Computational models [17, 18] have thus emerged to ease this process, yet they rarely capture high-level semantics.

In the following, we describe a machine learning-driven approach to create custom maps of visual saliency by prompting semantic and qualitative characteristics of CVI to multimodal vision models. We first outline a set of saliency markers related to CVI and an experiment setup to study those through measures of eye tracking. We engineer suitable prompts for attributes of interest and show that the derived saliency maps offer nuanced insights that are clinically relevant.

Cortical, or cerebral, visual impairment (CVI) is a neurological condition that occurs due to damage or injury to the visual pathways in the brain, resulting in a range of visual deficits [19]. Common causes of CVI include prematurity with periventricular leukomalacia, hypoxic-ischemic encephalopathy, trauma, hydrocephalus, metabolic or genetic disorders [9]. CVI is the leading cause of pediatric visual impairment in developed countries, yet there is no standardized method of quantifying the impacted visual characteristics [20], and no evidence-based treatment option is available.

Despite the prevalence of CVI, the precise definition remains a topic of debate in the literature. Traditionally, the diagnosis of CVI required decreased visual acuity and/or visual field defects [21]. Most children diagnosed using this definition had profound visual impairment or blindness at diagnosis. More recently, the term CVI has been used more broadly to include children with deficits of higher-order visual function (such as recognition of abstract objects or faces), with or without intact visual acuity and visual fields. This has led to the consensus definition proposed by Sakki et al. [22] of CVI as “a verifiable visual dysfunction which cannot be attributed to disorders of the anterior visual pathways or any potentially co-occurring ocular impairment.”

Still, several unifying characteristics among children with CVI have been reported in the literature for decades, mainly based on qualitative descriptions. These include reduced contrast sensitivity [23] and spared color discrimination [24], selective sparing or limitation of motion processing [25], difficulties with visual crowding or complexity [15], specific deficits in recognizing abstract objects or faces (e.g. prosopagnosia) [16], difficulties with visually-guided orientation in space (topographic agnosia) [26], and variability in visual function based on individual and environmental factors [19]. Since most children with CVI have developmental delays that impair communication, identifying these deficits relies on assessment of visual behavior. Eye tracking is an attractive option to that end because, in addition to being objective and quantitative, the tracking protocols may be designed to assess a multitude of lower- and higher-order visual characteristics.

Models of visual saliency traditionally include a set of handcrafted features extracted from the visual scene (images) through complex image processing techniques. Such features include color patterns or opponencies (e.g., red against green), edges or orientation patterns that have been shown to be relevant for human vision [27]. With the emergence of deep learning, there are more high-level attributes to account for, including shapes, depth and other human-inspired patterns. Neural network models like DeepGaze [17] have been trained on human fixations to predict an average image saliency map.

However, these bottom-up approaches are based solely on stimulus features. On the other hand, top-down human saliency takes into account human conditions, past experiences and intent to estimate fixation patterns [28]. Hence, CVI patients would be expected to diverge from baseline fixation estimations due to the associated visual impairment. Moreover, conditions like CVI require the extraction of saliency features that are challenging to extract with those methods, e.g., maps based on specific object categories, or based on non-material attributes like social interaction, motion, and complexity.

Multimodal learning, particularly vision-language models, have been proposed to ground the visual modality to text descriptions. CLIP [29] is among the first efficient multimodal models which include two unimodal encoders for visual and textual inputs respectively. CLIP is trained by minimizing the latent distance of paired images and text descriptions, whereas maximizing unpaired sample distances using the InfoNCE [30] contrastive objective. Here we adopt SegCLIP [31], a recent variant of CLIP which is specifically trained for text-driven image segmentation. This task is particularly suited for our study since the model output is a saliency map that corresponds to the segmentation of the prompted attributes.

Prompt Engineering  SegCLIP incorporates a dual-encoder architecture containing a text and image encoder. The authors suggest three losses, including a reconstruction loss that trains the generated segmentation output and a contrastive loss that aligns text and image inputs. We use the pre-trained model in its inference mode and provide image inputs (our stimuli) along with free-form text descriptions of the attributes of interest (prompts). The model then aligns the two inputs and outputs a segmentation map that highlights the corresponding attribute in the image (hence, a saliency map). While SegCLIP is trained primarily on object-centric prompts, authors show in [31] that the model can generalize to action attributes like flying, driving, etc. Here we investigate the extent to which it can generalize to qualitative attributes, such as complexity or social concepts, and we engineer prompts to better reflect the nuances of the CVI characteristics.

We first verify that CVI patients demonstrate atypical visual saliency compared to the matched controls. We use DeepGaze II [17] to create saliency maps that estimate typical human gaze patterns for our stimuli. Indeed, CVI patients report an average fixation saliency of 53, compared to 158 for controls ($p<10^{-17},d=2.12,p_{\text{perm}}=0.0002$), which verifies the assumption of atypical gaze patterns in CVI. In the following we analyze each of the prominent characteristics of CVI in terms of average fixation saliency measured across groups, and summarize the results in Figure 3.

Depth and Background For this experiment we extracted depth saliency maps by prompting for depth, with foreground information denoted with higher saliency. As shown in Figure 3a, patients with CVI focus significantly more to distant information in the image. In specific, the computed average is 107 for the control group, compared to 140 for CVI ($p<10^{-5},d=1.18,p_{\text{perm}}=0.0002$). This result is also verified using depth estimation maps, extracted using MiDaS [18]. By plotting the corresponding fixation density (Figure 4) we observe that the depth bias comes primarily from CVI tendency to focus less on foreground than more to background saliency. Despite this tendency, CVI subjects need more time to produce a valid background fixation ($p=0.0005,d=0.70,p_{\text{perm}}=0.0002$), while their overall latency to salient fixations does not differ ($p=0.372$).

Orientation and Motion Consistent with the hypothesis, CVI subjects focus on still rather than moving objects in an image ($p<10^{-9},d=2.28,p_{\text{perm}}=0.0002$), as shown from the differential saliency reported in Figure 3b. Regarding orientation information, SegCLIP outputs the most representative maps when prompted with bars or stripes, as compared to more abstract prompts like orientation, or direction and directional patterns. This is expected as the model performs better on object-centric prompts. As seen in Figure 3c, CVI patients seem to fixate more to such patterns than the control group ($p=0.006,d=0.81,p_{\text{perm}}=0.004$).

Contrast and Complexity For this experiment we tested the assumption that CVI produces functional challenges regarding scenes of high complexity, with a generally reduced contrast sensitivity [23]. As shown in Figure 3d, CVI subjects fixate significantly less on complex (positive saliency) rather than smooth (negative saliency) parts of the corresponding stimuli ($p<10^{-8},d=2.12,p_{\text{perm}}=0.0002$). This symmetric difference is evident also from the fixation density produced by the 2 groups (Figure 4). Similarly, testing on a subset of stimuli characterized by texture reveals that CVI subjects tend to avoid those regions, reporting an average saliency of 98 compared to 110 for the controls ($p=0.003,d=0.66,p_{\text{perm}}=0.002$).

Faces and Social Stimuli CVI is assumed to distract gaze from social stimuli. For this experiment we isolate multiple groups of stimuli images corresponding to different concepts of human interaction: faces, hands, eyes, etc. In this case, SegCLIP was prompted with single-object descriptions to yield representative saliency maps. Prompting with abstract concepts like human interaction, communication or emotion did not produce consistent maps for our stimuli. The results of these experiments are summarized in Figure 3e-f. CVI subjects are shown to fixate less than average on human faces ($p<10^{-10},d=2.86,p_{\text{perm}}=0.0002$), while also the latency of these fixations is longer ($p<10^{-4},d=0.87,p_{\text{perm}}=0.0002$). No difference emerges when it comes to fixations on eyes or mouth ($p>0.75$). Interestingly though, as shown in Figure 3f, children with CVI tend to focus more on the non-social human characteristics ($p=0.001,d=0.59,p_{\text{perm}}=0.0016$). An additional within-group difference is that controls produce longer fixations at higher saliency (pearson $\rho=0.21$, $p=0.001$), whereas CVIs report uniform durations ($p=0.675$).

A novel, machine learning-based approach to create maps of visual saliency was proposed, focusing on the assessment of qualitative markers of eye-tracking activity in children with CVI. To that end, we leveraged the robustness of large vision models in segmenting image patches conditioned on textual descriptions of CVI characteristics. The availability of those saliency maps assisted us in verifying and reinforcing our understanding of a challenging neurological impairment with no standardized diagnostic methods. In the future, the proposed approach could be used in designing individualized interventions and assessing their efficacy in clinical trials.