What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation

Vision-Language Models (VLMs) have become central to both computer vision and natural language processing (NLP) due to their ability to complete complex multimodal tasks like visual recognition (Han et al., 2023; Menon and Vondrick, 2022), visual question answering (Liu et al., 2024; Bai et al., 2023; Li et al., 2022, 2020), and image captioning (Lu et al., 2019; Yang et al., 2023; Zhang et al., 2021). Many foundational VLMs, such as BLIP (Bootstrapping Language-Image Pre-training) (Li et al., 2022), use cross-attention to capture interactions between modalities Tan and Bansal (2019); Lu et al. (2019); Li et al. (2022), while others, like LLaVA Liu et al. (2024), successfully use self-attention for multimodal integration. Despite recent works investigating the differences in cross-attention and self-attention in VLMs (Ilinykh and Dobnik, 2022a; Parcalabescu et al., 2020; Djamila-Romaissa et al., 2022), the specific role of different attention heads in vision-language is not fully understood, particularly in large VLMs such as LLaVA Yuksekgonul et al. (2022).

In NLP, mechanistic interpretability (MI) has proven effective for unraveling the complex structure of LLMs by identifying pathways, or circuits, that enable behaviors like logical reasoning and indirect object identification Hanna et al. (2023); Wang et al. (2022). A core method in MI is causal mediation analysis (CMA), which measures output changes by intervening on inputs and tracking effects. To apply CMA in Transformers, Meng et al. (2023) use activation patching, which identifies key model components for a given task by replacing the hidden states of a corrupt run with those from a clean run. A “clean run” refers to a standard forward pass on an input. A “corrupt” run either alters the input by swapping input tokens with semantically related tokens or applies Gaussian noise to token embeddings so that the model can no longer produce the correct answer from the clean pass. While successful in NLP, mechanistic interpretability methods are understudied in VLMs due to the need for an effective semantic-based corruption scheme for images.

Previous works applying activation patching to VLMs only study BLIP, focus on Multi-Layer Perceptron (MLP) layers in BLIP’s decoder, and apply Gaussian noise to image embeddings to create corrupt runs Palit et al. (2023). However, multimodal interaction occurs in BLIP’s image-grounded text encoder through the cross-attention layers, meaning findings from Palit et al. (2023) do not provide insights on how VLMs integrate modalities. Further, recent works have shown that creating corrupt runs by applying Gaussian noise to token embeddings can produce illusory results Zhang and Nanda (2024). Zhang and Nanda (2024) advocate for symmetric token replacement (STR) that swaps tokens in a clean prompt with semantically related tokens to create a corrupt run as it keeps the corrupted run within distribution and provides more reliable insights Current approaches to image corruption lack an equivalent method to STR that does not use Gaussian noise for image corruption, leaving a gap in reliable multimodal analysis.

We propose Semantic Image Pairs (SIP) corruption, which leverages existing datasets to create clean and corrupt images that vary by one semantic property (Figure 2). For instance, in the SVO-Probes dataset (Hendricks and Nematzadeh, 2021), subject-verb-object triplets are already curated to vary by just one element (e.g., ’puppy, lying, grass’ vs. ’goat, lying, grass’), allowing us to create “clean” and “corrupt” image pairs from the existing image labels (Figure 2 panel 1). SIP is not limited to curated datasets and can be extended to any image dataset with easily interchangeable attributes, such as cars, shapes, animals, etc. Further, we show in Section 4.1 that SIP corruption can be performed using generative models to create corrupt images.

Our framework, NOTICE (Gaussian-NOise-free Text-Image Corruption and Evaluation), is a mechanistic interpretability pipeline for VLMs. NOTICE uses our novel SIP method to corrupt images and STR to corrupt text. Using NOTICE, we uncover “universal cross-attention heads” in BLIP and “universal self-attention heads” in LLaVA that consistently produce large patching effects across all tasks. We show that universal cross-attention heads implement human-interpretable functions categorized into three classes: object detection, object suppression, and outlier suppression. We find that both self-attention and cross-attention heads implement outlier suppression. However, only cross-attention heads in BLIP perform the image-grounding functions of object detection and object suppression, highlighting key differences in how VLM architectures handle multimodal learning.

Multimodal Interpretability. Despite the successes of VLMs, the role of attention in multimodal integration remains under-studied Xu et al. (2023). Most works either design probing tasks to identify what vision-language concepts models struggle to capture (Shekhar et al., 2017a; Hendricks and Nematzadeh, 2021; Lu et al., 2023) or assess whether models rely more on one modality over the other Cao et al. (2020); Salin et al. (2022); Hessel and Lee (2020). Probing tasks evaluate whether a model encodes a concept by training a linear classifier on its hidden states. Salin et al. (2022) create visual, text, and multimodal probing tasks to show that models heavily rely on text priors and struggle with object, size, and count concepts. Gradient-based methods analyze gradients to understand how changes in input features influence outputs, assigning importance scores to features and providing insights into the decision-making process. Liang et al. (2023) use second-order gradients, building on LIME Ribeiro et al. (2016) and Shapley values Merrick and Taly (2020), to disentangle cross-modal interactions and determine how much each modality contributes to decisions. While these works have provided insights into multimodal models, recent research shows that 1) probing methods can produce misleading results (Belinkov, 2022) and 2) gradient-based methods for feature attributions can fail to predict behavior (Bilodeau et al., 2024). Instead, Giulianelli et al. (2021); Conneau et al. (2018) demonstrate that causal approaches are more informative than probing representations.

Multimodal models typically integrate their inputs through one of two main approaches: (1) early fusion Liu et al. (2024); Li et al. (2019), where self-attention processes combined image patches and text representations, or (2) cross-attention fusion Tan and Bansal (2019); Lu et al. (2019), where queries from one modality interact with representations from the other. Frank et al. (2021) investigate cross-modal integration using input ablation techniques and conclude that self-attention models, such as VisualBERT, show less sensitivity to missing visual input compared to cross-attention models like ViLBERT, which exhibit more significant reliance on cross-modal interactions.Ilinykh and Dobnik (2022b) finds that vision-text cross-attention learns visual grounding of noun phrases into objects, while text-to-text attention captures low-level syntactic word relations.Parcalabescu et al. (2020) probes dual-stream VLMs through tasks like image-sentence alignment and counting, revealing strong alignment performance but limited grounding of visual symbols, resulting in poor counting capabilities. However, there has yet to be a comprehensive study applying mechanistic interpretability techniques to investigate how recent architectures like BLIP and LLaVA 1) use attention mechanisms to integrate multimodal information and 2) determine the specific functions that attention heads in these models learn to perform.

Causal Mediation Analysis and Mechanistic Interpretability. Causal mediation analysis (CMA) uncovers how interventions in one part of a system cause changes in another. Recent works have adapted CMA to study language models by using activation patching Meng et al. (2023) to identify causal pathways, or circuits, responsible for certain behaviors in the model. Activation patching shows the indirect effect of a component on a model’s output by corrupting key input tokens and then restoring outputs by patching the internal modules from the clean run to the corrupted run. This approach has revealed that attention heads in transformers implement human-interpretable algorithms for “greater than” tasks Hanna et al. (2023), indirect object identification Wang et al. (2022), and learning mathematical group operations (Chughtai et al., 2023). Evidence suggests that models implement similar circuits across various tasks, indicating that studying a small set of computational units may provide insights into model behavior (Gould et al., 2023; Merullo et al., 2024). Although works in mechanistic interpretability have provided critical insights into LLMs, works applying activation patching to multimodal models have been limited due to the absence of an effective semantic-based method for image corruption. Given that Zhang and Nanda (2024) demonstrate Gaussian noise corruption can lead to “illusory” patching results, a semantic-based corruption scheme for images is needed to apply causal mediation analysis to VLMs. In this work, we introduce a semantic-based image corruption method (SIP) and token replacement (STR) for text, enabling meaningful causal mediation analysis to reveal distinct roles of self-attention and cross-attention in multimodal models like BLIP and LLAVA, advancing our understanding of vision-language integration.

NOTICE: Overview The first step in activation patching is developing a method to corrupt the input, preventing the model from predicting the correct answer without causing representations to be out-of-distribution. We propose the first semantic-based dual corruption scheme for VLMs to enhance understanding of each modality’s contribution to VQA tasks. We assess the impact of corrupting input text with symmetric token replacement Wang et al. (2022) and images using our novel Semantic Image Pairs (SIP) framework.

Text Corruption: Symmetric Token Replacement. Corrupting text inputs with symmetric token replacement (STR) swaps out the correct answer token of a prompt with another similar token that has the same length Wang et al. (2022). For indirect object identification, Wang et al. (2022) swap the names of the indirect and direct objects with randomly sampled names. The goal of STR is to corrupt inputs so the language model cannot complete the task correctly while avoiding Gaussian noise corruption that pushes token embeddings off-distribution. For our VQA tasks, we swap both multiple-choice options in the input prompt with two randomly sampled incorrect multiple-choice options with the same tokenized length from a different sample in the dataset. Figure 2 illustrates how STR alters the input text for a prompt.

Image Corruption: Semantic Image Pairs. While STR is a widely used text corruption method, extending semantic corruption to the vision domain presents significant challenges as it requires a semantically aligned image pair. To address this, we propose SIP corruption, a visual corollary to STR, which modifies only one semantic concept in an image (see Figure 2). We implement SIP in three diverse image-classification datasets: SVO-Probes (Hendricks and Nematzadeh, 2021) (an extension to FOIL it! Shekhar et al. (2017b)), MIT States (Isola et al., 2015), and Facial Expressions (Goodfellow et al., 2013). While natural variation in backgrounds can happen between semantic image pairs, we select datasets where nouns, subjects, verbs, and objects are known and can be controlled. The Facial Expressions dataset was curated to include only black-and-white photos of faces expressing different emotions. In MIT-States, object-state pairs are known and can be used to maintain objects (e.g., “building”) while varying their states (e.g., “modern”). The SVO-Probes dataset is designed to vary only one element (subject, verb, or object) at a time (e.g., only varying “dog” and “goat”, while maintaining “grass” and “lying” constant, as seen in Figure 2 panel 1). Thus, for each dataset, we collect the known elements (emotions, objects/states, subjects/verbs/objects) and create image pairs that differ on that element. Our framework can be extended to any image dataset with clearly defined attributes, such as cars, shapes, animals, etc., using the class labels to find clean and corrupt pairs.

Previous works have shown vision transformers can correctly identify objects in various contexts regardless of the foreground or background Vilas et al. (2024). Since image encoders in multimodal models, such as BLIP and LLaVA, are robust to minor variations to the context of an object, only the semantic contents of the image need to be controlled to obtain reliable patching results Vilas et al. (2024).In contrast, masking pixels, adding Gaussian noise to image patches, or replacing pixels would create unnatural images, leading to unreliable results. In Section 4.1, we demonstrate the effectiveness of SIP corruption over Gaussian-noise image corruption and experiment with using generative models to create SIP.

NOTICE VQA Task Design. We adapt the SVO-Probes, MIT States, and Facial Expressions image-classification datasets into VQA tasks. We prompt models to choose between a correct answer that matches the image and an incorrect, semantically different alternative, as seen in Figure 2. For example, using the image-class labels “happy” and “angry,” we create a VQA prompt: “Is this person feeling happy or angry?”. For LLaVA, we adapt the standard prompting format to ensure the model selects one of the multiple choice options: “USER: <image> [Prompt]. Answer with one word. Assistant:”. While in SVO-Probes, each subject, verb, or object comes with its own semantically different counterpart, we constructed such pairs for MIT-States and Facial Expression. In MIT-States, we separated states into color, shape, and texture and selected counterparts from the respective categories for each question. We grouped similar states (e.g., “huge”, “large”) together and ensured that the selected counterpart would not be similar in meaning to the correct answer (using cosine similarity and word2vec word embeddings Mikolov et al. (2013)). Similarly, in the Facial Expressions dataset, we grouped the seven emotions into positive, negative, and neutral and selected counterparts from opposing emotions. We include more details on the VQA design and the datasets in Appendix A and Appendix B, respectively.

Activation Patching. Let $M$ be a vision-language model and $M_{l}$ its hidden states at layer $l\in\{0,1,...,n\}$. The forward pass on an image ($I$) and text ($T$) pair at layer $l$ is denoted as $M_{l}(I,T)$. We define $I^{*}$ and $T^{*}$ as a corrupted image and text. Note that we only corrupt one modality at a time. In a forward pass with $I^{*}$ and $T$, activation patching replaces the hidden states of the corrupted run $M_{l}(I^{*},T)$ with the clean run $M_{l}(I,T)$ to determine which components have the most significant indirect effect on the output. The hidden states at all subsequent layers, $M^{{}^{\prime}}_{j}(I^{*},T)$ for $j\in\{l+1,...,n\}$, are called patched states. In Figure 3, we provide an example illustrating patching using SIP image corruption. The image of the puppy is the clean image, $I$, and the goat is $I^{*}$. Patching the correct answer token “puppy” at $M_{l}(I,T)$, from the clean image to the “puppy” token at $M(I^{*},T)$ creates the patched states $M^{{}^{\prime}}(I^{*},T)$ shown as orange diamonds in Figure 3.

Measuring Patching Effects. The most common ways of measuring patching effect are restoration probability and logit difference. Let $L^{*}$ and $L^{{}^{\prime}}$ denote the logits from the corrupted and patched runs, and let $\tau,\tau^{\text{inc}}$ denote the correct/incorrect multiple-choice answer. Logit difference measures the indirect impact patching has on the model’s logits. Let $L(\tau,\tau^{\text{inc}})=\text{Logit}(\tau)\text{-}\text{Logit}(\tau^{\text{inc}})$. The logit difference is defined as $L^{{}^{\prime}}(\tau,\tau^{\text{inc}})\text{-}L^{*}(\tau,\tau^{\text{inc}})$. Restoration probability measures the impact of patching clean states into a corrupt run on predicting the correct answer token: $P^{\text{'}}(\tau)\text{-}P^{*}(\tau)$.

We begin our investigation by conducting module-wise activation patching to understand the global patterns of self-attention, cross-attention, and MLP modules under both image and text corruption. We demonstrate that SIP image corruption produces results closely aligned with STR text corruption and effectively highlights the role of middle-layer MLPs, which have repeatedly proven crucial in a wide range of tasks Meng et al. (2023). Building on this, we perform a fine-grained analysis of cross-attention heads in BLIP and self-attention heads in LLaVA, the critical point of multimodal integration for each model. As we identify the most significant cross-attention heads in BLIP and most significant self-attention heads in LLaVA, we categorize them into three distinct functional roles, offering deeper insights into their contributions and behaviors.

In this work, we study the roles of cross-attention and self-attention in multimodal integration. To accomplish this, we introduce NOTICE, a Gaussian-Noise-free Text-Image Corruption and Evaluation pipeline for mechanistic interpretability in VLMs. Using our SIP image corruption framework and STR text corruption, NOTICE enables causal mediation analysis across both modalities. Experiments across three tasks reveal “universal attention heads” with consistent patching effects across BLIP and LLaVA. We show that cross-attention supports three functions: object detection, object suppression, and outlier suppression. However, self-attention primarily handles outlier suppression, indicating cross-attention’s unique role in image grounding. While attention presents a plausible, but not necessarily faithful rationale for model prediction Wiegreffe and Pinter (2019), our findings show patterns of attention heads that consistently, and causally, influence model logits. With the support NOTICE, applications using VLMs can benefit from more transparent and adaptable models, facilitating deeper insights and enhanced performance.

The NOTICE framework, combined with SIP corruption, offers valuable insights into the interpretability of vision-language models. In this paper, we applied our dual corruption scheme to three diverse datasets and experimented with generative models for image corruption. However, many more datasets remain unexplored. For instance, shape-based datasets like ShapeWorld Kuhnle and Copestake (2017) and CLEVR Johnson et al. (2017) allow variations in color and shape for corruption. Similarly, icon datasets like IconQA Lu et al. (2021) and Emojis Kralj Novak et al. (2015), as well as specialized datasets such as DeepFashion Liu et al. (2016), Caltech-UCSD Birds-200-2011 Wah et al. (2011), and the Stanford Cars dataset Kramberger and Potočnik (2020), can be turned into “clean” and “corrupt” pairs. In future work, we will expand our experiments to include a wider range of datasets and models, further enhancing the robustness and generalizability of our findings.