Zero-Shot Interactive Text-to-Image Retrieval via Diffusion-Augmented Representations

Before Diffusion Models (DMs), the most successful visual generative models were Generative Adversarial Networks (GANs) (Goodfellow et al., 2020, 2014; Creswell et al., 2018), which introduced adversarial training for image synthesis (Goodfellow et al., 2020). While GANs achieved notable success in tasks like image generation and style transfer (Goodfellow et al., 2020), they suffer from limitations such as mode collapse, training instability, and the need for carefully tuned architectures, making them less robust for diverse tasks (Gui et al., 2020).

DMs address these challenges by using a noise-adding and noise-removal process, enabling stable and high-quality generation (Croitoru et al., 2023). Initial approaches like Denoising Diffusion Probabilistic Models laid the groundwork (Ho et al., 2020), followed by advancements such as Stable Diffusion (Rombach et al., 2022) and Imagen (Saharia et al., 2022), which enhance efficiency and scalability (Croitoru et al., 2023; Saharia et al., 2022). Diffusion models excel in generating high-fidelity, diverse outputs without mode collapse and are adaptable to multimodal tasks like text-to-image generation (Ramesh et al., 2021). In this work, we aim to leverage the prior knowledge of DMs to bridge the domain gap between the pretraining tasks of Multimodal Large Language Models (MLLMs) and retrieval objectives, achieving superior zero-shot performance and better handling of unseen samples.

Interactive text-to-image retrieval (I-TIR) overcomes the limitations of conventional single-turn methods by iteratively clarifying a user’s information need across multiple dialog turns. This iterative process is especially effective for image retrieval, as a single initial query often fails to capture the fine details of target images. By incorporating user feedback over successive turns, I-TIR progressively aligns with user preferences, improving both retrieval accuracy and user satisfaction. However, this flexibility also adds complexity; the representation of a user’s information need becomes more elaborate due to the diversity of multi-turn dialogs.

Recent studies show the potential of LLMs and MLLMs for I-TIR. For instance, ChatIR (Levy et al., 2023) employs LLMs to simulate dialogues between users and answer bots, compensating for the scarcity of specialized text-to-image datasets tailored to I-TIR. Although this approach opens a promising direction for research, it does not address the unique training hurdles posed by I-TIR—specifically, handling highly diverse dialogue inputs. PlugIR (Lee et al., 2024) further advances I-TIR by improving the diversity of top-k results using $k$-means clustering to group similar candidate images and identify representative exemplars. However, this technique incurs additional computational overhead. Moreover, ChatIR and PlugIR, as well as other I-TIR strategies, rely on fine-tuning on small, curated datasets to achieve good results on a specific benchmark (Zhu et al., 2024; Li et al., 2024; Yoon et al., 2024), limiting their ability to generalize to the broad range of real-world dialogues. Consequently, they often fail on out-of-distribution queries, leading to diminished performance in practical settings, as shown in Section 4.3.

To address these challenges, we propose the DAR framework, which prioritizes zero-shot I-TIR performance. By avoiding dataset-specific fine-tuning altogether, DAR avoids the reduced distribution space introduced by smaller, finetuned datasets, thereby achieving superior generalizability.

Interactive text-to-image retrieval is formulated as a multi-turn task that begins with an initial user-provided description, $D_{0}$. The objective is to identify a target image through an iterative dialogue between the user and the retrieval framework. At each turn $t$, the retrieval framework generates a question $Q_{t}$ to clarify the search, and the user responds with an answer $A_{t}$. This interaction updates the dialogue context $C_{t}=(D_{0},Q_{1},A_{1},\ldots,Q_{t},A_{t})$, which is processed—such as by concatenating all textual elements—to form a unified search query $S_{t}$ for that round. The retrieval framework then matches images $I\in\mathcal{I}$ in the image database against $S_{t}$, ranking them based on a similarity score $s(I,S_{t})$. This process iterates until the target image $I^{*}$ is successfully retrieved or the maximum number of turns is reached. Formally, this process can be defined as:

where $S_{T}$ is the final search query after $T$ dialogue turns.

While multi-turn interactions help capture user intent, raw dialogue data can introduce noise and complexity that degrades retrieval performance. Both encoders and diffusion models struggle with lengthy or ambiguous dialogue context, particularly because DMs have limited capacity for long or complex descriptions. Nevertheless, the quality of images generated by DMs is crucial for DAR to achieve superior performance. Moreover, discrepancies between the training distributions of encoders and DMs make it difficult to generate images that accurately reflect user intentions. To address these challenges, we propose two targeted approaches: refining the dialogue for textual representations used in retrieval; and optimizing the prompts for the DM generation process:

1. (1)

   Dialogue Context Aware Reformulation: This pipeline adapts the dialogue context to better align with the input expectations of encoders and the user’s intent. Instead of directly using the raw dialogue context $C_{t}=\{D_{0},Q_{1},A_{1},\ldots,Q_{t},A_{t}\}$ as textual representations, we follow a multi-step process to refine the input.

   1. (a)

      Summarizing the Dialogue: We first ask an LLM to summarize the entire dialogue context $C_{t}$, providing a coherent and concise overview of the conversation up to turn $t$.

   2. (b)

      Structuring the Input: The summarized dialogue is then reformulated into a specific format that clearly distinguishes the initial query ($D_{0}$) from the subsequent elaborations in the dialogue. This ensures that the LLM understands the progression of the conversation and the relevance of each turn.

   3. (c)

      Generating the Refined Query: Using this structured input, we prompt the LLM to generate a refined query $S_{t}$ that adheres to the encoders’ input distribution. This ensures the generated query captures the user’s intent in a way that facilitates accurate retrieval.

   The reformulation process can be expressed as:

   $$S_{t}=\mathcal{R}_{1}(C_{t})$$

   where $\mathcal{R}_{1}$ denotes the reformulation function utilizing LLMs. An example prompt for $\mathcal{R}_{1}$ is: "The reconstructed [New Query] should be concise and in an appropriate format to retrieve a target image from a pool of candidate images." This approach allows the dialogue context to be transformed into a more structured and relevant form for the retrieval pipeline, optimizing the alignment between user intent and the model’s output.

2. (2)

   Diffusion Prompt Reformulation: This pipeline generates multiple prompts $P_{t,k}$ for use by the subsequent diffusion models based on the reformulated dialogue $S_{t}$. By producing diverse prompts, we ensure that the generated images align with the diffusion model’s training distribution, capturing various linguistic patterns and semantic nuances. The reformulation process follows these steps:

   1. (a)

      Structuring the Prompt Template: We begin by structuring a prompt template that captures key elements from the reformulated dialogue $S_{t}$, including the primary subject, setting, and important details. This structured format helps guide the diffusion model to generate images that are semantically coherent with the user’s intent.

   2. (b)

      Generating Diverse Prompts: Using the structured template, we generate multiple distinct prompts $P_{t,k}$ by varying linguistic patterns, modifiers, and details, ensuring a variety of interpretations that reflect the full scope of the dialogue. This diversity helps cover different possible details in the image generation process.

   3. (c)

      Adapting to the DM’s Distribution: The generated prompts are further adjusted to align with the diffusion model’s training distribution. This adaptation ensures that the prompts match the model’s expectations and improve the relevance of the generated images.

   The reformulation process is expressed as:

   $$P_{t,k}=\mathcal{R}_{2}(S_{t},k)\quad\text{for }k=1,2,\ldots,K$$

   where $\mathcal{R}_{2}$ denotes the prompt generation function, and $K$ is the number of prompts generated per turn. An example template prompt for $\mathcal{R}_{2}$ is: “[Adjective] [Primary Subject] in [Setting], [Key Details]. Style: photorealistic."

   By generating diverse prompts, this approach ensures that the diffusion models produce images that act as multiple intermediate representations of the user’s information need, which are both semantically rich and better aligned with the user’s query.

Overall, since we focus on zero-shot scenarios where retrieval models are not finetuned on a particular domain or dataset, reformulating the dialog can supply additional context or details. This helps bridge the semantic gap between the user’s language and the system’s learned representation space, boosting performance without further training. Code for these reformulation pipelines is available at https://anonymous.4open.science/r/Diffusion-Augmented-Retrieval-7EF1/README.md.

Following the reformulation step, we obtain a refined textual dialog for retrieval (Section 3.2) and multiple prompts that capture different aspects of the user’s intent. We now proceed to the imagine step, where these prompts are used to generate images that augment the retrieval process. We term this approach Diffusion Augmented Retrieval (DAR).

The core idea of DAR involves utilizing DMs to generate synthetic images that serve as multiple intermediate representations of the user’s information needs, thereby enhancing the retrieval process. DMs excel at producing high-quality, visually realistic images from textual descriptions, enabling them to closely align with the user’s intent. By leveraging the prior visual knowledge embedded in DMs through image generation, DAR establishes many-to-one mappings between queries and target images instead of one-to-one mappings. This multi-faceted representation of queries addresses the challenges posed by incomplete or ambiguous textual inputs, offering a richer and more diverse set of visual representations for retrieval. Consequently, DAR achieves robust zero-shot performance, particularly in I-TIR scenarios where labeled data is scarce.

Specifically, for text-guided diffusion generation, the reverse process is conditioned on a text embedding $\mathbf{t}_{d}$ from the diffusion text encoder:

where the function $\bm{\mu}_{\theta}(x_{t},t,\mathbf{t}_{d})$ represents the learned denoiser, which predicts the mean of the posterior distribution for $x_{t-1}$ given the noisy input $x_{t}$, timestep $t$, and additional context $\mathbf{t}_{d}$.

Let

be the set of $K$ diffusion-ready prompts produced by the reformulation process. We feed each prompt $P_{t,k}$ into the diffusion model $D(\cdot)$ to generate a corresponding set of images:

Each generated image $I_{t,k}$ reflects one possible interpretation of the user’s intent based on the prompt $P_{t,k}$. By leveraging the large-scale pretraining of the diffusion model and generating multiple images per turn, DAR captures diverse visual features relevant to the query.

We evaluate our proposed DAR framework in interactive retrieval settings using four well-established benchmarks. Specifically, we employ the validation set of Visual Dialog (VisDial) (Das et al., 2019) dataset and three dialog datasets¹¹1Available at: https://github.com/levymsn/ChatIR constructed by (Levy et al., 2023), named ChatGPT_BLIP2, HUMAN_BLIP2, and Flan-Alpaca-XXL_BLIP2. For the latter three dialog datasets, the first part of each name indicates the questioner model used to generate the dataset, and the second part refers to the answer model. Note that human refers to professional human assessors. Further details on these three datasets can be found in (Levy et al., 2023). All four dialog datasets consist of 2,064 dialogues, each with 10 dialogue turns.

Following previous work in the interactive cross-modal retrieval domain, we use BLIP (Li et al., 2022) as our default MLLM encoder, given its established zero-shot performance and to ensure a fair comparison with prior studies. Unless otherwise specified, we report Hits@10 as our primary evaluation metric.

We adopt the Stable Diffusion 3 model (SD3) (Esser et al., 2024) as the default diffusion model (DM) in our experiments. Additionally, BLIP-3 (Xue et al., 2024) is employed as the Large Language Model (LLM) for reformulating textual content. Using specially designed prompts, BLIP-3 operates under two distinct reformulation pipelines: one for adapting the dialogue and another for producing aligned prompts for the DM, as described in Section 3.2. We empirically set the weighting factors for textual and visual content to $0.7$ and $0.3$, respectively, for the first two dialogue turns. Starting from turn 3, the weights are both set to $0.5$. This strategy balances the influence of textual vs. visual embeddings as the dialogue becomes more dynamic. In all experiments, we fix the number of generated images per turn at three. Code for all experiments is available at https://anonymous.4open.science/r/Diffusion-Augmented-Retrieval-7EF1/README.md.

Baselines and DAR variants. We compare our proposed DAR framework with three baselines, namely ChatIR, ZS, and COCOFT:

• •

  ChatIR: We adopt the BLIP-based variant of ChatIR (Levy et al., 2023) as our baseline. Since it is finetuned on the Visual Dialog dataset, ChatIR represents a finetuned model that helps us compare both the effectiveness and efficiency benefits of DAR.

• •

  ZS: The zero-shot (ZS) baseline uses BLIP with its original, publicly available pretrained weights ²²2Available at: https://github.com/salesforce/BLIP, without any fine-tuning on retrieval datasets. This setup captures the common scenario where researchers or practitioners directly rely on publicly released weights without task-specific training.

• •

  COCOFT: The COCOFT baseline denotes the BLIP model finetuned on the popular MSCOCO (Lin et al., 2014) retrieval dataset. Although it benefits from MSCOCO-specific training, it remains non-finetuned for interactive text-to-image retrieval (I-TIR). Consequently, dialogues including questions and answers are applied directly as queries for retrieval. This serves as an indicator of how previously finetuned single-turn retrieval models perform in an interactive retrieval setting.

We integrate each of the three baseline encoders into our DAR framework to evaluate improvements across different scenarios, denoting them as DAR_xxx. For example, DAR_chatir adopts the BLIP-based ChatIR model as the encoder. Notably, comparisons should primarily focus on the corresponding pairs—such as DAR_chatir versus ChatIR—whose lines are represented by the same colors in Figure 3. This is because these pairs are initialized from the same pre-trained weights, ensuring a fair comparison that shows how the addition of DAR affects performance.

We first investigate how our proposed DAR framework performs under zero-shot I-TIR conditions. Specifically, BLIP_zs denotes the baseline where BLIP is used with its original, publicly available pretrained weights—without any fine-tuning on retrieval datasets. Our DAR_zs setup similarly employs pretrianed BLIP as the encoder, thereby reflecting a common scenario in which researchers or practitioners rely solely on publicly released weights.

Each subfigure (a–d) in Figures 3 presents the performance of DAR variants and baseline models across the four benchmarks outlined in Section 4.1, evaluated over multiple conversational turns.

In this section, we compare the dashed blue lines (baseline BLIP_zs) with the solid blue line (DAR_zs) across the four subfigures (a–d) in Figures 3. We can tell that DAR_zs consistently outperforms BLIP_zs across all four evaluated benchmarks. Notably, the FLAN_BLIP2 dataset exhibits the largest improvement, with DAR_zs achieving a 7.61% increase in Hits@10 after 10 dialog rounds. Indeed, the performance gap between DAR_zs and BLIP_zs consistently widens as the dialogue extends over multiple turns.

These findings demonstrate that DAR effectively boosts zero-shot performance in interactive retrieval tasks where our “query" is derived from a complex and diverse dialog, and it can do this without incurring additional tuning overhead.

Since our proposed DAR framework is aimed at tackling the challenges of complex and diverse dialogue queries in I-TIR, evaluating its robustness under such conditions is important. Therefore, we focus our analysis on the most challenging benchmark among the four we evaluated—specifically the one where the best-performing model achieves the lowest Hits@10 score. In this section, we analyze the solid green lines (DAR_chatir) as the best-performing model, comparing them with the dashed lines representing the three baselines across the four subfigures (a–d) in Figures 3.

We observe a similar performance trend across three of the benchmarks, whereas FLAN_BLIP2 differs considerably (see Figure 3.d). On the FLAN_BLIP2 benchmark, three baseline models (dashed lines) without DAR tend to see their performance plateau around the fifth dialogue turn, indicating they cannot effectively leverage additional complexities introduced later in the conversation. In contrast, our DAR framework continues to improve after turn 5, showing no signs of saturation. Additionally, even our best-performing model, (DAR_chatir), achieves the lowest Hits@10 on the FLAN_BLIP2 benchmark. We attribute this to the highly diverse dialogues in FLAN_BLIP2 compared to the other datasets, making it an important test case for evaluating out-of-distribution performance.

This suggests that DAR can better exploit the extra information presented in later dialogue turns and generate images more closely aligned with the user’s target. Therefore, DAR demonstrates greater robustness to complex and diverse dialogue queries, fulfilling the requirements of I-TIR.

In this section, we investigate both the effectiveness and efficiency of the zero-shot, non-finetuned version of our proposed DAR framework, DAR_zs, by comparing it against the finetuned ChatIR baseline. Thus, we compare the solid blue line (DAR_zs) with the dash green line (ChatIR) across the four subfigures (a–d) in Figures 3. Overall, DAR_zs demonstrates competitive performance on all evaluated benchmarks.

As analyzed in Section 4.3, FLAN_BLIP2 is the most challenging dataset in our experiments, owing to its particularly complex and dynamic interactive dialogues. Notably, DAR_zs (solid blue line) outperforms the finetuned ChatIR (dash green line) model by 4.22% in Hits@10 at turn 10 (Figure 3.d), supporting our hypothesis that fine-tuning MLLMs on limited I-TIR datasets can undermine their ability to handle out-of-distribution queries. Finetuned models like ChatIR perform best on the VisDial dataset (see Figure 3.a), given their extensive fine-tuning on 123k VisDial training samples. Even in this favorable setting, the worst-case of non-finetuned DAR model, DAR_zs, lags behind ChatIR by only 3.01% in Hits@10, all while saving 100% of the fine-tuning time.

This suggests that ChatIR’s narrower finetuned distribution makes it more prone to out-of-distribution errors during inference, leading to underperformance. In contrast, DAR excels in these failure scenarios and remains competitive even on ChatIR’s own finetuned dataset.

Our proposed DAR can also be combined with various encoders, including those already finetuned for dialogue-based text. In this context, the main comparison is between DAR_chatir and ChatIR, where DAR_chatir employs the finetuned ChatIR model as its encoder. Thus, we compare the solid green line (DAR_chatir) with the dash green line (ChatIR) across the four subfigures (a–d) in Figures 3.

Notably, as shown in Figures 3, DAR_chatir consistently outperforms ChatIR without additional training. Because ChatIR is finetuned on the VisDial dataset, it holds an obvious advantage in that benchmark. Building upon this strong baseline, DAR_chatir still achieves a 2.42% improvement in Hits@10 over ChatIR on the VisDial dataset.

More importantly, as discussed earlier, finetuned models often struggle with out-of-distribution data in interactive retrieval due to the inherently dynamic nature of multi-turn dialogues. This challenge arises in the other three benchmark datasets we evaluate. Consequently, DAR_chatir exhibits even greater performance gains in these scenarios, achieving up to a 9.4% increase in Hits@10 compared to ChatIR.

These findings support our claim that DAR can enhance zero-shot retrieval performance without further training—particularly in the face of dynamic dialogues in interactive retrieval—even when the underlying encoder is finetuned on a specific dataset.

Given that single-turn and interactive text-to-image retrieval share some similarities at turn 0, this section investigates whether fine-tuning on a single-turn retrieval dataset—specifically, the widely used MSCOCO dataset (Lin et al., 2014)—enhances performance in interactive retrieval tasks.

As shown in all four subfigures of Figure 3, the benefit of fine-tuning on a single-turn retrieval dataset is relatively limited. While finetuned models such as BLIP_cocoft (dash red lines) and DAR_cocoft (solid red lines) exhibit noticeable performance improvements at earlier turns compared to their zero-shot counterparts (BLIP_zs and DAR_zs), this advantage diminishes as the dialogue progresses. By turn 8, the zero-shot model DAR_zs even surpasses the COCO finetuned model DAR_cocoft in performance, as illustrated in Figure 3.d.

These findings show that conventional finetuning strategies on single-turn datasets are insufficient to address the increasing query diversity and complexity inherent in interactive text-to-image retrieval. This underscores the limitations of fine-tuning-based approaches in handling evolving multi-turn interactions. In contrast, our proposed DAR framework effectively bridges this research gap by enhancing zero-shot performance without requiring additional fine-tuning, making it a more scalable and adaptive solution for real-world retrieval scenarios.

Beyond the numerical results presented in Section 4, we conduct an in-depth analysis of the images generated by DAR and uncover several key insights. As illustrated in Figure 4, the initial generated images tend to lack fine details and are easily distinguishable as synthetic. For instance, as the dialogue progresses and additional contextual information is incorporated, the generated images become increasingly photorealistic, with details aligning more closely with the target image—such as the accurate color of the player’s clothing. A similar trend is observed in the second-row example, where the diffusion model dynamically adjusts clothing colors based on dialogue updates, ultimately enhancing retrieval accuracy.

These findings highlight DAR’s ability to iteratively refine image generation based on evolving dialogue cues, enabling more precise semantic alignment with the target image. This adaptive generation process not only improves retrieval performance but also demonstrates the potential of integrating generative models into cross-modal retrieval frameworks.

To assess the compatibility of DAR with different MLLMs as encoders and DMs as visual generators, we evaluate its performance using two widely adopted MLLMs—CLIP (Radford et al., 2021) and BEiT-3 (Wang et al., 2022)—as encoders, as well as Stable Diffusion v2-1 (Rombach et al., 2022) as an alternative visual generator.

The CLIP model, constrained by its maximum input length of 77 tokens, struggles to handle complex and lengthy dialogue-based queries compared to simpler caption-based queries in single-turn cross-modal retrieval tasks. Consequently, CLIP achieves a peak performance of 56.64% Hits@10 on the Visual Dialog benchmark. Despite this limitation, incorporating CLIP as the encoder within DAR still yields an improvement of 5.31% Hits@10 after 10 dialogue turns, demonstrating the robustness of our framework even with models that have constrained input capacity.

BEiT-3, a stronger encoder than CLIP and BLIP models, further enhances performance. When employed as the encoder within DAR, the framework achieves a 6.28% improvement in Hits@10 after 10 turns on the Visual Dialog benchmark, compared to the standalone pre-trained BEiT-3 model, highlighting the adaptability of DAR in leveraging stronger backbone encoders.

In addition to evaluating different encoders, we also assess the impact of using Stable Diffusion v2-1 as the visual generator. While DAR continues to yield notable improvements, the performance gain is slightly lower compared to using Stable Diffusion 3 (SD3), aligning with SD3’s superior generative capabilities. Specifically, DAR achieves a 6.37% Hits@10 with Stable Diffusion v2-1, compared to 7.61% Hits@10 with SD3.

These results demonstrate that DAR is flexible and compatible with various encoder and generation models. Regardless of the choice of encoder or visual generator, DAR consistently enhances retrieval performance, reinforcing its generalizability as an effective framework for zero-shot cross-modal retrieval.

In our evaluation, even when generating only a single image within our framework, we observe a 6.43% improvement in Hits@10 on the FLAN_BLIP2 benchmark compared to the ChatIR baseline. When increasing the number of generated images to three, performance further improves to 7.61%. However, beyond this point, we observe diminishing returns, with Hits@10 reaching saturation as the number of generated images continues to increase.

Considering the trade-off between effectiveness and computational efficiency, we set three generated images as the default configuration in DAR to achieve an balance between performance and inference cost.

While DAR eliminates the need for expensive fine-tuning, it introduces additional inference-time overhead due to the reformulation and generation of visual content. However, this trade-off is small compared with the benefits in retrieval performance and adaptability. In our experiments using a single Nvidia RTX 4090 GPU, the image generation process takes approximately 5 seconds, while the query reformulation step incurs only 0.5 seconds of additional processing time.

To put this into perspective, ChatGPT-o1 and other ‘reasoning’ LLMs use additional reasoning steps, which enhances response quality at the cost of increased inference time (often exceeding 10 seconds). Analogously, DAR achieves substantial gains in zero-shot retrieval while entirely eliminating training costs, making it an efficient and scalable alternative. Consequently, the modest inference overhead is a worthwhile trade-off, particularly in applications where adaptability and retrieval quality are crucial.