Enhancing Multimodal Query Representation via Visual Dialogues for End-to-End Knowledge Retrieval

Fine-grained Late-interaction. To preserve abundant information of knowledge, we employ token-level embeddings for both modalities, applying the MaxSim operation (Khattab and Zaharia 2020). Given the representation of a multimodal query $Q$ and a document $D$, we estimate the relevance score $r_{Q,D}$ between $Q$ and $D$ via late interaction between token-level embeddings from the multimodal retriever $\mathcal{R}$ as follows:

$$r_{Q,D}=\sum_{i=1}^{l_{Q}}\max_{j=1}^{l_{D}}(\lVert E_{Q_{i}}\rVert_{2}\cdot\lVert E_{D_{j}}^{T}\rVert_{2})$$

(1)

where $E_{Q}\in\mathbb{R}^{l_{Q}\times d}$ and $E_{D}\in\mathbb{R}^{l_{D}\times d}$ denote the representations of $Q$ and $D$ embedded by the retriever $\mathcal{R}$, with each embedding having a dimension of $d$. Here, $l_{Q}$ and $l_{D}$ represent the number of embeddings for $Q$ and $D$, respectively. This operation chooses the highest relevance scores over all document tokens for each query token. In our approach, we incorporate the late-interaction mechanism for pre-indexing documents, ensuring no interaction between the query and the documents during the encoding process.

Visual Embeddings for Multimodal Queries. Building upon previous approaches (Alayrac et al. 2022; Liu et al. 2023) that integrate visual comprehension into language models, Ret-XKnow adopts the Vision Transformer (ViT) (Dosovitskiy et al. 2021) as its vision encoder. We utilize token-level visual embeddings $F_{v}$ from the penultimate layer of the ViT, along with a global visual embedding $F_{g}$ obtained from a special token (i.e., CLS token). For the global embedding $F_{g}\in\mathbb{R}^{d_{v}}$, we directly project the visual features into the latent space of the text retriever $\mathcal{R}_{T}$ via a two-layer perceptron to align the vision and text modalities. Subsequently, the projected $F_{g}$ is reshaped into token-level embeddings $E_{g}\in\mathbb{R}^{l_{g}\times d}$, where $l_{g}$ is the pre-defined number of tokens. For token-level visual embeddings $F_{v}\in\mathbb{R}^{l_{v}\times d_{v}}$, each embedding corresponds to distinct visual elements or regions within the input image and involves similar semantics among adjacent patches. Thus, we can acquire regions of interest without region proposal networks since the visual embeddings $F_{v}$ encompass granular visual information. We aim to extract visual information pertinent to the textual query $T$ while diminishing the information on unrelated aspects, with rich visual representations. To achieve modality interaction, we employ the MaxSim operation and the partial convolution mechanism (Liu et al. 2018). The concept of partial convolutions was originally designed for image inpainting tasks to fill missing or unwanted pixels by using the surrounding pixel information. In the context of Ret-XKnow, this mechanism is adeptly repurposed to refine visual embeddings by filling the irrelevant embeddings with adjacent visual features. The visual embeddings $F_{v}$ first undergo a projection via a two-layer perceptron, which maps $F_{v}$ into a space of $\mathcal{R}_{T}$, i.e., $F_{v}\in\mathbb{R}^{l_{v}\times d_{v}}\to F_{vt}\in\mathbb{R}^{l_{v}\times d_{t}}$, where $d_{t}$ denotes the dimension of embeddings $F_{t}$ of $T$ from the penultimate layer of $\mathcal{R}_{T}$. Then, we assign a relevance score for $F_{t}$ to each visual embedding in the projected space as follows:

$$r_{I,T}=\sum_{i=1}^{l_{t}}\max_{j=1}^{l_{v}}(\lVert F_{vt_{i}}\rVert_{2}\cdot\lVert F_{t_{j}}^{T}\rVert_{2}),$$

(2)

where $l_{t}$ denotes the length of the textual query. To align the visual embeddings $F_{v}$ and their relevance scores in a spatial arrangement, $F_{v}$ and $r_{I,T}$ are reshaped to have spatial dimensions $p\times p$, resulting in $F^{\prime}_{v}\in\mathbb{R}^{p\times p\times d_{v}}$ and $r^{\prime}_{I,T}\in\mathbb{R}^{p\times p\times 1}$. The reshaped $F^{\prime}_{v}$ and $r^{\prime}_{I,T}$ are then concatenated along the channel dimension to form a combined feature map. We subject this feature map to a strided convolution operation $C$, defined as:

$$C([F^{\prime}_{v};r^{\prime}_{I,T}])=\sum_{m=1}^{k}{\sum_{n=1}^{k}{([F^{\prime}_{v};r^{\prime}_{I,T}]_{i+2m,j+2n}\cdot W_{mn}+B)}},$$

(3)

where $W_{mn}$ and $B$ denote the weights and bias of the convolutional filter, respectively. The convolution is applied with a stride that effectively downsamples the feature map, extracting and condensing the most salient visual features based on their relevance to the textual query. The output of this convolution $C([F^{\prime}_{v};r^{\prime}_{I,T}])$ yields a reduced set of selective visual features that are highly relevant to the corresponding textual content, represented as $F^{\prime}_{v}\in\mathbb{R}^{p^{\prime}\times p^{\prime}\times 2d_{v}}$, where $p^{\prime}$ is the reduced spatial dimension post-convolution. The $F^{\prime}_{v}$ is reshaped to embeddings with 4 tokens per embedding after being projected into $4d$ dimensions by a linear layer. Finally, we integrate the token-level text embeddings with two types of token-level visual embeddings to form the query embeddings $E_{Q}$ as follows:

$$E_{Q}=[E_{g};E_{v};E_{t}]$$

(4)

where $E_{t}$ and $E_{v}$ represent the final embeddings obtained by applying a linear layer to $F_{t}$ and the output of the convolution layer, respectively, to achieve embeddings with dimension $d$. Finally, we compute a relevance score $r_{Q,D}$ between $E_{Q}$ and $E_{D}$ with the MaxSim operation.

Pre-processing. First, we divide the dialogues into individual turns. Subsequently, to maintain informative content within the dataset, we filter out turns that are unsuitable for retrieval tasks based on responses and tasks given by queries. We exclude tasks that do not contribute to knowledge-based retrieval, such as queries requiring or providing the location of objects, which are unrelated to knowledge content. To reduce bias in training, responses containing simple affirmations or negations, such as “yes” or “no,” are edited to remove these elements.

Neural Filtering. To guarantee that the model learns to utilize visual information in multimodal retrieval, we apply neural filtering to our construction process. Using a text retriever, we perform the top-5 retrieval with questions from a knowledge base compiled with the responses in the visual dialogue datasets. Through this process, we automatically identify responses that can be retrieved based solely on textual queries. To avoid impeding the learning of image representations for retrieval, we filter out the matched query-response pairs from the text retrieval process.

Response-to-Passage Conversion. In the context of retrieval, passages may contain both relevant and irrelevant information, unlike responses in dialogues. Thus, we transform responses into passages typically found in knowledge retrieval tasks by unifying passages related to multimodal queries. To identify the relevant passages, we utilize the responses instead of the queries since the responses contain image-related information conditioned on the given query and image while the textual queries have restricted information. We obtain top-3 passages retrieved from Wikipedia using the responses as textual queries. However, the text retriever may retrieve inappropriate passages due to potential inaccuracies in the retriever. To ensure that converted passages are relevant to both the image and the question, we combine three retrieved passages with the responses. We simply concatenate the response behind the top-1 passage, thereby maintaining the relevance of the context.

ViD2R. Our pre-training dataset is synthesized from two visual dialogue datasets, as detailed in works by (Liu et al. 2023) and (Wang et al. 2023). These datasets merge tasks necessitating image comprehensions for instruction-following tuning (Ouyang et al. 2022). After the preprocessing stage, we yielded 1.35 million QA pairs, each associated with a single image. Subsequent neural filtering, employing ColBERTv2 (Santhanam et al. 2022b) trained on the MS MARCO Passage Ranking task (Nguyen et al. 2016), refined the preprocessed pairs to 0.98 million QA pairs including queries that require visual context for accurate retrieval. In the final stage, we use 6 million Wikipedia released in (Chen et al. 2023) as our data pool to convert responses to passages.

Ret-XKnow. We adopt CLIP ViT-base (Radford et al. 2021) as our vision encoder, alongside ColBERTv2 (Santhanam et al. 2022b), rooted on BERT-base (Devlin et al. 2019), serving as our text retriever. We reduce the number of visual embeddings $F_{v}$ to 16 embeddings via $C(\cdot)$ with 5 of kernel size. $F_{g}$ is converted into 16 embeddings via projection using two-layer perception. The dimension $d$ of final embeddings $E_{Q}$ and $E_{D}$ is set to 128.

Baselines. To demonstrate the effectiveness of our end-to-end multimodal retrieval framework, we compare our approach against the advancements including the following baselines. ColBERTv2 (Santhanam et al. 2022b), a text-based retrieval model that employs a fine-grained late-interaction mechanism. In experiments, this model retrieves relevant passages using only textual queries. FLMR+WiT (Lin et al. 2023), a multimodal retriever that incorporates external visual information. They use vision and text encoders identical to our Ret-XKnow. Their pre-training on a subset of the WiT dataset (Srinivasan et al. 2021) aims to align visual embeddings with the linguistic space of the text retriever. ReViz+VL-ICT (Luo et al. 2023), an end-to-end knowledge retrieval model for multimodal queries, which uses pre-trained ViLT and BERT-base as its query and document encoders, respectively. With ViLT optimized for the image size of $384\times 384$, their model introduces a pre-training strategy, VL-ICT. In the absence of publicly available code and data, we reconstructed the model and dataset based on descriptions in their publication, resulting in a VL-ICT dataset of 2,997,354 samples. PreFLMR (Lin et al. 2024), an end-to-end multimodal knowledge retrieval model, which utilizes a cross-attention mechanism with the outputs of the penultimate layer for fine-grained modality interaction. Unlike Ret-XKnow, their architecture directly fuses the text features with visual features.

Results. In Tab. 1, our approach achieves superior zero-shot retrieval performance across four datasets, significantly outperforming baseline models. Despite utilizing smaller image sizes and a smaller pre-training dataset than ReViz, our method significantly outperforms baselines across all retrieval metrics under zero-shot settings in an end-to-end manner. FLMR, which utilizes the same foundation models, showed performance degradation compared to the base text retriever even after pre-training with WiT, unlike our retriever. Note that we concatenated the aligned visual features with the final embeddings of the text retriever during the inference of FLMR. Despite using the same pre-training dataset, PreFLMR underperforms compared to our RetXKnow on state-of-the-art multimodal retrieval benchmarks due to clear differences in the modality interaction approach. Furthermore, Tab. 2 showcases that our pre-training dataset requires unifying visual information to textual query in contrast to the previous approach, resulting in improving zero-shot performance.

Results. The results from our experiments, detailed in Tab. 3, clearly demonstrate the efficacy of our approach. First, Ret-XKnow outperforms other baseline models on key metrics across knowledge bases of different sizes, even before any specialized pre-training. This performance advantage is meaningfully extended by applying transfer learning on the Ret-XKnow model pre-trained with our ViD2R dataset. Moreover, our investigations reveal that our end-to-end retriever reaches performance levels comparable with the baseline that utilizes caption. As shown in Tab. 4, we present the performance of models fine-tuned with image captions within the OK-VQA (GS) dataset. Ret-XKnow approximates these achievements without employing captions in Tab. 3. Such outcomes underscore the effectiveness of Ret-XKnow in enhancing multimodal query representations in an end-to-end manner.