VDialogUE: A Unified Evaluation Benchmark
for Visually-grounded Dialogue

To a certain extent, the emergence of unified benchmarks has driven the development of general models and provided a fair platform for subsequent work comparisons. GLUE Wang et al. (2018) and SuperGLUE Wang et al. (2019) have suggested a unified evaluation framework that simplifies the evaluation of pre-trained language models across various tasks. Additionally, DialoGLUE Mehri et al. (2020) and dodecaDialogue Shuster et al. (2019) offer a benchmark designed specifically for assessing dialogue systems.

Moreover, VALUE Cao et al. (2020) proposes probing tasks to delve into the inner workings of vision-language pre-training models, while Li et al. (2021) develops a video-and-language-focused VALUE benchmark. GEM Su et al. (2021) benchmark evaluates multilingual multi-modal models, including image-text retrieval and image captioning. Recently, Zhou et al. (2022) proposed the VLUE benchmark, which is a multi-task and multi-dimensional assessment of vision-language pre-training (VLP) models.

Although progress has been made in multi-modal and dialogue systems, the exploration of a unified multi-modal dialogue system remains relatively unexplored, despite its significant research value. To fill this gap, we suggest a unified benchmark for visually-grounded dialogue evaluation.

In recent years, there has been a proliferation of multi-modal dialogue datasets. VisDial Das et al. (2017) is one such dataset where workers generate conversation about a shared image grounded in corresponding images. IGC Mostafazadeh et al. (2017) is more realistic, but its limited size makes it challenging for model training. Image-Chat Shuster et al. (2018) is a larger dataset consisting of 202K image-grounded dialogues. PhotoChat Zang et al. (2021) is the first human-human multi-modal dialogue dataset from a real conversation scenario, while MMChat Zheng et al. (2021) features a large-scale Chinese multi-modal dialogue. MMDialog Feng et al. (2022) is a million-scale multi-turn dialogue dataset. Multi-modal datasets also include task-oriented conversation datasets, such as MMD Saha et al. (2018) with over 150K fashion domain conversation sessions between shoppers and sales agents, SIMMC_1.0Moon et al. (2020) with 13K dialogues on furniture and fashion domains, SIMMC_2.0Kottur et al. (2021) with more complex interaction scenarios than SIMMC_1.0, and MMConv Liao et al. (2021) with fully annotated role-playing dialogues covering multiple domains and tasks related to traveling scenarios.

Based on the aforementioned multi-modal dialogue datasets, numerous advanced works have been proposed and developed. Some modeling works, such as those by  Niu et al. (2019),  Gan et al. (2019) and Qi et al. (2020) have been conducted to improve the performance of conversational agents in image-grounded dialogue. Besides,  Zang et al. (2021) proposed a dual-encoder model that utilizes object labels to encode image features in order to perform a dialogue-based image retrieval task. Later on, Yang et al. (2021) and Chen et al. (2021) enhanced the textual expressions of generated dialogue responses through associative vision scenes. Zheng et al. (2021) proposed a multi-modal dialogue generation model based on Seq2Seq architecture.  Lee et al. (2022) created a multi-modal encoder-decoder that incorporates visual inputs and performs all sub-tasks via joint learning. More recently,  Sun et al. (2021) proposed the first multi-modal dialogue response generation model that understands multi-modal contexts and produces informative text and image responses. As previously mentioned, existing models have achieved success in specific sub-tasks within a given dataset, but they may not be sufficient for addressing diverse multi-modal dialogue tasks.

Given a set of $n$ multi-modal dialogue samples $\mathcal{D}=\left\{\left(U_{i},R_{i}\right)\right\}_{i=1}^{n}$, where $U_{i}$ and $R_{i}$ represent the dialogue context and response, respectively. Compared to traditional textual dialogue, both $U_{i}=\{u_{k}^{m}\}_{k=1}^{K}$ and $R_{i}=\{r_{q}^{m}\}_{q=1}^{Q}$ can incorporate various types of information including textual texts and visual images, where $K$ and $Q$ are the number of elements, and $m\in\{\mathfrak{t},\mathfrak{v}\}$ denotes the modality of $U_{i}$ (or $R_{i}$). The notation $\mathfrak{t}$ indicates textual utterances, while $\mathfrak{v}$ indicates visual images.

The aim of the Multi-Modal Intent Prediction task is to identify the specific intent of the user in the multi-modal context. More specifically, it predicts the probability of photo sharing in the upcoming conversation turn. In equation, it’s formulated as a binary classification task:

where $\mathcal{M}(\cdot,\cdot)$ is the intent prediction model taking the dialogue context $u_{\leq j}$ and response elements $r_{<j}$ of all the previous turns as the input and outputs a binary value. It should predict 1 when photo sharing behavior occurs in the next conversation turn and otherwise 0. Note that whether the model make use of all the previous turns is contingent upon the design of the model itself. We use F1 score, precision, and recall as the evaluation metrics for this task following Zang et al. (2021). We describe two different datasets for the task of intent prediction as below.

This task requires the model to retrieve the most relevant image from a candidate image set $\left\{r_{z}^{\mathfrak{v}}\right\}_{z=1}^{Z}$ while given the dialog history, where $Z$ is the size of the set. It requires models to extract key object information from the conversation and exclude irrelevant distractors. Additionally, the model needs to be capable of aligning the semantic space of visual and linguistic modalities, such that two perspectives of a scene are similarly represented in the vector space. Following Zang et al. (2021), we use $Recall@K(R@K)$, computed as “the fraction of times a correct item was found among the top K results” as the evaluation metrics.

We select PhotoChat and MMDialog datasets for multi-modal dialog retrieval (T2I) task in the VDialogUE benchmark. These two datasets have some differences in the selection of candidate image sets for evaluation. Since PhotoChat has a small number of test set images, the entire test set images are used as the candidate set for each dialogue. On the other hand, MMDialog has a large number of test set images (23k), making it difficult to use the entire test set as the image candidate set. Therefore, the developers manually selected 999 images as negative samples for each dialogue in the test set. As a result, the candidate set size for both datasets is 1,000.

It requires the model to take as input an image and a multi-turn, free-form, open-ended, natural language question about the image and produces or selects a natural language answer as the output. As with previous work, the task is defined to answer the question by selecting the most compatible target sample from the text candidate answers $\left\{r_{z}^{t}\right\}_{z=1}^{Z}$ in VDialogUE, where $Z$ is the size of the set. The candidate answers contain both ground truth samples and indistinguishable negative samples, and the model is asked to return a sorting of the candidate answers. The performance for Multi-Modal Dialog Retrieval (I2T) is measured by $R@K$. We introduce two different datasets for this task.

Multi-modal dialog state tracking (MMDST) extend the traditional notion of the textual dialog state tracking, where pre-defined slots $S=\left\{S_{1},\ldots,S_{N}\right\}$ are grounded on the multi-modal context. The model $\mathcal{M}$ determines for every turn whether any of the slot pairs in present and to predict the slot values. We select MMConv and SIMMC_2.0 datasets for MMDST in VDialogUE benchmark. $Accuracy$ is used for MMConv evaluation and the SIMMC_2.0’s performance is measured by the joint F1 for the cumulative intent, slot predictions.

Given the multi-modal dialogue context $U$, the multi-modal response generation task aims to generate the textual response $R$ by modeling $\boldsymbol{P}(R\mid U;\theta)$, where $\theta$ is the model $\mathcal{M}$ parameters. We select MMConv and SIMMC_2.0 datasets for multi-modal response generation task in the VDialogUE benchmark. We use the widely used BLEU Papineni et al. (2002) to measure the response generation quality.

With the aim of constructing a unified multi-modal dialogue system, we utilized AHP method (Analytic Hierarchy Process), a quantitative analysis tool, to determine the relative importance of different tasks. The most crucial demand for multi-modal dialogue systems is to produce responses that are coherent and contextually relevant, while intent recognition is considered a prerequisite for retrieval tasks, and dialogue tracking is just an intermediate task in continuous dialogue. Based on this assumptions, we created a pairwise comparison matrix as shown in Table 2. Then, we calculated the corresponding weights for each task and performed a consistency check to validate our assumptions. More details are stated in Appendix B. The final score of VDscore is calculated as:

$$\begin{split}\boldsymbol{VDscore}&=\boldsymbol{0.41}\cdot\includegraphics[width=7.96674pt,height,valign={m}]{fig/triangle}+\boldsymbol{0.25}\cdot\includegraphics[width=7.96674pt,height,valign={m}]{fig/star}\\ &+\boldsymbol{0.14}\cdot\includegraphics[width=7.96674pt,height,valign={m}]{fig/circle}+\boldsymbol{0.14}\cdot\includegraphics[width=7.96674pt,height,valign={m}]{fig/hexagon}+\boldsymbol{0.06}\cdot\includegraphics[width=7.96674pt,height,valign={m}]{fig/square}\\ \end{split}$$

(2)

where each icon represents the average measure of a specific task, obtained by averaging internal metrics and then averaging across different datasets.

As illustrated in Figure 2, VISIT has a succinct architecture as a baseline model for VDialogUE, which adopt the standard Transformer as modality interaction backbone and the simplest visual and text embedding scheme.

As shown in Table 3, the availability of sufficient paired image-text data enables the model to learn the fundamental inter-modal alignment. Additionally, pre-training the model on multi-modal dialogue data enhances its ability to process not just simple text but also complex, context-dependent dialogues. Therefore, as illustrated in Figure 2, we divided the pre-training process into two phases. In phase-I, VISIT is trained on image-text paired data using two standard pre-training objectives Kim et al. (2021), namely image-text matching (ITM) and masked language modeling (MLM). we integrate response generation modeling (RGM) into the pre-training process, which builds upon the foundations laid out in phase-I and is trained on multi-modal dialogue data.

Once the pre-training of VISIT is finished, we perform fine-tuning on specific downstream tasks. To tackle the multi-modal intent prediction and dialog retrieval tasks, we start by initializing the similarity score head with the ITM head that was pre-trained, specifically the component responsible for computing the true-pair logits. We then randomly select 15 text samples to act as negative examples and fine-tune the model using cross-entropy loss, with the goal of maximizing scores on positive pairs. To facilitate multi-modal dialog state tracking in the MMConv dataset, we augment the model architecture with a CATEGORICAL head and an SPAN head. The former handles categorical slots, while the latter is responsible for non-categorical ones. For the MMDST task in SIMMC_2.0 and multi-modal response generation, we fine-tune the model by calculating the standard negative log-likelihood loss $\mathcal{L}_{\mathrm{rgm}}$ in an end-to-end manner.

We initialize the Transformer weights with ViT-B/32 Dosovitskiy et al. (2020) pretrained on ImageNet. The embedding size $E$ is 768, layer depth $L$ is 12, patch size $P$ is 32, MLP size is 3,072, and the number of attention heads is 12. We use bert-base-uncased tokenizer to tokenize text inputs. Patch projection of VISIT yields $12\times 20=240$ patches for an image with a resolution of $384\times 384$. For all experiments, we use AdamW optimizer Loshchilov and Hutter (2017) with base learning rate $10^{-4}$ and weight decay of $10^{-2}$. The learning rate is warmed up for 10% of the total training steps and is decayed linearly to zero for the rest of the training. In the pre-training process, we conduct 200K and 25K steps for each of the two phases respectively with a batch size of 4,096. For all downstream tasks, we train for ten epochs with a batch size of 256.

Since there is currently no general model available to solve all tasks in VDialogUE, we choose the models that exhibit the best performance on specific tasks to serve as our baseline models. We compare VISIT with previous state-of-the-art models, including T5 Raffel et al. (2020), Divter Feng et al. (2022), ViLT Kim et al. (2021), SCAN Lee et al. (2018), DE++ Feng et al. (2022), TransResNet Shuster et al. (2018), UTC Chen et al. (2022), BART Lewis et al. (2019), DS-DST Zhang et al. (2019) and SimpleTOD Hosseini-Asl et al. (2020). Appendix A contains further descriptions of the baseline models.

The experimental results on the VDialogUE benchmark are presented in Table 4. We observe that our VISIT model can achieve competitive performance compared to strong baselines for a broad variety of visually-grounded dialogue tasks. More precisely, our VISIT model demonstrated state-of-the-art performance on certain tasks, including multi-modal intent prediction, dialog retrieval(T2I), dialog state tracking and response generation, across almost all metrics. Specifically, we found that our model also achieved consistent improvement in the comprehensive evaluation of VDscore.

However, we noticed that the VISIT model does not perform as well on text retrieval tasks as it does on other tasks, particularly on the VisDial dataset. We speculate that there are several reasons for this. Firstly, VisDial shows a clear distribution bias towards image content over real dialogue scenarios, allowing dialogue agents to rely solely on image features and ignore dialogue context Kim et al. (2020); Le et al. (2022). Secondly, there is an annotator bias that can lead to harmful causal links between dialogue context and output response as demonstrated in Qi et al. (2020). Thirdly, in multi-modal text retrieval, the model needs to prioritize candidate answers over the dialogue history, which is often much longer than the candidate answers consisting of only a few characters. However, our model failed to take this into account and treated the dialogue history and answer equally by simply concatenating them as the input on the text side.

In addition, we found that there is still significant room for improvement in the accuracy of our model on image retrieval tasks, particularly on the $R@1$ metric. To better analyze the limitations of VISIT, we carry out an analysis of the errors made by VISIT on the PhotoChat and MMDialog test dataset. As Figure 3 shows that due to the existence of many similar images in the datasets, VISIT struggles to differentiate some correct images from similar candidates. This limitation may be attributed to the lack of an explicit fine-grained reasoning module that can effectively capture the nuances in both images and texts.

Furthermore, we perform an ablation study to examine the effect of different pre-training data on both the PhotoChat and Image-Chat datasets. The models that are exclusively pre-trained on multi-modal non-dialog and dialog data are denoted as VISIT_Phase-I and VISIT_Phase-II, respectively. The ablation test results on PhotoChat and Image-Chat are provided in Table 5. It is evident that both the multi-modal non-dialog and dialog pre-training corpora significantly enhance the performance of VISIT. This outcome is not surprising as the multi-modal non-dialog data aids the model in acquiring exceptional image-text representations and their alignment, while the multi-modal dialog data stimulates VISIT to capture the contextual information of the dialog.