Scene-Aware Prompt for Multi-modal Dialogue Understanding and Generation^††thanks: Supported by the National Key R&D Program of China (2018YFB1305200), the National Natural Science Fund of China (62171183).

This multi-modal dialogue understanding and generation task includes three tracks:

1. 1.

   Dialogue scene identification: predict the boundaries of different dialogue scenes given a set of continuous dialogue utterances and a related video.

2. 2.

   Dialogue session identification: predict the boundaries of different dialogue sessions given a set of continuous dialogue utterances and a related video (which is identical to Track 1).

3. 3.

   Dialogue response generation: generate a response based on scene and session predictions, while coherently catching up with the conversation.

For pursuing these tasks, we formulated these tasks as follows, where the notation $V$ represents the video clips, and notation $C$ is the input dialogue context.

1. 1.

   For dialogue scene identification and dialogue session identification tasks, the final prediction (i.e., 0 or 1) is obtained through the input of $V$ and $C$, where we adopt a multi-task end-end framework to jointly perform these tasks.

2. 2.

   For dialogue response generation, given the scene ($S_{i}$) and session ($T_{i}$) predictions, the final response is generated with the pre-trained language model, where the clip captioners and the identified labels used as the prompt for providing extra context knowledge.

For the dialogue scene identification and dialogue topic identification tracks, we mainly use the accuracy metric (i.e., Acc_s and Acc_t) for the final evaluation ranking. The calculation equation is shown as follows.

where $l$ is the sample number, the S_i represents each predicted sample, and S${}_{i}^{\prime}$ is the ground truth label. Similarly, we can present the Acc_t below.

Also, the F1 metric is adopted for reference, where the definitions of F1 are shown as follows:

For the track 3, we adopt the BLEU [13], ROUGE [14], METEOR [15], and CIDER [16] scores of the generated response for further evaluations¹¹1https://github.com/tylin/coco-caption.

The NLPCC shared task 4 presents three shared tasks [6], namely, dialogue scene identification, dialogue session identification, and dialogue response generation. The ultimate goal is to generate a response that is coherent to the dialogue context and relevant to the video context.

The dataset of the competition²²2https://github.com/patrick-tssn/NLPCC-2022-Shared-Task-4 contains 40,006, 1,955 and 1,934 video clips as the visual context. The dialogue contains 1,000,079, 50,032 and 50,131 utterances in the train, dev and test sets respectively. The source of the videos and dialogues for this task are crawled from online American TV series, which are split into the training, validation, and test sets. Each sample contains a series of dialogue utterances, which is associated with the video clip (downsampled to 3fps) during the dialogue duration. Each clip is processed in the “jpg” format for further modeling.

Multi-modal dialogue understanding is still a great challenge since the visual and textual modalities share different information [1]. Both shared task 1 and shared task 2 share the same task objectives, which is to perform sequence prediction for the multi-modal dialogue understanding. Considering the intrinsic co-occurrence between the two labels (i.e., scene and session labels), we can obtain Table 1. From this table, we can conclude that both tracks share some common label information about the co-occurrence relationship. As a result, we propose the multi-tasking method for jointly training both tracks.

As shown in the Figure 1, we use the I3D model [17] to extract and process video features to promote deep multi-modal information fusion. After that, we change the structure of the original language model. Specifically, we use the timeline to align the dialogue span and visual frame, where each input text span is fused with the current video frame. The embedding layer can produce the fused features by adding fusion with the output text span and the corresponded frame. This fusion strategy can make the multi-modal interaction more precise and efficient [18].

For the final multi-task modelling, we design a linear layer for the output layer in each task. Specifically, the [SEP] feature vector of each sentence can produce the output with the two different binary linear layers. The multi-task modelling is designed to perform the scene and session sequence prediction, where two prediction results are obtained meanwhile for each input.

Traditional text-based generation methods have limitations in multi-modal scenarios [2]. On the one hand, it is because the interaction of characters in a multi-modal scene not only relies on the textual information of the dialogue context, but also needs to depend on the prompts of the environment scene and dialogue session [5]. On the other hand, the ability of a single text modality to perceive the multi-modal dialogue context is limited, and it is a wise choice to augment and enrich the dialogue context with other modalities [19].

Therefore, we propose a multi-modal dialogue generation method based on the scene-aware prompt, which is shown in the Figure 2. Specifically, we use the pre-trained video captioner model (i.e., UniVL [20]) and image captioner model (i.e., BLIP [21]) to obtain the caption information of the environmental scene, which further enhances the multi-modal dialogue generation. At the same time, we design a fixed-type templated prompt based on the scene- and session-aware labels to further improve the controllability of the generated responses.

For the training step, each sample is concatenated with the visual captions and the fixed prompt, where the visual captions are generated by the zero-shot video and image captioners, and the fixed prompt is produced by the corresponded scene and session labels.

For inference, we first predict the scene and session labels of the last turn, and then we translate the predicted labels into the fixed prompt for further concatenation. Also, the video clips and last image frame are implemented for obtaining the visual captions. Finally, the dialogue context, visual captions and the fixed prompt are concatenated for the response generation.

We conduct different experiments in three tasks in the MDUG dataset. Specifically, we use some pre-trained language models as the baseline methods. In subtask 1/2, we use the BERT³³3https://huggingface.co/bert-base-uncased [24] which a language mask model (MLM) [25] task pre-training model. The RoBERTa⁴⁴4https://huggingface.co/roberta-large [26] has conducted MLM pre-training for a long period time. The ELECTRA⁵⁵5https://huggingface.co/google/electra-large-discriminator [27] uses the replaced token detection task instead of the MLM task to obtain higher training efficiency. The DeBERTa-v3 [28] implements gradient decoupling on the basis of Electra to avoid the tug-of-war procedure [29].

In subtask 3, we select some strong baseline models in the generation domain for comparison. The BART [30] and the T5 [31] use the denoising and mask restoration task for pre-training respectively. The BART has achieved SOTA in generation tasks like translation, while T5 has SOTA performance in understanding and summarization.

In order to get a strong baseline, we use DeBERTa-v3-large as the backbone network in the understanding task (subtask 1 and 2) and Blender [3] as the backbone network in the generation task (subtask3).

All the hyperparameters are adjusted in the dev set to ensure fairness. In all our experiments, at the end of each epoch of training, we will test in the development dataset, and select the highest model (mainly depending on Acc or BLEU) to predict in the test dataset. All the tables report the highest score in the development set except for the final official score table. All the experimental results are repeated three times, and the highest and lowest scores are removed.

We set the maximum token length to 512 and delete the excess text. We have fine-tuned 10 epochs of training in three A100 GPUs on the Pytorch⁶⁶6https://pytorch.org and the hugging-face⁷⁷7https://github.com/huggingface/transformers framework, with a batch size of 10. We implement distributed training with mixed precision based on the DeepSpeed [32]. We use an AdamW optimizer [33] with a maximum learning rate of $1\times 10^{-5}$, followed by linear attenuation and warm-up optimizing schedules [34].

We conducted experiments on subtasks 1 and 2, which are shown in Table 2 and 3 respectively. We compare the performance of the pre-trained language model with random selection. From the table, we can see that the accuracy rate in random mode is less than 50%, but it has a high recall rate. This is because the label distribution of the MDUG dataset is not balanced. For the poor F1 performance of BERT, we consider that it is due to the extreme imbalance of labels so that the BERT cannot overfit these datasets, where the output recall and f1 results of the BERT are all 0. With the expansion of the pre-training scale, the effect of the model gradually improves. Compared with single-task modeling, our method with multi-tasking modeling can perceive and understand the visual information at a deeper level. Compared with other competitive methods, we can find that for subtask1, our accuracy and F1 index can be improved by 0.833% and 2.006% respectively. For subtask2, our accuracy and F1 index can be improved by 0.336% and 0.446% respectively.

The subtask 3 is a generation task, so we selected some generated baseline models for comparison, which is shown in the Table 4. We can find that our model can be greatly improved, which is because our method can make full use of multi-modal feature information on the basis of the pre-training language model. Compared with other competitive methods, our method improves the average score by 7.01, which proves the effectiveness of our method.

We also implement the ablation study for the proposed method, which is shown in Table 2 and 3. The image and video captions can improve the final results, and the prompt can enhance the performance of the dialogue generation. Also, with the aid of multi-tasking modeling, our method can be further improved.

Specifically, we make a more in-depth analysis of the experiment. We test the effect of cancelling multi-task in subtasks 1 and 2. We find that the effect will decrease significantly after single-task modelling is cancelled. We consider it is a great correlation between scene and session. Through multi-task learning, the model can better understand the actual meaning of the textual and visual information. It helps to increase the generalization ability of the model and improve the final effect.

Moreover, we perform ablation experiments in subtask 3, which is shown in Table 4. We make the model without some visual perception ability by canceling the original image caption or video caption, so as to evaluate the effectiveness of different methods. In our ablation experiments, we can find that the performance degradation of the model is not obvious if any visual information is missing. However, if all visual cues are missing, the model will lack the ability of visual scene modeling, which will greatly affect the final prediction performance.

As for the online results, we reported the final results of our system. In Tabel 5 and 6, our system showed very convincing performance. We have achieved the first place in all subtasks, which fully demonstrates our method’s effectiveness.