AV-EmoDialog: Chat with Audio-Visual Users
Leveraging Emotional Cues

Recent efforts have aimed to leverage the capabilities of large language models (LLMs) [52, 16, 45, 8] for enhancing spoken dialogue systems. They aim to autoregressively model the semantic and acoustic content of raw speech to process and generate speech. d-GSLM [35] models two-channel conversations to produce natural turn-taking conversations. SpeechGPT [50] initially converts speech to discrete speech tokens, followed by a three-stage training pipeline involving paired speech data, speech instruction data, and chain-of-modality instruction data. AudioGPT [24] guides LLMs to generate commands for interacting with external tools before processing these commands within the models. Qwen-Audio-Chat [17] is trained on extensive audio understanding tasks and instruction fine-tuned on dialogue framework.

Some spoken models [30, 49, 2] have incorporated emotional cues from speech to enhance the model’s ability to tailor responses to diverse emotional contexts. By predicting emotion labels alongside each response, it helps the LLM better understand the user’s emotional state and enables it to generate responses that are more emotionally attuned and contextually appropriate. While these studies have discerned emotional cues within speech, they overlook non-verbal cues present in the visual modality. As human interaction is inherently multimodal, with visual cues such as facial expressions and eye contact playing a crucial role in conveying emotions and intentions, our work aims to integrate the audio and visual information of the user to better capture the communicative signals.

There has been a surge of Multimodal Large Language Models (MM-LLMs) [28, 32, 31, 34, 47, 41, 48, 26, 44] that are capable of processing multimodal input. BLIP-2 [28] introduces a Q-Former module to bridge the modality gap between the visual and text. Llava [32, 31] aligns the visual and text modality by utilizing a large-scale multimodal instruction-following dataset on image QA, image description, and complex reasoning. Video-LlaMA [51] and Video-ChatGPT [34] process the visual and audio content in a video to engage in a conversation describing the visual scenes and actions in a spatio-temporal context. HuggingGPT [41] conducts task planning to connect various AI models upon receiving a user request and NextGPT [48] builds an end-to-end system that is capable of understanding and generating audio, visual, and text modality. However, they primarily concentrate on multimodal grounding tasks. They excel in integrating multimodal data for content reasoning but do not leverage this information to enhance interaction with the human user.

Recently, a few multimodal large language models [4, 18, 21] tailored for emotion-aware dialogue system attempted to incorporate emotional context. FaceChat [4] employs a pre-trained facial emotion classifier [42] to predict textual emotion labels from the user’s facial expression and incorporate this label into the LLM’s template prompt. It also needs an ASR module to transcribe speech into text, and utilizes the zero-shot capabilities of a pre-trained text LLM for response generation. SMES [18] utilizes a pre-trained Video-Llama [51] to extract emotion labels of each time step via prompting. These emotional labels are then combined with text inputs and fed into an LLM for response generation. EmpathyEar [21] utilizes an ImageBind [22] to extract multimodal embeddings, which are also combined with text embeddings to be fed into an LLM. Notably, these approaches follow a cascade approach, relying on pre-trained components such as emotion classifiers and ASR modules, which are not optimized end-to-end on audio-visual dialogue. Moreoever, they depend on emotion classifiers that assign emotions to one of only seven broad categories, restricting emotional understanding and leading to a simplified interpretation of temporally varying and complex user emotions.

In contrast, our work presents an audio-visual spoken dialogue system that directly processes both linguistic and non-linguistic content from audio-visual input, without relying on intermediate text. Our system is trained end-to-end on audio-visual dialogue data, ensuring that the communicative signals from the input modalities are jointly learned. We demonstrate our proposed method’s strong ability to generate responses that are both contextually and emotionally aligned with the user’s current state, leading to more natural, empathetic interactions.

From the raw audio input, we extract both the linguistic content (i.e. what the speaker says) and verbal emotional cues. This stage equips an audio encoder with the ability to understand such communicative signals directly from speech such that the LLM can directly understand. We employ a strong audio encoder, Whisper model [40] which has shown powerful capabilities in speech-text processing tasks such as speech recognition (ASR), speech translation, voice activity detection. The audio encoder is connected to the high-performing chat-LLM, Llama-3-Instruct¹¹1https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct, as illustrated in Figure 1(a). We directly feed audio features $f_{a}$ from Whisper model to the LLM and align the feature space between the audio and text using an ASR training objective as in [50, 17, 48, 43] on extensive audio-text datasets; Gigaspeech [15], CommonVoice [6], and LibriSpeech [36]. By training on ASR objective, the model can directly understand the speech content and align the speech feature that can be interpreted by the LLM.

Additionally, we incorporate speech emotion recognition (SER) objective to extract emotional tone from speech. By inputting speech features into the LLM instead of transcribed text, the LLM can infer deeper verbal aspects of speech such as the tone and pitch. We use speech data with emotion labels; RAVDESS [33], CREMA-D [13], MultiDialog [37]. The LLM remains frozen while we fine-tune the audio encoder. This dual training objective empowers the audio encoder to extract both linguistic content and emotional context that can be directly understood by the LLM. The hard prompt, namely the speech understanding prompt, given during this stage is shown in Figure 2(a).

The face video contains rich non-verbal cues essential for nuanced human communication. To capture these subtle non-verbal cues, we deploy a face encoder with the ability to recognize the fine-grained facial emotion that can be directly interpreted by the LLM. As shown in Figure 1(b), the face encoder consists of two components; the frame encoder and the temporal encoder. The frame encoder, CLIP-ViT [39], known for strong visual-text alignment, extracts facial features at the frame level. Then, the temporal encoder aggregates the frame-level features to learn facial dynamics as emotion varies over time. Similar to [25, 14], we employ learnable queries that cross-attend to the frame-level features to output a fixed-length video feature $f_{v}$. This allows for the efficient processing of videos of varying lengths, which not only reduces computational costs but also facilitates handling longer dialogue sequences. The output visual features are directly fed into the LLM, and trained to recognize emotion labels on RAVDESS [33], CREMA-D [13], and MultiDialog [37] datasets.

To optimize the training of the face encoder, we incorporate detailed descriptions of the facial expressions. Specifically, emotion-labeled face videos in the training datasets are further annotated with comprehensive descriptions of facial dynamics using GPT-4, as illustrated in Figure 2. These descriptions include granular details such as movements of the cheeks, lips, and eyes, as well as the progression of emotional expressions over time. By incorporating comprehensive facial descriptions along with the emotion labels as the training objectives, the face encoder learns to recognize the nuanced and dynamic aspects of human expressions that go beyond simplified emotion categories. Emotions during communication are inherently time-varying, often not consistently maintaining the same emotion. The time-considered detailed descriptions help the model recognize the evolving emotional states that go beyond the static and singular emotion categorization. It also guides to focus on specific facial attributes for accurate emotion detection, akin to how humans process emotion from the face. The face encoder is fine-tuned while the LLM remains frozen. The hard prompt, namely the face video understanding prompt, given to LLM during this stage is shown in Figure 2 (b).

We build an emotion-aware audio-visual dialogue framework, namely AV-EmoDialog, by integrating the speech encoder, face encoder, and a LLM as shown in Figure 1 (c). Since the speech encoder and the face encoder are equipped with strong capabilities to process the linguistic and non-linguistic cues from audio-visual user input from the previous stages, the LLM is simply adapted to incorporate the multimodal communicative signals to generate contextually appropriate responses. In our setting, each dialogue consists of $R$ rounds of turns between the AI and the user, with the AI producing textual responses to audio-visual provided by the user. We use audio-visual dialogue language modeling with the following objective:

$\displaystyle\mathcal{L}_{\text{dialog}}=\sum_{r=1}^{R}-\log p(e_{r}^{ai},T_{r}^{ai}\mid f_{a,r}^{user},f_{v,r}^{user},\mathcal{H}),$

(1)

where $e_{r}^{ai}$. $T_{r}^{ai}$ are the emotional state and textual response of the AI at round $r$, which the LLM has to predict given the audio feature $f_{a,r}^{user}$ and visual feature $f_{v,r}^{user}$ of the user input at round $r$, and the dialogue history $\mathcal{H}$. We keep the audio and visual encoders frozen and LoRA [23] fine-tune the LLM on the target dialogue data, preserving the linguistic prowess while tailoring on the target audio-visual dialogue task. Through this focused adaptation, AV-EmoDialog not only responds accurately in text but also aligns its responses with the emotional and contextual nuances captured from the user’s audio-visual cues. The hard prompt, audio-visual dialogue prompt is shown in Figure 2 (c).

We evaluate the dialogue generation in terms of semantic quality and emotional quality. For the semantic quality, we employ standard metrics used for text-based dialogue generation: BLEU-1, BLEU-4 [38], ROUGE [29], METEOR [9], and PPL [10]. PPL is calculated using Dialog-GPT [54] and it is calculated across the test set. For the emotional quality, we measure EmoBERT score [53] which calculates semantic similarity between the generated response and reference response using a BERT model finetuned on GoEmotion dataset [19]. We additionally measure the diversity score with DISTINCT-1 [27] which quantifies how many unique unigrams (individual words) appear in the generated text relative to the total number of unigrams. For a more comprehensive evaluation, we additionally conduct GPT-4 evaluation and human evaluation. GPT-4 evaluation rates response given the dialogue history from 0 to 5 in terms of fluency and emotional intelligence. Likewise, human evaluation directly ranks the models in terms of contextual appropriateness and emotional intelligence. The detailed instructions for the evaluations are in the Appendix.

For the speech encoder, we adopt a pre-trained Whisper model [40] of Qwen-Audio [17] and fine-tune with ASR and speech emotion recognition objectives on a total of 6,485 hours of paired audio-text data, including Gigaspeech [15], CommonVoice [6], LibriSpeech [36], RAVDESS [33], CREMA-D [13], and MultiDialog [37]. Note that we utilize the extra metadata available such as emotion, gender, age, and ethnicity to enrich the non-linguistic aspect of the speech feature.

For the face encoder, we utilize a pre-trained CLIP-ViT as the frame encoder, followed by a transformer encoder with 6 layers and 8 attention heads as the temporal encoder. The videos are preprocessed into face crops of size 96$\times$96 using a face detector [20] and a facial landmark detector [12]. Each video is sampled every 10th frame to extract the frame-level features, which are further encoded into a fixed-length visual representation using the learnable queries of size 128. For obtaining the facial emotion description, we created a collage of images from the video frames as the GPT-4 is more adept at analyzing an image than a video.

For the LLM, we use LlaMA3-Instruct1 which has a strong ability to understand and generate responses based on user instructions. In the first two stages, it is frozen, and in the last stage, we LoRA fine-tune [23] on the audio-visual dialogue data, MultiDialog [37]. Multidialog is a recently introduced large-scale multimodal dialogue corpus that includes audio, visual, and text modalities. All three stages are trained on 4 A6000 GPUs with AdamW optimizer and a cosine learning rate decay with a warm-up period. The lora rank is set to 16 and we enable bias and norm tuning. The max token length is 4096 with a batch size of 32. Please refer to the Appendix for the detailed statistics of the data.

We compare with the state-of-the-art multimodal language models that use various combinations of the modality. Llava-next (video) [31] takes a video and text as input to generate a response in text. SpeechGPT [50] takes speech input to generate a response in both text and speech. Since these baseline models are not tailored for dialogue generation tasks but instead on video understanding, captioning, speech recognition, and generation, we finetuned them on the testing dataset, MultiDialog [37]. We also construct a cascaded baseline by sequentially stacking state-of-the-art pre-trained models; audio speech recognition [17], speech emotion recognition [17], and Llama3 1, where the audio model first transcribes the speech and recognizes the emotion which then goes into the LLM to generate emotion-aware response. We also finetune the cascaded baseline on the testing data for a fair comparison.

We quantitatively compare the dialogue generation results in Table 1. Our experiments were conducted on the test rare split of the MultiDialog dataset. For each conversation, we randomly selected four turns from the assistant’s side. Compared with state-of-the-art multimodal large language models, our method significantly improves the EmoBERT score, which assesses the emotion alignment of the generated responses with the ground truth emotion. In addition, the semantic scores were also boosted, as evidenced by the highest scores in BLEU-1, BLEU-4, and METEOR. The high semantic performance confirms that our method generates a contextually coherent response. This result is particularly notable given that other comparison methods rely on text input, a format with which large language models are trained extensively. Compared with the cascaded baseline, which stacks a high-performing ASR, Emotion Recognition (ER) model, and LLM, our method does not require separate conversion of input speech into text and emotion labels. Nevertheless, our model achieves superior emotion alignment and comparable semantic performance, highlighting the efficiency of our end-to-end approach. Finally, AV-EmoDialog also demonstrates the highest diversity scores. These quantitative results suggest that our approach to systematically exploiting communicative cues from audio-visual inputs not only enhances emotional alignment but also improves the overall semantic quality and diversity of the generated responses. This result implies that having a better emotional context enables the model to produce responses that are more attuned to the user’s state and dialogue context.

We incorporated GPT-4²²2https://chatgpt.com/?model=gpt-4 evaluation to further assess the generated response in Table 2. Since EmoBERT only assesses alignment with ground truth emotional labels, a more thorough evaluation was necessary to understand the model’s ability to address emotions. Henceforth, we evaluated two aspects of emotional intelligence using GPT-4: empathy which is how well the response identifies and reflects the emotion of the user, and emotional context which is whether the emotional tone of the response is suitable for the given context. The high ratings in emotional intelligence particularly suggest that our system excels at interpreting and responding to the emotional states conveyed by users.

Also, although our system directly interprets audio-visual as the input, it has even superior fluency compared to other methods that rely on textual inputs. This suggests that using audio-visual modality to directly process both linguistic and non-linguistic content not only enhances emotional understanding but also linguistic capability.

We conducted a human evaluation through Prolific³³3https://www.prolific.com/ to compare the quality of responses generated by various dialogue models in Table 3. 10 gathered English-speaking participants were instructed to rank generated responses from four different models according to predefined criteria: relevance, coherence, contextual appropriateness, emotional coherence, and engagement. Table 3 shows the percentage with which each model was ranked first. The results indicate that AV-EmoDialog was most frequently selected as the top-performing model, which coincides with our automatic and GPT evaluations.

We analyzed the emotional context of response enhanced by AV-EmoDialog in Figure 4. In (a), the user speaks in a cheerful tone with a happy smile, asking the AI about its preference for football. The proposed method effectively recognizes the user’s happy emotion through audio-visual cues, which would not have been possible with text input alone. AV-EmoDialog responds accurately to the happy emotional tone of the user by saying "Are you excited for the Super Bowl?". On the other hand, the baseline methods fail to reflect the user’s emotional tone, resulting in less resonant responses such as "I’m sorry. I’m not familiar with this topic" or "Good evening! I’m a big fan of football.". In (c), the user talks with a sad and depressed face about how he feels stuck in the snow. Although it may sound like the user is frustrated from the text alone, the proposed method most accurately empathizes with the user’s sadness, responding with "Ughh…", and suggests one way to get out of the bad feeling. In contrast, baseline methods misinterpret the emotional context, responding with surprise or changing the topic. Likewise in (d), the user expresses surprise about the explosion through face, tone, and the content itself. While the baseline methods moves on to a different aspect of the topic, the proposed method mirrors the user’s surprise and engages further in the same emotional tone, by responding "That is just crazy money… That is more than the budget of the Star Wars trilogy.". In each example, the proposed method effectively recognizes the user’s emotion from the speech tone and facial expressions, providing the most emotionally appropriate responses that would have been difficult with pure text-only input. Please refer to appendix for more generation results. Note that we provide responses in speech using an off-the-shelf TTS model⁴⁴4https://github.com/2noise/ChatTTS for demonstration.

We tested AV-EmoDialog with emotion excluded from the last stage of training on the audio-visual dialogue task. The results are shown in the second to last row in Table 1. There was a huge drop in emotion performance - from 0.3 to 0.2, which suggests that having the emotion label in the dialogue allows the model to keep track of the understanding of the emotional context.

We evaluated the training schemes used for the speech and face encoders in Table 3. We observed that incorporating the detailed descriptions of the facial expressions greatly increased the emotion recognition performance of the face encoder by 70%. Also, the emotion recognition performance of the speech encoder is higher than from the face. This indicates that vocal expressions, often being more expressive and straightforward, can capture emotional nuances more effectively than visual expressions. But since they distinctively have their effects, we leverage both cues to boost the performance in the dialogue generation task.

In the dialogue generation task, speech encoder trained with emotion recognition objective has better performance (rows 1 and 2). This demonstrates that having the emotion label gives better context to generate a response that is more emotionally and semantically aligned. Furthermore, the face encoder, trained with detailed facial descriptions, also showed enhanced emotion recognition capabilities, which impacted its dialogue generation performance (rows 3 and 4). While there was a notable increase in the EmoBERT score, a slight decrease in semantic scores was observed. This reduction may be due to the complexities introduced by the detailed emotion descriptions, which could complicate the dialogue modeling. Nevertheless, its effect on emotion performance is evident, and semantic performance remains competitive, surpassing that of the baselines.

In Table 5, we conducted experiments with various modality inputs to validate the efficacy of each modality in dialogue generation. The text-only input has the highest semantic quality overall, which can largely be attributed to the fact that the underlying llm has been extensively trained on textual data. Yet, utilizing both audio-visual modalities was able to achieve on-par semantic performance while achieving superior emotional intelligence.

Another finding is that using both audio-visual modalities results in enhanced semantic and emotional performance compared to using audio modality alone. This improvement underscores the value of visual cues such as facial expressions and eye contact, which provide complementary information to audio cues such as pitch and tone of voice. Our approach reflects natural human communication, where verbal cues are typically supported and enriched by non-verbal expressions, creating a more holistic and effective interaction.