ERIT Lightweight Multimodal Dataset for Elderly Emotion Recognition and Multimodal Fusion Evaluation

Emotion recognition plays a crucial role in understanding human behavior and improving human-computer interaction. With the growing elderly population, there is an increasing need for effective emotion recognition systems tailored to the specific characteristics of this demographic. ERIT is a multimodal dataset that aims to address this need by providing a rich source of data for training and evaluating emotion recognition models for elderly individuals. This paper describes the motivation behind creating the ERIT dataset, the methodology used to obtain and validate the emotion labels, and the potential applications of the dataset in the field of elderly emotion recognition.

The primary motivation for creating ERIT dataset is the lightweight evaluation of multimodal fusion. The task chosen for research on multimodal fusion needs to be able to perform across different modalities, possibly with a limited amount of classes for an easy evaluation. Emotion recognition meets the demands of such a task, with only seven classes and the possibility of performing emotion recognition among different modalities, such as facial expresion Leong et al. (2023); Huang et al. (2023), pose Yang et al. (2020), text Acheampong et al. (2020); Nandwani and Verma (2021) or audio Zhao and Shu (2023); George and Muhamed Ilyas (2024) or non-contact physiological signals Li and Peng (2024).

The secondary motivation behind the creation of the ERIT dataset is to facilitate research in elderly emotion recognition and contribute to the development of more accurate and robust emotion recognition systems for this demographic. The dataset can be used for various applications, including healthcare, elderly care, and assistive technologies Sharma et al. (2021).

The ERIT dataset contains lightweight text and image data collected from videos of elderly people reacting to various stimuli. The dataset includes transcriptions of the speech and emotion labels extracted from the video frames. The emotion labels are provided for seven basic emotions: anger, disgust, fear, happiness, sadness, surprise, and neutral. The dataset was built from frames extracted from the ElderReact video dataset by Ma et al. (2019). The reasons behind creating an image and text-based dataset were: lightweight processing of text and image, accuracy of evaluating fusion compared to frames randomly extracted from the video and filling the gap in emotion recognition among different age groups.

Text and image are better for lightweight fusion evaluation than computationally expensive audio or video processing and can be processed by most of the large multimodal models (LMMs) such as GPT4v OpenAI (2023b) and GPT4o OpenAI (2023a), multimodal versions of LLaMA Zhang et al. (2023), or Flamingo Alayrac et al. (2022) etc.

Due to the labeling of specific frames in the video, the prediction is more accurate than in cases of videos labeled and evaluated by sampling a few frames, the method used on evaluated MER-MULTI dataset by Lian et al. (2024). The authors admit that this method could potentially ignore key samples and decrease the performance.

Finally, the emotion recognition images try to fill the age gap in the emotion recognition community by creating of an openly public dataset for facial emotion recognition. While some of the elderly facial emotion recognition image datasets Ebner et al. (2010) (1068 images of older faces) and Minear and Park (2004) provide acted emotions, emotions of the elderly users in ERIT were their natural reaction to the presented material.

This analysis focuses on how different models perform across various data types, particularly using the ERIT dataset as a benchmark for evaluating multimodal fusion compared to single-mode text and image analyses. By assessing models’ abilities to integrate both textual and visual inputs effectively, the study underscores the importance of leveraging combined data sources for enhanced sentiment analysis and recognition tasks. Models that excel in multimodal fusion demonstrate a clear advantage in leveraging the complementary strengths of text and image inputs, showcasing their capability to deliver nuanced and accurate analyses across diverse datasets like ERIT.

Prompts used for evaluation of LMMs were similar for each case, and varied only in the type of modal information passed. The correct prompts were generated with the help of the GPT 3.5 prompt generator which largely improved the number of predictions compared to hand-engineered prompts.

Finally, we analyze the performance of LLMs on emotion recognition tasks using only textual data across various emotion datasets, including ERIT. Models such as GPT-4 and LLaMA-7B demonstrate competitive performance on datasets like EmoWOZ and DAIC-WOZ, indicating strong proficiency in text-based emotion recognition tasks. However, their effectiveness varies when applied to the ERIT dataset, which poses challenges due to its complex emotional expressions and different age groups that might necessitate the integration of multimodal inputs for more accurate analysis.

The ERIT dataset is a valuable resource for researchers and practitioners working in the field of both multimodal fusion as well as elderly emotion recognition. The dataset provides a rich source of multimodal data, that is images paired with text and labeled with emotions, which can be used to research multimodal fusion performance. It can also be used for emotion recognition models specifically tailored to the elderly population. The comprehensive experiments on LMMs ensure the accuracy and reliability of the emotion labels and make the dataset applicable to a wide range of lightweight multimodal experiments.

The datasets generated and analyzed during the current study are available in the Zenodo repository, DOI: 10.5281/zenodo.12803448. The data includes text transcriptions, labels, and images used for training and evaluation of the models. The code for data preprocessing is available at https://github.com/khleeloo/ERIT.