MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics

We collect our deception dataset using: a sports camera Gopro Hero9 with a resolution of 1920x1080 and a frame rate of 60 fps. The voices of the subjects are also recorded by the built-in microphone of the camera. A Thinkpad laptop was provided to subjects for watching emotion induction videos during the emotion experiment. The experimental place is in a professional recording stdio, and during the data collection process, only the subject and interviewer stay in the room. Some materials were prepared by the Data Collection Coordinator (DCC). The experimental setup is shown in Figure 1.

There were 193 subjects in this study, of which 130 were female and 63 were male. They are all native Chinese speakers from different backgrounds. Their age ranged from 18 to 69 years old, and they had various professions including students, workers, teachers, retirees, etc.

Firstly, we segmented the raw video, resulting in 1808 minutes of deceptive video and 4401 minutes of emotional video, totaling 6209 minutes. Each of the 193 subjects provided 24 responses, 9 of which were deceptive. The length of deception videos ranged from 4 minutes to 27 minutes. Each subject had 16 emotional videos, with lengths ranging from 19 minutes to 38 minutes (including the time spent watching emotional induction videos).

Before the experiment began, the subjects were informed of all experimental procedures. The subjects explicitly consented to record their conversation and publish the video data in a scientific conference or journal. And we do not publish any privacy sensitive data, and the anonymity of participants will be guaranteed. All data were collected under a protocol approved by the authors’institution’s Human Subjects Institutional Review Board.

Personality Characteristics Collection: Subjects were required to fill out a Big Five personality questionnaire [45], which consists of 60 questions. Each question was marked with a score indicating whether the descriptions match their own, with 1 indicating strong disagreement and 5 indicating strong agreement. Details can be found in the appendix A.

Emotional Experiment: Subjects were asked to watch 16 emotional induction videos, including two induction videos for each emotion of sadness, happiness, relaxation, surprise, fear, disgust, anger, and neutral. There are a total of 39 induction videos, of which 17 are from the Chinese Emotional Video System (CEVS) [46]. Each video segment has been labeled and evaluated to ensure that it can induce corresponding emotions. Another 22 are from our online collection. Because the CEVS only include six emotions: sadness, happiness, fear, disgust, anger, and neutral, and some videos of the CEVS are outdated and cannot successfully induce corresponding emotions in our pre-experiments. Each video we collected online was annotated by 12 data annotators based on CEVS selection criteria and evaluation methods, and the results showed that each video triggered strong emotions.

Before the emotional experiment began, the DCC randomly selected 16 induction videos (ensure two videos for each emotion) for the subjects to watch. After watching each video, subjects were required to describe their feelings and then fill out an “Emotional Scale", which quantified 8 emotions. Subjects rated their 8 emotions, ranging from 1 to 5, with 1 indicating no such emotion and 5 indicating the strongest emotion. Details of the emotion scale can be found in the appendix B.

Deception Experiment: The deception data collection process follows DDPM [27]. The interviewer conducted an interview with the subject, asking a total of 24 questions. These 24 questions were jointly formulated by 5 psychology researchers with over 5 years of experience. Before the interview, details of the emotion scale can be found in the appendix C. The DDC randomly selected 9 questions that must lie and hand them over to the subject (the interviewer does not know which 9 questions). The first 3 questions will not be selected, which means that the first three “warm up” questions were always to be answered honestly. They allowed the subject to get settled, and gave the interviewer an idea of the subject’s demeanor when answering a question honestly.

The subject have a maximum of 15 minutes to prepare, and during the preparation process, they must remember these 9 questions and think about how to deceive in the upcoming interview process. During the interview process, when asked these 9 questions, the subject must lie, and when asked the remaining 15 questions, they must tell the truth. Subjects were motivated to deceive successfully through two levels of bonus compensation: if they were able to deceive the interviewer in five or six of the nine deceptive responses, they were given a 150 percent of a base incentive payment; the base payment was doubled if they were successfully deceptive in seven or more questions. In order to collect more indistinguishable deception answers, we encourage subjects to incorporate some truth into lies when answering these deceptive questions.

During the interview process, the interviewer asked 24 questions in random order, and provide their judgment of truthful or deceptive answers to each question. And the interviewer filled out the "Interviewer Judgment Scale", which record the trust level the interviewer thinks of each answer. The trust level was divided into 1-5 points, where 1 represents definitely true and 5 represents definitely false. After the interview, the subject also filled out the "Subject Lie Confidence Scale" and be asked to rate the answer they just lied to. The same score is 1-5, where 1 represents that I have definitely deceived successfully and 5 represents that I have definitely not deceived successfully.

Each subject was required to complete the above three experiments, so that we can obtain their personality , emotional and deceptive characteristics.

For the visual modality, we first unify the raw video to 30 fps of the frame rate, and crop and align faces via DLib Toolkit [47]. Figure 2 shows some examples of real and deceptive faces. Then, we use visual encoders to extract frame-level , followed by average pooling to compress them to the video level. For the audio modality, we use FFmpeg to separate the audio from the video and unify the audio format to 16kHz and mono. For the textual modality, we first extract transcript using Paraformer [48], an open-source ASR toolkit. After data preprocessing, for each sample $x_{i}$, we extract acoustic features $f_{i}^{a}\in\mathbb{R}^{d_{a}}$, textual features $f_{i}^{l}\in\mathbb{R}^{d_{l}}$, and visual features $f_{i}^{v}\in\mathbb{R}^{d_{v}}$, where $\left\{d_{m}\right\}_{m\in\left\{a,l,v\right\}}$ is the feature dimension for each modality.

Different features lead to distinct results. To guide feature selection, we evaluate the performance of different features under the same experimental setup.

Visual Modality: Compared with handcrafted features, deep features extracted from supervised models are useful for facial expression recognition [49]. CLIP [50] is a multimodal model based on contrastive learning, where training utilizes text and images to construct positive and negative sample pairs. Pre-training is conducted on a dataset comprising 400 million pairs, resulting in strong generalization capabilities. And Vision Transformer (ViT) [51] is a transformer encoder model , pre-trained in a supervised manner on a large dataset of images. Images are presented to the model in the form of sequences of fixed-size patches (resolution of 16x16) and undergo linear embedding.

Acoustic Modality: We extracted the handcrafted feature extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [52], which contains 88 acoustic parameters designed specially for speech emotional recognition tasks, covering spectral, cepstral, and prosodic features . And Wav2vec [53] demonstrates the power of learning robust representations solely from speech audio, fine-tuning on transcribed speech, surpassing the best semi-supervised methods. It masks speech inputs in the latent space and addresses a contrastive task defined on quantized latent representations. It has been widely applied to downstream speech tasks. HUBERT [54] utilizes offline clustering steps to provide aligned target labels for prediction losses. A key component of this approach is applying prediction losses only in masked regions, forcing the model to learn representations that combine acoustic and language models on contiguous inputs. To better distinguish between speakers, a sentence mixture training strategy WavLM [55] is proposed, allowing for the unsupervised creation and merging of additional overlapping sentences during the training process.

Text Modality: Among all language models, BERT [56] and its variants are widely utilized, and it uses the masked language model and next sentence prediction objectives to learn word embeddings. Here, we extract the sbert-chinese-general-v2 features, which is based on the bert-base-chinese version of the BERT model and is trained on the million-level semantic similarity data set SimCLUE. ChatGLM [57] is an open-source conversational language model supporting bilingual question answering in both Chinese and English, based on the General Language Model (GLM) architecture. It demonstrats exceptional contextual understanding and more efficient inference capabilities. Baichuan [58] is an open-source large-scale model with 13 billion parameters. It features a larger size, more extensive training data, and more efficient inference capabilities.

For unimodal features, we utilize the fully-connected layers to extract hidden representations and predict deception:

where $h_{i}^{m}\in\mathbb{R}^{h}$ is the hidden feature for each modality, $d_{i}\in\mathbb{R}^{2}$ is the estimated deception probabilities. For multimodal features, different modalities contribute differently to deception detection. Therefore, we compute importance scores $\alpha_{i}\in\mathbb{R}^{3\times 1}$ for each modality and exploit weighted fusion to obtain multimodal features:

Similarly, for personality and emotional expression features, we use concatenation for feature fusion. We directly use personality scale scores as personality features. For emotional features, we train an emotion recognition model first, input all emotional expression samples into the emotion recognition model, and take the last fully connected layer features for average pooling as the emotion expression feature.

We select the dimension of latent representations from $\left\{64,128,256\right\}$. During training, we use the Adam [59] optimizer and choose the learning rate from $\left\{10^{-3},10^{-4}\right\}$. We set the maximum number of epochs to 300 and the weight decay to $10^{-5}$. Dropout [60] is also employed, and we select the rate from $\left\{0.2,0.3,0.4,0.5\right\}$ to alleviate the over-fitting problem. We randomly select 5 answers (3 truths and 2 deceptions) from 24 answers in all samples as the validation set, and the remaining 19 answers as the training set. To mitigate randomness, we run each experiment five times and report the average result. And we choose the cross-entropy as loss function and the accuracy and AUC as the evaluation metric.

Unimodal Results In this section, we establish the unimodal benchmark for MDPE and report results in Table 2. We hope this benchmark can provide guidance for feature selection and point the way to developing powerful feature extractors. For the visual modalities, VIT achieves better results than CLIP, possibly because VIT is trained on supervised datasets and can reveal more deceptive clues than CLIP features that use text as a supervisory signal. For the acoustic modality, the deep features outperform handcrafted features. This may be because egemas are features related to emotions, using a emotion-related handcrafted acoustic feature may limit performance. In contrast, deep features can capture more universal acoustic representations for deception detection. For the textual modality, we focus on textual encoders that support Chinese (large language models are generally trained on multilingual corpora containing Chinese). And the textual modality can achieve best performance than the visual and acoustic modalities, which indicates that our dataset textual reveals more deceptive clues than other modalities.

Of course, we have also incorporated personality and emotional features into deception detection. Models trained with personality features almost often exhibit superior performance. These findings demonstrate the importance of personality in deception detection tasks. And the addition of emotional features has also improved the performance of deception detection models, but the improvement is not comparable to personality features, even some results have decreased. This may be because personality traits are directly usable features, and emotional features are extracted by emotion recognition models. The quality of features is also influenced by the emotion recognition models. In the future, better methods for using emotional expression features can be explored. By incorporating both personality and emotional features, the deception detection model achieved the highest performance in unimodal results, These results show that personality and emotional expression characteristics are indeed important for deception detection tasks, and using them can truly achieve deception detection modeling based on individual differences. It is worth mentioning that among all the unimodal results, Baichuan always achieved the best results. It is the model with the largest number of parameters among the features we used, and it further demonstrates the potential of large language models in deception detection.

Multimodal Results In Table 3, we select several well-performing unimodal features and report their fusion results. Experimental results demonstrate that multimodal fusion consistently improves performance. The reason lies in the fact that deception cues can be conveyed through multiple modalities. The integration of multimodal information allows the model to better comprehend the video content and accurately detection deception. Firstly, almost all features have achieved performance improvements in multimodal feature fusion, but the fusion of visual and acoustic modalities has hardly improved or even declined. It indicates that the addition of text features make the performance of the model tend to stabilize. This is consistent with human judgment. When people judge whether others are lying, they tend to express the truth or falsehood of the content, because it difficult to judge from visual or acoustic features. In the future, further exploration and research should be conducted on deceptive clues in visual and acoustic features.

Similar to unimodal results, models trained with personality features typically exhibit excellent performance. The addition of emotional features makes the performance of the model unstable. Some features have been improved, while others have decreased. Finally, the best result always comes from the three modalities fusion, and the three modalities fusion with personality and emotional characteristics has achieved the best performance.

Firstly, dimensional emotions are also important. In fact, we are labeling MDPE with dimensional emotions (including deception and emotion), which can not only study the impact of dimensional emotional features on lie detection at the individual level, but also explore more emotional clues in the deception process. Secondly, we extracts deep features from some pre-trained models and uses simple models for deception detection. In the future,larger models should be designed to be used for deception detection tasks. Thirdly, this article simply concatenates personality and emotional expression features to assist in deception detection tasks. In the future, more complex feature fusion or model fusion algorithms can be used for deception detection. Finally, in this paper, we only focuses on deception detection tasks, but MDPE includes individual personality, deception, and emotional expression information, which can not only support deception detection tasks, but also tasks such as personality recognition and emotion recognition. Even these individual level information can be used to assist other tasks, as shown in Figure 3. In fact, there have been many studies on the relationship between personality and emotion [61, 62, 63], but the lack of relevant datasets has led to slow research progress in this field, and we believe that MDPE can provide valuable resources for these research directions.

Firstly, although the subjects were required that they must lie about the deception questions, and verified the deceptive questions and content with the Interviewer after the deception experiment, we do not know whether the subjects have actually deceived on the deception questions. Secondly, although our induction videos have been annotated by professionals to demonstrate their reliability and validity, there are still some subject whose feelings are inconsistent with the emotions we expected to induce. This is because different people have different understandings of the video content, triggering different emotions. Thirdly, relying on self-assessment scales for data annotation is a subjective process for subjects, which may lead to bias in subsequent analysis. Different subjects may have significant differences in their perception of emotions. In addition, MDPE only collects native Chinese speakers, there may be cultural differences in deception detection. Finally, gender imbalance among subjects in MPDE is a common issue in human data collection [64, 65].