Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks

Deception detection has been a topic of interest across many research fields – ranging from psychology [1] to computer science [2]. With an ever-growing accessibility to multimodal media, for instance social media like YouTube and Snapchat, the detection of deceit based on multimodal data becomes increasingly necessary.

While deception detection is widely used in police interrogation, law enforcement, and employee security screening, the methods used often have a large time-requirement and rely highly upon physiological sensors and human experts, leading to bias and poor accuracy [3]. There have been efforts to eliminate the need of human experts and introduce automated approaches. Machine learning methods have been used for the purpose of deception detection in the past, and efforts have been made to leverage multiple modalities to make predictions on the truthfulness of unseen data [2].

These previous studies relied either on a single modality or on integrated multiple modalities in order to detect deceit using regular classification methods. The usage of a single modality might not provide enough information in order to detect deceit. On the other hand, the usage of multiple modalities means more information, and accordingly provides improved performance in many cases, reaching approximately 60-70% accuracy [4, 5].

This implies that there is still room for improvement, and provides the opportunity to take advantage of the availability of multiple modalities to apply advanced learning techniques. Recent studies have shown that convolutional neural networks (CNNs) [6] can improve the state-of-the-art performance on various tasks, including image analysis [7, 8], image classification [9, 10], localization and mapping [11, 12, 13], natural language processing [14, 15], sentence classification [16, 17], which most recently inspired researchers’ interests in utilizing deep learning into the deception detection problem. For instance, [18] implemented a fake review detection model using CNNs. However, a single modality was used to construct the network. An additional concern with the usage of multimodal data is the difficulty of collecting such data compared to a single modality. This fact causes the size of multimodal datasets to be limited, which may negatively affect the performance of deep learning methods, which do traditionally use very large datasets for training.

This paper addresses the problem of deception detection using multimodal neural networks. The paper makes three important contributions. First, we use neural networks to learn from two separate modalities, namely the linguistic and physiological modalities. Second, we construct a fused neural network that learns from both modalities, which to our knowledge has not been attempted before. Third, we compare our approach with earlier approaches that used regular machine learning techniques. Furthermore, we address the issues that arise using a CNN with a small training dataset by using a simple approach to solve the overfitting and large variance problems, namely using majority voting. We additionally devise a new procedure to deal with small datasets, including choosing an appropriate number of parameters as well as fixing the previous trained network weights to form a modality-wise training process.

This paper is organized as follows. Section II surveys some related work. Section III describes the dataset we used. Section IV illustrates the proposed deep learning approaches utilized for submodules as well as the whole framework. Section V explains the experimental setup including data processing and feature extraction. Section VI discusses our experimental results. Finally, concluding remarks and future work are provided in Section VII.

Traditional methods mainly focused on the physiological indicators of deceit as the case with polygraph tests, such as blood pressure, respiration rate, and skin conductance. Different factors can affect the reliability of polygraphs including the fear of being perceived as a liar and the stress of being tested [19]. Additionally countermeasures to fake innocence can be used, such as lying in the pretest questions and muscle tensing [20].

Another alternative to detect deception, for instance, is extracting features from the speaker’s speech. Different studies have analyzed whether verbal cues were good indicators of deceptive behaviour. Examples of these clues included the speaker’s pitch and speaking rate [21]. Other linguistic features have been extracted as well, such as the quantity, diversity, complexity, and specificity of messages, the word count and number of self-references, the keystroke dynamics and typing patterns, the corpus statistics and syntactic patterns, and the writing styles. There have also been efforts to use thermal imaging features for the purpose of detecting deception [22].

The availability of multiple modalities offers the opportunity of extracting more information by considering the correspondences that exist naturally between multiple data sources [23]. In the domain of deception detection, feature fusion between linguistic, thermal, and physiological features has been explored for crowd-sourced data [4].

For the purpose of automated detection of deceit, there has been research into applying traditional machine learning techniques. Such initiatives cast deception detection as a classification task, and use the available data to learn parameters for the model to be used for classification [24].

A more recent direction is the application of deep learning algorithms in this problem domain. Deep Learning methods have been used in natural language processing problems. For instance, Convolutional Neural Networks (CNNs) were used to produce state-of-the-art results on several problems in NLP [16]. Deep Learning for deception detection is more scant. Recent attempts were proposed to detect fake news and spam [25]. A new dataset for fake news has been benchmarked and released [26].

Our dataset includes two scenarios, namely “Abortion” and “Best Friend”. The subjects were asked to sit comfortably on a chair in a lab and were connected to four physiological sensors including blood volume pulse, skin conductance, skin temperature, and abdominal respiration sensors. The participants were informed of the topic matter before each individual recording. In the two scenarios, subjects were allowed to speak freely first truthfully and then deceptively.

Subjects. The multimodal dataset includes recordings collected from 104 students, including 53 females and 51 males. All subjects expressed themselves in English, had several ethnic backgrounds, and had an age range between approximately 20 and 35 years.

Abortion. In this scenario participants were asked to provide first a truthful and then a deceptive opinion about their feelings regarding abortion and whether they think it is right or wrong. The experimental session consisted of two independent recordings for each case.

Best Friend. In this scenario subjects were instructed to provide an honest description of their best friend, followed by a deceptive description about a person they cannot stand. In the deceptive response, they had to describe an individual they cannot stand as if he or she was their best friend. Hence, in both cases, the person was described positively.

Deep learning is an approach that has seen rapid growth in terms of popularity and usage, especially for classification tasks. We opted to use it for our classification task, where we aim at classifying the data as “truthful” or “deceptive”.

Our data is from two sources, namely the transcripts of the participants’ responses, and the physiological data collected during the recordings. Accordingly, we utilize a linguistic CNN (LingCNN), a physiological CNN (PhysCNN), and a BiModal CNN network. The latter one fuses the previous two networks. In addition, a word2vec model devised by [27] is used to transfer the transcripts to vectors as the input to our LingCNN.

Considering the size of our dataset is only $416$ instances, we set suitable hyperparameters, which correspond to reasonable numbers of weights in our networks. Furthermore, we utilize a modality-wise training fashion for our BiModal CNN, where we first train the linguistic and physiological CNNs, then use their output features as input for the BiModal CNN. We test our design using different experiments.

In this section we describe our experimental setup, including the data preprocessing as well as the training and testing procedures.

Our entire dataset consists of 416 samples including the “Abortion” and “Best Friend” topics. We evaluate the performance of the features extracted from each of the two topics as well as both topics combined using the overall accuracy and class recall. Moreover, we compare the performance of individual modalities to that of their combination. Furthermore, we compare the performance of our proposed networks to that of learning using regular classifiers such as Decision Tree, Support Vector Machine (SVM), and Logistic Regression.

This paper devised a method for using deep learning along with linguistic and physiological data for deception detection. The paper is the first, to our knowledge, to use multimodal data and deep learning to detect deception.

From the experimental results, we observed that the linguistic modality worked significantly better than the physiological modality. One of the reasons is that the linguistic modality used all the information in the transcripts, while the physiological modality simply averaged the data over the whole time period, which could result in loss of some physiological patterns in the learning process.

It can also be noticed that in the majority of the cases, the bimodal network achieved better performance that the unimodal ones. This indicates that the proposed fused neural network can integrate and learn discriminative features from multimodal data, which results in improved and more reliable performance.

For training and testing on the same topic, we note that by combining both modalities, the overall accuracy is higher than that obtained using the individual modalities. The same trend is observed for cross-topic learning, as well. We can therefore conclude that bimodal fusion has an overall advantageous effect over using individual modalities. This may be explained by considering that the fused network was provided by richer information using the two modalities.

Our experiments also indicated that cross-topic learning leads to a decrease in the performance for our model especially for the linguistic modality, which indicates that the performance is topic-dependent.

For future work, we will consider performing a time-series analysis to potentially discover time-dependent relationships among the data. For the BiModal CNN, we will also extract different sizes of hidden layers from the LingCNN and PhysCNN, and then concatenate them to form new feature vectors.