“You made me feel this way”: Investigating Partners’ Influence in Predicting Emotions in Couples’ Conflict Interactions using Speech Data

Understanding the emotions partners feel during and after conflict interactions is important because of its long-term effects on couples’ relationship quality and stability (Gottman, 1994). Happy couples, for example, experience more positive and less negative emotions during conflict interactions compared to unhappy couples (Levenson et al., 1994). The current study focuses on one fundamental aspect of the conflict mechanism — how the emotional experience within each partner is influenced by the behavioral exchange between partners.

A crucial aspect of conflict interaction in couples is how the behavioral exchange makes each person feel during and after the interaction (Ruef and Levenson, 2007). But although both partners experience the same interaction, they can feel very differently about it. For example, if we assume that partner A shows contempt and criticizes partner B, we can assume that partner A might feel angry or superior whereas partner B might feel hurt or humiliated. Thus, the experience can be very different for the partner who communicates something compared to the partner who perceives it (Butler, 2011). This differentiation allows us to reflect on another mechanism, namely that the emotions a person experiences are the results of two kinds of influences. Obviously, a person’s emotional experience is constantly influenced by a partner’s behavior as a kind of co-regulating force, talk turn by talk turn. In addition, however, each person has the ability to regulate one’s own emotional response (e.g., cognitive appraisal, emotion regulation) (Gross, 2014), which then affects one’s own subsequent behavioral response. Thus, what partners experience emotionally during and at the end of a conflict interaction is a reflection of the co-regulation and self-regulation processes (Boker and Laurenceau, 2006).

To better understand emotions in couples and their impact on relationships, often self-report assessments are used in which each partner is asked to provide a rating of their own emotions right after an interaction, or partners are asked to watch a video recording of the interaction and provide continuous ratings using a joystick, for example, (Ruef and Levenson, 2007; Hilpert et al., 2020). Self-reports are burdensome to complete and may not be collected frequently. This means that the relationship between behavior and emotions cannot be studied often. Thus, an automatic emotion recognition system would allow scaling of couples research.

Various works have used linguistic features (i.e., what has been said) and paralinguistic features (i.e., how it was said) to predict the emotions of each partner in couples interactions more broadly (Black et al., 2010; Lee et al., 2010; Black et al., 2013; Lee et al., 2014; Xia et al., 2015; Li et al., 2016; Chakravarthula et al., 2015; Tseng et al., 2016; Tseng et al., 2017; Chakravarthula et al., 2018; Tseng et al., 2018) and in conflict interactions in particular (Chakravarthula et al., 2019; Boateng et al., 2020). Most of these works have used observer ratings (perceived emotions) as labels rather than self-reports (one’s actual emotions). Hence, the prediction task becomes that of recognizing external individuals’ perception of each partner’s emotion rather than each partner’s emotion per their own assessment. Though similar, the latter is more challenging than the former for a number of reasons. First, the rating might be biased and may not reflect their actual emotions over the period the rating is for (e.g., the past 5 minutes). Whereas for observer ratings, coders are generally trained over several weeks, it is done by more than one person and various approaches are employed to resolve ratings that are not in agreement and ensure the validity of the labels. Second, the self-reported emotion may not be reflected in that partner’s behavior in comparison to observer ratings which are purely based on behavioral observation.

Despite these challenges, insights from psychology research could be leveraged to make the prediction task easier. Specifically, given that partners’ behaviors influence each other’s emotions in conflict interaction, the behavior of both partners could be considered to better predict each partner’s end-of-conversation emotion. However, it is yet to be investigated how doing so compares to using each partner’s own behavior only in terms of emotion prediction performance. In this work, we used a dataset collected from 368 couples who were recorded during an 8-minute conflict interaction, extracted linguistic and paralinguistic features, and used machine learning approaches to predict how each partner felt directly after the conflict interaction (self-reported emotion). We answer the following research questions (RQs)

RQ1: How well can the end-of-conversation emotion of each partner be predicted by their own behavior — a combination of linguistic and paralinguistic data? (self-regulation)

RQ2: How does the prediction performance change when including the other partner’s behavior — (a) linguistic only, (b) paralinguistic only, and (c) combination of linguistic and paralinguistic data? (co-regulation)

Our contributions are (1) an evaluation of how well a partner’s own linguistic and paralinguistic features predict one’s own end-of-conversation emotion (2) an investigation of how the prediction performance changes when including one’s partner’s features (linguistic, paralinguistic, and both) (3) the use of a unique dataset — spontaneous, real-life, speech data collected from German-speaking, Swiss couples (n=368 couples, N=736 participants), which is the largest ever such dataset used in the literature for automatic recognition of partners’ end-of-conversation emotion. The insights from our work would advance the use of methods to automatically recognize the emotions of each partner which could enable research and applications to better understand couples’ relationships in therapy and the real world.

The rest of the paper is organized as follows: In Section 2, we describe our data collection, preprocessing and feature extraction, in Section 3, we describe our experiments and evaluation, in Section 4 we present and discuss our results, in Section 5, we present limitations and future work, and we conclude in Section 6.

We run experiments using scikit-learn (Pedregosa et al., 2011) the following machine learning models: support vector machine (SVM) algorithm with linear and radial basis function kernel, and random forests. We trained models to perform binary classification of each partner’s self-reported positive and negative emotion using different feature sets. We used the four feature sets described in the previous section. To train and evaluate the models, we used nested K-fold cross-validation (CV). The nested procedure consisted of utilising an “inner” run of 5-fold CV for hyperparameter tuning, followed by an “outer” run of 10-fold CV which utilizes the best values for each hyperparameter found by the “inner” run. We prevented data from the same couple from being in both the train and test folds, thereby evaluating the model’s performance on data from unseen couples. As the data was imbalanced, we used the metric balanced accuracy which is the unweighted average of the recall of each class and confusion matrices for evaluation. We used the “balanced” hyperparameter for all the models to mitigate the class imbalance while training. We compare to a random baseline of 50% balanced accuracy.

Our results are shown in Table 1. The baseline approach with multimodal fusion was not better than chance in predicting men’s emotions at the end of the conflict interaction (49.8%). This is unexpected as this indicates that men’s own behaviors during the interaction are not related to how they feel at the end of the interaction. This might be due to self-regulation processes or not showing much emotions during the interaction. This is clearly different from women as their emotions can be predicted by their own behavior (59%). Thus, it seems that women seem to express their emotions more clearly in their behavior. These results of poorer prediction performance for men compared to women is consistent with the results of (Boateng et al., 2020).

Including features from the interacting partner improved the results for both men (52.3%) and women (63.2%). These results are consistent with psychology research that the behavior of partners’ have an effect on each other’s emotions in conflict interaction (Gottman, 2014; Butler, 2011). This is a crucial finding because (i) previous research shows that the behavior of one person influences the behavior of the other person (Gottman, 1994) (ii) as well as that the emotional changes of one person affect the emotions of the other (Butler, 2011). However, this is the first study showing that behavioral features assessed during the conflict interaction can be used to predict one partner’s emotion at the end of the conversation. In addition, the improvement in women’s emotion prediction at the end of interaction is greater when including their partner’s linguistic data (64.8%) whereas there is hardly any difference when including partner’s paralinguistic features. This is a surprising finding as women generally pay more attention to paralinguistic cues (Gottman and Levenson, 1992). Notably, the results are different for men. The prediction for men’s emotions slightly increases when including their partner’s linguistic features but the prediction improves substantially when including women’s paralinguistic features. Although we do not know which specific paralinguistic features are the main drivers for predicting the emotions, this finding is in line with prior findings — when women “nag”, men experience strong negative physiologic reactions and tend to withdraw (Gottman and Levenson, 1992; Gottman, 1994). Future research is needed to investigate which aspects of one partner’s behavior exactly cause their own emotion prediction performance to decline. In addition, these results have implications for the kind of behavioral information to consider to best predict each partner’s end-of-conversation emotions.

We show the confusion matrices for the best models for the men and women in Figures 1 and 2 respectively. They reveal the models’ challenges at recognizing positive emotions (1), resulting in them being misclassified as negative emotions (0).

Further work is needed to investigate if these results are the same for couples in a different cultural context and also explore the effect of the interacting partners’ behavior at a more granular level such as on a talk-turn basis. More fine-grained emotion ratings may be needed to investigate that. Other fusion approaches like late fusion can be explored.

We used BERT as a feature extractor in this work. Generating domain-specific sentence embeddings via fine-tuning the BERT model and exploring deep transfer learning models for the paralinguistic features may improve the results. BERT models have been shown to encode gender and racial bias because of the models they are trained on. Further investigations are needed in the future of potential biases in prediction (Bender et al., 2021).

Finally, we used manual annotations and transcripts. To accomplish true automatic emotion prediction, speaker annotations need to be done automatically and our approach needs to use and work for automated transcriptions. Current speech recognition systems do not work for this unique dataset given that couples speak Swiss German, which is (1) a spoken dialect and not written, and (2) varies across different parts of the German-speaking regions of Switzerland. Further work is needed to develop automatic speech recognition systems for Swiss German.

In this work, we investigated one’s own and the partner’s behavior in predicting end-of-conversation emotions in the context of conflict interactions in German-speaking Swiss couples. We extracted linguistic features using BERT and paralinguistic features using openSMILE. We fused both features in a multimodal approach for each partner. We also fused the features of both partners to predict the emotions of each partner. Our results show that including the behavior of the other partner improves the prediction performance. Furthermore, for men, considering how their female partners spoke is most important, and for women considering what their male partners said is most important in getting better prediction performance. These insights have implications for the behavioral information to (not) include to better predict each partner’s end-of-conversation emotions which will enable a better understanding of couples relations in research, therapy, and the real world.