Commonsense-Focused Dialogues for Response Generation: An Empirical Study

We study commonsense-focused response generation for dialogues. Commonsense can be defined as “the basic level of practical knowledge and reasoning concerning everyday situations and events that are commonly shared among most people” Sap et al. (2020). Dialogue response generation is the task of generating a response turn $r$ in a conversational setting given previous history turns $h$. Thus by combining these two together, we want to examine models’ ability to produce responses that make sense or is plausible in terms of commonsense.

We propose a simple process for filtering commonsense in dialogues and present our analysis of three dialogue datasets with different focuses: DailyDialog Li et al. (2017), EmpatheticDialogues Rashkin et al. (2019), and MuTual Cui et al. (2020). The general idea is to refer to a commonsense knowledge graph (CSKG) such as ConceptNet Liu and Singh (2004) to identify potential commonsense triples ($e_{1},r,e_{2}$) expressing a commonsense assertion between turns in a dialogue. The following describes the detailed process.

After the process, we find that in the training sets, around 7k out of the 11k dialogues (63%) from Dailydialogue contain at least one matched triple between their turns, and 9.5k out of the 18k for EmpatheticDialogues (53%), and 5k out of 7k (73%) for MuTual dialogues. For the valid and test sets, the proportion of such dialogues is similar to that in the training sets for these three data sets.

Note that there are some limitations in our ConceptNet based data selection approach. First, we match concept entities based on just surface form, rather than semantic meaning or word senses in the context. Second, we are only using single word concepts, not phrases. Third, we are only considering one-hop concept relation identified in ConceptNet. The first one may affect the precision of the selected dialogues, and the other two reasons affect the recall. Without human annotated commonsense reasoning for dialog turns, we can not compute the exact performance of our filtering method. We plan to conduct some human annotation in our future work. Among the three data sets used in this study, the fact that there is a higher percentage of dialogues selected in MuTual may indicate that data focuses more on reasoning and thus is more likely to contain commonsense relations.

To facilitate commonsense-guided response generation training, we collect more dialogues with a focus on getting responses that require commonsense. Specifically, we make use of an existing commonsense multiple-choice benchmark SocialIQA Sap et al. (2019b) to crowdsource dialogues. This section provides background on SocialIQA, the crowdsourcing process, and the resulting dialogues.

We choose to use SocialIQA contexts because of three reasons: (1) they are specific instantiations of the event phrases found in the knowledge graph ATOMIC, which guarantees that there is at least one potential commonsense inference that can be made from the event; (2) ATOMIC covers a wide range of commonsense motivations and reactions and thus the contexts also embed diverse commonsense; (3) the rewriting process from SocialIQA ensures that the context sentences are well-formed and similar to natural sentences, which we expect is not hard for crowd workers to come up with a dialogue.

We pose three requirements for turkers in order to work on our task: locate in US, UK, or Canada; successful HITS are over 1000, and with more than 95% HIT acceptance rate. We pay MTurk workers $0.5 for each instance, roughly translating to 10 dollars per hour, well above the minimum wage of US.

To account for multiple plausible dialogues expanded from the context event, we assign each context to five different MTurk workers. We randomly sample 5k context sentences out of 13.5k filtered ones and collect five dialogues for each context, resulting in 25k dialogues. Examples of our collected dialogues are shown in Table 1.

For our collected data, we follow the same filtering steps as used for other existing data (Section  3.1). This ConceptNet filtering identifies about 11K dialog from the entire collection.²²2Our released data is based on these ConceptNet filtered conversations. We recruited in-house crowd workers to manually check the dialogs for profanity, speaker name, and other issues in the data. Note the experiments conducted in this paper used the initial collection, not this released version. Though we expect the SocialIQA contexts are from ATOMIC and may trigger more commonsensical dialogue, we find this is not the case since the percentage of dialogues containing ConceptNet triples is even lower than what we observed for the other existing data sets. This may be because of the limitations of the filtering method we are using as described earlier: matching to ConceptNet is based on surface textual form and concepts are on word-level, which omits deeper and more contextual commonsense relationships.

For response generation models, we take one of the state-of-the-art pre-trained language models, GPT2 Radford et al. (2019), and further train it on our training data sets. Specifically, the model is trained in a multitask fashion that minimizes the LM loss as well as the multiple choice loss following Wolf et al. (2019), and generates responses for a given dialog history.

We consider the follow three types of training data setups.

• •

  Existing data sets, including DailyDialog Li et al. (2017) (DD), EmpatheticDialogues Rashkin et al. (2019)(ED), and Topical-Chat Gopalakrishnan et al. (2019), a knowledge-grounded open-domain dataset with around 11k dialogues. MuTual Cui et al. (2020) is not included since it is designed for response selection.

• •

  As described in Section 3.1, we use ConceptNet to search for potential triples in response turns and filter three dialogue datasets, DD, ED, and MuTual. We combine the three filtered dialogues from these datasets to form our training data, named ‘filter existing’ (FE, total around 21K dialogues).

• •

  The third category includes our collected dialogues using SocialIQA contexts. This is used along with the FE data above: FE and all of the 25k collected dialogues (FE+new crowdsourced), and FE plus the 11K filtered dialogues of our collected data (FE+filtered crowdsourced).

To evaluate models’ response generation capabilities, we sample 10% of the FE+new data, resulting in 4.6k testing dialogues with no overlap with the training set of any of the settings above. We use GPT2 trained on different versions of dialogue data (6 trained GPT2 models in total) to generate a randomly sampled response for each turn of our test set dialogues.

We perform automatic evaluation on the test set and human evaluation on sampled dialogs.

Human evaluation is expensive to obtain, especially when the dataset is large. In addition, they are also subjective and hard to reproduce. Aiming to provide a reliable and scalable automatic metric focusing on commonsense in response generation, we propose an unreferenced automatic metric, which is a regression model trained from the human annotation scores for different responses. The metric is reference-free, meaning that it does not require human ground truth response when scoring a model-generated response, unlike referenced metrics such as BLEU, ROUGE, Meteor.

Table 2 shows results according to automatic metrics on our 4.6K testing dialogues. We find that perplexity scores for the GPT2 model trained on filtered existing dialogue data (FE), or plus new collected data (FE+Crowdsourced), are much lower than that just trained on existing datasets as is. There are several reasons for this. One is that since the testing dialogues are from the filtered version, training on those better matches the evaluation scenario. In addition, the test set is a sample of multiple data sets, and thus training on just one data set does not perform well. Finally the combined data (the last three rows in the table) is larger in size (see training size in Table 3). However, note the gain from the increasing training data size decreases in comparison to the difference between using the filter data settings and those single data sets. Meteor and ROUGE scores for all the trained models are quite low, and show less differences, probably indicating the limitation of these metrics for dialog response evaluation. BERTScore shows a similar pattern as perplexity in terms of model quality.

Table 3 shows the human evaluation scores on 300 responses for models trained with different types of data. The most obvious and perhaps expected finding is that GPT2, no matter trained on what types of data, is still way behind human performance (6.86 with high variance versus 9.3 with low variance). By analyzing different variables that cause performance difference, we find the following patterns, some of which are similar to using automatic metrics. (1) Using the Filtered Existing dialogue data (FE) helps improve the average of commonsense scores (more than 1 point improvement comparing to using individual data sets), but variance remains high; (2) Including our collected dialogues further increases the average (FE+Crowdsourced), and also decreases the variance in response quality in terms of commonsense plausibility; (3) Regarding our collected data, using the filter subset of it yields slightly better performance than using the entire data collection. This suggests that even though our data is collected using SocialIQA events, some dialogues may not be commonsense rich, which is also reflected by the percentage of dialogues that contain ConceptNet triples as discussed earlier. In addition, it shows that though overall increasing training data size benefits model performance, the quality of data plays a more important role. We plan to perform more sophisticated data selection and commonsense annotation for our data set in the future. We include examples of responses from humans and models trained on these different types of data as well as annotation scores in Appendix A Table 5. It shows some different characteristics of the responses, for example, empathy in the responses using ED model, and richer information (though inappropriate since they are off topic) using TC model.

We now examine the correlation of our proposed automatic metric (MLP regressor) with human scores on the testing portion of our annotations. We cross-validate on the collected dialogues with 0.8/0.1/0.1 proportions. For comparison, we consider three baselines: our MLP with only symbolic features, our MLP with only neural features, and FED Mehri and Eskenazi (2020a), which uses DialoGPT to score how likely the next turn after the response expresses confusion. It requires no training nor human references, and has been shown to correlate with humans judgements on different criteria (commonsense not included). Table 4 shows the Spearman’s correlation of the system computed scores and human annotation scores using all the annotated data in a cross-validation setup. We can see that our simple MLP-based regressor reaches the highest spearman’s correlation with human scores, outperforming other baselines significantly. However, such a correlation result still suggests a large gap for a reliable scorer targeting commonsense evaluation for dialogue response generation. We also notice that FED performs poorly in terms of commonsense evaluation. Furthermore, there is a large correlation drop when considering either symbolic or neural features alone in our model, indicating that they might each capture a different aspect for evaluating commonsense.

The majority of recent commonsense reasoning benchmarks Zellers et al. (2018); Talmor et al. (2019); Bisk et al. (2020); Sap et al. (2019b) test a model’s ability to choose the correct option given a context and a question; pre-trained language models have reached high performance on these benchmarks after fine-tuning. There have been many benchmarks that focus on reasoning abilities in multiple tasks such as reading comprehension Huang et al. (2019); Yu et al. (2020), dialogue systems Cui et al. (2020), and natural language inference Williams et al. (2018), which involve inferences on language. Recent work also aims to probe models in these tasks to see if reasoning is actually achieved Richardson and Sabharwal (2020); Richardson et al. (2020); Zhou et al. (2020). In this study we tackle the response generation problem in dialogues, with a focus on collecting commonsense rich dialog data and evaluating commonsense quality of model responses.

Recently open domain dialog systems have been modeled using end-to-end approaches, more specifically encoder-decoder architectures (Sordoni et al., 2015; Serban et al., 2017, 2016; Vinyals and Le, 2015). Recent work focused on finetuning large pre-trained transformer models Radford et al. (2019); Zhang et al. (2020) on dialog data. Many dialog datasets have been collected with different focuses such as incorporating knowledge (Gopalakrishnan et al., 2019; Dinan et al., 2018), empathy Rashkin et al. (2019), task completion Budzianowski et al. (2018), consistency Nie et al. (2020), personality Zhang et al. (2018) and reasoning Cui et al. (2020) within dialog systems. There has also been work on combining a variety of datasets to exhibit multiple attributes Roller et al. (2020).

Due to the diverse responses that a dialog system can output, referenced automatic metrics (such as BLEU, ROUGE, Perplexity) do not correlate well with human judgement of these systems (Deriu et al., 2020; Liu et al., 2016). As a result, human evaluation has become the de-facto standard to evaluate dialog systems. However human evaluation is costly. Recently model-based metrics have been proposed with good correlation with human annotations (Zhang et al., 2019; Sellam et al., 2020; Mehri and Eskenazi, 2020b, a; Tao et al., 2018; Lowe et al., 2017). Most metrics focus on evaluating the coherence or appropriateness of a response with respect to its dialog context. Mehri and Eskenazi (2020a) identified 18 different dialog qualities such as interesting and topic depth. However none of these metrics evaluate the commonsense of a response, which is the focus of this work.