SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation

Before dialogue segmentation, text segmentation was already a promising task for analyzing document structures and semantics by segmenting a long document into several pieces or sections. To the best of our knowledge, cue phrases were used for text segmentation as early as 1983 Brown and Yule. (1983); Grosz and Sidner (1986); Hirschberg and Litman (1993); Passonneau and Litman (1997). Subsequently, Hearst (1997) proposed TextTiling to distinguish topic shifts based on the lexical co-occurrence. Choi (2000) and Glavaš et al. (2016) also proposed clustering-based and graph-based methods for text segmentation. Before large-scale datasets existed, these unsupervised methods performed relatively well but easily reached the ceiling. Facing this challenge, Koshorek et al. (2018) collected 727k documents from Wikipedia and segmented them based on the natural document structures. With the Wiki727K dataset, they formulated text segmentation as a supervised learning problem and consequently led to many outstanding algorithms Arnold et al. (2019); Barrow et al. (2020); Glavaš and Somasundaran (2020); Gong et al. (2022). However, previous work Xie et al. (2021) experimented that it is inadvisable to directly apply these studies to dialogue systems because dialogue possesses unique characteristics.

Recently, limitations such as lacking common sense Shang et al. (2015) and knowledge hallucination Roller et al. (2021) have been identified in dialogue systems. To solve these issues, previous studies focused on extracting useful information from knowledge graphs Jung et al. (2020), images Das et al. (2017), and documents Zhou et al. (2018); Dinan et al. (2019), yielding informative responses. Feng et al. (2020, 2021a) proposed two DGDS corpora with rich annotations, including role information, dialogue act, and grounding reference of each utterance, which enable knowledge identification learning and reveal how interlocutors think and seek information during the conversation.

Unlike documents that have natural structures (e.g., titles, sections, and paragraphs), dialogue segmentation involves several challenges, including the vague definition of segmentation points, which increases the difficulty of data collection even for humans. Recently, Xia et al. (2022) manually collected 562 annotated samples to construct the CSTS dataset, and Xie et al. (2021) constructed the TIAGE dataset containing 300 samples for training. Not only was the size of annotated datasets too small for supervised learning, they also observed a similar issue: such datasets suffered from boundary ambiguity and noise frequently, even when the segmentation points were determined by human annotators. Zhang and Zhou (2019) claimed that they collected 4k manually labeled topic segments and automatically augmented them to be approximately 40k dialogues in total; however, until now, these annotated data are still not publicly available.

For evaluation, Xu et al. (2021) constructed the Dialseg711 dataset by randomly combining dialogues in different domains. Automatically generating large-scale datasets in this way seems plausible, but the dialogue flow around the segmentation point is not natural, resulting in semantic fractures. Though the dataset can still serve as an evaluation dataset, this characteristic can be interpreted as shortcuts Geirhos et al. (2020), leading to poor generalization in real-world scenarios. Based on a prevalent DGDS dataset, doc2dial Feng et al. (2020), Xing and Carenini (2021) used the training and validation sets to construct an evaluation dataset for dialogue segmentation but missed some crucial details. For example, as shown in Figure 1, the user begins the conversation with a general query referring to the document title but the following utterance from the agent refers to the first section. In this case, Xing and Carenini (2021) mistakenly assigned them to different segments, even though they are still discussing the same topic. Furthermore, most existing datasets neglected useful dialogue-related annotations like role information and dialogue act, while we inherit these annotations from the existing DGDS corpora to support future research. We will describe our dataset construction approach in Section 3.2. Table 1 summarizes the datasets mentioned above.

Doc2dial Feng et al. (2020) and MultiDoc2Dial Feng et al. (2021a) are two popular and high-quality document-grounded dialogue corpora. Compared to other DGDS datasets Zhou et al. (2018); Dinan et al. (2019), they contain fine-grained grounding references for each utterance, as well as additional dialogue-related annotations like dialogue acts and role information. Based on these annotations, we can construct a supervised dialogue segmentation dataset without human effort. Detailed information about doc2dial and MultiDoc2Dial can be found in Appendix B.

Since doc2dial and MultiDoc2Dial provide the grounding texts for utterances, such that we can understand what the utterances are about according to the grounding references in documents. Take Figure 1 as an example. The user query ‘Will my child receive the full payment up front or monthly payments’ refers to the information from the section ‘Benefits For Your Family’. Then, the agent reads the document, seeking information from the same section, and provides a proper response. In this case, we know they are discussing a same topic because the subject of these utterances locates in the same section of the document. During this interaction in the DGDS scenario, we find that it provides an alignment between documents and dialogues. In this manner, we can define the segmentation points of dialogue based on the inherent structure of documents considering the grounding reference of each utterance. Furthermore, the grounding reference can reflect what interlocutors are thinking about when talking, which precisely matches the definition discussed in Section 1. Thus, this validates the rationality of our data collection method. In this paper, we focus on segmenting this type of dialogue, which is also dominant in recent ChatGPT-like dialogue systems (e.g., ChatPDF⁴⁴4https://www.chatpdf.com/). Surprisingly, our empirical studies indicate that models trained on our datasets can also perform well in other types of dialogue (i.e. non-document-grounded dialogue). We will discuss it later in Section 5.4.

Based on this idea, we determine that one utterance starts a new topic when its grounding reference originates from a different section in the grounded document compared to its previous utterance. Besides, there may be multiple grounding references for one utterance. In such a case, if all references of the current utterance are not located in the same section of any reference of the previous utterance, we regard it as the beginning of a new topic. Otherwise, these two utterances refer to the same topic. Note that MultiDoc2Dial has already had document-level segmentation points, but our definition is more fine-grained and can describe the fine-grained topic shifts in conversations.

Furthermore, we carefully consider some details in the DGDS datasets. In the doc2dial and MultiDoc2Dial datasets, users sometimes ask questions that require agents’ further confirmation, like ‘Hello. I’d like to learn about your retirement program’ in Figure 1. However, in this case, though these queries and their following utterances are discussing the same topic, their grounding spans may not locate in the same section, leading to mistakes. Therefore, we corrected it as a non-segmentation point if the dialogue act of utterances from users is ‘query condition’ (the user needs further confirmation). Besides, we also removed the dialogues with meaningless utterances like ‘null’/‘Null’ to further improve the quality of our supervised dataset.

Moreover, the dev/test sets of the DGDS datasets contain some unseen documents with respect to the training set. Consequently, some dialogues in the dev/test sets discuss unseen topics with respect to the training set. Therefore, following the official splitting can serve as an excellent testbed to develop dialogue segmentation models with high generalization ability. Algorithm 1 shows the procedure of determining the segmentation points for a document-grounded dialogue. Note that the $g_{i}$ is a set containing single/multiple grounding reference(s).

Besides, doc2dial and MultiDoc2Dial provide useful dialogue-related annotations about the role (user or agent) and dialogue act of each utterance (e.g., respond solution). We inherit them to enrich our supervised dialogue segmentation corpus. Intuitively, dialogue segments are characterized by specific patterns of roles and dialogue acts. For example, when users ask questions, the agents will provide answers or ask another question to confirm some details. The discussed topics are generally maintained until the problem is solved. Therefore, we hypothesize that introducing dialogue act information is helpful for dialogue segmentation. We conducted a pilot experiment on the relationship between the segmentation points and the dialogue acts, proving our hypothesis. Figure 2 illustrates the distribution of dialogue acts for all utterances and those serve as the segmentation points in the test set of SuperDialseg. We observe that most dialogue acts are highly related to the segmentation points. Specifically, utterances with dialogue acts DA-1, DA-5, DA-6, and DA-7 probably act as segmentation points, which indicates that a topic usually ends when solutions are provided, or the previous query has no solution. Therefore, these extra dialogue-related annotations are significant for the task of dialogue segmentation. We also conducted an analysis in Section 5.4.4 to further confirm our hypothesis.

Table 2 summarizes the statistics of the SuperDialseg dataset. Detailed distribution of the number of segment is shown in Appendix C.

To verify the quality of our proposed dataset, we conducted human verification to confirm the rationality of our data collection method. We randomly selected 100 dialogues from the test set of SuperDialseg and each dialogue was annotated by two human annotators. We collected the annotations from the crowd-sourcing platform, Amazon Mechanical Turk ⁵⁵5https://www.mturk.com/, and 20 valid workers participated in our annotation task. Each assignment contains 10 dialogues to be annotated.

Since it is a verification process, it should be conducted more carefully than collecting data for training. Therefore, we only allowed workers whose HIT approval rate is greater than 95% to access and also rejected some invalid results, for example, including too many consecutive segmentation points. Detailed rejection rules and the user interface of our verification task can be found in Appendix D. For each dialogue, we asked two unique workers to do the annotation and calculated the Cohen’s Kappa scores between their annotations and the labels constructed by our data collection method. Finally, the Kappa score is 0.536, which is higher than TIAGE’s Kappa score. Therefore, we can conclude that SuperDialseg has similar or even higher quality than the manually collected corpus (i.e., TIAGE) so that it can serve as a good training resource and testbed for the task of dialogue segmentation. Note that our automatic data collection method does not require human effort and can be easily scaled up as long as extra annotated document-grounded dialogue corpora are available.

SuperDialseg is our proposed dataset, containing 9,478 conversations with rich annotations. Details are provided in Section 3.

TIAGE Xie et al. (2021) is a manually annotated dataset extended from PersonaChat Zhang et al. (2018) comprising 500 samples, where 300, 100, and 100 samples are for the training, testing, and validation sets, respectively.

DialSeg711 Xu et al. (2021) is an evaluation dataset containing 711 multi-turn conversations, which are synthesized by combining single-topic dialogues sampled from the MultiWOZ Budzianowski et al. (2018) and the Stanford Dialogue Dataset Eric et al. (2017).

We construct an extensive benchmark involving 18 models across five categories, including random methods, unsupervised traditional methods, unsupervised DL-based methods, LLM-based methods, and supervised DL-based methods.

Random Methods. We design two random methods to show the bottom bound of the dialogue segmentation task. In an $n$-turn dialogue, we randomly select the number of segmentation points $c$ from $\{0,\cdots,n-1\}$. Then, Random places a segmentation point with the probability $\frac{c}{n}$, whereas Even distributes the segmentation boundaries evenly based on the interval $\lfloor\frac{n}{c}\rfloor$.

Unsupervised Traditional Methods. We select three representative algorithms in this category. BayesSeg Eisenstein and Barzilay (2008) is a Bayesian lexical cohesion segmenter. GraphSeg Glavaš et al. (2016) segments by solving maximum clique problem of a semantic association graph with utterance nodes. TextTiling Hearst (1997) determines the segmentation point when the calculated depth score of a pair of adjacent utterances is beyond a given threshold.

Unsupervised DL-based Methods. Song et al. (2016) enhanced TextTiling by word embeddings. We use Glove embedding Pennington et al. (2014) here and denote it as TextTiling+Glove. TextTiling+[CLS] Xu et al. (2021) further enhanced TextTiling with [CLS] representation from pre-trained BERT Devlin et al. (2018). Xu et al. (2021) improved the TextTiling algorithm to be the GreedySeg algorithm to provide more accurate dialogue segments for response selection. Xing and Carenini (2021) designed TextTiling+NSP, which uses the next sentence prediction (NSP) score of BERT as the similarity between two adjacent utterances to enhance TextTiling, and also pre-trained a coherence scoring model (CSM) on large-scale dialogue corpora with 553k samples to compute the coherence score of two adjacent utterances instead. Koshorek et al. (2018) proposed an LSTM-based supervised text segmentation model called TextSeg. Since it is trained on the Wiki727K dataset without using annotated dialogue segmentation datasets, we regard it as an unsupervised method for dialogue segmentation, denoted as TextSeg_text.

LLM-based Methods. Recently, LLMs achieve remarkable success in various fields. We investigate how well they can segment dialogues by applying the InstructGPT Ouyang et al. (2022) and ChatGPT based on simple prompt templates.

Supervised DL-based Methods. We design some strong baselines based on pre-trained language models (PLMs), including BERT, RoBERTa Liu et al. (2019a), and TOD-BERT Wu et al. (2020). Besides, we train TextSeg but using SuperDialseg or TIAGE dataset, denoted as TextSeg_dial. Xie et al. (2021) proposed RetroTS-T5, which is a generative model for dialogue segmentation.

We try our best to reproduce the reported performances in published papers with official open-sourced resources and our implementation. Most of the results are similar or even superior to those reported. Further details about all the methods and the implementation are provided in Appendix A for reproducing the results in our benchmark.

To evaluate the model performances, we use the following metrics: (1) $P_{k}$-error Beeferman et al. (1999) checks whether the existences of segmentation points within a sliding window in predictions and references are coincident; (2) WindowDiff (WD) Pevzner and Hearst (2002) is similar to $P_{k}$, but it checks whether the numbers of segmentation points within a sliding window in predictions and references are the same; $P_{k}$ and WD compute soft errors of segmentation points within sliding windows, and lower scores indicate better performances. (3) $F1$ score; (4) mean absolute error (MAE) calculates the absolute difference between the numbers of segmentation points in predictions and references; (5) By considering the soft errors and the $F1$ score simultaneously, we use the following score for convenient comparison:

suggested by the ICASSP2023 General Meeting Understanding and Generation Challenge (MUG).⁶⁶6https://2023.ieeeicassp.org/signal-processing-grand-challenges/

For the $P_{k}$ and WD, previous works adopted different lengths of sliding windows (Xu et al. (2021) used four, but Xing and Carenini (2021) used different settings), leading to an unfair comparison. Following Fournier (2013), we set the length of the sliding window as half of the average length of segments in a dialogue, which is more reasonable because each dialogue has a different average length of segments. In contrast to Xing and Carenini (2021), we use $F1$ (binary) instead of $F1$ (macro) in the evaluation, because we only care about the segmentation points.

We conduct extensive experiments to demonstrate the effectiveness of our proposed dataset and reveal valuable insights of dialogue segmentation. Table 3 shows the results of our benchmark.