Topic Shift Detection in Chinese Dialogues: Corpus and Benchmark

Previous studies explored the dialogue topic tasks and published the annotated topic dialogue corpus. For English, Xie et al. [7] annotated the TIAGE consisting of 500 dialogues with 7861 turns based on PersonaChat [11]. Xu et al. [8] built a dataset including 711 dialogues by joining dialogues from existing multi-turn dialogue datasets: MultiWOZ Corpus [12], and Stanford Dialog Dataset [13]. Both corpora are either small or limited to a particular domain, and neither applies to the study of the natural dialogue domain.

For Chinese, Xu et al. [8] annotated a dataset including 505 phone records of customer service on banking consultation. However, this corpus is likewise restricted to a few specialized domains while natural dialogues are more complicated. Natural dialogues have a range of topic shift scenarios, unrestricted topics, and more free colloquialisms in the utterances. The above corpus is insufficient to fill the gap in the Chinese natural dialogue topic corpus.

The task of dialogue topic shift detection is also in its initial stage and only a few studies focused on this task. As we mentioned in Introduction, topic segmentation is a similar task. Hence, we first introduce the related work of topic segmentation in dialogues. Due to the lack of training data on dialogue, early approaches of dialogue topic segmentation usually adopted an unsupervised approach using word co-occurrence statistics [14] or sentence topic distributions [15] to measure sentence similarity between turns to achieve detection of thematic or semantic changes. Recently, with the availability of large-scale corpora sampled from Wikipedia, supervised methods for monologic topic segmentation have grown rapidly by using partial tokens as ground-truth segmentation boundaries, especially neural-based methods [16, 17, 18]. These supervised solutions are favored by researchers due to their more robust performance and efficiency.

Dialogue topic shift detection is strongly different from dialogue topic segmentation. For the dialogue topic detection task, Xie et al. [7] proposed a detailed definition with two lines: the response-known task considering both the context and the response, and the response-unknown task considering the context only. However, methods based solely on the context are still scarce. Only Xie et al. [7] predict the topic shift or not based on the T5 model. Sun et al. [19] introduce structural and semantic information to help the model detect topic shifts in online discussions, which is similar to response-known task. It is imperative to address the dialogue topic shift detection. In general, the dialogue topic shift detection task is still a challenge, as it can only rely on the context information of the dialogue. In this paper, we solve the lack of response information by utilizing knowledge distillation and hierarchical contrastive learning.

Each dialogue in our corpus has a piece of news as a base document, which is not available in other corpus and can be used as additional information for further research and expansion. The news is from six domains, which brings our conversations closer to natural dialogue. Besides, the speakers in our corpus are not restricted in any way, which also makes it closer to natural dialogues. In addition, we annotated the fine-grained dialogues topics, refer to Section 3.2. Fine-grained labels are beneficial to promote further research on dialogue topics.

Compared with the existing Chinese topic corpus annotated by Xu et al. [8], the dialogues in our corpus do not have meaningless and repetitive turns. Also, the corpus is more than twice the size of the other corpus. In addition, the news in the corpus can be studied as additional information for the dialogues.

Following the annotation guidelines in TIAGE[7], we distinguish each dialogue turns whether changed the topic compared with the context. The response of a speaker to the dialogue context usually falls into one of the following cases in dialogues where the examples can be found in Table 1.

• •

  Commenting on the previous context: The response is a comment on what is said by the speaker previously;

• •

  Question answering: The response is an answer to the question that comes from the speaker previously;

• •

  Developing the dialogue to sub-topics: The response develops to a sub-topic compared to the context;

• •

  Introducing a relevant but different topic: The response introduces a relevant but different topic compared to the context;

• •

  Completely changing the topic: The response completely changes the topic compared to the context.

Among them, we uniformly identify the two cases of greeting and farewell specific to CNTD as the topic shift.

We chose the NaturalConv dataset [20] as the source corpus, which contains about 400K utterances and 19.9K dialogues in multiple domains. It is designed to collect a multi-turn document grounded dialogue dataset with scenario and naturalness properties of dialogue.

We consider NaturalConv as a promising dataset for dialogue topic detection for the following reasons: 1) NaturalConv is much closer to human-like dialogue with the natural property, including a full and natural setting such as scenario assumption, free topic extension, greetings, etc.; 2) NaturalConv contains about 400K utterances and 19.9K dialogues in multiple domains; 3) The average turn number of this corpus is 20, and longer dialogue contexts tend to exhibit a flow with more topics; 4) The corpus has almost no restrictions or assumptions about the speakers, e.g., no explicit goal is proposed [21].

We have three annotators for coarse-grained annotations and two for fine-grained annotations. Both annotations are divided into three stages as follows.

Co-annotation Stage First, for coarse-grained annotations, we draw a total of 100 dialogues from each domain of the NaturalConv dataset proportionally for a total of 2014 dialogue turns. In this stage, three annotators are asked to discuss every 20 dialogues they annotated, and each annotator is asked to give a reason for the annotation during the discussion. Finally, the Kappa value of all annotators for coarse-grained annotations at this stage is 0.7426. In addition, we annotated the fine-grained information based on the results of the complete coarse-grained annotations. Two annotators annotated the same 150 dialogues and discussed them several times for consistency. Finally, the kappa value of all annotators for fine-grained annotations at this stage is 0.9032. These kappa values confirm that our annotators already have sufficient annotation capabilities for independent annotation, as well as the high quality of our corpus.

Independent-annotation Stage We ensured the quality of each annotator’s annotation and judging criteria before starting the second phase of annotation. For both granularity annotations, we randomly assign the dialogues drawn from each domain to each annotator for independent annotation. At this stage, we annotate 1208 dialogues for coarse-grained annotations and 1158 dialogues for fine-grained annotations.

Semi-automatic Rechecking Stage Finally, we use a semi-automatic rechecking process to ensure that the corpus is still of high quality. On the one hand, we automatically format the dialogues with annotations to detect formatting problems caused by manual annotation. On the other hand, we automatically match the related news to each dialogue and check that the topic attributes are consistent with the dialogue to rule out any possible errors.

Due to the limited time, we randomly select 1308 dialogues from the NaturalConv dataset and annotate them with four annotators. Finally, we construct a Chinese natural topic dialogues corpus containing 26K dialogue turns.

As shown in Table 2, we randomly split them into 1041 train, 134 validation, and 133 test dialogues respectively, according to the percentage of different categories. In addition, we show the details of CNTD in Table 3, which shows that our corpus has enough topics and long turns which is suitable for dialogue topic detection. Finally, there are the statistics of our fine-grained labels, as shown in Table 4.

We count the number of dialogues with different numbers of topics, as shown in Fig. 2. On another side, we count the distribution of topic shift signals in dialogues, shown in Fig. 3. We can see there are a total of 21 turns and three peaks of topic shift signals, which occur in $2^{nd}$, $4^{th}$, and $18^{th}$ turns, respectively. The reason is that the dialogue in our corpus usually starts with a greeting and ends with a farewell, which leads to more topic shifts at the beginning and end of the dialogues. In addition, the NaturalConv corpus gives a piece of news as the base document of the dialogue, so there are more frequent transitions from news to derived topics, leading to the third highest peak in $4^{th}$ turn. However, we think this is consistent with a natural dialogue scenario because people often talk about recent news after daily greetings.

Existing studies cannot effectively predict topic shifts due to the lack of future information. To address this problem, we introduced a teacher-student framework for dialogue topic shift detection. The student side learns an implicit way of topic prediction from the teacher side through knowledge distillation.

In this framework, we employ a pre-trained model as an encoder on both sides to obtain semantic representations of the dialogues. Besides, we share the encoder weights on the teacher when encoding the context and the response information. For instance, the representation of $[CLS]$ of the last hidden layer state is taken as a dialogue-level representation $H$ as follows.

where $H\in R^{N\times d},i\in I=\left\{1,2,...,N\right\}$ denotes the index of samples in the batch, $C\in\left\{P,F,A\right\}$ where $P$ represents the context, $F$ represents the response, $A$ represents the full dialog. $E$ represents the encoder, $T$ represents the teacher, and $S$ represents the student.
On the teacher side, we connect the dialogue-level representations and then feed them into the linear layer to obtain the final detection results while on the student as follows.

where $W_{T}$ and $W_{S}$ denote the linear layers on the teacher and student sides, respectively.

In addition to calculating the cross-entropy loss of the final detection results, we establish the mean squared error loss between each hidden layer of the teacher and student encoders.

We consider that the response information learned by knowledge distillation solely is insufficient. And the unbalanced proportion of shift or not in the dialogue corpus makes the model perform poorly in distinguishing different shift cases. Therefore, we propose hierarchical contrast learning, consisting of information simulation loss at the high level and semantic coherence-aware loss at the low level, respectively. Both losses are based on contrast learning, but the former is to strengthen the learning of features and the latter is to alleviate the imbalance of labels.

For information simulation loss, this loss enables active learning for each context representation. And this effectiveness has been demonstrated by several works [1, 22, 23], the incorporation of global information permits local information representation with some predictive information. This helps our encoder obtain a representation with more reliable predictive information.

To alleviate the unbalanced proportion of labels in the dialogues, we propose a semantic coherence-aware loss based on supervised contrast learning (SCL). The main concept of SCL[24, 25] is to regard the samples of different categories as positive and negative samples from each other to address the issue of significant quantitative imbalance in the dataset[26]. This loss effectively alleviates the imbalance of shift cases and helps the model further distinguish between different shift cases.

High-level Information Simulation Loss We build the information simulation loss on both sides so that the context representations can be mapped to the same high-dimensional space. Thus, it is easier to learn the response information to improve final detection results. The following equation can be used to describe this loss.

where $A(i)=I-\left\{i\right\}$ denotes the samples in the current batch other than itself, $P$ denotes the context, $A$ denotes the full dialog, and $M\in\left\{T,S\right\}$ where $T$ denotes the teacher, $S$ denotes the student.

Low-level Semantic Conherent-aware Loss For a batch with $N$ training samples, a copy of the dialogue’s last hidden state $H$ is made to obtain $\overline{H}$ that is considered as the positive, and its gradient is detached. This results in $2N$ samples, then the semantic coherence-aware loss of all samples in a batch can be expressed as follows.

where $U\in R^{2N\times d},i\in I=\left\{1,2,...,2N\right\}$ denotes the index of samples in a batch, and $P\left(i\right)=I_{j=i}-\left\{i\right\}$ denotes samples of the same category as $i$ but not itself.

We train our model in two steps. The teacher is trained first, and its loss consists of two parts: cross-entropy loss between predictions and manual annotation labels, named $L_{NCE}$. And the information simulation loss for learning response representation is named $L^{T}_{ISL}$. The overall training loss is as follows.

Then, we train the student, which consists of four parts: cross-entropy loss between predictions and manual annotation labels named $L_{NCE}$, knowledge distillation between each hidden layer of both sides named $L_{KD}$, information simulation loss for the dialogue context named $L^{S}_{ISL}$, and the semantic coherence-aware loss for different shift cases named $L_{SCL}$. The above equation represents its overall training loss. As it proves to be arduous to fine-tune the weights assigned to the four losses, we ultimately opt for equal weighting across all of them.

Based on the train/validation/test dataset of CNTD we partitioned in Table 2 and previous work on TIAGE [7], we extract (context, response) pairs from each dialogue as input and the label of response as a target for the response-unknown task. In our experiments, every utterance except the first utterance of the dialogue can be considered as a response. As for evaluation, we report Precision (P), Recall (R), and Micro-F1 scores.

We use BERT as an encoder and fine-tune it during training. For both the TIAGE and CNTD corpus, all pre-trained model parameters are set to default values. We conduct our experiments on NVIDIA GeForce GTX 1080 Ti and NVIDIA GeForce GTX 3090 with batch sizes of 2 and 6 for both CNTD and TIAGE, with the initial learning rates of 2e-5. And we set the epochs of training to 20, and the dropout to 0.5.

For the pre-trained models in the experiment, we apply BERT-base-Chinese and MT5-base to obtain the semantic representation of the dialogues in CNTD, and we apply BERT-base-uncased and T5-base to obtain the semantic representation of the dialogues in TIAGE.

Dialogue topic shift detection is a new task and there is no complex model available, besides a simple T5 [7] that can be considered as the SOTA model. Since we employ BERT as our encoder and the T5 model is used in TIAGE, we use the pre-trained models of T5 [9] and BERT [27] as baselines. For BERT, we connect the utterances in the context and separate the last utterance with $[SEP]$. For T5, we also connect utterances in the context and classify the undecidable predicted results to the ‘not a topic shift’ category.

Table 5 shows the performance comparison between our model and the baselines, in which TS denotes our teacher-student model without the hierarchical comparative learning (HCL) and Ours denotes our final model, i.e., the addition of SCL on the student side based on the addition of ISL on both the teacher and student sides.

It can be found that on CNTD, our model achieves a good improvement and improves both precision and recall in comparison with the baselines. Although T5 does not perform poorly on recall, its precision is inadequate in comparison with BERT, and it is clear that T5 is not effective in predicting topics. In contrast, TS improved by 1.0 in Micro-F1 in comparison with BERT, which confirms that the teacher-student framework is effective in introducing response information. As well, Ours improved by 4.0 in micro-F1 in comparison with TS, and also showed significant improvement in P and R, which fully demonstrates that our HCL can improve the model’s ability to discriminate between different topic situations. In particular, our model improves on CNTD by 5.0 in comparison with the best baseline BERT, which shows the effectiveness of our proposed model.

To verify the effectiveness of the components used in our model, we conduct ablation studies on CTND, and the experimental results are shown in Table 6.

If we remove ISL on the teacher side ($-ISL_{S}$) or the student side ($-ISL_{T}$), the performance of the model decreased by 1.5 and 1.3 on the Micro-F1 value, respectively, with the largest decrease after removing the ISL on the student side. Although $-ISL_{T}$ has the highest precision in predicting topics and lower error probability than $Ours$ and $-ISL_{S}$. However, it can be seen that adding ISL at both the teacher and student sides can better improve the correct prediction rate. Moreover, if we remove ISL both on the teacher and student side ($-ISL_{TS}$), it achieves a similar performance on Micro-F1, in comparison with $-ISL_{S}$ and $-ISL_{T}$. However, it achieves the highest precision (58.8%).

If we remove SCL (-SCL) or HCL (-HCL) from our model, the Micro-F1 value of the models -SCL and -HCL drop from 53.9 to 52.4 (-1.5) and 49.9 (-4.0), respectively. These results show that our Semantic Conherent-aware Loss(SCL), and Hierarchical Contrastive Learning(HCL) are effective for this task, especially HCL.

In addition, we explore the performance of the dialogues with different numbers of topics to analyze our model in comparison with BERT, as shown in Table 7. It can be found that our model has a better performance than BERT on dialogues with fewer topics. Our model gets at least a 6% improvement in topic shift prediction on dialogues with 2 to 5 topics and obtains above-average performance. And when the number of topics increases to 9, the performance improves because the conversation length is still about 20 and the topics shift more significantly.

In Table 8, we also investigate the recall of the topic shift detection for various topic turns. Our model is improved for varying degrees across topic turns, with the most significant improvements in turns 7-9. Even in long topic shift cases, our model can obtain an effective boost. However, the performance of our model inevitably decreases compared to short topic shift cases. When there are fewer topic turns, the topic shift situation is simpler, so it is easier to determine. When the length of turns becomes longer and the situation becomes complicated, the topic of long turns has more information so it is easier to identify.

As shown in Table 9, it can be found that our model also achieves a good improvement on English TIAGE. Although our model is not the best on precision, we obtain the best performance on both recall and Micro-F1 values, especially on micro-F1 with a 5.8% improvement over T5. This proves that our model achieves the best performance both in English and Chinese.

We also conducted a case study. The prediction made by our model, the BERT model on the instance, and the manual labels are shown in Table 10. Compared with the BERT model, it is obvious that our model can accurately anticipate the change of topic in the instances corresponding to the utterances "Yes, that’s right.","It is, indeed, should pay attention to it." etc., belonging to the question-answering scenario. However, if you respond "Well, the policy has been implemented in place this time." and "And now we are promoting the development of children’s creative and practical skills." etc. belonging to the commenting on the previous context scenario, our model or BERT cannot accurately predict the topic shift in this scenario. This shows that detecting the topic shifts in natural dialogue is still challenging.

We further analyze the errors of the prediction produced in our experiments. Specifically, we analyzed the example to explore whether the error in the results of this example is prevalent in other dialogues. From Table 10, we can find that the wrong predictions at $14^{th}$ and $18^{th}$ turn. We predict "The teaching equipment must be updated, right?" as ‘not a topic shift’ and "Well, thanks to the government!" as ‘topic shift’.

We counted the appearance of many errors, and the errors are mainly divided into two categories. One is for the "Introducing a relevant but different topic" type of utterance. It was predicted that no topic shift occurred due to the lack of information about the future of the conversation. The other is the "commenting on the previous context" category. Since this type of response does not affect the integrity of the previous topic, it is mostly predicted to be a topic shift.