The Conversational Short-phrase Speaker Diarization (CSSD) Task: Dataset, Evaluation Metric and Baselines

Speech data is essential for data-driven spoken language processing methods [5, 6, 7, 8]. For speaker diarization task, there are some multi-speaker datasets collected in diverse scenarios, e.g. meeting scenario and dialog scenario. For meeting scenario, CHIL [9] and AMI [10] corpora are composed of audio recorded during the conferences held in academic laboratories. AISHELL-4 [11] is a 120-hour Mandarin speech dataset collected in conference scenarios. The number of attendees in each session is between four and eight. The indoor conversation corpus CHiME-6 [12] consists of unsegmented recordings of twenty separate dinner-party conversations among four friends captured by Kinect devices. It has a more casual and natural speaking style yet relatively low recording quality. For dialog scenario, Switchboard [13] is a classic dataset of English telephony conversations with similar settings and different scales. HKUST is a Mandarin conversational corpus [14] made up of spontaneous telephone conversations. Audios in these three telephony conversational datasets are recorded with a sampling rate of 8 kHz, which is incompatible with the demand of some speech processing systems nowadays.

The most commonly used metric for speaker diarization is DER [1], which is introduced for the NIST Rich Transcription Spring 2003 Evaluation (RT-03S). Specifically, DER is defined as:

where $FA$ is the total hypothesis speaker time not attributed to a reference speaker, $MISS$ is the total reference speaker time not attributed to a hypothesis speaker, $ERROR$ is the total reference speaker time attributed to the wrong speaker, and $TOTAL$ is the total reference speaker time, i.e. the sum of the duration of all reference speaker segments [15]. In most challenges and studying, a 0.25 second of collar is set around each segmentation boundaries, which aims at mitigating annotation error of reference transcript [1]. The calculation of DER is following.

Besides, Jaccard error rate (JER) is introduced by [16], which is developed as a metric for DIHARD II. Inspired by the evaluating method in image segmentation area, JER uses Jaccard similarity index that is the ratio between the sizes of the intersections and unions of two sets of segments.

However, all these evaluation metrics fail to give enough importance to the short conversational phrases. For comparison, in our proposed evaluation metric, Conversational Diarization Error Rate (CDER), all type of mistakes are equally reflected in the final evaluation metric, regardless of the length of the spoken sentence.

Recent speaker diarization systems could be devided into two types according to whether to use accurate timestamp annotations, i.e. clustering based approaches and fully supervised approaches [3]. Clustering-based methods generate diarization results in an unsupervised way. A typical procedure is firstly conducting speech activity detection. Then embeddings such as x-vectors are extracting and finally clustering [17]. Based on this, several sophisticated clustering models are designed. Fox et al. [18] introduce the Dirichlet processes mixture model into the diarization task. Diez et al. [19] designed a Bayesian model, i.e. VB-HMM, to simulate conversational state. And graph neural networks are used for clustering by Wang et al. [20] to model the conversation of multi-speakers. Besides, semantic information are introduced to assist speaker embeddings clustering by using word boundary information [21] and utilizing endpoints of sentences [22]. However, clustering-based method does not optimized diarization object directly [23] and the basic assumption of clustering-based method is that a chunk belongs only to one person, which restricts the capacity of handling overlapped conversation.

Fully supervised approaches need data with accurate timestamp annotation, thus optimizing diarization objects in a fully supervised way. And it could achieve promising performance, especially when speech from different speakers is highly overlapped [24, 25]. UIS-RNN [26] uses the recurrent neural networks (RNN) to model the transition probabilities among different speakers in conversation, which could output diarization results in an online fashion. EEND [24, 27] discards the use of the speakers’ embeddings and directly optimizes the diarization task in an end-to-end manner. Based on EEND, EDA-EEND [28] decodes the diarization results of different speakers recursively, in order to tackle the challenge of an uncertain number of speakers. Maiti et al. [27] handle this challenge by using the variable-number permutation-invariant training. Xue et al. [29] introduce a speaker-tracing buffer in order to solve the across-chunk permutation issue when extending the EEND to online conversation scenario. All these methods treat speaker diarization as a sequence-to-sequence problem, aiming to decode a sequence of speaker labels from an initial sequence of features. Another perspective is to regard speaker diarization as a detection task, where the goal is to predict the position of each person in a conversation. RPNSD [30] utilizes the Faster R-CNN to detect activities of each person in conversation. And TS-VAD [25] compares the embedding of a chunk of audio with the embedding of a target speaker to detect whether the corresponding person speaks or not.

MagicData-RAMC [4] contains 180 hours of dialog speech in total. The dataset is divided into 149.65 hours training set, 9.89 hours development set, and 20.64 hours test set (not the final evaluation set of the CSSD challenge), consisting of 289, 19, and 43 conversations, respectively. The original partition of the speech data is provided in TSV format and the 180-hour data will be released to the participants at the beginning of the challenge. Each conversation is of 30.80 minutes duration on average. The numbers of participants involved in three subsets are 556, 38, and 86, respectively. The gender and region distribution is roughly proportional to the entire dataset. Table 1 provides a summary of the partition of the corpus. It is recommended to provide a result based on MagicData-RAMC’s test set which helps to promote the reproducibility of the research. The dataset is collected indoors. The domestic environments are small rooms under 20 square meters in area, and the reverberation time (RT60) is less than 0.4 seconds. The environments are relatively quiet during recording, with ambient noise level lower than 40 dB. The audios are recorded over mainstream smartphones, including Android phones and iPhones. The ratio of Android phones to iPhones is around 1:1. All recording devices work at 16 kHz, 16-bit to guarantee high recording quality.

There are a total of 663 speakers involved in the recording, of which 295 are female and 368 are male. Each segment is labeled with the corresponding speaker-id. All participants are native and fluent Mandarin Chinese speakers with slight variations of accent and participants in each group are acquaintances. The accent region is roughly balanced, with 334 Northern Chinese speakers and 329 Southern Chinese speakers. Besides, each speaker participates in up to three conversations. All speech data are manually labeled. Sound segments without semantic information during the conversations, including laughter, music, and other noise events, are annotated with specific symbols. Phenomena common in spontaneous communications, such as colloquial expressions, partial words, repetitions, and other speech disfluencies, are recorded and fully transcribed. The precise voice activity timestamps of each speaker are provided. All transcriptions of the speech data are prepared in TXT format for each dialog. The start and end times of all segments are specified to within a few milliseconds. The statistics are presented in Table 2. Topics are freely chosen by the participants and the conversations in MagicData-RAMC are classified into 15 diversified domains, ranging from science and technology to ordinary life.

The test set contains 41 conversations, which amount to 20 hours. The number of participants is 82 with 43 males and 39 females. There are 9 conversations between women, 11 between men and 21 between the opposite sex.

Despite the MagicData-RAMC dataset, we prepare an individual 20-hour conversational speech test dataset with artificially verified speakers timestamps annotations for the CSSD task. Each segment is labeled with the corresponding speaker-id. Besides, all participants are native Mandarin Chinese speakers with slight variations of accent and participants in each group are acquaintances. The accent region and gender is roughly balanced. There are a total of 82 speakers involved in the recording, of which 39 are female and 43 are male. There are two speaker in each conversation, and 41 conversations are recorded. Among the 41 conversations, the number of conversations in which speakers consists of a man and a woman is 21. And the number of conversations between women is 9, while the number of conversations between men is 11.

Apart from speaker diarization datasets that could be used in fully supervised methods, there are several speaker recognition datasets needed in clustering-based methods, i.e. CN-Celeb [31] and VoxCeleb [32]. CN-Celeb includes two large-scale Chinese speaker recognition corpora. CN-Celeb1 consists of 274 hours audio from 1000 speakers, while CN-Celeb2 contains 1090 hours audio from 2000 speakers. These datasets could be used to train the speaker embedding extractor.