Multi-target Extractor and Detector for Unknown-number Speaker Diarization

Inspired by TS-VAD [31], we propose a two-step SD method called MTEAD, which can be adapted to different numbers of speakers. MTEAD not only addresses the main weakness of TS-VAD (i.e., handling only sessions with a fixed number of speakers, e.g., four in [31]), but also uses improved speaker representations. As shown in Fig. 1, the first step of MTEAD (i.e., Extractor) relies on a traditional x-vector/AHC SD method to initially detect the speech regions for each speaker (i.e., speaker occurrence labels). Then, the frames corresponding to each speaker are used to extract his/her speaker representation. Here, the representation may be a traditional i-vector or x-vector, or our specially designed z-vector for SD. The second step of MTEAD (i.e., Detector) requires two inputs: the frame-level MFCCs and the representation of each speaker. The frame-level MFCCs are first passed through a four-layer convolutional neural network (CNN), and then concatenated with each speaker representation. The output of Detector is the final diarization result for each speaker.

In TS-VAD, the second step is implemented using BiLSTM. The first two layers of the BiLSTM take the frame-level MFCCs and four i-vectors as input, and output four speaker detection ($\rm S_{d}$) vector sequences. Then, the four $\rm S_{d}$ vector sequences are concatenated along the feature dimension and passed through the third layer of the BiLSTM to generate the final diarization result for each speaker. It is this concatenation that causes TS-VAD to only handle four-speaker recordings. Furthermore, it uses the i-vector as the speaker representation, which is defeated by the x- and z-vectors in our experiments.

The Detector consists of a Feature Mixer and three consecutive Time-Speaker Contextualizers implemented using BiLSTM. The MFCCs are passed through CNNs and concatenated with each speaker representation. These concatenated features are then processed by Feature Mixer to generate the corresponding $\rm S_{d}$ frames $\mathbf{X}^{i}$ for the $i$-th speaker, as shown in Fig. 1. As illustrated in Fig. 2, the stack of the $\rm S_{d}$ frames of all speakers, $\mathbf{X}_{in}\in\mathbb{R}^{D\times T\times N}$, is the input of the Time-Speaker Contextualizer, where $D$, $T$, and $N$ denote the dimension of the $\rm S_{d}$ vector, number of frames, and number of speakers, respectively. Inspired by DPRNN, the Time-Speaker Contextualizer is designed to handle different numbers of speakers in one session. It contains two stages. In the first stage, the $\rm S_{d}$ vector sequence of each speaker is passed through the Across-time Contextualizer separately to generate the temporal contextual information using

The output of the first stage, $\tilde{\mathbf{X}}$, is the stack of $\tilde{\mathbf{X}}^{i}$, $i=1,...,N$. The input of the second stage, $\mathbf{Y}$, is generated by reshaping $\tilde{\mathbf{X}}$. Slicing the input by timeframes yields $\mathbf{Y}^{j}\in\mathbb{R}^{D\times N}$, $j=1,...,T$, which is treated as the activity of individual speakers in a single frame $j$. Then, $\mathbf{Y}^{j}$ is passed through the Across-speaker Contextualizer to generate the speaker contextual information using

$\tilde{\mathbf{Y}}^{j}$, $j=1,...,T$, is stacked and reshaped to $\mathbf{X}_{out}\in\mathbb{R}^{D\times T\times N}$. $\mathbf{X}_{out}$ is used as the input, $\mathbf{X}_{in}$, of the subsequent Time-Speaker Contextualizer. Finally, for each speaker, the corresponding $\rm S_{d}$ vector sequence from the output of the last Time-Speaker Contextualizer is passed through a linear-sigmoid layer to generate the final diarization result.

In Eq. (2), the number of speakers $N$ is the length of the input sequence; therefore, it is variable. The Across-speaker Contextualizer allows the sharing of speakers’ information within a single frame, which can help Detector to discern differences among speaker representations. On the other hand, by scanning along the frame, the Across-time Contextualizer can help Detector to determine whether each speaker is active in each frame. Therefore, MTEAD achieves information sharing among all speakers and frames through the Across-speaker and Across-time Contextualizers, which not only preserves the advantages of TS-VAD, but also does not have the limitation of handling only conversations of a fixed number of speakers.

We argue that the quality of speaker representations is critical to SD performance. Therefore, we design an extractor specifically for extracting speaker representations suitable for SD. As shown in Fig. 1, using the speaker speech regions provided by x-vector/AHC diarization and the MFCCs of the session, the Z-vector Extractor generates the corresponding speaker representations¹¹1While we were preparing this paper, there was a similar work extending TS-VAD by jointly training a speaker embedding network [34]. However, it does not handle a variable number of speakers.. The Z-vector Extractor comprises ResNet and Attentive Statistics Pooling (ASP) [35]. The ResNet here is similar to ResNet50, but the last layer is replaced by the ASP layer. The input to ResNet is a sequence of 500 frames of MFCCs from each speaker, and the output shape is set to be the same dimension as the x-vector²²2The number of parameters for the x- and z-vector extractors is about 8M and 30M, respectively.. Therefore, the z-vectors can be used as input to Detector instead of x- or i-vectors. In MTEAD, speaker representation extraction and speaker detection are combined into a unified process by integrating Extractor with Detector. Also, since Extractor and Detector are trained jointly, it is expected that the z-vector is more suitable for SD than the x- and i-vectors.

When using z-vectors, Extractor and Detector are jointly trained from scratch using a binary cross-entropy loss between speaker labels and predicted SD results. The per-speaker losses are summed directly and back-propagated. When using x-vectors and i-vectors, Extractor is replaced by other extractors pretrained by the Kaldi recipe, and only Detector is trained using the same binary cross-entropy loss.

The total number of speakers in the 683 hours of data from the SRE and Switchboard datasets was 6,392. We simulated the training and evaluation data by Algorithm 1 in [28]. The speakers in the two sets did not overlap. To achieve 20% overlap, for the cases of two, three, and four speakers, the parameter $\beta$ in the algorithm was set to 3, 6, and 9 seconds, resulting in 137, 226, and 320 hours of data, respectively.

During training, the ground truth in Rich Transcription Time Marked (RTTM) format was used as the speaker occurrence labels for each session. During inference, we used x-vector/AHC to produce the RTTM file for each test session. The Threshold in x-vector/AHC was set based on the performance on the training set. Generation of speaker representations followed the methods described in Secs. II-A and II-D.

Results and discussion. First, we investigated the impact of the quality of speaker representations on SD performance. Extractor in MTEAD (cf. Fig. 1) was replaced with a pretrained x-vector extractor. During MTEAD training, each target speaker’s x-vector was extracted from all his/her original utterances in Switchboard and NIST-SRE (denoted as G (global)) or from the corresponding speech frames in a session based on the ground truth RTTM (denoted as L (local)). The ratio $|G|/(|G|+|L|)$ indicates the degree to which global speaker representations were used for training. The Threshold and Oracle tests denote that the number of clusters in AHC is automatically determined and set to the true number of speakers, respectively. Ideal denotes the (cheating) case where the target speaker’s x-vector was extracted from the test session based on the ground truth RTTM. As shown in Table I, the ratio has little impact on SD performance, showing that x-vectors extracted from a limited amount of speech in a training session are as effective as x-vectors extracted from a large amount of speech in the entire training dataset. Regardless of the ratio, all Ideal test conditions outperformed their Threshold and Oracle counterparts. These results demonstrate the importance of accurate speaker representations, showing that extracting better speaker representations is key to producing excellent SD results.

Next, we compared the performance of different speaker representation models, including z-vector, i-vector, and x-vector. As shown in Table II, while the MTEAD model with i-vectors or x-vectors showed better performance than the baseline x-vector/AHC method, the MTEAD model with z-vectors achieved the best performance. The results confirm that the z-vector jointly trained with the MTEAD model is more effective than the x-vector and i-vector obtained from pretrained models. Speaker representations targeting SD did yield better SD performance.

CALLHOME is a telephony dataset containing conversations in multiple languages. The dataset consists of a total of 500 conversations recorded at a sampling rate of 8 kHz. The number of speakers in each conversation varies from 2 to 7. Since the CALLHOME dataset is too small to train our model, we used the SWB+SRE dataset for pretraining.

During training, we pretrained the MTEAD models on SWB+SRE. We divided the CALLHOME set equally into two parts as Kaldi instructed. CALLHOME-1 was used for fine-tuning the pretrained models, while CALLHOME-2 was used for evaluation. We determined the Threshold in x-vector/AHC based on the performance on CALLHOME-1. During inference, we used x-vector/AHC to produce RTTM files on CALLHOME-2. We used the same methods as the experiments on SWB+SRE to generate speaker representations.

As the number of speakers in a CALLHOME session varies from 2 to 7, it is necessary to pretrain and fine-tune a TS-VAD model for each speaker number separately. To this end, the SWB+SRE and CALLHOME-1 datasets were split into 2-, 3-, and 4-speaker subsets for training the corresponding TS-VAD models. For each TS-VAD model, we also trained the corresponding MTEAD* model using the same training data and procedure for fair comparison. MTEAD was trained on all training data containing different numbers of speakers.

Results and discussion. First, we compared MTEAD with TS-VAD. TS-VAD, MTEAD*, and MTEAD were all based on the initial diarization of x-vector/AHC, and were all implemented with i-vectors. For each specific number of speakers, MTEAD* and TS-VAD were trained on the same training data. The experiments were conducted under the 2-, 3-, and 4-speaker conditions, assuming the number of speakers was known. The evaluation results on CALLHOME-2 are shown in Table III. It is clear that MTEAD* consistently outperformed the corresponding TS-VAD and x-vector/AHC baselines under each identical condition. Moreover, MTEAD outperformed MTEAD* because it was trained with all training data containing different numbers of speakers, while each MTEAD* model was trained with only a subset of training data with a specific number of speakers. Results demonstrate the advantage of the MTEAD Detector: MTEAD significantly outperformed TS-VAD because its detector could be trained using all training data with different numbers of speakers. After overcoming the weakness of TS-VAD, MTEAD with i-vector has already surpassed SA-EEND-EDA [29].

Next, we compared the performance of different speaker representation models, including z-vector, i-vector, and x-vector. As can be seen from Table IV, all three MTEAD models outperformed not only the x-vector/AHC and x-vector/AHC+VB baselines, but also the strong SA-EEND-EDA model [29]. Furthermore, z-vector-based MTEAD outperformed i-vector-based and x-vector-based MTEAD under both Threshold and Oracle conditions, and larger improvements are observed under the Oracle condition. We speculate that this is because when the number of speakers is correct in the initial diarization, the model can estimate a more accurate z-vector for each speaker. In contrast, under the Threshold condition, the incorrect number of speakers predicted in x-vector/AHC may cause some z-vectors to not match actual speakers, resulting in less reductions in DER and JER. The results in Tables III and IV show that, with its well-designed Extractor and Detector, MTEAD is a flexible and effective SD model that can extract more accurate speaker representations and handle conversations with different numbers of speakers. Compared to x-vector/AHC, x-vector/AHC+VB, and SA-EEND-EDA, MTEAD with z-vector achieved relative DER reductions of 30.9%, 26.7%, and 6.4%, respectively.

Finally, we compared MTEAD with other models. Since most NN-based SD models were only evaluated in 2-speaker experiments, we compared different models on the 2-speaker CALLHOME task. From Table V, MTEAD outperformed all models except DIVE [27], with a 12.2% relative reduction in DER compared to x-vector/AHC [29]. According to Tables IV and V, MTEAD outperformed most of the models compared in this study in both 2-speaker and multi-speaker tasks. This study focuses on improving the shortcomings of TS-VAD. Although there are many other advanced end-to-end or unsupervised SD models [41, 42], we did not include them in the comparison due to the different training conditions of these models and paper length constraints.