Robust End-to-end Speaker Diarization with Generic Neural Clustering

A neural end-to-end diarization model receives a sequence of acoustic features $F\in\mathbb{R}^{D\times T}$ as the input, and outputs the active probabilities of each speaker $Y\in(0,1]^{S\times T}$, where $T$ is the length of input frames and $S$ is the number of speakers. A two-stage diarization framework normally consists of two parts: the chunk-level predictor processes the chunk-level subsequences and calculates local speaker representations called attractors. These attractors are then fed into the clustering module which concatenates the diarization result of each chunk with the speakers being reordered correctly. Such two stages are jointly optimized with a loss function $L$:

$$L=L_{pre}+L_{post}.$$

The proposed neural clustering approach can be integrated with any chunk-level predictor. Without loss of generality, in this paper we will focus on its integration with the the EEND-EDA predictor with local attractors [13, 18]. In the EEND-EDA framework, to deal with the mismatch of number of speakers, the input frames are first fed into a stack of Transformers. And the output of encoder $E$ is then split into $N$ chunks, $E=\{E_{n}\}_{n=1}^{N},E_{n}\in\mathbb{R}^{D\times L}$, where $L$ is the size of chunk. A chunk-level predictor is applied on each chunk separately. The details of this predictor will be given in Section 2.2. After the chunk-level prediction stage, the proposed neural clustering module then re-concatenates the outputs together, which will be described in Section 2.3.

Similar to [18], the chunk-level predictor is a local encoder-decoder-based attractor (EDA) calculator aiming to diarize subsequences and to calculate speaker attractors. it consists of an EDA module and a Transformer decoder layer.

The details of an EDA module can be found in [13]. It can be defined as:

$$\hat{a}^{n},\hat{z}^{n}={\rm EDA}(E_{n}),$$

where $\hat{a}^{n}\in\mathbb{R}^{D\times S_{n}}$ is the speaker representation and $\hat{z}^{n}\in(0,1]^{S_{n}}$ is the corresponding existence probabilities. The chunk-level active probability is computed by:

$$y^{n}=\sigma(E_{n}^{T}\cdot\hat{a}^{n})\in(0,1]^{T\times S_{n}}.$$

The chunk-level predictor is trained with two losses, namely the attractor existence loss and diarization loss. The attractor existence loss optimizes the estimation of each speaker’s existence and the diarization loss optimizes the active probabilities with a permutation-invariant training method [19]:

	$\displaystyle L_{attr}$	$\displaystyle=\frac{1}{N}\sum_{n=1}^{N}{\rm BCE}(\hat{z}^{n},z^{n}),~{}~{}~{}z^{n}=[\underbrace{1,\cdots,1}_{S_{n}},0],$		(1)
	$\displaystyle L_{diar}$	$\displaystyle=\min_{{\phi}\in\Phi}({\rm BCE}(y^{n},l^{n}_{\phi})).$		(2)

Here $\Phi$ is the set of all possible permutations of $S_{n}$ speakers in the current chunk and $l^{n}_{\phi}\in\{0,1\}^{T\times S_{n}}$ is the corresponding diarization label. The total loss of chunk-level predictor are calculated as:

$$L_{pre}=L_{diar}+L_{attr}.$$

(3)

Since the local attractors are optimized to minimize the diarization error, they can not be used as the input for the neural clustering model directly. As in [18], the local attractors, $\hat{a}^{n}$, are converted using a Transformer decoder before clustering:

$$a^{n}=\mathrm{TransformerDec}(\hat{a}^{n},E).$$

On the basis of the chunk-level predictor, we get pairs of attractors and diarization results for each chunk $\{(a^{n}_{i},y^{n}_{i})\}_{i=0}^{S_{n}-1}$. The goal of the neural clustering module is to reorder them by the optimal correspondence among chunks.

In this paper, we explore a recurrent neural network (RNN) for the neural clustering because of its sequence modelling ability. Gated recurrent units (GRUs) [20] are leveraged for the clustering. As shown in Figure 2, the hidden states of the GRU cell can play a similar role as clusters in the unsupervised clustering algorithms [9], which are used to model the global speakers. However, instead of being fixed during each iteration, these hidden states are updated adaptively according to the estimation of the attractors for each chunk. Thus, the estimation of the diarization outputs can be further split into two steps, namely the prediction and update steps. The details of these two steps are described as follows.

The prediction step aims to associate chunk-level attractors with the global hidden states, for which the RNN clustering (RC) model is optimized to estimate the probability $p^{n}_{i}=(p^{n}_{ij})_{j}\in(0,1]^{C}$ that the input attractor $a_{i}^{n}$ belongs to each speaker, which can be calculated as:

$$p^{n}_{i}=\mathrm{softmax}\left((a_{i}^{n})^{T}h^{n-1}\right),$$

where $h^{n-1}=(h^{n-1}_{j})_{j}\in\mathbb{R}^{D\times C}$ are the previous hidden states. $C$ is the number of clusters present before. During the training phase, all hidden states have been initialized in the beginning, so $C$ is equalled to $S+1$.

Although the attractors are still out of order at this stage, we can link each attractor to a hidden state vector with the optimal permutation $\phi^{n}$ using Equation (2). Thus, the targets for training such a sequential clustering model are formed as:

$\displaystyle r_{ij}^{n}$

$\displaystyle=\left\{\begin{array}[]{ll}1&\phi^{n}_{i}=j\\ 0&otherwise\hbox to0.0pt{.\hss}\end{array}\right.$

(6)

Because the optimal permutation has been estimated using Equation (2), permutation-invariant training is not required at this stage. Thus, a cross entropy (CE) loss is used to optimize the RNN clustering model:

$$L_{post}=\frac{1}{N}\sum_{n=1}^{N}\left(\frac{1}{S_{n}}\sum_{i=0}^{S_{n}-1}\sum_{j=0}^{C-1}-r^{n}_{ij}\cdot\log(p^{n}_{ij})\right).$$

After the prediction step, the goal of the update step is to adaptively adjust the hidden states at each step given the estimate of the attractors. Because $r_{ij}^{n}$ provides a mapping from the $i$th attractor, $a_{i}^{n}$, to the $j$th hidden state, $h_{j}^{n}$, the hidden states can be updated using a teacher-forcing strategy [21]:

$$h_{j}^{n}=\left\{\begin{array}[]{ll}h_{j}^{n-1}&\forall i~{}~{}~{}r^{n}_{ij}=0\\ \mathrm{GRU}(a_{i}^{n},h_{j}^{n-1})&\exists i~{}~{}~{}r_{ij}^{n}=1.\end{array}\right.$$

(7)

For the inference, the number of speakers is unknown. To dynamically estimate the attendance of new speakers, an initialized $h_{0}$ is concatenated with the previous states at every step. The optimal speaker permutation $\phi^{n}$, is estimated by: $\phi^{n}=\arg\min_{\phi\in\Phi^{\prime}}\prod_{i=0}^{S_{n}-1}p^{n}_{i,\phi_{i}}$, where $\Phi^{\prime}$ is the set of all possible permutations. It should be noticed that a cannot-link exists between every pair of attractors in the same chunk, therefore each hidden state is linked to no more than one attractors except for the last one. Then the hidden states are updated according to Equation (6) and (7).

After obtaining each $\phi^{n}$, the final output of the diarization is reordered by replacing the chunk-level output $y^{n}_{i}$ with $y^{n}_{\phi^{n}_{i}}$.

To evaluate the performance of the proposed RNN nerual clustering integrated EEND- EDA approach, namely the EDA-RC approach, training and test datasets with different number of speakers were generated. The simulated datasets were generated using the data from the LibriSpeech[22] 360 hours of clean speech. The sampling frequency is 8kHz. A fixed number $n$ of speakers were first selected from this dataset, then for each speaker, utterances in the range of $[30/n,60/n]$ were randomly selected. This simulating procedure was identical to the one that was introduced in [11]. It worth noting that the average length of silence intervals between utterances varied in order to maintain the overlap ratio in a reasonable range. The simulated training set includes 100,000 samples with 3 speakers. The details of the test sets for the simulation data, LS_3, LS_4 and LS_5, are given in Table 1.

The CALLHOME dataset, i.e., NIST SRE 2000 (LDC2001S97, Disk-8) [23], was used to further evaluate the performance on the real data. This dataset has been widely used for speaker diarization experiments, which contains 500 sessions of multilingual telephonic speech. Although each session has 2 to 6 speakers, there are 2 dominant speakers in each conversation. For the experiments, it was split into training and test sets using the tools provided in the Kaldi Toolkit diarization recipe [24]. Samples with $\leq 3$ speakers in the training set were picked to train the models. The test set was further split with various numbers of speakers in order to evaluate the performance of approaches for both matched and unmatched conditions.

We use 23-dimensional log-Mel-filterbanks with a 25-ms frame length and 10-ms frame shift as the input feature. Feature vectors were concatenated with their neighboring $[-7,7]$ frames and the concatenated features were subsampled by a factor of ten. Therefore, the total input dimension was 345 and the interval between frames was 100ms. This configuration was the same as that used in [11]. The training samples were fed into models every 500 frames (50s) while the test samples remain un-split.

The baseline approaches include both one-stage and two-stage end-to-end approaches: the EEND approach [11], the EEND-EDA approach [13] and the EEND-EDA with a COP-K-means unsupervised clustering on the local attractors (EDA+COP-K-means) [18]. For the EEND and EEND-EDA approaches, a 4-layer Transformer with 256 hidden units were used as the encoder. Each layer had 4 heads and a dropout rate of 0.1. The end-to-end models were trained for 50 epochs with a batch size of 32. An Adam optimizer [25] with the same learning rate scheduler in [26] was used during training.

For the EDA+COP-K-means and EDA-RC approaches, the chunk size was set to 50. Parameters of the EDA chunk-level predictor were the same as those in EEND-EDA. The models were trained for another 20 epochs by an Adam optimizer with a learning rate of $10^{-5}$. Diarization Error Rate (DER) was used as the evaluation metric. Errors less than 0.25s around boundaries were tolerated. Please note that for the DER computation, all of the errors are evaluated including the overlapping speech segments.

Table 2 shows the DERs for the Librispeech-based simulated datasets. Please note that the switching strategy described in [18] is not adopted here, because the main purpose is to compare the unsupervised and supervised clustering methods instead of an overall performance. As expected, it can be seen that EEND and EEND-EDA approaches can yield performance for matched conditions but it gets worse with the increase of speakers. On the other hand, the EDA+COP-K-means approach takes an obvious advantage over the EEND-EDA approach and it outperforms the EEND-EDA by about 10% DER under mismatched conditions. For our proposed EDA-RC approach, it can be seen that EDA-RC gives a DER reduction of 1.66% and 1.56% compared to the baseline approaches under mismatched conditions while the performance does not degrade much for the matched conditions.

Additionally, a refine method is also applied to the EDA-RC, where an extra decoding process was conducted with the hidden states initialized as the first-pass decoded hidden states. Table 2 shows that this strategy can yield small performance gains when the number of speakers is large. Finally, the performance of the EDA-RC with the oracle permutations (oracle) is also given as a reference. This shows the performance upper bound of the clustering integrated EEND approaches as the correct clustering methods are used to reorder the speakers of each chunk.

In order to compare the performance on real audio recordings, models were also finetuned on CALLHOME. The beam size during decoding was set to 3. Models of the last $1/10$ epochs were averaged to obtain a better performance.

Table 3 shows the DERs of various methods on the CALLHOME real datasets. It can be the proposed EDA-RC approach yields 25.46% and 35.79% DER for the 4 and 5 speakers’ conditions, which correspond to about 12.8% and 9.0% relative reduction compared to the EDA+COP-K-means approach. This validates the effectiveness and robustness of the proposed RNN-based clustering model.

To make a more comprehensive comparison, a switching strategy that can dynamically decide whether to decode at chunk level or global level according to the potential number of speakers is also applied  [18] in this case. Thus, a global EDA loss is further added during the pretraining and finetuning of the EDA-RC model.

We notice that the gap between the clustering-integrated approaches and the conventional ones are rather small. This may be caused by the mismatch of data. As mentioned in Section 3.1, CALLHOME consists of real conversational speech where there are only two dominant speakers while all speakers have the same priority in the simulated data. Even though, with the switching method, clustering-integrated approaches can also have a better performance in some cases.

We further conduct ablation studies to analyze the performance of the proposed EDA-RC approach for various conditions. Considering an unsupervised clustering algorithm does not require a temporal order, it would be interesting to investigate the sensitivity of the RNN-based neural clustering to the order of the input chunks. The results are given in Table 4 with various shuffled ratios of samples. It shows that DER decreased by about 1% when the order is completely shuffled, which indicates the EDA-RC is not sensitive to the order of inputs. This may be resulted from the position independence of the Transformer encoder. In addition, Table 4 shows that the impact of using a smaller decoding beam size on the performance of EDA-RC is also very small.