Multi-scale Speaker Diarization with Dynamic Scale Weighting

The largest difference between a conventional single scale clustering method and multi-scale method is in the way in which the speech stream is segmented. The proposed system uses the same multi-scale embedding extraction and clustering mechanism as in [5], except that the scale weights are computed differently in this work. We employ a uniform segmentation scheme that originally appeared in [2, 4], and such segmentation is applied for multiple scales. Fig. 2 shows an example of a multi-scale segmentation. As shown in Fig. 2, we refer to the finest scale, i.e., 0.5 s, as the base scale because the speaker label estimation is applied at the finest scale. We denote the number of scales as $K$.

After segmentation is applied for all scales, grouping among the segments at each scale is then applied. The grouping process is conducted by assigning the segments from each scale for the corresponding base scale segment. The segments from the lower temporal resolution scales (in this example, 2.0, 1.5, and 1.0 s in length) are selected and grouped by measuring the distance between the centers of the segments and choosing the closest ones. By grouping the segments as in Fig. 2, each group will generate one weighted cosine similarity value when a weighted affinity matrix is calculated. A detailed description of multi-scale clustering can be found in [5]. We use the speaker embedding extractor model proposed in [13], which generates 192-dimensional embedding.

Because we previously proposed a multi-scale system in [5], we employ the auto-tuning spectral clustering approach proposed in [14], which is referred to as normalized maximum eigengap spectral clustering (NME-SC). From the clustering result, we obtain the estimated cluster label per each base scale segment and the estimated number of speakers $S$.

Based on the clustering results, we take the average of all the speaker embeddings obtained from the initial clustering result as in Eq. (1). We obtain cluster-average embedding vectors $\textbf{v}_{k}^{s}$ from the initial clustering result as follows:

$$\textbf{v}_{k}^{s}=\frac{1}{N_{k}}\sum_{i=1}^{N_{k}}\textbf{v}_{i,k}^{s},$$

(1)

where $k$ is the scale index, $N_{k}$ is the number of $k$-th scale embeddings during the session and $s$ is the cluster index (speaker index) from the clustering result.

To dynamically adjust the scale weights during the inference phase, the proposed diarization system should look into the input speaker and reference embeddings simultaneously. Fig. 3 shows how scale weights are calculated in the case of $K$$=$$4$ scales for $S$$=$$2$ speakers. We stack the embeddings from the input signal and the average embeddings from the clustering results of two speakers ($s$$=$$1$,$2$) in the following manner:

$$\vspace{-1.0ex}\textbf{D}_{i}=[\textbf{u}_{i,0};\text{\ldots};\textbf{u}_{i,K-1};\textbf{v}_{0}^{1};\text{\ldots}\textbf{v}_{\text{$K$-$1$}}^{1};\textbf{v}_{0}^{2};\text{\ldots};\textbf{v}_{K-1}^{2}]^{\intercal},$$

(2)

where $\textbf{u}_{i,k}$ are column-wise input speaker embeddings, and $\textbf{v}_{k}^{s}$ are cluster-average speaker embeddings, as described in Section 3.2. Thus, the CNN input becomes a $3K\times N_{e}$ matrix $\textbf{D}_{i}$, where $N_{e}$ is the embedding dimension, and $K$ is the number of scales.

To estimate the scale weight, we propose 1-D CNN with 1-D filters. The motivation behind using a 1-D filter is to compare the embedding vectors bin-by-bin such that the learned weights in the filters focus only on the difference between other embeddings for each bin. Two CNN layers are followed by two linear layers and the softmax layer, i.e.,

$$w_{k}=\frac{e^{f(z_{k})}}{\sum_{k=0}^{K-1}e^{f(z_{k})}},$$

(3)

where $k$ is the scale index, and $f(z_{k})$ is the output of the linear layer (see Fig. 3). The scale weight values $w_{k}$ in Eq.  (3) are multiplied with the cosine similarity in an element-wise manner and create the context vectors for each speaker $\textbf{c}_{i}^{s}$. We concatenate these two speaker-specific context vectors for each $i$-th time step into the global context vector $\mathbf{C}_{i}=[\textbf{c}_{i}^{1};\textbf{c}_{i}^{2}]$ of length $2K$ (see Fig. 2).

Although clustering based diarization shows a competitive performance, it lacks the ability to assign multiple speakers at the same time step, which is needed for overlap-aware diarization. In addition, the clustering-based multi-scale diarization requires a separate training step for multi-scale weights or a high-dimensional grid search. This negatively affects the accuracy of the model for unseen domains during training. Moreover, the performance of a clustering-based multi-scale diarization system is heavily dependent on the scale weights, as shown in [5].

Hence, we derived a scale weighting mechanism that dynamically calculates the scale weights at every time step when a trained sequence model estimates the speaker labels. In the scale weighting system, during the training process, the scale weights are multiplied to the cosine similarity values to create a context vector that is fed to the sequence model (LSTM). The scale weighting mechanism is described in Fig. 2. Let $\textbf{u}_{i,k}$ be the $k$-th scale embedding of the input speech signal and $i$ be the time-step index. In addition, $\textbf{v}_{k}^{s}$ is the $k$-th scale average embedding vector of cluster $s$ from the initial clustering result in Eq. (1), and let ${w}_{k,i}$ be the $k$-th scale weight at the $i$-th time-step index. The scale weight $w_{k,i}$ is applied to the cosine distance between the inputs $\textbf{v}_{k}^{s}$ and $\textbf{u}_{i,k}$, i.e.,

$$\textbf{c}_{i}^{s}[k]=w_{k}\cdot\frac{\textbf{v}_{k}^{s}\cdot\textbf{u}_{i,k}}{\|\textbf{v}_{k}^{s}\|\|\textbf{u}_{i,k}\|},$$

(4)

where $\textbf{c}_{i}^{s}$ is the context vector for the $s$-th speaker (see Fig.5).

The weighted cosine similarity vectors are fed to the sequence model. In our proposed system, we employ a two-layer Bi-LSTM. Fig. 5 shows how the context vector is fed to the LSTM for estimating the speaker-wise label. Note that the output layer goes through the sigmoid layer such that the output value ranges from zero to 1 independently from the neighboring values. We use the binary cross-entropy loss to train the model. The binary ground truth label is generated by calculating the overlap between the ground truth speaker timestamps and the base scale segment: If the overlap is greater than 50%, the segment is assigned a value of 1 (see Fig.4).

The speaker label inference for the sessions with more than two speakers is obtained by selecting pairs from the estimated number of speakers and averaging the prediction for the corresponding speaker from the two-speaker models. For example, in the three speaker cases in which speakers A, B, and C exist, we average the two results for A from (A, B) and (A, C). Letting S be the total number of speakers in a session, then the proposed model infers $\binom{S}{2}$ number of speakers and then take average of all $S$$-$$1$ output pairs from a certain speaker through the following equation:

$$p(s,i)=\frac{1}{\text{$S$ $-$ $1$}}\sum_{q\in U\text{$-$}\{s\}}\sigma(\mathbf{z})^{s,(s,q)}_{i}$$

(5)

where $U$ is a complete set of the speakers, and $\sigma(\mathbf{z})^{s,(s,q)}_{i}$ is the sigmoid output of the $i$-th base scale segment of the $s$-th speaker, which is extracted from the pairing of the $s$-th and $q$-th speakers. We use a threshold value $T$ on the average sigmoid value $p(s,i)$ to obtain the final speaker label output for each base scale segment. If both sigmoid values are below the threshold $T$ for the given time step $i$, we then choose the label from the initial clustering result.

We train the meeting and telephonic models (3.4M parameters) separately using the same dataset but different parameters. Table 1 shows the length of sclae and the speaker embedding extractor models. In this study, we use TitaNet-M (13.4M parameters) and TitaNet-L (25.3M parameters) models [13]. The shift lengths for each scale are half of each scale window length. The scale information in Table 1 is applied to both the neural network model and the clustering as initialization. The initial scale weights are set using the following equation:

$$w_{k}=r-\frac{(r-1)}{K-1}k,$$

(6)

where the weights are linearly increased or decreased from the parameter value $r$ and the base scale (the highest index) is always assigned a value of $1.0$. In this way, the initial clustering scale weight vector $\mathbf{w}=[w_{0},w_{1},\dots,w_{K-1}]$ is formed. Note that the overall scale of the scale weights $w_{k}$ for clustering does not affect the result because the weighted sum of the cosine similarity values are min-max normalized during the clustering process [14]. The proposed system requires parameter tuning on $r$ in Eq. (6) for clustering and threshold $T$ described in Section 3.3 for the MSDD.

An evaluation of a speaker diarization system depends on numerous conditions regarding the use of the development set, whether information on the oracle number of speakers is used, and the diarization error rate (DER) evaluation conditions such as collar and overlap inclusion. We only report the results based on the estimated number of speakers and oracle voice activity detection (VAD). We report two different DER evaluation settings following the name of the setups in [12]: (1) Forgiving, $\text{DER}_{A}$ , 0.25-s collar and ignored overlap speech; and (2) Full, $\text{DER}_{B}$, 0-s collar and included overlap speech.

We use following datasets:

• •

  CH(NS): NIST-SRE-2000 (LDC2001S97) is the most popular diarization evaluation dataset and referred to as CALLHOME in the speaker diarization papers. To make a fair comparison with other studies and our previous research, a two-fold cross validation is used for tuning the parameters using the split appeared in [15, 16].

• •

  CH109: Call Home American English Speech (CHAES, LDC97S42) is a corpus that contains only English telephonic speech data. We evaluate 109 sessions (CH109) which have two speakers per session. The parameters are optimized on the remaining 11 sessions in the CHAES dataset.

• •

  AMI-MH: AMI meeting corpus [17] is a meeting speech dataset containing up to five speakers. For the evaluation, we use annotation files and the file lists of splits provided by [18] from the MixHeadset part. All parameters are optimized on the dev set.

• •

  The Fisher corpus: The Fisher corpus contains 10 minute English conversations. We use Fisher corpus [19] to train the proposed model using the two-speaker setting. We use our own random split that has 10,000 sessions for training and 1,500 sessions for validation.

In terms of the hyperparameters, we use 256 nodes of hidden layer units for both hidden layers in the scale weight generation and the LSTM. We use 16 filters for the 1-D CNN described in 3.1. F1 score is used for stopping the training or select the model. The results reported herein can be reproduced using the NeMo¹¹1https://github.com/NVIDIA/NeMo open-source toolkit [20].

Table 2 shows the performance of the previously published speaker diarization performance on the same dataset. In Table 2, note that $\text{DER}_{A}$ does not reflect the overlap detection capability. SS-Clus is our previous approach [13], where SS clustering-based diarization systems with TitaNet models [13] were used. The Equal-$\mathbf{w}$-Clus system uses equal weights, which means $w_{k}$$=$$1$ in Eq. (6) for all elements in the scale weight vector $\mathbf{w}$. Accordingly, Equal-$\mathbf{w}$-MSDD is the result obtained from the MSDD based on the initializing clustering with an equal scale weight. By contrast, Opt-$\mathbf{w}$-Clus indicates a clustering-based diarization result with the optimized scale weight vector $\mathbf{w}$ on each development set, and Opt-$\mathbf{w}$-MSDD refers to the system based on the clustering result Opt-$\mathbf{w}$-Clus. Whereas Opt-$\mathbf{w}$-MSDD shows an overall better result, we want to emphasize the importance of the Equal-$\mathbf{w}$-MSDD result because Equal-$\mathbf{w}$-MSDD does not require parameter tuning for the initializing clustering.