Probabilistic back-ends for online speaker recognition and clustering

General formulation. In this study we focus on a variant of PLDA known as the two-covariance model [15]. Let $\boldsymbol{\mathrm{x}}_{i,j}\in\mathbb{R}^{d}$ denote the $j$th speaker embedding of speaker $i$. Also, let $\boldsymbol{\mathrm{y}}_{i}$ be the latent speaker identity of speaker $i$. Then, the model is specified by two Gaussian distributions:

Here, $\boldsymbol{\mathrm{\mu}}$ is a global mean, and $\boldsymbol{\mathrm{B}},\boldsymbol{\mathrm{W}}\in\mathbb{R}^{d\times d}$ are the between- and within -speaker covariance matrices, respectively.

Being a linear Gaussian model, PLDA allows making inferences about speaker identities in closed form. Given a set of observations (embeddings), one can compare different hypotheses about the partition of this set by computing the corresponding hypothesis likelihoods. This is often referred to as by-the-book scoring in the literature [10, 16].

PLDA with spherical covariances. Despite being a gold standard for previously popular i-vectors [17], one could recently observe a gradual shift towards replacing PLDA with a simpler parameter-less cosine scoring back-end [18]. As discussed in [14], the high intra-speaker compactness of the large-margin embedding makes the conventional full-rank PLDA model superfluous. It was also observed in [14] that discarding off-diagonal elements in the within-speaker covariance matrix can bring considerable performance gain. Here, we analyze a much more constrained version of the PLDA model, to our knowledge, firstly proposed in [1, 19]. Specifically, we consider PLDA with spherical covariances, $\boldsymbol{\mathrm{B}}=\sigma_{\text{B}}^{2}\boldsymbol{\mathrm{I}}$, $\boldsymbol{\mathrm{W}}=\sigma_{\text{W}}^{2}\boldsymbol{\mathrm{I}}$, where $\sigma_{\text{B}}^{2}$ and $\sigma_{\text{W}}^{2}$ are between- and within-speaker variances and $\boldsymbol{\mathrm{I}}$ denotes an identity matrix. In the following text, we will refer to this model as the spherical PLDA.

Relationship with cosine scoring. As was shown in [18], for length-normalized and centered embeddings, the verification likelihood ratio of the spherical PLDA can be written as a scaled and shifted cosine similarity measure. Since an affine transformation of scores is order-preserving, the two scoring rules are equivalent. This brings up a question about the usefulness of spherical PLDA. As we discuss further, spherical PLDA has several advantages over cosine scoring. For instance, we show that the PLDA by-the-book scoring outperforms different cosine based heuristic scoring methods in multi-enrollment verification.

Relationship with PSDA. Another closely related scoring back-end is the so-called probabilistic spherical discriminant analysis (PSDA) recently proposed in [20]. It can be viewed as PLDA model with Gaussian distributions replaced by von Mises-Fisher (VMF) distributions that are defined on the $d-1$ dimensional unit hypersphere $\mathbb{S}^{d-1}$ [21]:

Here, $\mathcal{V}(\boldsymbol{\mathrm{y}}|\boldsymbol{\mathrm{\mu}},\kappa)$ denotes the density of the VMF distribution with mean direction vector $\boldsymbol{\mathrm{\mu}}\in\mathbb{S}^{d-1}$ and scalar concentration $\kappa\geq 0$ parameter. Similar to spherical PLDA, this model is parameterized by the mean direction vector $\boldsymbol{\mathrm{\mu}}$ and two scalars: between-speaker, $b$, and within-speaker, $w$, concentrations.

The relation to spherical PLDA follows from the fact that restricting any isotropic Gaussian density to the unit hypersphere gives a VMF density, up to normalization. However, the two models are not equivalent, though their behavior is very similar as we show in the experiments.

We use both spherical PLDA and PSDA as a basis for a proposed online speaker clustering algorithm described in Section 3.

When available, multiple enrollment utterances may represent various acoustic environments, or channels, that could be useful to better disentangle speaker identity from other irrelevant factors.

The study in [10] analyzes different methods of aggregating information from multiple speech segments for the PLDA scoring. Among them were embedding averaging, score averaging, and by-the-book scoring. Their experiments with i-vectors revealed that embedding averaging systematically outperforms other methods, including by-the-book scoring.

However, these observations were made for previously popular i-vector embeddings and have not been yet confirmed for modern large-margin embeddings. In fact, in our experiments, we observe that by-the-book scoring with spherical PLDA or PSDA outperforms embedding averaging.

The general pattern behind many online clustering algorithms is solving a series of successive open-set identification tasks [4, 3, 6, 22]. The basic idea is to compare each new observation to the existing clusters, and either alter the closest cluster or create a new cluster. The generic Algorithm 1 demonstrates this for a single time step $t$.

First, the observation $\boldsymbol{\mathrm{x}}_{t}$ is compared to all existing clusters, each represented by a set of observations, $\boldsymbol{\mathrm{X}}_{i}$. If similarity to the closest cluster is above the threshold, $\tau$, then $\boldsymbol{\mathrm{x}}_{t}$ is assigned to this cluster. Otherwise, a new cluster is formed.

In this algorithm, clusters are represented by subsets of observations sharing the same label. Therefore, computing similarity to a cluster involves many-to-one comparison, also referred to as multi-enrollment verification in the context of speaker recognition. As discussed in [11], varying cluster sizes may result in miscalibrated scores leading to sub-optimal decisions with a fixed threshold $\tau$.

We aim at addressing this issue and propose an algorithm suitable for online clustering. Specifically, the underlying scoring model should be robust to varying cluster sizes naturally occurring in the online scenario. The proposed algorithm can be seen as a probabilistic extension of the Algorithm 1 constructed upon PLDA or PSDA models. As a result, it benefits from the advantages of PLDA (or PSDA) for multi-enrollment verification.

We start with a brief description of a generative model-based clustering [23, 24].

Model-based clustering builds upon a generative model that specifies how a set of data points $\boldsymbol{\mathrm{X}}=\{\boldsymbol{\mathrm{x}}_{1},...,\boldsymbol{\mathrm{x}}_{N}\}$ is generated from the hidden parameters of $K$ clusters $\boldsymbol{\mathrm{Y}}=\{\boldsymbol{\mathrm{y}}_{1},...,\boldsymbol{\mathrm{y}}_{K}\}$, given the cluster assignments $\boldsymbol{\mathrm{Z}}=\{\boldsymbol{\mathrm{z}}_{1},...,\boldsymbol{\mathrm{z}}_{N}\}$. A typical clustering model is given by the following joint distribution: $p(\boldsymbol{\mathrm{X}},\boldsymbol{\mathrm{Y}},\boldsymbol{\mathrm{Z}})=p(\boldsymbol{\mathrm{X}}|\boldsymbol{\mathrm{Z}},\boldsymbol{\mathrm{Y}})p(\boldsymbol{\mathrm{Y}})p(\boldsymbol{\mathrm{Z}})$.

The clustering problem requires finding the most likely partition of the data $\boldsymbol{\mathrm{Z}}_{*}=\arg\max_{\boldsymbol{\mathrm{Z}}}p(\boldsymbol{\mathrm{Z}}|\boldsymbol{\mathrm{X}})$. Our approach is based on the “mean-field” variational Bayesian approximation [25, 26] assuming that the approximate posterior factorizes as $p(\boldsymbol{\mathrm{Z}},\boldsymbol{\mathrm{Y}}|\boldsymbol{\mathrm{X}})\approx q(\boldsymbol{\mathrm{Z}})q(\boldsymbol{\mathrm{Y}})$. This assumption leads the algorithm consisting of iterative updates of the factors $q(\boldsymbol{\mathrm{Z}})$ and $q(\boldsymbol{\mathrm{Y}})$.

However, such updates are designed for the conventional clustering setup, where all observations are available at once. We modify the standard inference algorithm to make it suitable for online clustering, where observations arrive sequentially. This algorithm can be seen as an online version of the VBx [27] with simplified prior on assignments $p(\boldsymbol{\mathrm{Z}})$. It is also similar to the algorithm from [7], where the authors modified the offline variational inference to make it suitable for online processing.

Let us denote the current observation at the time step $t$ as $\boldsymbol{\mathrm{x}}_{t}$, and use the notation $\boldsymbol{\mathrm{X}}_{1:t}=\{\boldsymbol{\mathrm{x}}_{1},...,\boldsymbol{\mathrm{x}}_{t}\}$ to denote causal observations.

The algorithm updates posterior distributions of latent identity variables $q(\boldsymbol{\mathrm{y}}_{k})\approx p(\boldsymbol{\mathrm{y}}_{k}|\boldsymbol{\mathrm{X}}_{1:t})$ after receiving a new observation $\boldsymbol{\mathrm{x}}_{t}$. In general, several update iterations can be done. Our experiments reveal that even a single update can be sufficient for reasonable performance. In this case only posterior for the current data point $q(\boldsymbol{\mathrm{z}}_{t})$ needs to be computed, followed by updating each of $q(\boldsymbol{\mathrm{y}}_{k})$:

Here, $\gamma_{t,k}$ is the $k$-th component of the vector of posterior probabilities $q(\boldsymbol{\mathrm{z}}_{t})$ over the cluster assignments and $\pi_{k}$ are the corresponding prior probabilities.

This algorithm continuously updates speaker models defined by $q(\boldsymbol{\mathrm{y}}_{k})$. Also, one can obtain speaker labels at each time step $t$ by finding $\arg\max_{k}\gamma_{t,k}$. For instance, if $\gamma_{t,k}=0$, then the posterior $q(\boldsymbol{\mathrm{y}}_{k})$ stays unchanged. In general, if the soft-assignments $\boldsymbol{\mathrm{\gamma}}_{t}$ were converted into hard decisions, then updating $q(\boldsymbol{\mathrm{y}}_{k})$ would be nothing more than the sequential application of the Bayes formula. Also, the algorithm would become very similar to a sequence of multi-enrollment recognition tasks, where predictions are obtained via by-the-book scoring.

These update equations can be used to construct different online recognition and clustering algorithms depending on a particular choice of the underlying generative model defined by $p(\boldsymbol{\mathrm{x}}|\boldsymbol{\mathrm{y}})$ and $p(\boldsymbol{\mathrm{y}})$. In this study, we use two models: spherical PLDA and PSDA. Table 1 demonstrates the update equations for both models.

To detect new speakers we introduce an extra class corresponding to an unknown speaker. For this class, the posterior for the speaker identity variable is equal to the prior.

Algorithm 2 outlines the time step $t$ of the proposed algorithms.

The advantage of the proposed algorithm over Algorithm 1 is that it uses soft decisions for updating clusters. This makes the algorithm more robust to classification errors.

As a baseline for our experiments, we use Algorithm 1 with cosine similarity scoring.

In this section, we compare different scoring back-ends in multi-enrollment speaker verification scenario. Specifically, we investigate calibration properties of the verification scores in the case where the number of enrollment and test segment varies within an evaluation protocol.

Experimental setup. We created several custom evaluation protocols from the VoxCeleb1 test set [30]. Specifically, we generated four trial lists with configurations $(1,1)$, $(3,1)$, $(10,1)$, and $(3,3)$, where the notation (#enrollments, #tests) represents the number of enrollment or test segments in a single trial. In addition, we combined all the trial lists to get the pooled protocol. The idea behind it is to reveal the robustness of scoring back-ends to the number of enrollment segments. To exclude the effect of utterance duration, the recordings were cropped to 2 seconds before extracting embeddings.

We compared several different scoring variants: cosine similarity with embedding averaging (CSEA) or score averaging (CSSA), PSDA [20], and three versions of PLDA with spherical, diagonal, and full covariance matrices. For PLDA and PSDA by-the-book scoring was used. The VoxCeleb1 dev set [30] was used for training the back-ends. We used two performance metrics: the equal error rate (EER) and the minimum normalized detection cost function (minDCF) with $P_{\text{target}}=0.01$ [31].

Results. Figure 1 demonstrates the distribution of verification scores for different numbers of enrollment segments.

One can see considerable distribution shifts for the target scores computed with CSEA. To be precise, a large variation of EER thresholds clearly makes each one sub-optimal for the other protocols. In contrast, the EER thresholds seem to be more stable for other scoring back-ends, even despite large differences in distribution means and variances for PLDA and PSDA. These observations are supported by objective metrics presented in Table 2.

Despite low EERs for each protocol individually, the performance of CSEA degrades significantly on the pooled protocol. In contrast, CSSA does not suffer from this problem, however, it has higher error rates on the other protocols. Finally, PLDA and PSDA perform the best, overall, handling well all the cases. They also have very similar metrics and distributions of scores. These results are also in line with findings in [1] where spherical PLDA outperformed cosine similarity in the household speaker recognition task. Note that CSEA, CSSA, and sph-PLDA have exactly the same metrics in the $(1,1)$ protocol because sph-PLDA is equivalent to cosine scoring. Another observation is that models with more parameters, diag- and full-PLDA, have comparable performance to sph-PLDA. This motivates choosing sph-PLDA as a simpler and faster alternative.

It should be noted that, unlike this study, PLDA model studied in [11] was not robust to the number of enrollment utterances. This probably can be explained by the nature of i-vector distribution which is different from the distribution of large-margin embeddings.

In this section, we describe experiments on online speaker diarization. We used the same PLDA and PSDA models as for the previous experiments.

Experimental setup. We used two popular datasets of multi-speaker recordings: AMI [32], and VoxConverse [33]. Again, due to space limitations, we report only the results for the first one, while similar observations were made for the VoxConverse.

We used the development/evaluation split for the AMI corpus from [27]²²2https://github.com/BUTSpeechFIT/AMI-diarization-setup. The development set was used for tuning the hyper-parameters of the back-end models, pretrained on the VoxCeleb data.

For AMI, the evaluation was performed on Mix-Headset channel. We extracted embeddings from segments of length $2.0$ sec with $1.0$ sec overlap within the boundaries obtained by the ground-truth annotation. These embeddings were sequentially processed by several online clustering algorithms, producing the output annotation. We did not use any special heuristics for handling segments with overlapped speakers, thus one speaker was assigned to each segment.

We compared three versions of Algorithm 1: with CSEA, CSSA, and PLDA scoring. Also, we evaluated two versions of the proposed Algorithm 2, with sph-PLDA and PSDA models. All of the algorithms have at least one hyper-parameter (e.g. decision threshold) that was tuned on the development split.

Results. For the evaluation metrics, we use the diarization error rate (DER) [34] and Jaccard error rate (JER) [35]. The forgiveness collar was set to $0.25$, and overlapped speech regions were excluded from evaluation for DER, however, JER is calculated with no forgiveness collar and includes overlapped speech [35]. Table 3 provides the evaluation results.

Unlike the previous experiment on speaker verification, we found that sequential PLDA scoring performs worse than cosine. As was discussed in [36] and [37], this probably can be explained by an inadequate assumption of statistical independence of the enrollment segments, which affects the score calibration. According to the theoretical model, enrollment segments are independent draws from the within-speaker distribution, while in diarization it is clearly not the case because of a shared acoustic environment and recording channel. However, unlike i-vector embeddings considered in [10, 36], this effect seems to be less evident for large-margin embeddings. Apparently, PLDA suffers from this effect only in diarization, while yielding adequate score calibration when embeddings are less statistically dependent, as in our previous experiment.

At the same time, the proposed clustering algorithm which uses the same PLDA model delivers lower error rates than Algorithm 1. In the future, we plan to further investigate the properties of this algorithm in other applications such as household speaker recognition, where speech utterances are also processed sequentially.