Speaker Diarization as a Fully Online Learning Problem in MiniVox

In an ideal online learning scenario without oracle, we start with a single arm, and when new labels arrive new arms are then generated accordingly. This problem is loosely modelled by the bandits with infinitely many arms [25]. For our specific application of speaker registration process, we applied the arm expansion process outlined in Figure 1: starting from a single arm (for the “new” action), if a feedback confirms a new addition, a new arm is initialized and stored (more details on the handling of growing arms can be found in section 4.4).

We proposed Background Episodically Rewarded LinUCB (BerlinUCB [26]), a semi-supervised and self-supervised online contextual bandit which updates the context representations and reward mapping separately given the state of the feedbacks being present or missing. As in Algorithm 2, the steps 1 through 12 of BerlinUCB are the same as the standard LinUCB algorithm [21], and in case of a missing reward, we introduced the steps 13 through 20 as the alternative strategy. We assume that: (1) when there are feedbacks available, the feedbacks are genuine, assigned by the oracle, and (2) when the feedbacks are missing (not revealed by the background), it is either due to the fact that the action is preferred (no intervention required by the oracle, i.e. with an implied default rewards), or that the oracle didn’t have a chance to respond or intervene (i.e. with unknown rewards). Especially in the Step 15, when there is no feedbacks, we assign the context $\textbf{x}_{t}$ to a class $a^{\prime}$ (an action arm) with the self-supervision given the previous labelled context history (section 4.3). Since we don’t have the actual label for this context, we only update the reward mapping parameter $\textbf{b}_{a^{\prime}}$ and leave the covariance matrix $\textbf{A}_{a^{\prime}}$ untouched. This additional usage of unlabelled data (or unrevealed feedback) is especially important in our model.

We construct our self-supervision modules given the cluster assumption of the semi-supervision problem: the points within the same cluster are more likely to share a label. As shown in many work in modern speaker diarization, clustering algorithms like GMM [9] and spectral clustering [8] are especially powerful unsupervised modules, especially in their offline versions. Their online variants, however, often perform poorly [12]. Nonetheless, we chose three popular clustering algorithms as the self-supervision: GMM, Kmeans and K-nearest neighbors (KNN), all in their online versions.

To adapt our BerlinUCB algorithm to the specific application of speaker recognition, we firdst define our actions. There are three major classes of actions: an arm “New” to denote that a new speaker is detected, an arm “No Speaker” to denote that no one is speaking, and N different arms “User n” to denote that user n is speaking. Table 1 presents the reward assignment given four types of feedbacks. Note that we assume that when the agent correctly identifies the speaker (or no speaker), the user (as the feedback dispenser) should send no feedbacks to the system by doing nothing. In another word, in an ideal scenario when the agent does a perfect job by correctly identifying the speaker all the time, we are not necessary to be around to correct it anymore (i.e. truly feedback free). As we pointed out earlier, this could be a challenge earlier on, because other than implicitly approving the agent’s choice, receiving no feedbacks could also mean the feedbacks are not revealed properly (e.g. the human oracle took a break). Furthermore, we noted that when “No Speaker” and “User n” arms are correctly identified, there is no feedback from us the human oracle (meaning that these arms would never have learned from a single positive reward if we don’t use the “None” feedback iterations at all!). The semi-supervision by self-supervision step is exactly tailored for a scenario like this, where the lack of revealed positive reward for “No Speaker” and “User n” arms is compensated by the additional training of the reward mapping $\textbf{b}_{a_{t}}$ if context $\textbf{x}_{t}$ is assigned to the right arm.

To tackle the cold start problem, the agent grows it arms in the following fashion: the agent starts with two arms, “No Speaker” and “New”; if it is actually a new speaker speaking, we have the following three conditions: (1) if “New” is chosen, the user approves this arm by giving it a positive reward (i.e. clicking on it) and the agent initializes a new arm called “User $N$” and update $N=N+1$ (where $N$ is the number of registered speakers at the moment); (2) if “No Speaker” is chosen, the user disapproves this arm by giving it a zero reward and clicking on the “New” instead), while the agent initializes a new arm; (3) if one of the user arms is chosen (e.g. “User 5” is chosen while in fact a new person is speaking), the agent copies the wrong user arm’s parameters to initialize the new arm, since the voiceprint of the mistaken one might be beneficial to initialize the new user profile. In this way, we can transfer what has been learned for a similar context representations to the new arm.²²2Potential problems might occur if the number of users grows steadily through misclassifications. In future work, we will investigate possible branch pruning strategies and the processing of very sparse reward feedback.

We applied MiniVox on VoxCeleb [4] to generate three data streams with 5, 10 and 20 speakers to simulate real-world conversations. We extracted two types of features (more details in section 6.2) and evaluated it in two scenarios (with or without oracle). The reward streams are sparsified given a revealing probability of 0.5, 0.1 and 0.01. In summary, we evaluated our models in a combinatorial total of 3 speaker numbers $\times$ 3 reward revealing probabilities $\times$ 2 feature types $\times$ 2 test scenarios $=$ 36 online learning environments. The online learning timescale range from $\sim$12000 to $\sim$60000 timeframes, with a frame shift of 10 ms. For notation of a specific MiniVox, in this paper we would denote “MiniVox C5-MFCC-60k” as a MiniVox environment with 5 speakers ranging 60k time frames using MFCC as features.

To evaluate performance in the above MiniVox environments, we reported Diarization Error Rates (DER), the standard introduced in the NIST Rich Transcription 2009 (RT-09). In addition, as a common metric in online learning literature, we also recorded the cumulative reward: at each frame, if the agent correctly predicts a given speaker, the reward is counted as +1 (regardless of whether the agent observes it or not).

We compared 5 agents: The baseline, LinUCB is the contextual bandit with extendable arms proposed in section 4.1. BerlinUCB is our standard contextual bandit model designed for sparse feedbacks without the self-supervision modules. To test the effect of self-supervision, we introduced three clustering modules in BerlinUCB (Alg 2, Step 15) denoted B-Kmeans, B-KNN (with K=5), and B-GMM, whose clustering modules are randomly initialized and updated online.

We utilized two feature embeddings for our evaluation: MFCC [6] and a Convolutional Neural Network (CNN). We utilized the same CNN architecture as the VGG-M [27] used in VolCeleb evaluation [4]. It takes the spectrogram of an utterance as the input, and generate a feature vector of 1024 in layer fc8 (table 4 in [4] for details about this CNN).

Why don’t we use more complicated embeddings? Although more complicated embedding extraction modules such as i-vectors [7] or d-vectors [8] can improve diarization, they require extensive pretraining on big datasets, which is contradictory to our problem setting and beyond our scope.

Why do we still include a pretrained CNN embedding? Indeed, if our end goal is to let the system learn from scratch without pretraining, why do we consider it in our evaluation? The CNN model was trained for speaker verification task in VoxCeleb and we are curious about the relationship between a learned representation and our online learning agents. Despite this note, we are most interested in the performance given MFCC, because we aim to push the system to the extreme of not having pretraining of any type before deployment.

Given MFCC features without pretraining, our online learning agent demonstrated a robust performance (Figure 3a,b,c,d): in most cases, it significantly outperformed the baseline. ³³3We wish to note the overall high diarization error rate in all MFCC benchmarks: it is important to keep in mind that the bandit feedback (correct or incorrect classification) makes the online speaker diarization problem significantly more challenging, as compared to the standard supervised learning in offline speaker diarization, since the true label is never revealed in bandit setting unless the classification is correct. Thus, the diarization error rate in a bandit online setting is expected to be much higher than in the supervised learning setting, which is not due to inferiority of bandit decision making algorithm versus other classifiers, but due to increased problem difficulty.

Learning without Oracle (Table 2). In both the MFCC and CNN MiniVox environments, we observed that BerlinUCB and its variants outperform the baseline most of the time. The discrepancy of performance of the MFCC and CNN environments can be explained by the innate difficulties of the two tasks: while the CNN embeddings are already well separated because they were pretrained with contrastive loss [4], in MFCC environments our online learning models need to learn from scratch both how to cluster and how to map the reward with the features, while maintaining a good balance between exploitation and exploration.

Learning with Oracle (Table 3). Given the number of speakers, the online clustering modules appears to be more effective. However, the behaviors vary: we observed that B-GMM performed the poorest in the oracle-free environments, but performed the best in some environments with oracle; we also noted that despite the consistent best model in many oracle-free environments, the standard BerlinUCB was surpassed by the baseline and its self-supervised variants in a few MFCC cases with oracle; in certain challenging cases where the reward is sparsely revealed (p=0.01 or 0.1), the self-supervised variants improve the performance of BerlinUCB.

Is self-supervision useful? To our surprise, our benchmark results suggested that for most cases, the proposed self-supervision modules didn’t improve upon our proposed contextual bandit model. Only in specific conditions (e.g. MiniVox C20-MFCC-60k p=0.1 with Oracle), the self-supervised contextual bandits outperformed both the standard BerlinUCB and the baseline. Further investigation into the reward curve revealed more complicated interactions between the self-supervision modules with the online learning modules (the contextual bandit): as shown in Figure 3e,f, B-GMM and B-KNN built upon the effective reward mapping from their BerlinUCB backbone, and benefited from the unlabelled data points to yield a fairly good performance.