Contrastive-Mixup Learning for Improved Speaker Verification

The original prototypical network is designed for few-shot learning tasks [12] where good generalization can be achieved by training on a small number of samples in each class. Recent work demonstrates that metric learning objectives, such as generalized end-to-end and angular prototypical networks, outperform conventional classification objectives for speaker recognition. In particular, Chung et al. [10] presented a comprehensive study for metric learning in speaker recognition and conclude that angular prototypical loss outperforms state-of-the-art methods. In this paper, we choose angular prototypical (AP) loss to establish our baselines.

During training, suppose each batch contains $M$ utterances from $N$ different speakers. The embedding of each speaker is $x_{j,i}$, where $1\leq j\leq N$ and $1\leq i\leq M$. For the prototypical network, each batch consists of a support set $S$ and a query set $Q$. The $M$th utterance is set to be the query utterance. The centroid of speaker $j$ is computed:

$$c_{j}=\frac{1}{M-1}\sum_{m=1}^{M-1}\mathbf{x}_{j,m}$$

(1)

Then, the distance between $\mathbf{x}_{j,M}$ and $\mathbf{c}_{k}$ are computed using trainable scale and bias coefficients, $w$ and $b$, as follows:

$$\mathbf{S}_{j,k}=w\cos{(\mathbf{x}_{j,M},\mathbf{c}_{k})}+b$$

(2)

Finally, the angular prototypical loss is computed using exponentiation as follows:

$$L_{p}=-\frac{1}{N}\sum_{j=1}^{N}\mathrm{log}\frac{e^{\mathbf{S}_{j,j}}}{\sum_{k=1}^{N}e^{\mathbf{S}_{j,k}}}$$

(3)

As shown in (3), and as reported in [23], angular prototypical loss outperforms the original prototypical networks and GE2E for speaker recognition tasks. As a result, we use AP loss as the baseline loss function for model training in our experiments.

Mixup training is based on the principle of vicinal risk minimization where the classifier is trained in the vicinity of each training sample [8]. Despite its simplicity, mixup has succeeded in a wide range of applications including computer vision [8], natural language processing [24], and semi-supervised learning [25]. In mixup, the input data and its associated label is modified as follows:

	$\displaystyle\overline{x}_{i}=\lambda x_{i}+(1-\lambda)x_{\mathrm{R}_{i}}$		(4)
	$\displaystyle\overline{y}_{i}=\lambda y_{i}+(1-\lambda)y_{\mathrm{R}_{i}}$

where $x_{i}(0\leq i\leq B-1)$ is the $i$th data in the batch of size $B$, and $y_{i}$ is the one-hot encoding of the label. $\mathrm{R}_{i}$ is the $i$th randomly shuffled index. For each training batch, the interpolation parameter $\lambda$ is sampled from a symmetric $Beta(\alpha,\alpha)$ distribution with $\alpha\in(0,\infty)$. Given fabricated input-label pairs, $(\overline{x},\overline{y})$, the loss function for the classification task is computed as follows:

$\displaystyle L_{mixup}(\overline{x},y,y_{\mathrm{R}},\lambda)=\lambda CE(\overline{x},y)+(1-\lambda)CE(\overline{x},y_{\mathrm{R}}),$

(5)

where CE denotes the cross-entropy loss. To apply mixup to AP loss, our first approach is to use cross entropy losses with respect to the ground truth and the shuffled labels as follows:

	$\displaystyle L_{CE\_mixup}=$
	$\displaystyle-\frac{1}{N}\sum_{i=1}^{N}\left[\lambda\mathrm{log}\frac{e^{\overline{\mathbf{S}}_{i,i}}}{\sum_{k=1}^{N}e^{\overline{\mathbf{S}}_{i,k}}}+(1-\lambda)\mathrm{log}\frac{e^{\overline{\mathbf{S}}_{i,\mathrm{R}_{i}}}}{\sum_{k=1}^{N}e^{\overline{\mathbf{S}}_{i,k}}}\right]$		(6)

by interpolating the losses with the ground truth and the shuffled labels. In the rest of this paper, (6) is referred to CE mixup. When applying mixup for raw speech signals, it is critical to normalize the volume when interpolating utterances from different speakers as it adjusts sound from diverse sources to the same volume level.

While mixup approach has been widely used for classification tasks, it remains a question how to implement mixup with metric learning objectives. Herein, we reformulate the AP loss by introducing a binary label function $d_{j,k}$:

$\displaystyle L_{c\_mixup}=-\frac{1}{N}\sum_{j=1}^{N}\mathrm{log}\frac{\sum_{k=1}^{N}d_{j,k}e^{\overline{\mathbf{S}}_{j,k}}}{\sum_{k=1}^{N}e^{\overline{\mathbf{S}}_{j,k}}}$

(7)

where $0\leq d_{j,k}\leq 1$. When $d_{j,k}=1$ for $j=k$ and 0 otherwise, this becomes identical to (3). In this paper, we set $d_{j,k}$ as follows:

$$d_{j,k}=\begin{cases}\lambda,&\text{if }k=j\\ 1-\lambda,&\text{if }k=\mathrm{R}_{j}\\ 0&\text{otherwise}\end{cases}$$

(8)

In this scenario, the binary label $d_{j,k}$ can act as the virtual label for speaker/utterance pairs. When applying mixup on reformulated AP loss, the convex interpolation can be performed on both original utterances and binary labels. We investigate the performance of contrastive mixup relative to the vanilla AP loss function.

Our models are trained using the development set of VoxCeleb2 dataset [23]. VoxCeleb2 dataset contains more that 1 million utterances collected from nearly 6,000 speakers under controlled conditions. In order to obtain a fair comparison with previous results, the model evaluation is performed based on the test set of VoxCeleb1 [26].

In our experiments, we employed PyTorch code [23] as a baseline. During training, 2s-long segments are randomly extracted from each utterance. The input feature, log Mel-spectrogram, was extracted every 10 ms using a 25-ms window size. As shown in [23], Fast ResNet34, which has a quarter-size channel compared to the original ResNet-34 [27], was used as a backbone model. The encoded output is aggregated using self-attentive pooling (SAP) [28] to generate utterance-level representations.

The models in this paper were trained using four NVIDIA V100 Tensor core GPUs with distributed configuration. We use the Adam optimizer with a learning rate of 0.001, decreasing by 5% for every 10 epochs. Unless specified elsewhere, all models are trained for 500 epochs using a distributed configuration with four GPUs.

As indicated in [23], large batch size leads to improved performance for metric learning methods due to the ability to sample hard negative samples within the batch. Therefore, we intentionally choose the largest batch size without exceeding the memory limits of the GPUs. For Quarter ResNet-34, the batch size is $400\times 2$, namely the number of speakers $\times$ the number of utterances per speaker. For fair and reliable comparisons, all experiments are repeated three times, with mean and standard deviation calculated.

In our effort to investigate the effect of contrastive mixup, given that utterances per speaker are limited, we build a small training dataset by randomly sampling a certain number of utterances from each speaker. As shown in Table 1, the reduced training datasets consists of 2, 3, 5, and 10 utterances per speaker, which is significantly smaller compared to the original VoxCeleb2. Then, we compare different training strategies using those down-sampled datasets.

The evaluation metric used in this work is equal error rate (EER) in %, where false acceptance rate (FAR) and false rejection rate (FRR) are closest. To demonstrate the effect of contrastive mixup across different training settings, we conduct experiments and compare the resulting EERs for the following cases:

1. 1.

   original AP loss without augmentation and mixup;

2. 2.

   original AP loss plus augmentation without mixup;

3. 3.

   contrastive mixup without augmentation;

4. 4.

   contrastive mixup plus augmentation.

Here augmentation refers to noise addition and room impulse response (RIR) application based on the widely-used MUSAN corpus [13], which should be distinguished from the proposed contrastive mixup approach.

We use the VoxCeleb1 [26] test set to evaluate model performance. Unlike the training stage, we sample ten 4-second temporal crops at regular intervals from each test segment. The similarities between all segment pairs across utterances are computed. The mean value of the similarities is calculated as the final score. A similar protocol can be found in [10].

We focus on two critical aspects. First, how does contrastive mixup compare to the vanilla AP loss, and second, how does contrastive mixup behave with limited training data? To this end, we train the models using different sizes of datasets. Note that all the other settings and hyperparameters are kept constant.

In Tables 2–4, the baseline EER refers to training using original AP loss without augmentation. The mixup EER refers to training using the proposed contrastive-mixup loss without augmentation. The augmentation EER refers to training using original AP loss with augmentation [13]. The mixup + augmentation EER refers to training using contrastive-mixup loss as well as augmentation. In addition, when employing contrastive-mixup loss, we fine-tune the mixing coefficient $\lambda$ to achieve the optimal performance.

In our preliminary analysis, we found that the performance of contrastive mixup is closely dependent on the choice of hyperparameter $\lambda$. Therefore, we have explored a range of $\lambda\sim Beta(\alpha,\alpha)$, where $\alpha$ equals 0.1, 0.2, 0.4, or 0.6, and compared the resulting EERs. Experimental results indicate that when training with the entire VoxCeleb2 dataset, setting $\alpha$ to be a smaller value (0.1) gives the best performance, whereas setting $\alpha$ to a larger value (e.g., 0.4) gives the smallest EER when training data is limited. A previous study [15] shows that relatively small values of $\alpha\in[0.1,0.4]$ produce the best performance for classification, whereas a large value of $\alpha$ results in significant under-fitting. Our results agree well with previous observations in different domains [8, 29]. Also, the mixup is performed at the raw waveform level instead of the Mel-spectrum level based on our preliminary results, where the waveform level and the Mel-spectrum level give EERs of 2.11 $\pm$ 0.02 and 2.44 $\pm$ 0.06, respectively. Unless specified, the EER and its standard deviation in this paper were computed by repeating the experiments three times.

Table 2 presents the EERs of baseline and two different versions (loss functions) of the mixup algorithm. Given the entire VoxCeleb2 training data set, we observed that mixup implementations improved with a modest gain of 4.52%. Based on this result, in the follow-up experiments, we used contrastive mixup (7).

Table 3 compares the EERs with limited training data, where the number of utterances per speaker is limited to 2, 3, 5, or 10, to support our hypothesis that mixup improves generalization. The gain with mixup was modest when the full VoxCeleb training data was used, as shown in Table 2. As the number of utterances per speaker is reduced, we observed that contrastive mixup shows larger performance improvements relative to the baseline. When training on only two utterances per speaker, contrastive mixup can reduce the EER by up to 16.3%. This supports our hypothesis that mixup works as a generalization enhancement, especially with limited training data.

In Table 4, contrastive mixup is applied on top of online augmentation. In these experiments, we fine-tune the mixing coefficient by choosing $Beta(0.6,0.6)$ for 2 utterances, $Beta(0.2,0.2)$ for 3 utterances, $Beta(0.4,0.4)$ for 5 utterances, and $Beta(0.1,0.1)$ for 10 utterances. While augmentation improves over the baseline, adding contrastive mixup on top of augmentation gives additional improvements, as shown in Table 4.

As indicated by the evaluation results, while performance improvement is modest when training on the full VoxCeleb dataset, contrastive mixup consistently outperforms the baseline when the number of utterances per speaker is limited. The relative improvement between contrastive mixup and the baseline increases as the number of training utterance is reduced. Thus, our experimental results support the hypothesis that mixup boosts the generalization of neural network models.