The HCCL system for VoxCeleb Speaker Recognition Challenge 2022

Training Data: We use VoxCeleb2-dev(vox2dev) [1] as training data, which consists of 1092009 utterances from 5994 speakers. To augment data, we first use the SoX speed function with speeds 0.9 and 1.1 to generate extra twice speakers [2]. In total, there are 17982 speakers and 3276027 utterances. Then, we use MUSAN [3] and RIRs noises [4] to perform online data augmentation. Similar to SpeakIn systems for VoxSRC2021 [5], we used a chain augment pipeline to generate samples:

• •

  MUSAN noise with probability 0.2

• •

  MUSAN music with probability 0.2

• •

  MUSAN speech with probability 0.2

• •

  RIRs noises with probability 0.6

SpeechBrain [6] was used to build the pipeline.

Developing Data: We use official validation sets [7, 1, 8, 9] to evaluate our models: Vox1-O, Vox1-E, Vox1-H, Vox20-dev, Vox21-dev and Vox22-dev.

Features: We extract 80-dimensional log-Mel Filter Banks (Fbank) as input features without any voice activity detection(VAD). The frame length is 25ms and the frame shift is 10ms. Cepstral mean normalization(CMN) is applied. Our implementation was based on TorchAudio [10].

Two main stream of current most popular model structures was used for the challenge: 1D-convolution-based ECAPA-TDNN [11] and its variants, 2D-convolution-based ResNet [12] series and RepVGG [13].

ECAPA-TDNN: We trained ECAPA-TDNN large with 1024 channels and its two variants, ECAPA-TDNN-X3(EX3) and ECAPA-TDNN-X4(EX4). We used SpeechBrain implementation with 1024 channels. To further boost its performance, we made it deeper and added branches in res2block to enhance representational ability. For EX3, we combine two se-res2block as one basic block. For EX4, we added one more basic block, and meanwhile, to restrict the receptive field, we did not use dilation in the first few blocks. Besides, we used group convolution to preserve the original feature map.

ResNet/SE-ResNet: ResNet is one of the most popular model structures currently. Here we used standard ResNet34 with 64 channels. Squeeze excitation module [14] uses an attention mechanism to re-weight feature map channels. Besides, we use a modified version of ResNet as described in [15] except that the SE module was added only in the first two blocks. We trained SE-ResNet101 with 32 channels and 64 channels.

HS-ResNet & Res2Net In order to model multi-scale features, we used Res2Net [16] and its variants hs-resnet[17] with dssa module. Both models are 50 layers deep, scale 8 and width 14 in Res2Net and scale 8 and width 6 in HS-ResNet.

RepVGG RepVGG use 3 branches of convolution and batch normalization when training and re-parameterize them as one for inference. Branch greatly boost capability to model multi-level features. We use the official version with configuration b1.

Deep neural network-based systems use a pooling layer to aggregate frame-level features into segment-level embeddings. We used two pooling methods: attentive statistics pooling(ASP) [18] and channel-wise correlation pooling(CWC) [19].

Margin-based loss functions have greatly improved system performance. In this challenge, we adopted circle loss [20]. Moreover, subcenter [21] and intertopk [22] are two plugin methods that could further improve the discrimination of embeddings. Circle loss is formulated as follows:

where $m$ is the margin and $s$ is the scale factor.

We trained models with a two-stage protocol. All experiments were based on the PyTorch [23].

Adam optimizer with weight decay 5e-5 was used in the first stage. Cycle learning rate scheduler [24] was adopted, where the minimum learning rate is 1e-8 and the maximum is 1e-3, we trained 2 cycles with one cycle of 100k steps. The batch size was 1024 and the segment duration was 2s. Margin and scale were set to 0.35 and 60 for circle loss, respectively. We used subcenter k=3 and intertopk k=5, m=0.1.

The second stage was large margin fine-tuning(LMF), we expanded segment duration to 6s, only removed intertopk from loss function, and increased weight decay from 5e-5 to 4e-4. We used a constant learning rate of 2e-5 for 10k steps.

After LMF, cosine distance was used for $4s\times 10$ scoring. Evenly cut 10 4-second long segments from utterances, the mean of score matrix with the size of $\mathbb{R}^{10\times 10}$ served as the score of a trial. Then, we used adaptive score normalization(AS-norm) [25] and quality measure functions(QMF) [26] to calibrate the scores. For AS-norm, speaker-wise averaged embeddings from vox2dev, leading to 5994 cohort speakers with top 400 imposter scores were used. By the way, we removed imposter variance. For QMF, we followed IDLAB’s method to generate 30k trials from vox2dev. Then we trained the logistic regression(LR) model to calibrate the AS-normed score. In the end, another LR model was used to combine all of the calibrated models to get the final fused score. While the generated trials could not perfectly fit the evaluation distribution, we manually tuned the model weights based on Vox22-dev trials.

Because of the lack of filtering when constructing the CN-Celeb2[27, 28], it contains much noisy audio. One of the most intuitive manifestations is that there is much-repeated audio in CN-Celeb2, and some are given different labels. We directly used md5sum to de-duplicate the speech and the number of audio decreased from 455,946 to 409,628.

To obtain high confidence edges by using the voting strategy, it is beneficial to select models with as much variance as possible, from the model structure to the training Protocol. We selected the following five models: (1) SE-ResNet34 with 32 channel; (2)ECAPA-TDNN with 1024 channel; (3) Conformer-MFA with 256 hidden dim [29]; (4) SE-ResNet101 with 32 channel; (5) Cot-Net[30] with 32 channel; Others settings are shown in Table 3.

During the evaluation process, domain adaptation is necessary due to a large domain mismatch. An important manifestation of domain mismatch on embedding is the difference between the mean and variance. Thus, the simplest approach is to align the centers of the different domains directly, and experiments show significant improvements. Further, aligning the variance can also achieve improvements in theory. We attempt to apply CORAL [31], CORAL+[32] and CORAL++[33] into embeddings of target domain directly and use cosine similarity for scoring. However, there is no performance improvement unless we use back-ends, LDA, and PLDA[34, 35]. Due to time constraints and inconvenient operations, we do not do this work systematically and will do these in the future.

Because AHC is computationally infeasible and k-means depend on the estimate of k, we propose a novel cluster algorithm, a progressive sub-graph clustering algorithm based on two Gaussian fitting and multi-model voting, denoted as GMVPG clustering. The key points of this algorithm are as follows: First, finding high-confidence positive trials, that is, edges, using a multi-model voting strategy based on the KNN affinity graph. Secondly, utilizing connected sub-graphs to obtain pseudo labels Then, using iterative top-k information to gradually merge sub-classes to prevent super-classes. Finally, two Gaussian distributions are introduced to fit the intra-class score distribution to further check for high-confidence edges. The detailed algorithm is shown as follows:

Since the GMVPG method brings some mislabeled and noisy samples, it is important to correct labels after the initial model training. The key points of this algorithm are as follows:

i. Split audio into three types according to its similarity to centers, high/median/low-confidence; ii. Stas the correlation between each class based on audio with median-confidence iii. Integrate the results from multi models to correct the labels.

The detailed algorithm is shown as follows: 1. calculate the similarity of all audio in CN-Celeb to the two most similar class centers, denoted as $sim_{i}^{top1}$ and $sim_{i}^{top2}$, the labels of most similar class centers are denoted as $y_{i}^{top1}$ and $y_{i}^{top2}$; 2. Split audios according to $sim_{i}^{top1}$ and $sim_{i}^{top2}$: $sim_{i}^{top1}>0.5$ and $sim_{i}^{top2}<0.4$ high-confidence; $sim_{i}^{top1}>0.5$ and $sim_{i}^{top2}>0.4$ median-confidence; $sim_{i}^{top1}<0.5$ low-confidence; 3. Use audio with median-confidence to find labels of samples, which comes from the same speaker but are given labels of multiple speakers by the GMVPG clustering algorithm, 4. Two classes are merged into one when multiple models all show that they need to be merged. 5. Filter out audio that is low confidence, other audio is labeled by using predicted posterior probability.

Score calibration has been an essential part in recent VoxSRCs. However, unlike track1&2, track3 provided only 50 labeled speakers. To build a matched developing set as far as we can, we filtered 70 speakers with 20 segments each from unlabeled data according to their clustering purity. We gave labeled speakers more weight when generating developing trials as pseudo-label could be wrong. Finally, we got 40000 trials which is equal to the validation trials. The results show that our developing set is slightly easier but still brings performance gain.

Models shown in Table 3 are used to cluster. Systems in Table 4 are developed for Track3. Results we submitted are shown in Table 5. After GMVPG clustering algorithm, we obtain 1711 speakers and 348861 utts. After model training and label correction, we obtain 1760 speakers and 387700 utts.