PCF: ECAPA-TDNN with Progressive Channel Fusion for Speaker Verification

The local receptive field is one of the basis when constructing deep convolution networks, in collaboration with stacked blocks, it expands sensor space from local to the whole feature map in deep blocks while avoiding over-fitting. One of the critical differences between ResNet and TDNN is that two-dimensional convolution preserves a local receptive field on both frequency and temporal channels. Because conventional TDNN has full access to all feature channels, the risk of over-fitting goes high when increasing the number of parameters. To alleviate this phenomenon, we propose a strategy called progressive channel fusion, which allows the model to gradually fuse the channel relationship across the forward propagation.

Assume input feature map $X\in\mathcal{R}^{F\times T}$, where $F,T$ is the feature dimension and temporal dimension respectively. We split $F$ into $N$ sub-bands, where $N=8$ for the first block, then we half $N$ after each block. It should be noted that in Res2Block, benefit from its hierarchical design, frequency bands still have the opportunity to communicate, while the difference is that our strategy assigns major frequency bands to channel splits. Besides, we introduce links between the spectrogram and each block with a TDNN block, it shares an identical split configuration with the target block. The receptive field of ResNet34, ECAPA-TDNN, and the proposed PCF-ECAPA are visualized in Fig 1. the first row shows the receptive field of block1 and block3 in ResNet34. It has a gradually increased sensor space. The second row gives the result of the original ECAPA-TDNN, which always has global sensor space across the channel dimension, while the last two rows present all blocks’ receptive field of our proposed progressive channel fusion strategy. It shares a similar behavior with the ResNet model, where the visible area of the spectrogram spread around as the block goes deep.

RepVGG [12, 14] has proven to be effective in recent challenges. Due to its re-parameterized structure design, the model can learn multi-scale features during training, and the convolution branches are merged into a single branch for inference. The key operation to improve the performance of RepVGG is the multi-branch structure, it introduces convolution branches with multiple kernel sizes so that the model can learn multi-scale representation. Therefore, we also introduce this structure by adding a branch with a convolution kernel size of 1 in the form of res2block. Although the two branches cannot be merged into one during inference, it still boosts the ability to capture input features at multiple levels.

It is usually more effective to make the model deeper rather than wider when scaling convolution networks because of the growing range of receptive field. In section 3.1, we introduce a progressive channel fusion strategy, which improves the model performance while reducing the model parameters. To make up for the weakening of the modeling ability caused by the reduction of parameters, after the first TDNN block, we extend the backbone to 4 blocks, each block containing 2 Res2Blocks with the same dilation. For the MFA module, we concatenate outputs from 4 blocks as the input.

We use voxceleb2-dev [15] as training set, containing 1,092,009 utterances from 5,994 speakers. For data augmentation, we use MUSAN [16] and RIRS-NOISES [17]. 80-dimensional log-Mel Filter Bank is used as input features with cepstral mean normalization applied, and voice activity detection is not used. For evaluation, we use official evaluation sets including Vox1o,e,h, and validation sets from the last three years of VoxCeleb speaker recognition challenges[18, 19].

We use the ECAPA-TDNN implemented by speechbrain [20] as the baseline model for the experiments, where two settings are used, a base model with 512 channels, and a large model with 1024 channels. Our model also adopts the same configuration. We fix the output channels of the MFA module to 1536 to be consistent with the original configuration. For ResNet, we use ResNet18 and ResNet34 with 32 channels and the same pooling as ECAPA-TDNN.

We use Adam [21] for optimization, and the learning rate curve adopt the cycle strategy [22]. We train for 3 cycles with one cycle lasting 100k steps, the learning rate in each cycle varies from 1e-8 to 1e-3, and the weight decay is 5e-5. The batch size is set to 256. Circle loss [23], which has stronger constraints on speaker embedding compared to AAM-Softmax loss [24], is used with $m=0.35,s=60$. All models are trained with the same setting.

To bridge the gap between the duration of segments for training and evaluation, we sample short clips from testing utterances and use the mean of the cosine similarity between a pair of embedding matrices as the final score. We test the model at the end of the final cycle and report all systems performance in terms of equal error rate(EER) and minimum Detection Cost Function(minDCF) with $p_{target}=0.01,C_{FA}=C_{Miss}=1$.

Table 2 summarizes the performance of PCF-ECAPA and the original ECAPA-TDNN together with the number of model parameters except for the classification layer. Our proposed architecture outperforms the baseline systems and gives an average relative improvement of 15.6% on EER and 15.2% on minDCF over the best baseline system. We conducted ablation experiments on the base model, and the results are shown in Table 3. From the ECAPA-TDNN base, we stack three methods one by one and finally get the proposed PCF-ECAPA.

We first evaluate the impact of the model depth. Simply increasing the number of blocks from 3 to 8 dramatically improves the performance, where EER improves from 1.058 to 0.792 and minDCF reduces to 0.0912, giving 26.6% and 10.7% relative improvement respectively. The almost doubled number of parameters greatly enhances the model’s ability to capture deep representations. Meanwhile, the deepened base model exceeds ECAPA-TDNN-large for 7.5% with less amount of parameters. It proves our assumption that it is usually more efficient for convolution networks to deepen the model rather than broaden it.

Secondly, the branched block pushes the EER to 0.744 but pulls the minDCF back to 0.1024. Parallel branch structure helps the model capture multi-scale features at the cost of 1.8% extra parameters. The performance loss may come from the circle loss based on our experience.

Finally, we apply the proposed PCF strategy. It further improves EER to 0.718 and minDCF to 0.0858. Restricting the sensor space of TDNN models over channel dimension is proved to be effective because of the fine-grained attention on each frequency sub-bands. Besides, it brings an 18.3% cut on the number of parameters as a result of the local receptive field.

Moreover, experiments show that simply scaling the model from 512 channels to 1024 channels brings little improvement, resulting in EER=0.718 and minDCF=0.0892 on vox1o. On other evaluation sets, the large model gets the biggest boost on Vox22-dev. Nevertheless, it has a less competitive parameter efficiency.