MFA-Conformer: Multi-scale Feature Aggregation Conformer for
Automatic Speaker Verification

Local features and global dependencies are both crucial for speaker representation learning. Local features, including the pitch, intonation style and pronunciation pattern, are essential to represent the speaker characteristics. And the good capacity of global context modeling to capture the long-range dependencies of variable-length speech will lead to robust speaker embedding extraction. Convolution neural networks are good at extracting local features but is weak at capturing global representations. Self-attention from Transformer can capture long-range global context dependencies but is lack of local details.

In order to both model the global and local features more directly and efficiently, we employ the Conformer block[22], a combination of CNNs and Transformers, to better capture the speaker characteristics. Multi-head self-attention (MHSA) and convolution modules are the key components for a Conformer block. The MHSA in the Conformer employs the relative positional encoding scheme proposed in Transformer-XL[16]. It encodes inputs according to the relative position deviation and takes account of both the global content offset and global position offset. Therefore, the relative positional encoding MHSA module is more robust for the utterances with variable input lengths. The convolution module after the MHSA consists of Pointwise convolution, 1D Depthwise convolution and BatchNorm after the convolution layer contributing to training deep models easier.

The structure of the Conformer block[22] is different from the Transformer block[15]. It contains two Macaron-like feed forward modules (FNN) with half residual connections sandwiching the MHSA and convolution modules (Conv). Mathematically, for the input feature $\bm{h_{i-1}}$ to the $i$ Conformer block, the output feature $\bm{h_{i}}$ is calculated by:

	$\displaystyle\widetilde{\bm{h}}_{i}$	$\displaystyle=\bm{h_{i-1}}+\frac{1}{2}{\rm FNN}(\bm{h_{i-1}})$		(1)
	$\displaystyle\bm{h^{\prime}_{i}}$	$\displaystyle=\widetilde{\bm{h}}_{i}+{\rm MHSA}(\widetilde{\bm{h}}_{i})$
	$\displaystyle\bm{h^{\prime\prime}_{i}}$	$\displaystyle=\bm{h^{\prime}_{i}}+{\rm Conv}(\bm{h^{\prime}_{i}})$
	$\displaystyle\bm{h_{i}}$	$\displaystyle={\rm LayerNorm}(\bm{h^{\prime\prime}_{i}}+\frac{1}{2}{\rm FNN}(\bm{h^{\prime\prime}_{i}}))$

Note that $\bm{h_{i-1}}\in\mathbb{R}^{d\times T}$ and $\bm{h_{i}}\in\mathbb{R}^{d\times T}$, where $d$ denotes the Conformer encoder dimension and $T$ denotes the frame length.

Previous studies[26, 27] indicate that the low-level feature maps can also contribute towards the accurate speaker embedding extraction. Based on this experience, in ECAPA-TDNN system, the output feature maps from all SE-Res2blocks are aggregated before the final pooling layer, and this aggregation leads to an obvious performance improvement. Likewise, we concatenate the output feature maps from each Conformer block and then feed them into a LayerNorm layer:

	$\displaystyle\bm{H^{\prime}}$	$\displaystyle={\rm Concat}(\bm{h_{1}},\bm{h_{2}},...,\bm{h_{L}})$		(2)
	$\displaystyle\bm{H}$	$\displaystyle={\rm LayerNorm}(\bm{H^{\prime}})$

where $\bm{H^{\prime}}\in\mathbb{R}^{D\times T}$ and $\bm{H}=[H_{1},H_{2},...,H_{T}]\in\mathbb{R}^{D\times T}$. $L$ denotes the number of Conformer blocks and $D=d\times L$.

Furthermore, we adopt the attentive statistics pooling [28, 6] to capture the importance of each frame and extract more robust speaker embedding. Specifically, for a frame-level feature $H_{t}$ at time step $t$, we firstly calculate scalar score $e_{t}$ and normalized score $\alpha_{t}$ as:

	$\displaystyle e_{t}$	$\displaystyle=\bm{v}^{T}f(\bm{W}H_{t}+\bm{b})+k$		(3)
	$\displaystyle\alpha_{t}$	$\displaystyle=\frac{\exp(e_{t})}{\sum_{\tau=1}^{T}\exp(e_{\tau})}$

where $\bm{W}\in\mathbb{R}^{D\times D}$, $\bm{b}\in\mathbb{R}^{D\times 1}$, $\bm{v}\in\mathbb{R}^{D\times 1}$ and $k$ are the trainable parameters for attention. $f(\cdot)$ denotes the Tanh activation function. After that, the normalized score $\alpha_{t}$ is adopted as the weight to calculate the weighted mean vector $\bm{\widetilde{\mu}}$ and weighted standard deviation $\bm{\widetilde{\sigma}}$, which are formulated as:

	$\displaystyle\bm{\widetilde{\mu}}$	$\displaystyle=\sum_{t=1}^{T}\alpha_{t}H_{t}$		(4)
	$\displaystyle\bm{\widetilde{\sigma}}$	$\displaystyle=\sqrt{\sum_{t=1}^{T}\alpha_{t}H_{t}\odot H_{t}-\mu\odot\mu}$

where $\mu=\frac{1}{T}\sum_{\tau=1}^{T}H_{\tau}$ and $\odot$ denotes the Hadamard product. The output of the pooling layer is given by concatenating the vectors of the weighted mean $\bm{\widetilde{\mu}}$ and weighted standard deviation $\bm{\widetilde{\sigma}}$.

Finally, the speaker embedding is extracted from a high dimension vector to a low dimension vector with BatchNorm using the fully-connected linear layer.

VoxCeleb1&2[29, 30] and SITW[25] are used in our experiments. VoxCeleb is an audio-visual dataset consisting of 2,000+ hours short clips of human speech, extracted from interview videos on YouTube. SITW is a widely-used standard evaluation dataset in real-world conditions, consisting of 299 speakers including two testing trials (SITW.Dev and SITW.Eval). For model training, we use the development set of VoxCeleb1&2, which contain 1,240,000+ utterances from 7,205 speakers.

To better illustrate the advantages of MFA-Conformer in different utterance duration conditions, we adopt 3 trials including Voxceleb1-O, SITW.Dev and SITW.Eval for recognition performance evaluation. Note that Voxceleb1-O is the test part of Voxceleb1 and the major durations of testing utterances are 5-8s, which can be regarded as the short-duration utterance scenario. As for SITW, the major durations are about 30-40s, therefore it can be regarded as the long-duration scenario.

The speaker embedding dimension of all systems is 192. For fair comparisons, we also re-implement the r-vector system proposed in [8] and the ECAPA-TDNN system proposed in [6] as the baselines. Other configurations are presented below:

We use Pytorch[31] framework to implement the proposed MFA-Conformer and the baseline systems²²2Original code of the baseline systems is not available. For a fair comparison, all the models are trained in the same framework. There may be a few mismatches between the re-implemented baseline systems and reference papers[8, 6]. We borrow code from WeNet toolkit[32] and the training details can be found in the accompanying open-sourced code. . A fixed length 3-second segments are extracted randomly from each utterance. The input features are 80-dimensional Fbanks with a window length of 25 ms and a frame-shift of 10 ms. No voice activity detection or augmentation are performed. All models are trained using additive margin Softmax (AM-Softmax) loss[33] with a margin of 0.2 and a scaling factor of 30. We use the Adam optimizer with an initial learning rate of 0.001 and decrease the learning rate by 50% every 4 epoch. We also set the weight decay as 1e-7 to avoid overfitting and perform a linear warmup at the first 2k steps. The batch size is 200.

We use cosine distance with adaptive s-norm[34] for scoring. Then we report the Equal Error Rate (EER) and minimum Detection Cost Function (minDCF) with $P_{target}=0.01$ and $C_{FA}=C_{Miss}=1$ for performance evaluation. Furthermore, we calculate the real-time factor (RTF) on an Intel(R) Xeon(R) Silver 4210R CPU (2.40GHz) to evaluate the inference speed.

In this section, we present the performance of the proposed MFA-Conformer with different subsampling rates, as well as the performance of the two baseline systems ResNet34 and ECAPA-TDNN. Table 1 reports the equal error rate (EER) and minimum Detection Cost Function (minDCF) together with the number of model parameters and real time factor (RTF). Firstly, it can be observed from the first and second lines that comparing with ResNet34, the ECAPA-TDNN system achieves an obvious advantage in recognition performance, yet the RTF is unsatisfied. Secondly, from the results of the proposed MFA-Conformer with different subsampling rates, we find that MFA-Conformer (1/1), MFA-Conformer(1/2) and MFA-Conformer (1/4) achieve promising results. Compared with the popular ECAPA-TDNN systems, the MFA-Conformer(1/2) obtains 21% relative improvement in EER and 32% relative improvement in inference speed (RTF). Thirdly, as described in Section 3.1, VoxCeleb1-O is a short-duration test scenario and SITW is a long-duration test scenario. We can find that the MFA-Conformer is able to obtain more competitive results than CNNs-based systems in long-duration utterance scenarios. This also indicates that MFA-Conformer can better handle long-range sequences and extract a more reliable speaker embedding for long utterances.

In the real-world applications, such as video processing, real-time online meeting, the utterance lengths may vary from less than 5 seconds to more than 30 seconds. Extracting robust global features for utterances with different lengths is important for speaker verification. The Multi-head self-attention (MHSA) is the key component to make Transformer or Conformer unique from the widely used CNNs-based models. In this section, we investigate the impacts of the global interactions modeling of MHSA by comparing the system performances with different utterance durations. We randomly split the test utterances of SITW according to different duration ranges to generate new trials, and report the EERs of the new trials in Fig.2. we can observe that when the test utterance duration becomes longer, the MFA-Conformer achieves larger relative improvements. It further indicates that MFA-Conformer is better at modeling long-range, global context information.

The Conformer inserts an additional Convolution module into the Transformer to better capture the local information. In this section, we study the impacts of the local feature modeling by comparing the performances between Transformer and Conformer blocks. We replace the Conformer blocks by Transformer blocks and make a group of experiments with different number of blocks $L$ in SITW test set. As shown in Table 2, it can be clearly observed that Conformer blocks have remarkable advantages over Transformer blocks. MFA-Transformer can hardly obtain a satisfactory performance even increasing the number of the blocks. This indicates that the local spatial modeling by convolution module plays a critical role in accurate speaker embedding extraction. And the results show that setting $L$ to 6 performs better than the rests.

In the final section, we remove the individual components introduced in Section 2 to study the effect of each component contributing to performance improvements. Due to the space limitation, we only present the results on VoxCeleb1-O shown in Table 3, and the results in other sets attain the same trend. The results in the first line are the MFA-Conformer with 1/2 subsampling rate. In the second line, we remove relative positional encoding scheme; in the third line, we remove the Macaron-style feed forward network and only keep the single feed-forward network after MHSA; in the fourth line, we discard the multi-scale feature aggregation strategy and only use the output representations from the last Conformer block; in the last line, we remove the convolution module to further measure the impact of the local feature modeling. It can be observed that multi-scale feature aggregation and convolution module play the most critical roles in achieving the promising performance Aggregation of the outputs from all blocks brings 48.3% relative improvement in EER. And convolution module leads to 54.9% relative improvement in EER.