XCB: an effective contextual biasing approach to bias cross-lingual phrases in speech recognition

As a fast and accuracy parallel transformer, Paraformer[11] is a powerful NAR(non-autoregressive) ASR model trained with a large amount (more than 20k hours) of data. To speed up inference, two key technologies are designed. One is a continuous integrate-and-fire-based (CIF) based predictor to accurately predict the number of output tokens, the other is a glancing language model (GLM) based sampler to enhance the NAR decoder with the ability to model token inter-dependence. Benefiting from the technologies, on many public Mandarin benchmarks, Paraformer delivers remarkable performance on par with state-of-the-art AR systems, with more than a 10-fold speedup.

SeACo-Paraformer[12] integrates the ability of hotword customization to the Paraformer backbone. By appending a semantic-augmentation contextual module aside the encoder, SeACo extracts hotword embeddings from the hotword list and sends them into the bias decoder to obtain biased acoustic embeddings and biased semantic embeddings, and finally yielding contextual biased probabilities. Due to its high accuracy and efficiency, in this study, we adopt the SeACo-Paraformer as our backbone for contextualized ASR system.

The proposed XCB module is sandwiched between Paraformer encoder and predictor, as illustrated in Fig.1(a). It comprises two core components: Language Biasing Adapter(LB Adapter) and Biasing Merging Gate(BM Gate), which are elaborated in Fig. 1(b).

Our proposed ASR model with XCB-module is trained and evaluated on an internal code-switching (Mandarin-English) data. This in-house dataset has 13 independently recorded industrial audio data, containing 14k utterances with duration of 20 hours. All these recordings are code-switching utterances, collected with diverse speech topics and recording environment.

The in-house data is randomly split, with 11 recordings for training and 2 for testing. Considering that most early works on code-switching ASR use the ASRU 2019 Mandarin-English code-switching challenge dataset, we also perform evaluation experiment on the ASRU dataset. Note that since it is not publicly available, we can only access its test-set (about 16K utterances). Due to limited data resource, we can not provide comparisons with those prior arts, but only to gauge the generalizability and robustness of our proposed model. That is, to apply the XCB model trained on our in-house dataset to the unseen ASRU test-set without any finetuning.

To prepare the hotword list, for each code-swtiching utterance, $N$ bi-lingual phrases are selected. Besides one target L_2nd phrase, the remaining $N-1$ phrases are selected from the whole test hotword list, composed of all extracted named-entities (either in L_1st or L_2nd). of size around 60 for each testing utterance. The English word and Mandarin key entities of the utterance is selected as target hotwords, and they are mixed with interference items collected from contextual contents of the same audio recording to form the hotword list.

The official published model ¹¹1https://github.com/modelscope/FunASR of SeACo-Paraformer is used as our ASR baseline system, which was pre-trained using up to 50k hours Mandarin speech data. Initialized by the baseline, our proposed XCB-enhanced ASR model is trained using the in-house training dataset for 10 epochs, with learning rate equalling 0.0002 and batch size of 30. The weight $\alpha$ is set as 0.3 and no averaging action is applied to model checkpoints.

The experimental results are presented in Table1. Besides the SeACo baseline, we also directly finetuned SeACo using the in-house training dataset, and add the supervised-finetuned version (termed SeACo:sft) as another baseline for performance comparison.

From the experiment results, we obtain a few valuable insights: 1) as expected, our proposed XCB system significantly outperforms both baselines in terms of BWER. On the in-house test dataset, our system shows an impressive 17.2% relative reduction comparing to the vanilla backbone, and 8% over the fine-tuned backbone; 2) comparing SeACo and XCB, slight performance differences are observed regarding the general MER and BCER, these confirm that our proposal does not compromise the existing recognition capability on the dominant language; 3) our proposed method improves both precision and recall of biased English phrases, with significant absolute gains over 9%; 4) the results of extending our approach to the unseen ASRU test-set without further fine-tuning on ASRU data, also show considerably performance boost, revealing the efficiency and generalization of our proposal. Lastly, on the ASRU test-set, the performance degradation is noted with comparison between the baseline and its fine-tuned version. It may caused by over-finetuning on the relatively small data. In contrast, without any fine-tuning, our proposed XCB systems achieves the best performance in terms of BWER and BMER.

As proved above, our proposed XCB system, by adding auxiliary XCB module on the ASR backbone, outperforms its baseline in both BMER and BWER. Herein, we conduct one more experiment by keeping the auxiliary XCB module inactive during the inference, with expectation to compare our system with the baseline at the condition of same computational complexity. The results are presented in Table2, where XCB:nBM refers to the inference system that inactivates the XCB module and directly feeds the hidden representation $H$ into the predictor. Interestingly, even bypassed the XCB module, the XCB:nBM system shows even better performance than the XCB, on both test datasets. It sounds appealing considering the XCB:nBM incurs no additional computational overhead. We speculate that our XCB training might encourage discrimination of L_2nd in the lower features produced by the encoder. Further in-depth investigation is needed in our future work.