Task Arithmetic can Mitigate Synthetic-to-Real Gap in
Automatic Speech Recognition

We assume a problem setting in which we have two domains: a source domain $D_{s}$ and a target domain $D_{t}$. The source domain $D_{s}$ consists of paired text and speech samples, denoted as $T_{s}$ and $S_{s}$, respectively. The target domain $D_{t}$ contains only text data, denoted as $T_{t}$. This problem setting is common as it is easy to generate synthetic text data, whereas collecting paired real speech data is labor-intensive.

To adapt ASR models to a previously unseen domain, we employ a common methodology Joshi and Singh (2022) that utilizes synthetic data generated from the target text $T_{t}$ for model adaptation.

Previous work in task arithmetic has demonstrated that vectors can encode distinct capabilities, such as language or domain-specific features. We hypothesize that the differences in acoustic properties between real and synthetic speech are also learnable and can be isolated through parameter arithmetic. Specifically, we assume that we have models fine-tuned on real and synthetic data from the source domain, denoted as $\theta_{real}^{S}$ and $\theta_{syn}^{S}$ respectively. The acoustic disparity between real and synthetic speech is quantified by subtracting the parameter sets of these models:

Once the SYN2REAL vector $\tau$ is computed, we apply it to the model parameters fine-tuned on the synthetic data in the target domain, $\theta_{syn}^{T}$, thereby enhancing its adaptation to the target domain:

Where $\lambda$ is the scaling factor of SYN2REAL task vector.

This adjusted model, $\theta_{syn\_new}$, is expected to perform more robustly in the target domain as it incorporates the acoustic characteristics of real speech, making it better suited for practical ASR tasks where real speech is present.

SLURP Bastianelli et al. (2020) is a spoken language understanding dataset containing 16521 utterances of human commands towards a virtual agent, based on 200 pre-defined prompts such as “How would you ask for the time.” The utterances are categorized into 18 domains (e.g., email, cooking, etc.). In each of our experiments, we select one of these domains as the target domain and combine the remaining 17 domains to form the source domain.

In our experiments, we prepare synthetic speech using two off-the-shelf TTS models for text from the target domains.

To answer Q1, we apply our method by comparing the word error rate (WER) across various target domains. We select one of these domains as the target domain and combine the remaining 17 domains to form the source domain Table 1 presents the WER results for both the baseline ASR model fine-tuned on synthetic speech data and the model enhanced with the SYN2REAL task vector.

The baseline model, fine-tuned solely on synthetic data, exhibits varying WERs across different target domains, with an average WER of 21.16. This performance highlights the challenge of adapting ASR models to real-world data when trained on synthetic speech, primarily due to acoustic mismatches.

The application of the SYN2REAL task vector significantly reduces WER across most target domains. The SYN2REAL-enhanced model achieves an average WER of 19.04, representing an average relative WER reduction of 10.03%. This improvement demonstrates the SYN2REAL task vector’s effectiveness in bridging the gap between synthetic and real speech data, enhancing the model’s adaptability to diverse real-world scenarios.

To further validate our method’s effectiveness in solving the synthetic vs. real speech mismatch, we also report two additional baselines. We conducted experiments evaluating the pretrained Whisper small model and the Whisper small model fine-tuned on the source domain (real + synthetic) on the target domain, with average WERs of 33.30 and 27.13, respectively. We address the acoustic variation mismatch by fine-tuning the pretrained Whisper on source domain speech. Further fine-tuning on target domain synthetic data addresses the textual topic mismatch, achieving a WER of 21.16. Finally, applying the SYN2REAL task vector mitigates the synthetic vs. real speech mismatch, resulting in a WER of 19.13.

The SYN2REAL task vector shows particularly notable improvements in domains such as ’Music’ (27.57% reduction), ’Takeaway’ (15.14% reduction), and ’Social’ (26.04% reduction). These results suggest that the task vector effectively captures domain-specific acoustic variations, enabling the ASR model to generalize better to unseen target domains. In the following experiments we select the four domains includes two highest improved domains (’Music’ & ’Social’), and the two lowest improved domains (’Weather’ & ’Cooking’) to conduct the experiments.

However, it is important to note that some domains, such as ’Cooking’ and ’Weather,’ exhibit marginal improvements or slight degradation in WER. These variations indicate that while the SYN2REAL task vector generally enhances performance, further fine-tuning and domain-specific adjustments may be necessary to optimize results across all target domains.

Overall, the results demonstrate that the SYN2REAL task vector is a promising approach for improving ASR domain adaptation, showing significant improvements over both the baseline and a recent work, AdaBERT-CTC Vuong et al. (2023), which only achieved a WER of 26.1 on the SLURP test set. By addressing the acoustic mismatches between synthetic and real speech data, our method significantly enhances the performance of ASR models in real-world applications.

To answer Q2, we analyze the effect of model size on the performance of ASR adaptation using the SYN2REAL task vector. Table 2 presents the relative word error rate improvements across different model sizes (Tiny, Base, Small) and various target domains.

The results indicate that the Base model achieves the highest average relative WER improvement of 14.70% across all target domains. This model size shows substantial gains, particularly in the ’Music’ (37.80%) and ’Social’ (5.00%) domains, demonstrating its robustness in adapting to diverse acoustic characteristics using the SYN2REAL task vector.

The Tiny model, while achieving a higher average improvement of 19.48%, shows considerable performance gains in the ’Cooking’ (41.11%) and ’Weather’ (30.42%) domains. However, it experiences a performance degradation in the ’Music’ domain (-13.47%). This suggests that while the Tiny model can benefit significantly from the SYN2REAL task vector in certain domains, its overall adaptability might be limited compared to larger models due to its reduced model size.

Interestingly, the Small model exhibits an average relative WER improvement of 12.43%, with substantial gains in the ’Social’ (26.04%) and ’Music’ (27.56%) domains. However, it shows a notable degradation in the ’Weather’ domain (-31.91%), indicating potential overfitting or sensitivity to specific acoustic variations.

These results highlight the importance of model size in ASR adaptation using the SYN2REAL task vector. The Base model consistently provides balanced performance across most domains, suggesting it strikes a good balance between model size and performance. In contrast, the Tiny and Small models show varying degrees of effectiveness.

Overall, the analysis demonstrates that while the SYN2REAL task vector significantly improves ASR performance across different model sizes, the extent of improvement is influenced by the model’s capacity.

To validate the effectiveness of the SYN2REAL task vector on other ASR models, we conduct additional experiments using the Wav2vec2-Conformer large model.

Table 3 presents the WER results across various target domains, including ’Cooking’, ’Music’, ’Social’, and ’Weather’, comparing the baseline model finetuned on synthetic speech with the model enhanced by the SYN2REAL task vector. The Table 3 shows a significant reduction in WER when the SYN2REAL task vector is applied. The average WER drops from 20.31 to 17.01, representing an overall relative improvement of 16.25%.

The most notable improvement is observed in the ’social’ domain, with a relative WER reduction of 16.87%. The ’Music’ domain also shows a substantial improvement of 17.66%, indicating that the task vector successfully captures and mitigates the acoustic variability associated with music-related speech.

In the ’Cooking’ and ’Weather’ domains, the WER reductions are 11.21% and 20.22%, respectively. While the improvement in the ’Cooking’ domain is more modest, it still indicates that the SYN2REAL task vector enhances the model’s adaptability to domain-specific acoustic characteristics.

Overall, the application of the SYN2REAL task vector significantly enhances the performance of the Wav2vec2-Conformer large model across all tested domains. These results validate the effectiveness of the SYN2REAL approach in bridging the gap between synthetic and real speech data, ultimately improving the robustness and versatility of ASR systems in diverse real-world scenarios.

To answer Q4, we conducted experiments using the Whisper Small model with synthetic data generated by the Speech T5 model and XTTS model. Table 4 presents the WER results across various target domains, including ’Cooking’, ’Music’, ’Social’, and ’Weather’, comparing the baseline model finetuned on synthetic speech with the model enhanced by the SYN2REAL task vector.

The results indicate that applying the SYN2REAL task vector leads to a reduction in WER across most of tested domains. The average WER drops from 25.65 to 25.17 for Speech T5 and from 17.40 to 16.86 for XTTS, representing an overall relative improvement of 1.86% and 3.10% respectively.

The ’Cooking’ domain shows the highest relative WER reduction of 5.57% and 8.11%, suggesting that the SYN2REAL task vector effectively adapts the model to this specific domain.

However, the improvement in the ’social’ domain is relatively modest for Speech T5, with a relative WER reduction of only 0.73 %. This could be attributed to the high baseline WER in this domain, suggesting that the synthetic data from Speech T5 might have limitations.

Overall, the application of the SYN2REAL task vector to the Whisper Small model with synthetic data from different TTS models demonstrates consistent improvements in average WER, albeit with varying degrees of improvement across different domains. These results validate the flexibility and effectiveness of our approach in improving ASR models trained with synthetic data from different TTS models.

This section investigates the effect of scaling the SYN2REAL task vector on the WER of different ASR models. Figure 4 illustrates the WER as a function of the scaling factor $\lambda$ for various ASR models and synthetic data, including Whisper Tiny with BARK, Whisper Base with BARK, Whisper Small with BARK, Whisper Small with Speech T5, and W2V2-Conformer with BARK.

The scaling factor $\lambda$ adjusts the magnitude of the SYN2REAL task vector applied to the ASR models. We evaluated a range of scaling factors from 0.1 to 1.0 to determine the optimal balance that minimizes WER.

The results show that different models respond variably to changes in the scaling factor. For Whisper Tiny+BARK, the curve is steeper, indicating that smaller models may be more sensitive to larger adjustments from the SYN2REAL task vector. In contrast, Whisper Base+BARK maintains relatively stable WER values across different scaling factors, suggesting a more robust performance.

Notably, Whisper Small+BARK and Whisper Small + Speech T5 exhibit a U-shaped trend, where moderate scaling factors (around $\lambda$ = 0.3 to 0.5) yield the lowest WER. This indicates that an optimal scaling factor exists for these models, which balances the incorporation of real speech characteristics without overwhelming the model with excessive parameter adjustments. The Wav2vec2-Conformer model consistently shows lower WER values across all scaling factors, with the best performance at $\lambda$ = 0.5.

Overall, the analysis suggests that the optimal scaling factor $\lambda$ varies depending on the ASR model’s architecture and size. While smaller models like Whisper Tiny+BARK may benefit from lower scaling factors, larger and more robust models like W2V2-Conformer+BARK can effectively leverage higher scaling factors. These findings highlight the importance of tuning the scaling factor to achieve the best domain adaptation performance for different ASR models.

To further validate the SYN2REAL approach, we conducted a cosine similarity analysis between SYN2REAL task vectors derived by different text-to-speech (TTS) models: BARK (denoted as B_) and Speech T5 (denoted as S_). Figure 5 presents the cosine similarity heatmap between these SYN2REAL task vectors.

The heat map reveals that SYN2REAL task vectors from similar TTS exhibit higher cosine similarity, indicating that the SYN2REAL task vector effectively captures the distributional shifts between different acoustic domains.

Moreover, the negative similarities between certain SYN2REAL task vectors, such as ’B_recommendation’ and ’S_music’ (-0.67), highlight the distinct acoustic features between these TTS synthetic data, further emphasizing the effectiveness of the SYN2REAL approach in distinguishing and adapting to different acoustic environments.

The overall trend observed in the heatmap supports the hypothesis that the SYN2REAL task vectors not only bridge the gap between synthetic and real data but also maintain consistency within similar acoustic environments. This consistency is crucial for enhancing ASR performance across diverse target domains, as it ensures that the task vectors can generalize well to new, unseen domains.

To evaluate the effectiveness of the SYN2REAL Ensemble task vector, we conducted experiments using the Whisper Small model with synthetic speech generated by the BARK TTS model. The experiments were carried out on four target domains: ’Cooking’, ’Music’, ’Social’, and ’Weather,’ using 17 source domains to create the SYN2REAL Ensemble task vectors. The results are presented in Table 5. The results indicate that the SYN2REAL Ensemble task vector provides significant improvements in WER for several target domains compared to the baseline method. The average WER is reduced from 20.70 to 16.92, representing an overall relative improvement of 18.25%. This highlights the effectiveness of using domain-specific task vectors to capture detailed acoustic characteristics, leading to enhanced model adaptation.

Comparing the SYN2REAL Ensemble task vector to the original SYN2REAL task vector, we find that SYN2REAL Ensemble task vector generally outperforms the original approach. The detailed domain-specific information captured by the distinct task vectors enhances model adaptation in most domains. However, in real-world scenarios, we often do not have access to labels finer-grained domain labels.

Figure 6 shows the average WER across four target domains (’Cooking’, ’Music’, ’Social’, and ’Weather’) when we use different numbers of source domain data to generate SYN2REAL Ensemble task vectors.

The results indicate that increasing the number of source domains generally improves ASR performance. The WER decreases from 17.8 to 16.8 as the number of source domains increases from 1 to 17. Notably, significant improvements are observed within the incorporation of the first 5 source domains. This trend suggests that incorporating more source domains helps capture diverse acoustic characteristics, leading to better domain adaptation.