Disentangling the Impacts of Language and Channel Variability on Speech Separation Networks

Mandarin Continuous Speech Prosody Corpora (COSPRO) [30] was designed, collected, and annotated at the Institute of Linguistics, Academia Sinica. As the name suggests, the main feature of this dataset is continuous or fluent speech prosody and its influence on speech production and perception. COSPRO includes a total of approximately 130 hours of read speech and only 0.3 hours of spontaneous narration in Mandarin Chinese. There are nine sets of speech corpora, summarized in Table 1. Each corpus was designed to bring out different prosodic features involved in fluent speech.

Before mixing the utterances in COSPRO, we first removed files that were inappropriate, such as utterances containing only a few characters or carrier sentences that differ a lot from actual conversations. Because most utterances are much longer than those in WSJ0, we used a pre-trained speech recognizer to segment the utterances. To this end, forced alignment was performed on all utterances with word-level transcriptions using the Kaldi toolkit [32]. A time-marked conversation file (CTM) containing time-aligned word transcriptions was obtained for each utterance. Then, we trimmed each utterance into multiple consecutive segments based on its CTM, subject to the limit of the maximum segment length (approximately 10 seconds). Every cut point must be at a word boundary.

After segmentation, we referred to how WSJ0-2mix was mixed to make a dataset comparable to WSJ0-2mix. Our procedure for making COSPRO-2mix is as follows. First, we selected 81 speakers (34 males and 47 females) from the training set, each with more than 400 seconds of speech. In COSPRO, there is a dataset consisting of more speakers, but some of them with fewer utterances. To avoid overfitting in training, we only selected speakers with longer speaking times. For the evaluation set, seven males and seven females were selected. Then, we followed the mixing procedure of WSJ0-2mix. For each mixed utterance in WSJ0-2mix, we constructed a corresponding mixed utterance based on the same gender combination and volume ratio. In this way, COSPRO-2mix and WSJ0-2mix have the same number of mixed utterances and similar statistics. COSPRO-2mix was used in language-related experiments.

The English Across Taiwan (EAT) corpus³³3http://www.aclclp.org.tw/doc/eat_brief_e.pdf contains microphone and telephone recordings of college students in Taiwan. We selected English utterances and English/Mandarin code-mixing utterances from the microphone recordings to avoid the channel mismatch as much as possible and created an English test set ($\textnormal{EAT-2mix}_{\textnormal{Eng}}$) and an English/Mandarin code-mixing test set ($\textnormal{EAT-2mix}_{\textnormal{E\&M}}$) in the same way as COSPRO-2mix. Both test sets contain the same speakers and similar utterance lengths. With this dataset, we can compare performance within the same channel, which means that the performance gap is mainly due to differences in content. The two test sets were used in the first part of our experiment.

The Taiwanese Across Taiwan (TAT) corpus [31] was originally intended for Taiwanese speech recognition. Taiwanese, also known as Southern Min, is a common dialect in Taiwan and belongs to the same language family as Mandarin. The 200 speakers contributing to the corpus include men, women, and children of all ages across Taiwan, with content ranging from written articles to everyday conversations. Most notably, the corpus was recorded simultaneously on six channels, including different microphones and smartphones. This feature helps us investigate the impacts of different channels. The way of synthesizing the datasets of mixed utterances is described below.

The six channels in TAT include: one close-talk (Audio-Technica AT2020), one distant X-Y stereo microphone (ZOOM XYH-6 stereo microphone, containing left and right channels), one lavalier (Superlux WO518+PS418D), iOS devices (including iPhones, iPads, and iPods), and Android phones (produced by ASUS and Samsung). According to these six channels, we created six datasets respectively: $\textnormal{TAT-2mix}_{\textnormal{condenser}}$, $\textnormal{TAT-2mix}_{\textnormal{XYH-6-X}}$, $\textnormal{TAT-2mix}_{\textnormal{XYH-6-Y}}$, $\textnormal{TAT-2mix}_{\textnormal{lavalier}}$, $\textnormal{TAT-2mix}_{\textnormal{iOS}}$, and $\textnormal{TAT-2mix}_{\textnormal{Android}}$.

Before mixing the utterances in TAT, we segmented the utterances using the same method as in COSPRO-2mix to trim the noise in the front and tail of the audio. Since the original length was already appropriate for speech separation tasks, we did not cut the recording into shorter utterances.

After segmentation, the mixing procedure is similar to that of WSJ0-2mix. We randomly selected 101 speakers (49 males and 52 females) for training and validation. On the other hand, 18 speakers (11 males and 7 females) were selected for testing. The number of speakers and the gender distribution are exactly the same as WSJ0-2mix. Then, we followed the mixing procedure of WSJ0-2mix to assure that TAT-2mix and WSJ0-2mix have the same number of mixed utterances and similar statistics. All datasets are comprised of the same mixtures of utterances but from different channels.

We trained the representative models on WSJ0-2mix and COSPRO-2mix and evaluated them on several datasets. The results are shown in Table 2, where the columns refer to the training sets and the separation models; the rows refer to the test sets. The metric used in this study is SI-SNRi (dB), the improvement of SI-SNR (cf. Eq. 1), defined by

where $\mathbf{m}$ and $\mathbf{x}$ denote the input mixture and output of the network, respectively.

First, comparing each column in Table 2 under the two training sets, the performance rankings are in line with past results: DPT-Net outperforms DPRNN, and DPRNN outperforms Conv-TasNet. Then, we examine the performance in each row. Looking at the first three rows (the group of WSJ0 & COSPRO), we can see that the results of WSJ0-2mix and COSPRO-2mix are comparable, under both in-domain (training/testing from the same dataset) and cross-domain (training/testing from the different dataset) testing conditions. In WSJ0$\times$COSPRO, each utterance is a mixture of a Mandarin utterance from COSPRO and an English utterance from WSJ0. Due to different languages and different recording conditions, WSJ0$\times$COSPRO is easier to separate than WSJ0-2mix and COSPRO-2mix.

The results in the fourth and fifth rows of Table 2 (the group of EAT) show the impact of language more explicitly since the only difference between the two test sets is the language. We observe that the performance on EAT is much worse than those on COSPRO-2mix and WSJ0-2mix, because the clean waveform contains some microphone noise, which may degrade SI-SNR, and the recording quality is not high, which causes many pop and explosive sounds. Compared to the performance in other rows, it is obvious that the performance gap between $\textnormal{EAT-2mix}_{\textnormal{E\&M}}$ and $\textnormal{EAT-2mix}_{\textnormal{Eng}}$ is almost negligible. That is, although language has an impact on speech separation, the impact is so small to neglect, complying with the results in [35].

The results in the remaining rows of Table 2 (the group of TAT) reveal the impact of channels since the only difference between these test sets is the channel. There is a large difference between the performance of each row, suggesting that channel variability seems to dominate the performance. We also found that when the test set is more difficult, such as $\textnormal{TAT-2mix}_{\textnormal{iOS}}$, the performance difference between models is less pronounced. To investigate the impact of channel variability and the relationship between channels, we performed the next set of experiments.

In this set of experiments, we trained a DPT-Net on each TAT-2mix recorded by a specific channel and evaluated on all TAT-2mix separately. The results are shown in Table 3. The columns stand for the training sets, and the rows present the test sets.

Overall, we can again see the severity of the impact of channel variability. Although all datasets are composed of the same speech content and speakers, the performance varies. In addition, the models perform best when the test data is in-domain (training and testing are from the same channel), in line with previous studies that emphasized the importance of in-domain training data [26, 27, 36].

If we examine the SI-SNRi values in Table 3, condenser and lavalier are the easiest channels, and XYH-6-X and XYH-6-Y are the hardest. The result is reasonable because condenser is closest to the speaker (5–10cm) with a pop filter, and lavalier is also very close to the speaker’s mouth. Android and iOS are 15–20cm from the speaker. The distance between XYH-6 and the speaker is about one meter. As reported in [29], speech separation is more difficult for distant microphones.

By comparing the columns of XYH-6-X and XYH-6-Y, it is clear that the performance between the left and right channels is almost equivalent, but XYH-6-X has a slight advantage. When comparing the rows of XYH-6-X and XYH-6-Y, XYH-6-Y is slightly harder than XYH-6-X. As expected, the left and right channels are somewhat similar. As for the two smartphone systems, whether comparing the rows of Android and iOS to see how other models perform on the two test sets, or comparing the columns of Android and iOS to examine the generalizability of the two models, the results show that the characteristics of the iOS microphones and the Android microphones differ considerably. However, for these two smartphone systems, their counterpart model still performs the best among all cross-domain models, e.g., the iOS model gives an SI-SNRi of 7.60 when tested on the Android test set, which is better than the SI-SNRi values of other cross-domain models.

Furthermore, the model trained on $\textnormal{TAT-2mix}_{\textnormal{Android}}$ achieves the highest average performance on all test sets, indicating its robustness. One possible reason is that Android contains smartphones from different companies, with different frequency responses for their built-in microphones. It is also worth mentioning that the model trained on $\textnormal{TAT-2mix}_{\textnormal{condenser}}$ does not generalize well to other channels. However, condenser microphones are the most popular devices for recording corpora, which poses a problem in real-world scenarios. In the next subsection, we proposed a solution that effectively selects additional training data to improve performance on the in-the-wild test data.

We performed pairwise principal component analysis (PCA) to visualize the relationship between channels. First, we encoded the results of individual test sets (i.e., the rows in Table 3) into row vectors $\{\mathbf{v}_{i}\}$. Then, we constructed the covariance matrix $\Lambda$ by

where $N$ denotes the number of vectors. We performed eigen-decomposition on $\Lambda$ to find the two eigenvectors with the largest eigenvalues as the two directions of the new feature space. Then, $\{\mathbf{v}_{i}\}$ was projected onto the new 2D feature space.

Fig. 1 shows the scatterness of the projected points. From the figure, we can see that XYH-6-X and XYH-6-Y are close. In order to evaluate the usefulness of distance, we conducted two experiments. We found that if the training set contains data close to the test set, the performance improves significantly. For example, as shown by the solid arrows, when the training set contains $\textnormal{TAT-2mix}_{\textnormal{condenser}}$ and $\textnormal{TAT-2mix}_{\textnormal{XYH-6-X}}$, the performance on $\textnormal{TAT-2mix}_{\textnormal{XYH-6-Y}}$ can be improved by at least 11% compared to the performance of the model trained on $\textnormal{TAT-2mix}_{\textnormal{XYH-6-X}}$ (8.17 vs 7.34). In contrast, if both training sets are far away from the test data, even doubling the size of the training data may not improve the performance much. For example, the dotted arrows show that the performance on $\textnormal{TAT-2mix}_{\textnormal{iOS}}$ is only improved by at most 7% comparing the model trained on $\textnormal{TAT-2mix}_{\textnormal{XYH-6-X}}$ & $\textnormal{TAT-2mix}_{\textnormal{lavalier}}$ and the model trained on $\textnormal{TAT-2mix}_{\textnormal{lavalier}}$ (4.23 vs 3.97).

The above results suggest that additional training data should be chosen wisely to effectively enhance the robustness of the model. Therefore, we propose the following solution. When encountering new data recorded by an unknown channel, we can first use these six models to perform speech separation on the data. Then, the test results are projected onto the 2D space, so that we can observe the distances between the new dataset and other points. Finally, we select existing datasets close to the new point and add them to the training data to improve the performance of the model on the new data.