Cross-lingual Transfer for Speech Processing
using Acoustic Language Similarity

Given the set of all languages $S$, cross-lingual transfer generally refers utilizing $n$ source languages $\{s_{1},\dots,s_{n}\}\subset S$ to improve task performance in target language $t\in S$, where $t$ is typically low-resource [9]. Assuming that we are using a data-driven approach, data $D_{i}$ would need to be chosen for each source language $s_{i}$. This definition reveals three challenging tasks: (1) choosing source languages $\{s_{i}\}$ from $S$; (2) choosing data $D_{i}$ for each $s_{i}$; (3) transferring the information in $\{D_{1},\dots,D_{n}\}$ to the task in target language $t$. We focus on the first challenge in this work, and discuss its relation to other tasks below.

Many speech- and text-based works have approached the aforementioned third challenge of improving performance in target language $t$ using source data $\{D_{i}\}$ [3, 10, 11, 12, 4, 7, 13, 5, 14]. To handle the second challenge of choosing the $D_{i}$’s, many works select all the source data from the same corpus [5, 13] or an arbitrary set [15, 12, 4]. Multiple works have approached the first challenge of choosing source languages $\{s_{i}\}$ in a similar manner [5, 13, 4, 15, 12]. These algorithms that utilize vast amounts of source language data are generally pre-training ones, and thus can be used in conjunction with our approach [3, 11, 12, 4, 16].

Many works have shown that certain subsets of source data are more suitable than others for cross-lingual transfer [5, 8]. At the dataset level, Li et al. [6] propose an approach to select suitable corpora by iteratively sampling from each dataset and gradually attending to less data using similarity scores. Since this work assumes a given initial set of candidate languages and datasets, it addresses the second challenge of selecting the data $\{D_{i}\}$. The output of our approach, which selects languages $\{s_{i}\}$, can thus be used as an input to theirs.

Within the text modality, Lin et al. explore using phylogenetic and typological features to select source languages [9]. Since speech data contains a wealth of additional information, we propose acoustic-based features and compare them with their features in speech recognition and speech synthesis. As discussed in Sections 5, 6, and 7, we observe unique advantages of our features in speech processing tasks. We describe all of these features and how we utilize them to measure language similarity below.

In this work, we explore eight different language similarity approaches to select cross-lingual transfer languages for speech processing tasks. Namely, we focus on addressing the first challenge posed in Section 2.1 of choosing source languages $\{s_{i}\}$. All eight approaches are dataset-independent. They can be used in any cross-lingual transfer task as a compact look-up table without requiring additional computation. We proceed by detailing non-acoustic similarity approaches in Section 3 and our proposed acoustic ones in Section 4.

For our non-acoustic language similarity approaches, we use the six distance metrics from Lin et al. [9, 17]: genetic, inventory, syntactic, phonological, featural, and geographic. Sections 3.2 and 3.3 describe the background and details of the first five approaches. Since we observed that their geographic distance method would sometimes map languages to the same location, we utilize an additional geographic measure, as discussed in Section 3.4.

Multiple sources have devised ways to categorize languages into families, using genealogical, typological, and other linguistic information [18, 19, 20]. Several works categorize languages under a hierarchy of language families, which we can visualize as a directed acyclic graph where family nodes have languages and subfamilies as children. Since categorization decisions differ between linguists, these language family trees can vary noticeably between sources, as shown in Figure 1. The large number of different linguistic attributes thus offers us many different ways of defining language similarity, which we discuss below.

As in Lin et al. and the URIEL typological database [9, 17], our five language distance metrics that incorporate phylogenetic and typological information are:

1. 1.

   Genetic Distance: The distance between languages in Glottolog’s world language family tree [19]. For reference, a subset of this tree is depicted in Figure 1 above.

2. 2.

   Inventory Distance: The cosine distance between binary vectors extracted from the PHOIBLE database [22]. Each vector dimension has phonetic information about the presence of sounds [17].

3. 3.

   Syntactic Distance: The cosine distance between syntax feature vectors utilizing information from the World Atlas of Language Structures (WALS) and Syntactic Structures of World Languages (SSWL) [20, 23].

4. 4.

   Phonological Distance: The cosine distance between vectors containing phonological information from Ethnologue and WALS [18, 20].

5. 5.

   Featural Distance: The cosine distance between vectors incorporating features from the above four [17].

For these five language similarity approaches, we use the pre-computed lang2vec distances provided in URIEL [17]. The genetic distance between any two languages is a real value in $[0,1]$, and all the aforementioned vectors are element-wise non-negative with norm greater than $0$. Thus, since cosine distance is given by

the other four distance functions have range $(0,1]$.

The sixth non-acoustic language similarity approach that we use is geographic distance, which we measure in two ways. Our first way is via the orthodromic distance approach described in Lin et al. and URIEL, using pre-computed lang2vec distances [9, 17]. We observed that languages may be mapped to the same location, and thus utilize an additional geographic distance measure. Namely, we obtain language coordinates using Wilderness metadata and compute the geodesic distance with the GeoPy library [24, 25].¹¹1https://github.com/geopy/geopy Unless stated otherwise, we report results in this work using the first approach.

We propose two acoustic-based approaches for measuring language similarity. In contrast to the non-acoustic approaches in Section 3, these methods are entirely data driven and thus do not rely on expert linguistic knowledge. Additionally, they leverage information directly from speech data, offering opportunities to learn language properties suitable for downstream speech tasks. Our acoustic-based approaches are motivated by i-vector and x-vector language recognition works [26, 27]. While these works study tens of languages, we show that ours can successfully support hundreds. As far as we are aware, this work is also the first to compare speech vectors with other language similarity approaches and systematically study their uses in downstream speech tasks.

Generally, both involve training a neural network to classify languages given multilingual speech data. We generate a vector embedding for a language by averaging the outputs of a designated embedding layer across speech samples from that language. Then, we measure language distance as the distance between the language embeddings. The primary difference between our approaches is that one aligns its speech representations with another space, as detailed below.

Formally, our approaches are each defined by a model $M$ and a size-$N$ dataset $D=\{(x_{n},y_{n})\}_{n=1}^{N}$, where the $x$’s are model inputs and the $y$’s are language labels. $x$ is a speech representation in our first approach and a (speech, text) pair in our second. Let $M_{e}(x)$ be the output of the embedding layer, and $M_{c}(M_{e}(x))$ be the output of the language classification layer. In our first approach, we train $M$ on a subset $D_{train}\subset D$ using a classification loss $f_{c}(M_{c}(M_{e}(x)),y)$. Then, we generate an embedding $e_{l}$ for language $l$ using samples $D_{l}\subset\{x_{n}|y_{n}=l\land(x_{n},y_{n})\in D\}$. We compute $e_{l}$ as the mean of the embedding layer outputs, or $\frac{1}{|D_{l}|}\sum_{x_{n}\in D_{l}}M_{e}(x_{n})$. In this work, we measure language similarity using cosine distance, given in Equation 1. We refer to this approach as the speech-based approach.

For our second approach, we add a term to our loss function and assume each $x_{n}$ is a pair $(v_{n},w_{n})$ of speech and text data. Our new loss is given by $f(x,y)=f_{c}(M_{c}(M_{e}(v)),y)+f_{a}(M,x,y)$, where $f_{a}$ is an alignment loss. In this work, we train a text encoder $T$ jointly with $M$ and define $f_{a}$ as

where $d_{cos}$ is the cosine distance function in Equation 1 and $\alpha$ is a hyperparameter. We create language embeddings in a similar manner as the first approach, feeding $v$ into $M_{e}$ instead of $x$. We refer to this approach as the multimodal-based approach and discuss specific architectures used for both approaches in Section 4.3.

We use the Wilderness dataset to train the neural models in our acoustic similarity approaches from Section 4.1 [24]. This dataset is suitable for our acoustic approaches since it contains parallel speech and text data for over 700 languages. In this work, we focus on using data that are considered as being good or very good quality in Wilderness paper. Namely, we choose languages with data that could train a random forest Clustergen speech synthesizer with Mel-cepstral distortion (MCD) under a threshold $c$ [28, 29, 30, 31]. We let $c=5.5$ since this is the middle value of the good quality range used in the Wilderness work [24]. As discussed in that paper, MCD serves as an objective measure of synthesis quality, which in turn has been shown to correlate with alignment and dataset quality [24, 32, 33]. Among this subset of languages, we choose the low-resource ones, which we define as those with limited amounts of public data outside of Wilderness. This yields $195$ languages, with $22.8\pm 5.9$ hours of speech per language. Figure 2 visualizes the location of these languages.

For both acoustic approaches proposed in Section 4.1, we use a convolutional network based on VGGVox as our speech encoder $M_{e}$, with an output dimension of $512$ [34] and two-dimensional adaptive max pooling. Each input to the speech encoder is an $80$-channel normalized Mel-scaled spectrogram with an FFT window length of $800$, hop size of $200$, window size of $800$, and sampling rate of $16000$ Hz. For all experiments, we use normalized embeddings as in Khosla et al. [35], Adam optimization with learning rate $10^{-3}$, and batch size $128$. We explore two classification losses for our speech-based approach: cross-entropy and SupCon [35], referred to as CE and SC in Tables 3, 4, and 5. Classifier $M_{c}$ is a ReLU activation followed by a fully connected (FC) layer with output dimension $195$ in the former, and an FC layer with 128 outputs in the latter. We replace the original SupCon augmentations with SpecAugment [36].

For the multimodal-based approach described in Section 4.1, we use an LSTM-based model as our text encoder $T$. Specifically, we feed each input into an embedding matrix with embedding dimension of $256$, followed by an LSTM with hidden dimension $128$. We then apply max pooling across the sequence dimension and feed the resulting vector into an FC layer with output dimension $128$. We chose max pooling instead of a more complex approach like attention, since the latter did not improve performance in the language family classification task in Section 5. To standardize text inputs across languages, we romanize them using UniTran, as described in the Wilderness paper [24, 37]. For all experiments with the multimodal-based approach, we set the alignment hyperparameter $\alpha$ to $3*10^{-2}$. Each model takes a few GPU days to train and a few GPU hours to extract all the language embeddings on an RTX 2080 Ti.

To check whether our language embeddings are interpretable, we compare their $k$-means clustering assignments with their geographic locations. Here, we train the classifier for the speech-based approach described in Section 4.1 on a random $80\%$-$10\%$-$10\%$ train-val-test split of our $195$-language Wilderness subset using cross-entropy. For each language, we make an embedding using all of the test data in that language. Figure 3 colors the locations of these languages based on their $k$-means clustering assignments. Since assignments to the same cluster generally seem close on the map, it appears that our embeddings contain some degree of geographical information. Additionally, Figure 2 and Figure 3 share many boundaries where language colorings change, suggesting that our embeddings capture some language family information as well.

We also analyze the performance of our language similarity approaches in a cross-lingual speech recognition (ASR) task. Namely, we train monolingual ASR models on a set of source languages and evaluate them on a set of target languages. Thus, experiments using source languages different from target ones are zero-shot. For all ASR experiments, we romanize text using Unidecode²²2https://pypi.org/project/Unidecode and evaluate performance using character error rate (CER). We note that the transliteration helps to make zero-shot ASR possible. To measure the suitability of each language similarity approach for cross-lingual ASR, we calculate the Spearman correlation between the language similarities and the CERs. Details on the specific sets of source-target language pairs used are described below. As mentioned in Section 2.1, we focus here on how to choose the source languages rather than how to improve the subsequent steps in the cross-lingual transfer procedure.

Unless mentioned otherwise, we use an ESPNet Transformer ASR model comprised of a $12$-block Transformer encoder, a $6$-block Transformer decoder, and a CTC module [40, 41]. Additionally, we use a byte pair encoding vocabulary size of $1000$, decoding beam size of $10$, and an optimization procedure similar to that of Vaswani et al. [42]. Further details for reproducibility are provided in the accompanying code.

In addition to data from Wilderness [24], we also conduct ASR experiments using other public datasets. Namely, we use Javanese and Sundanese data from Kjartansson et al. [43], Iban data from Juan et al. [44], and Indonesian, Hindi, Vietnamese, and Hakha Chin data from Common Voice Corpus 6.1 [45]. We chose these datasets since they are all public, their languages are relatively near each other, and most of these languages are low-resource. Train-validation-test splits follow those defined in the respective works [43, 44, 45]. We train our cross-lingual models using the source language training and validation sets and evaluate them on the target language test sets. Further details on how we use these datasets are described below.

Our first ASR experiment compares cross-lingual models with monolingual and multilingual ones, all evaluated on the Indonesian and Iban test sets described in Section 6.1. We train each monolingual model using data from the same dataset as the respective test set. Each cross-lingual model trains on a monolingual dataset in a language different from the test ones. Here, these monolingual datasets are the Javanese dataset by Kjartansson et al. [43], LibriSpeech [46], and GigaSpeech [47]. Our two multilingual models are trained on the $8$-language MLS [48] and the $52$-language version of Hou et al. [4, 40]. The training data for all models are the same as those defined by the original works. The monolingual and Javanese models are described in Section 6.1, and the rest are Conformers and ESPnet Transformers detailed in the ESPnet Model Zoo [49, 40, 41].³³3https://github.com/espnet/espnet_model_zoo Table 2 contains the CERs of all models on the two test sets.

By leveraging more training data, multiple models outperform the monolingual ones, which train on the training sets in the target datasets. Also, despite using less training data than multilingual and other cross-lingual approaches, Javanese performs the best on both test sets. Since Javanese is closer to the target languages than the other sources, these results suggest the usefulness of leveraging data from similar languages to build ASR systems for low-resource languages. The language similarity approaches in Sections 3 and 4 offer ways to automatically identify such similar languages. For example, Javanese is closer to Indonesian than English for both our acoustic measures and the majority of the non-acoustic ones. Thus, we proceed to study the correlation between language similarity and performance in downstream speech tasks.

To compare the language similarity methods in Sections 3 and 4, we first perform zero-shot ASR using Javanese and Sundanese as our source languages and Wilderness languages as our targets. Here, we randomly select ten languages from our $195$-language Wilderness subset and evaluate each language using all of its data.⁴⁴4Wilderness ASR $6$-letter codes are ACCIBS, AGUNVS, BOATBL, CSOTBL, FRDWBT, JACWBT, NODWBT, POHPOC, STNBSP, and VUNBST. Table 3 contains the Spearman correlations between ASR performance (CER) and different similarity measures, where each number uses a single source language and all the targets. Our multimodal-based similarity approach performs the best overall, followed by the genetic, speech-based, and geographic measures. Also, our speech-based approaches perform about the same. This suggests that cross-entropy may be sufficient to train our acoustic measure, potentially since its training dataset is large and consists of non-noisy labels. Another benefit of cross-entropy is its low computational costs during training relative to objectives like contrastive loss. To explore the effects of ensembling, we combined the best linguistic language similarity measure for each source language with our best acoustic one by rescaling their language similarities to $[0,1]$ and then averaging the rescaled values. This combination performs the best overall, suggesting the usefulness of leveraging both non-acoustic and acoustic information. We note that both our speech-based approach and the target languages here use the $195$-language Wilderness subset described in Section 4.2. Thus, in the next section, we study how well our similarity measure generalizes to cases where both the source and target language data are not from Wilderness.

To study the generalizability of our speech-based similarity measure, we perform cross-lingual ASR using source and target language data that are both not from Wilderness. Namely, our source languages are Javanese and Sundanese, and our target languages are these two plus Iban, Indonesian, Hindi, Vietnamese, and Hakha Chin [43, 44, 45]. Section 6.1 provides more information about these data. Table 3 contains the Spearman correlations between ASR performance (CER) and different language similarity measures, where each number uses a single source language and all the targets. The cross-entropy speech-based approach performs the best overall, followed by the geographic, genetic, and SupCon speech-based measures. These approaches also performed well in the Wilderness ASR task discussed in Section 6.3. Moreover, ensembling in the same manner as that task also yields the highest average correlation, suggesting that these trends may hold even in non-Wilderness data settings.

We also explore the performance of our similarity measures with fine-tuned ASR models. For each source-target pair in our zero-shot experiments above, we fine-tuned the ASR model on a random ten-minute subset of the target language dataset that is disjoint from the evaluation data. We again observe that the speech-based, multimodal-based, geographic, and genetic measures perform the best, as well as the ensemble having the highest mean correlation.

We also analyze our similarity measures in cross-lingual text-to-speech (TTS) tasks. Namely, we train monolingual TTS models on a set on source languages and evaluate them on a set of target languages. Thus, experiments using source languages different from target ones are zero-shot. Section 7.2 describes our experiments using a statistical model with Wilderness data, and Section 7.3 describes our experiments using a neural model with non-Wilderness data. Similarly to our ASR experiments in Section 6, we romanize text using Unidecode, which helps make zero-shot TTS possible. Details on evaluation metrics and the specific sets of source-target language pairs used are described below. As mentioned in Section 2.1, we focus here on how to choose the source languages rather than how to improve the subsequent steps in the cross-lingual transfer procedure.

We first check the suitability of our similarity measures for cross-lingual TTS with Wilderness data [24]. Namely, we randomly select $15$ languages from our $195$-language Wilderness subset described in Section 4.2 and use all $15$ as both sources and targets, yielding $225$ source-target language pairs.⁵⁵5Wilderness TTS $6$-letter codes are APRWBT, JACWBT, KEKSBG, KJBSBG, MAKLAI, MOPWBT, NASPNG, NPLWYI, POHPOC, QEJLLB, QULSBB, QWHLLB, TZBSBM, TZCSBM, and VUNBST. We found that the data in each language was insufficient to build a neural TTS model [1], and thus instead used the random forest Clustergen speech synthesizer as our TTS model [28, 29]. Our training steps are the same as those in the Wilderness paper, and we evaluate models on the first chapter texts in each target language [24]. As discussed in Section 4.2, we measure TTS performance objectively using Mel-cepstral distortion (MCD).

Table 5 shows the Spearman correlations between TTS performance (MCD) and the language similarity measures. For each language similarity approach, we calculate the mean of $15$ Spearman correlations, each using one source and all the targets as in Section 6. Our acoustic approaches described in Section 4.1 perform the best, followed by the genetic distance measure. We note that the trends observed here are similar to those from our ASR tasks in Section 6.