Joint Speech Transcription and Translation:
Pseudo-Labeling with Out-of-Distribution Data

Our task of speech transcription and translation (STT) is closely related to script recognition (ASR) and speech translation (ST). ASR is the task of generating the text equivalent to an audio speech signal. Meanwhile, ST aims to generate the text equivalent to the signal in a target language other than the language of the speaker. In contrast, STT generates both the transcript and the translation jointly in an end-to-end fashion. STT is particularly appealing in cases where both the transcript and translation are to be displayed to the user.

Formally, STT can be modeled as follows: given a speech signal ($s$), the model generates the transcript ($tc$) and translation ($tl$) concatenated together in the output as one single sequence: $s\rightarrow tc\_tl$ Sperber et al. (2020). This formulation is simple to implement as it casts STT as an instance of the well-known seq2seq modeling and results in a single end-to-end model to be stored on device. Furthermore, as reported by Sperber et al. (2020), this formulation results in a reasonably consistent transcripts and translations as the coupled inference ensures that translations are conditioned on transcripts. In our experiments, we adopt this STT formulation.

However, the major challenge that such modeling presents is insufficient data resources: three-way parallel samples of form $(s,tc,tl)$ are expensive to annotate. Annotation would require multilingual annotators and would be time-consuming. To alleviate this limitation, we study how pseudo-labeling can be employed effectively to combat data scarcity in this setting. We provide a background on pseudo-labeling in the next section.

Pseudo-labeling, which is also often referred to as self-training in the literature, addresses the data insufficiency issue by taking advantage of much larger amounts of unsupervised data. More precisely, assume a labeled set $L=\{x_{i},y_{i}\}$ and an unlabeled set $U=\{x_{j}\}$, where $|U|\geq|L|$, are available (note that in the case of STT, $y_{i}$ is actually a tuple consisting of the transcript and the translation: $y_{i}=(tc_{i},tl_{i})$). Pseudo-labeling starts with training an initial model $M$ in a supervised manner using $L$. Then, using $M$, it generates pseudo-labels (predictions) for $U$. It then incorporates the pseudo-labels to create a new model $M^{+}$, which hopefully supersedes $M$ in performance. $M^{+}$ can then replace $M$ to repeat this process for as many rounds as desired, or until no further gains are observed. Although conceptually simple, several key decisions need to be made before pseudo-labeling can be applied:

• •

  How should $M^{+}$ be created? $M^{+}$ can be trained from scratch (e.g., as done by Park et al. (2020)) or alternatively obtained by continuously fine-tuning $M$ (e.g., as done by Xu et al. (2020)) using the labeled set combined with the pseudo-labeled set. As we later report in §4, in our preliminary experiments, fine-tuning consistently outperforms training from scratch. Hence, we opt for fine-tuning in our experiments.

• •

  Should pseudo-labeling be applied to supervised sets? For the pseudo-labeling stage, we consider and experiment with labeling the supervised set in addition to the unsupervised set and monitor for any potential improvements. Similar to the previous item, as we later show in §4, using the pseudo-labels for the supervised set does not prove to be beneficial in our preliminary experiments. Therefore, we generate predictions only for the unlabeled set.

• •

  In what way should the pseudo-labels be used to update existing models? For instance, He et al. (2020), at each round, first train a model from scratch on the pseudo-labeled set, and then fine-tune it on the labeled set to obtain the final model for that round. Alternatively, Xu et al. (2020) combine the two sets and use a hyper-parameter to have control over the relative weight of the labeled portion against the pseudo-labeled portion. To keep our setup simple, we opt for combining the sets and treating them equally.

With the key factors outlined above, Algorithm 1 shows how we carry out vanilla pseudo-labeling for our experiments. All results we report in §4.1 follow this algorithm.

In this work, we use two publicly available multilingual speech translation datasets which, thanks to the nature of their creation, also include transcripts: CoVoST V2 Wang et al. (2020) and MuST-C Cattoni et al. (2021). CoVoST V2 is created by amending the validated audio clips and transcripts from the Common Voice crowd-sourced ASR corpus Ardila et al. (2020) with professional translations. It covers translations from English into 15 languages and from 21 languages into English. MuST-C is created by automatically aligning the audio segments from TED talks to corresponding manual transcripts and translations (available from the TED website), which are also aligned. It covers translations from English into 14 languages.

We conduct our experiments across two language pairs : English–German (En–De) and English–Chinese (En–Zh), which are available in both CoVoST and MuST-C. In all our experiments, we designate CoVoST as the supervised set, and MuST-C as the unsupervised set. Note that this means our objective is to reach the best performance possible on the CoVoST evaluation set. While we also have the gold transcripts and translations (labels in the STT problem) for MuST-C, we do not use them and practically treat MuST-C as an unlabeled set. We only use MuST-C gold labels for analysis and pseudo-label quality assessment. We provide the statistics of our data in Table 1.

Our model is an end-to-end Transformer Vaswani et al. (2017) with a hidden dimension of 1024, and five and three layers of encoder and decoder respectively (following Sperber et al. (2020)). For the input, on the encoder side, we first use wav2vec 2.0 Base Baevski et al. (2020) as provided by Hugging Face Transformers Wolf et al. (2020) (specifically, facebook/wav2vec2-base-960h) to extract speech representations before feeding them into the Transformer encoder. On the output side, as described in §2.1, the decoder generates one sequence consisting of the transcript and the translation concatenated together.

In terms of input prepossessing, we remove instances where speech is either shorter than 0.5 seconds or longer than 15 seconds, or either the transcript or the translation is longer than 50 words. After that, we use SentencePiece Kudo and Richardson (2018) for subword tokenization. We use a vocabulary size of 1020 and 8188 in the case of En–De and En–Zh, respectively. The transcription and translation vocabulary is shared in both cases.

The objective function during optimization is a weighted sum of the CTC loss Graves et al. (2006) on the encoder side, and the cross-entropy loss on the decoder side. Both during training a base model and fine-tuning an existing checkpoint on the union of the labeled set and the pseudo-labeled set, we use Adam optimizer Kingma and Ba (2015) with peak learning rate of 0.0005 after 500 warmup steps, coupled with inverse square root learning rate scheduling. We train for a total of 100 epochs.

For both language pairs, we use the dev sets provided by the corpora as the held-out evaluation set. We remove diacritics and punctuation before scoring and report our performance in terms of WER of transcripts and BLEU of translations using beam size of five with SacreBLEU.²²2Hash: case.lc+numrefs.1+smooth.4.0+tok.{13a,zh} for {En–De,En–Zh}.

Our implementation is built upon PyTorch Paszke et al. (2019), xnmt Neubig et al. (2018), and Lightning Falcon and The PyTorch Lightning team (2019).

In Table 2, we include the results of vanilla pseudo-labeling, as in Algorithm 1, with no modifications. We report WER and BLEU for En–De and En–Zh across both corpora. To reiterate, CoVoST (distinguished by the magnifying glass symbol \faSearch) is our designated supervised set, and hence, what we are trying to boost performance on. MuST-C scores, on the other hand, are reported for the sake of analysis; the metrics are to assess the quality of the pseudo-labels.

We report the performance of the initial model (the fully supervised baseline, Model $M$ on line 1 of the Algorithm 1) in the “Base Model” column. Scores from each pseudo-labeling round, thereafter, appear on the corresponding “R” column. To have an upper bound of what is possible with the collective data that we have, we run an experiment training a single model using both corpora in a supervised manner. Those numbers are provided in the “Bound” column. Note that this is the only case for which MuST-C gold labels are used, solely to obtain an upper bound performance: what would be possible if we predicted labels perfectly for the unsupervised set? Is pseudo-labeling helping us close the gap between the base model and that upper bound for \faSearchCoVoST?

First and foremost, in confirmation with the literature, vanilla pseudo-labeling is effective. On \faSearchCoVoST, it is able to improve the base model by 0.4% absolute WER and 1.7 BLEU points on En–De, and 0.2% absolute WER and 2.0 BLEU points on En–Zh. However, with a closer look at the quality of pseudo-labels at each round (i.e., MuST-C scores), it is evident that the generated labels are far from ideal quality. Specifically, even if we train plain machine translation systems on \faSearchCoVoST transcripts and translations (and take the audio out of the picture), the En–De system scores 12.4 BLEU on MuST-C En–De, and the En–Zh system scores 9.6 BLEU on MuST-C En–Zh.

Our investigation into the reasons as to why that is the case points to two root causes that indicate \faSearchCoVoST and MuST-C are significantly different in domain in the following aspects:

• •

  Length mismatch between corpora: As shown in Figure 2, MuST-C speech sequences are generally longer, which also results in longer transcripts and translations.

• •

  Vocabulary mismatch between corpora: Additionally, we were able to identify discrepancies between the vocabulary of words between the two corpora. For instance, on the English side, MuST-C and CoVoST each have roughly 64k and 121k unique types, respectively. Of those, only 38k types are in common, with CoVoST having more probability mass on rare (tail-end of the Zipfian distribution) vocabulary types.

Following this observation, we next demonstrate that it is possible to counteract the domain mismatch and enhance the quality of labels to boost the effectiveness of pseudo-labeling.

Per §2.2, in vanilla pseudo-labeling, we use all the generated labels to update the model. Alternatively, pseudo-labels can be filtered to remove predictions of less quality. Recent works Park et al. (2020) rely on confidence scores from the model to filter the pseudo-labels, which require careful and proper normalization. Kahn et al. (2020) use a combination of heuristic-based and confidence-based filtering. In our case, similar to Likhomanenko et al. (2021), we propose and only rely on data-centric metrics to specifically target domain-mismatch and select a subset of pseudo-labels to use in the next round: transcript length to speech length ratio and transcript and translation LASER embeddings cosine similarity.

Our previous filtering methods remove pseudo-labels so that the remaining subset has a higher quality. However, if we can generate better labels, to begin with, we can discard none and retain all the labels. Here, to improve the quality of the labels generated by the base model at no extra supervision cost, we use data augmentation by concatenation to directly target the reported length mismatch between corpora in §4.1. To do so, we create an augmented set from our supervised set by randomly selecting a pair of samples and constructing a new sample by concatenating the speech signals as the input and concatenating corresponding transcripts and translations as output. In our experiments, we build a set of 20k augmented samples as such using the original \faSearchCoVoST data. After training the base model, before generating pseudo-labels, we first further fine-tune the base model on the union of the original supervised set and the augmented set. We then proceed as in vanilla pseudo-labeling with the union of the original data and the augmented set as our supervised training set.

As shown in Table 3, row “Augmentation”, we can see that although no generated labels are thrown away, the quality of pseudo-labels is indeed increased in the subsequent round. This is especially pronounced in the case of translations.

With retaining all pseudo-labels, not only does bootstrapping the supervised set using concatenation expedite the gains from pseudo-labeling, but it is also the most effective in terms of the final performance before saturation by improving the score in three cases: it improves the performance of vanilla pseudo-labeling on \faSearchCoVoST by 0.4 and 0.2 BLEU points on En–De and En–Zh, respectively, and by 0.3% absolute WER on En–Zh. Therefore, it further closes the gap between pseudo-labeling and the upper bounds.

This ends our discussion of how domain mismatch can be addressed. We find filtering methods, which discard labels, to be only effective when due to any resource limitation, only a few, one or two rounds of pseudo-labeling can be run. This finding also echoes insights from Bansal et al. (2022) that studies data scaling laws for MT and shows while filtering may benefit computational efficiency, more unfiltered data can replace filtered data. As an alternative to filtering, we show that improving the quality of all generated labels through augmentation so that all can be kept, is the most effective, especially when as many rounds as needed can be run to reach saturation.