Leveraging Large Text Corpora for End-to-End Speech Summarization

Our E2E SSum model receives three training datasets: paired data of speech and transcription $(\mathbf{x},y_{\mathrm{asr}})\in\mathcal{R}_{\mathrm{ASR}}$, paired data of speech and summary $(\mathbf{x},y)\in\mathcal{R}_{\mathrm{Sum}}$, and external text summarization data $(d,y)\in\mathcal{R}_{\mathrm{Ext}}$. Here, $d$ is a TSum input text. To leverage a variety of knowledge, we adopt a training strategy consisting of multiple stages as follows:

1. (i)

   An AED model is trained as an ASR model using the pairs in $\mathcal{R}_{\mathrm{ASR}}$ to acquire the transcribing capability internally.

2. (ii)

   To build the E2E SSum model, the aforementioned model is additionally fine-tuned with $\mathcal{R}_{\mathrm{Sum}}$.

3. (iii)

   The model is further fine-tuned with the combined dataset of $\mathcal{R}_{\mathrm{Sum}}$ and artificial data derived from $\mathcal{R}_{\mathrm{Ext}}$.

We define the model after stage (ii) as a baseline model and after stage (iii) as an augmented model by the TTS- or phoneme-based method.

To utilize the TSum data for E2E SS, we propose two approaches: TTS- and phoneme-based data augmentation. For the TTS-based approach, we synthesize the raw waveforms from the input texts $d$ of the TSum dataset and extract acoustic features for the stage (iii).

Although this TTS-based method is straightforward to augment speech summarization data, it requires high computational costs to synthesize speech. Therefore, we also propose an alternative approach that feeds phoneme sequences into the E2E SSum model instead. This approach is motivated by the performance improvement in [16], in which a multi-modal (i.e., acoustic and textual) data augmentation technique for E2E ASR was proposed. As a procedure, a word sequence $d$ is first automatically converted into a phoneme sequence $d_{p}$ by utilizing a grapheme-to-phoneme model. The phoneme duration is then modeled by repeating each phoneme in $d_{p}$ in accordance with separately calculated statistics. The repeated phoneme sequence is denoted by $d^{\mathrm{rep}}_{p}$. To input the $d^{\mathrm{rep}}_{p}$ together with $\mathbf{x}$, we also add and optimize the phoneme pre-encoder $\mathrm{PreEnc}_{\mathrm{p}}(\cdot)$ in addition to $\mathrm{PreEnc}$ simultaneously as

Both inputs $\mathbf{x}$ and $d^{\mathrm{rep}}_{p}$ share other modules as depicted in Fig. 2. Here, we choose bidirectional LSTMs followed by a LN layer and subsampling CNNs for $\mathrm{PreEnc}_{\mathrm{p}}$.

We used the How2 dataset, which has approximately 2,000 hours of speech and is suitable for the E2E SSum task because of its relatively short speech inputs and abstractive summaries with a high compression rate. The training, validation, and evaluation sets contain 68,336, 1,600, and 2,127 pairs, and the average durations of their spoken documents are 84.7, 76.0, and 98.7 seconds, respectively. The input speech is truncated up to 100 seconds and composed of 40-dimensional log Mel-filterbank outputs with 3-dimensional pitch features. According to [21], the summaries are automatically collected from “description” tags on YouTube.

For the data augmentation, we utilized the Gigaword corpus [19, 20] composed of 3.8M pairs of the first sentences and titles of news articles. The average word counts on this dataset for the input text and summaries are 31.4 and 8.3, respectively. Since the average input length is much smaller than that of other text summarization corpora, such as CNN Daily Mail [29], the synthesizing cost for the Gigaword corpus is relatively lower. Even though the domain of this external corpus mismatches that of the How2 dataset, we expect that the model could learn additional words and expressions.

We adopted a 12-layer Conformer encoder and a 6-layer Transformer decoder for the model architecture. The pre-encoder was composed of 4-layer CNNs with a subsampling rate of 4. We evaluated two model sizes: the base and large models with 512- and 768-dimensional embeddings, respectively. Their encoders commonly had 2048 feed-forward (FF) units, 8 attention heads, and a kernel size of 31. The decoder of the base model had 2048 FF units and 4 attention heads, and that of the large model had 3072 FF units and 12 attention heads.

In the training stage (i) described in Section 4.1, we trained the E2E ASR model with the Adam optimizer [30] and Noam scheduler [26] with a learning rate of $2\cdot 10^{-3}$, warmup steps of 40k, a weight decay rate of $10^{-5}$, and a batch size of 512. The word error rates ($\downarrow$) of the base and large models on the evaluation set were 9.8% and 9.5% with a beam size of 16, respectively.

For the training stage (ii), the model was fine-tuned as the E2E SSum model with a learning rate of $10^{-4}$ with a reduction factor of 0.5, and batch size of 30.

The model was further fine-tuned with augmented data in the training stage (iii). For the TTS model, we used VITS [31] trained with the LJSpeech corpus ²²2https://github.com/espnet/espnet/tree/master/
egs2/ljspeech/tts1#pretrained-models-1. The synthesized speech does not have a diversity of speakers because the LJSpeech contains only a single speaker. This enables us to evaluate only the effectiveness of additional linguistic information excluding the effect of augmenting speaker varieties. After generating the raw waveform, we extracted 43-dimensional acoustic features with a window size of 25ms and a shift of 10ms from the waveform and applied cepstral mean-variance normalization (CMVN) to match the original features in the How2 corpus. When training the model in the stage (iii), we adopted a learning rate of $10^{-4}$ with a reduction factor of 0.5, and the learning rate of its decoder was $10^{-3}$. The maximum total length of input sequences in one batch was set to 300,000. Each batch contains only real or artificial samples.

For the phoneme-based method, we used the g2pE toolkit³³3https://github.com/Kyubyong/g2p to convert word sequences into phoneme sequences. The number of phoneme units was 42. The duration of phoneme $p$ was determined following the normal distribution $\mathcal{N}(\mu_{p},\sigma_{p})$ at each time it appeared. Here, the mean $\mu_{p}$ and standard deviation $\sigma_{p}$ of $p$ were estimated with the TIMIT corpus [32]. The phoneme pre-encoder had 3-layer bidirectional LSTMs followed by LN and 4-layer CNNs with a subsampling rate of 4. The learning rate of the phoneme pre-encoder was $10^{-3}$. The other configurations were the same as those of the TTS-based method. In the stage (i), (ii), and (iii), we used byte-pair encoding and set the vocabulary size to 1,000.

When evaluating model performance, we chose the checkpoints with the best validation accuracy and decoded them with a beam width of 4 for all conditions. For evaluation metrics, we selected the ROUGE [33], METEOR [34], and BERTScore [35] scores commonly used in TSum tasks.

We also evaluated the conventional cascade model. The ASR model was the same as the large model obtained after the training stage (i). The TSum model was a BART-based model fine-tuned with the CNN/Daily Mail corpus⁴⁴4https://huggingface.co/ainize/bart-base-cnn [29] and the How2 dataset. During the fine-tuning with the How2 dataset, we trained the model for 1M steps with the Adam optimizer, a learning rate of $5\times 10^{-5}$ with linear decay, and a batch size of 8. We used ESPnet2⁵⁵5https://github.com/espnet/espnet for all of the implementation, training, and evaluation.

Table 1 shows the evaluation scores of the augmented model (T-1, T-2) with prior state-of-the-art (P-1) [11], the cascade model (C-1), and our baselines (B-1, B-2) on the How2 evaluation set. Compared with Model P-1, Model B-1 with full attention, LN, and LPE remarkably improved the scores and achieved the current SoTA performance. There was a slight improvement by enlarging the model size as in Model B-2, and this strong model is the baseline for the proposed data augmentation methods. The result of Model T-1 shows that the TTS-based data augmentation method further improved the METEOR, ROUGE-L, and BERTScore scores by as much as 2.1, 3.0, and 0.6 points, respectively. In addition, despite its low cost, Model T-2 trained with the phoneme-based method was also improved by 1.1, 2.1, and 0.4 points in terms of the METEOR, ROUGE-L, and BERTScore scores, respectively. Although the domain of the augmented data is completely different from that of the How2 dataset, it helped the model expand its vocabulary. For instance, as seen in Table 2, Model B-2 could not output a word gladiator, which does not appear in the How2 training set, while Model T-1 and T-2 seemed to have learned it from the Gigaword training set.

While we showed the effectiveness of our system in Section 5.3, we observed several evaluation examples with extraordinarily high scores only for the E2E SSum models compared with the cascade model. We analyzed these samples and found that the training dataset contained samples with similar or perfectly matched summaries to the evaluation set. For example, the summary of ID: 6D0hfwYIwm4 from the training set:

Therefore, we propose to filter the evaluation set to omit such samples by utilizing a quantitative criterion. First, we computed the ROUGE-L scores between each summary in the evaluation set and the whole How2 dataset. We defined the highest score as the leakage score for each summary in the evaluation set. For example, the leakage score of ID: 2NdV94T-JNc is 0.95. Then, we removed the samples whose leakage score was larger than a threshold $\alpha$ from the evaluation set. Table 3 shows the number of samples contained in each filtered evaluation set for different threshold values.

Figure 3 shows the METEOR and BERTScore scores of Models B-2, T-1, T-2, and C-1 with a different threshold $\alpha$. While the scores for all models tend to be worse as $\alpha$ decreases, Model B-2 shows the clearest trend. This result suggests that the baseline E2E system overfits the training data and is vulnerable to unfamiliar samples. Model C-1 shows robustness for unseen samples and the slowest score reductions, probably owing to the abundant linguistic knowledge of its TSum model. For our proposed method, Models T-1 and T-2 achieved the highest accuracy in all conditions and seemed to have gained linguistic information from the external TSum corpus.