Attention-based Multi-hypothesis Fusion for Speech Summarization

We use a state-of-the-art Transformer model for ASR [21, 22]. The ASR system predicts the word sequence, $\hat{S}_{k}$ associated with the speech signal $X_{k}$ using beam search. This is achieved by combining the scores from the transformer and language model,

$\displaystyle\hat{S}_{k}$

$\displaystyle=\mathrm{argmax}_{S}\left(\log p_{\text{transf}}(S|X_{k})+\lambda\log p_{\text{lm}}(S)\right),$

(1)

where $p_{\text{transf}}(S|X_{k})$ is the posterior probability of $S$ given $X_{k}$ obtained with the transformer model, $p_{\text{lm}}(S)$ represents the language model, and $\lambda$ is the shallow fusion weight of the language model.

In this paper, we assume that we can obtain multiple ASR hypotheses. These hypotheses can be the $N$-best recognition hypotheses obtained using beam search decoding or $N$ hypotheses obtained with different ASR systems as often done with system combination [23]. In the following, $\hat{S}^{n}_{k}$ represents the $n$-th hypothesis obtained for a list of $N$ hypotheses.

In practice, ASR operates on sub-word units such as byte pair encoding (BPE), and thus a recognition hypothesis can be expressed as $\hat{S}^{n}_{k}=[\hat{s}^{n}_{k,1},\ldots,\hat{s}^{n}_{k,M_{k}^{n}}]^{\mathrm{T}}\in\mathbb{R}^{M_{k}^{n}\times V}$, where $\hat{s}^{n}_{k,m}$ is a one-hot vector representing the $m$-th token of the $n$-th hypothesis and the $k$-th utterance, $M_{k}^{n}$ is the length of the hypothesis, $V$ is the vocabulary size (the number of sub-word units), and $\mathrm{T}$ is the transpose operation. To obtain recognition hypotheses for the entire spoken document, we can simply concatenate the hypotheses of all utterances as $\hat{S}^{n}=[\hat{s}^{n}_{1,1},\ldots,\hat{s}^{n}_{K,M_{K}^{n}}]^{\mathrm{T}}$. By abuse of notation, we remove the utterance index $k$ and redefine $\hat{S}^{n}=[\hat{s}^{n}_{1},\ldots,\hat{s}^{n}_{M^{n}}]^{\mathrm{T}}\in\mathbb{R}^{M^{n}\times V}$, where $M^{n}$ is the total length of the concatenated $n$-th hypotheses of the the spoken document, i.e, $M^{n}=\sum_{k}M^{n}_{k}$. With beam search, we can also obtain the posterior probabilities associated with each token in the hypothesis, which we denote as $\hat{p}^{n}_{m}$.

Early works on text summarization used combinatorial optimization approaches to find important content in a text document [24, 25, 26]. It was difficult to generate consistent abstractive summaries with such approaches, so most research focused on extractive summaries. The success of the deep learning-based language models greatly improved the quality of abstractive summarization [27, 28, 27, 29]. Recently, Transformer-based models have become state-of-the-art models for abstractive text summarization [30]. For example, BERTSum [29] leverages the strong language modeling capability of a pre-trained BERT [1] model to achieve high-quality abstractive summarization through transfer learning. Since, the original BERT model assumes single sentences as input instead of a sequence of sentences as in summarization tasks, BERTSum introduces BERT’s classification (CLS) tokens at the start of each sentence of the input document. BERTSum uses the BERT model as an encoder and adds a Transformer-based decoder to generate the summary. The model is then fine-tuned on the text summarization task.

The TS system accepts the entire transcription of the spoken document as an input. In general, we can simply use the 1-best hypothesis $\hat{S}^{1}$. We employ a BERTSum model to generate a summary from $\hat{S}^{1}$ as

	$\displaystyle E$	$\displaystyle=\text{Emb}(\hat{S}^{1}),$		(2)
	$\displaystyle Z$	$\displaystyle=\text{Enc}^{\text{bert}}(E),$		(3)
	$\displaystyle Y$	$\displaystyle=\text{Dec}^{\text{transf}}(Z),$		(4)

where $\text{Emb}(\cdot)$ is an embedding layer that converts the one-hot token sequence into a sequence of embedding vectors $E=[e_{1},\ldots,e_{M^{1}}]^{\mathrm{T}}\in\mathbb{R}^{M^{1}\times B}$, $B$ is the size of the embedding, $\text{Enc}^{\text{bert}}(\cdot)$ represents a BERT encoder, $Z$ is an intermediate representation of the document, and $\text{Dec}^{\text{transf}}(\cdot)$ represents a transformer-based decoder that generates a summary based on $Z$. BERTSum takes advantage of the strong language modeling capability of the pre-trained BERT model to generate high-quality summaries. By inserting CLS symbols between consecutive sequences, the BERT encoder can process a sequence of multiple utterances that forms a long document.

There are two issues with the cascade connection of ASR and BERTSum. First, the BERT model assumes discrete features representing the IDs of the sub-word units as inputs. Thus it cannot accept an uncertain input such as posteriors of the ASR system directly, and recognition errors will directly affect the summary. We discuss some proposals to address this problem in Section 3.

Second, although both systems use sub-word units, the definition of sub-word units usually differs. Typically, ASR performs better with much smaller BPE units than the conventional BERT system.. However, since we would like to exploit a pre-trained BERT model, we consider two options. With the first option, we re-tokenize the word sequence after ASR to match the sub-word definition of the BERT model. With this approach, recognition performance may be optimal, but it offers limited possibilities for the interconnection of the ASR and TS systems. The other option is to use an ASR system trained with the same sub-word unit definitions as the BERT model. This training will degrade ASR performance but allow more flexibility in combining the two systems. We will analyze the impact of these options in the experiments of Section 5.2.

Many ASR+NLP systems such as speech summarization, translation, and spoken dialog systems use a cascade of ASR and NLP sub-systems built independently. The performance of the downstream NLP task is directly affected by the recognition errors [31]. Previous studies improved the robustness of the NLP back-end to ASR errors using various auxiliary information sources from the ASR system, e.g., probabilities, recognition hypotheses, and hidden states [32, 33, 34, 35, 8, 11, 36, 6, 10, 4, 37].

For speech summarization, Weng et al. proposed including a confidence embedding in the input of BERTSum [29] to achieve robust speech summarization [4]. They modified the input embedding vectors of BERTSum to be the sum of the sub-word embedding vector and a confidence embedding. Here, we implemented a similar method. We use $\hat{p}^{1}_{m}$, as introduced in Section 2.1, as a confidence value, which we map to a hidden vector using a linear mapping as

$$c_{m}=\text{Emb}^{\text{conf}}(\hat{p}^{1}_{m}),$$

(5)

where $c_{m}\in\mathbb{R}^{B}$ is a confidence embedding and $\text{Emb}^{\text{conf}}(\cdot)$ is a linear embedding layer, where $B$ is the dimension of the projected embedding vectors. Then the modified embedding vector $e_{m}^{\text{conf}}$ is obtained as the summation of confidence and word embeddings:

$$e_{m}^{\text{conf}}=c_{m}+e_{m}.$$

(6)

The posterior fusion consists of summing the embedding vectors of all sub-words weighted by their posterior probabilities, $\hat{p}_{m}^{n}$, as

$\displaystyle e_{m}^{\text{post}}=$

$\displaystyle\sum_{n=1}^{N}\hat{p}_{m}^{n}e_{m}^{n},$

(8)

where $e_{m}^{\text{post}}$ is a modified embedding vector and $e_{m}^{n}$ is the $m$-th embedding vector of the $n$-th hypothesis obtained with Eq. (2). The modified embedding $e_{m}^{\text{post}}$ can include the uncertainty in the ASR system. Computing $e_{m}^{\text{post}}$ requires that all hypotheses are aligned and have the same length. We thus create the $N$ hypotheses as follows. First, we save the sequence of sum of output log-softmax values from the ASR and the language models for the best beam-search path. After decoding, for each step in the 1-best path, we select the $N{=}10$ tokens with the top 10 values of saved output vectors in the path. This way of creating $N$ hypotheses may generate more diverse hypotheses than simply obtaining the $N$-best list from beam search decoding. Note that since the posterior-based fusion modifies the input of the TS model, we need to retrain the BERTSum model using ASR hypotheses.

Posterior probability fusion trusts the ASR weighting and may mitigate the influence of unreliable tokens when performing summarization. However, in the case of the ASR outputs the correct word at $10$-th hypotheses, it may be difficult to recover information from the modified embedding of Eq. (8) because the correct word’s weight $\hat{p}^{10}_{m^{1}0}$ is too small. On the other hand, our proposal re-calculates for all hypothesis weight based on BERT representation. Thus, even if the correct word is $10$-th hypotheses, our proposal can provide high weight to the correct word.

Attention-based fusion consists of two steps. In the first step, we pick up a representative ASR hypothesis and align the other hypothesis to it. We use the most confident ASR hypothesis, i.e., the 1-best hypothesis, $\hat{S}^{1}$. If we use multiple ASR systems, we use a hypothesis of a possible best performing ASR system as $\hat{S}^{1}$.

We obtain the embedding vectors for the $n$-th hypothesis, $\tilde{E}^{n}=[\tilde{e}^{n}_{1},\ldots,\tilde{e}^{n}_{M^{1}}]\in\mathbb{R}^{M^{1}\times B}$, which is time-aligned with the $M^{1}$-length hypothesis $\hat{S}^{1}$, based on the attention mechanism as:

$$\tilde{E}^{n}=\text{softmax}\left((E^{1}W^{Q})(E^{n}W^{K})^{\mathrm{T}}\right)E^{n}W^{V},$$

(9)

$E^{n}=[e_{1}^{n},\ldots,e_{M^{n}}^{n}]^{\mathrm{T}}\in\mathbb{R}^{M^{n}\times B}$ is the sequence of embedding vectors associated with the $n$-th hypothesis. $E^{1}$ is the sequence of the 1-best embedding vectors and is used as a query. $\text{softmax}(\cdot)$ is the softmax operation and $W^{Q}\in\mathbb{R}^{B\times B^{\prime}}$, $W^{K}\in\mathbb{R}^{B\times B^{\prime}}$, $W^{V}\in\mathbb{R}^{B\times B^{\prime}}$ are the query, key, and value projection matrices, respectively.

In the second step, we perform attention over the different hypotheses for every aligned sub-word position $m$ in a similar way as the hierarchical attention [14, 34, 38]. Let $C_{m}=[\tilde{e}^{1}_{m},\dots,\tilde{e}^{N}_{m}]\in\mathbb{R}^{B^{\prime}\times N}$ be a matrix containing the $N$ aligned embedding sequences for the $m$-th sub-word position in a sequence. We can perform attention over the hypotheses to obtain a modified embedding vector $e^{\text{att}}_{m}$ as

	$\displaystyle\alpha_{m}=$	$\displaystyle\text{softmax}\left((e^{1}_{m})^{\mathrm{T}}W^{Q}C_{m}\right),$		(10)
	$\displaystyle e^{\text{att}}_{m}=$	$\displaystyle\alpha_{m}C_{m}^{\mathrm{T}},$		(11)

where $\alpha_{m}\in\mathbb{R}^{1\times N}$ are attention weights over the recognition hypotheses. Eq. (11) performs a similar summation over embedding vectors as in Eq. (8), but using the attention mechanism to compute the weights and time-aligned embedding vectors.

The proposed attention fusion can also be extended to multi-head attention. In that case, we can obtain an aligned hypothesis for each attention head, $\tilde{E}_{h}^{n}$, using a similar equation as Eq. (9), with different projection matrices for each head, i.e., $W^{Q}_{h}$, $W^{K}_{h}$ and $W^{V}_{h}$, where $h$ is the head index. We can then also define $C_{m,h}$ for each head and compute attention weights $\alpha_{m,h}$ and fused hypotheses ${e}^{\text{att}}_{m,h}$ for each head as in Eqs. (10) and (11). We then obtain the fused hypothesis as,

$\displaystyle e^{\text{att}}_{m}=$

$\displaystyle\text{concatenate}(e^{\text{att}}_{m,1},\ldots,e^{\text{att}}_{m,H})W^{o},$

(12)

where $H$ is the number of attention heads and $W^{o}\in\mathbb{R}^{(HB^{\prime})\times B}$ is an output projection matrix as used in previous work [39]. We used the multi-head implementation in our experiments. Note that we also used the cosine similarity instead of the dot product to compute the attention weights in Eqs. (9) and (10).

Unlike the posterior fusion, we can use hypotheses of different lengths with the proposed attention fusion thanks to the time-alignment step of Eq.(9). Moreover, although we derived the attention fusion assuming it was performed at the input of the BERT encoder, we can also perform attention fusion at any layer within the BERT encoder. When performing the fusion within the BERT encoder, the multiple hypotheses are embedded in the intermediate representation of the BERT model. Therefore, we can exploit BERT’s strong language modeling capabilities to combine the hypotheses. Figure 1 illustrates how we apply the proposed attention fusion within the BERT encoder.

Note that when we introduce a randomly initialized attention fusion layer, it may corrupt the BERT encoder, making the training slow or unstable. In this paper $B$ equals $B^{\prime}$, therefore, we initialize the projection matrices $W^{Q}_{h}$, $W^{K}_{h}$, and $W^{V}_{h}$ to identity matrices, so at the beginning of the training, Eq. 11 provide highest value to first hypotheses, and the $e^{\text{att}}_{m}$ is close to the 1-best hypothesis $e^{1}_{m}$.

CNN-DailyMail (CNNDM) is a large-volume corpus used for text summarization of news documents. We use this corpus to pre-train the TS system. How2 is a publicly available corpus for speech summarization [40]. It consists of summarizations of How2 videos taken from YouTube. The target for summarization consists of the brief video description provided on YouTube. Although the corpus also includes videos, allowing multi-modal summarization, here we only use the audio content of the corpus.

The TED corpus consists of a summarization of TED Talks. We created this corpus by associating TED Talks included in the TEDLIUM corpus with their summaries obtained from the TED website²²2https://www.ted.com/talks. For the TED summarization task, we add the speaker name to the speech document to allow the TS systems to output the speaker names, which commonly appear in the reference summaries. The target summary consists of the title and abstract of the talk. Note that others have also used TED Talks from speech summarization [41], but those corpora were small. Furthermore, they did not publicly release their corpus.

Table 1 shows the number of text and speech documents, the compression rate, the source and target lengths, and the word error rate (WER) of the three corpora. The compression rate is expressed as the ratio between the output and input document lengths. Thus, a lower value means higher information compression. This table shows that our proposed TED summarization task is challenging because it has a relatively small amount of training data and consists of long input speech documents that require higher information compression than needed for existing datasets. Note that the WER is about 8.5%, which makes it slightly easier than the How2 corpus in terms of the ASR performance.

Another way in analyzing the complexity of a summarization task is whether it is to apply extractive summarization. We can measure ideal extractive summarization scores using the reference summary to select the set of sentences from the input document that achieves the highest summarization score. Table 2 shows the ROUGE scores [42] and word overlap of the ideal extractive summary for the three datasets. The word overlap measures the percentage of words from the target summaries that are in the source documents. CNNDM, which consists of a summary of news articles, is well suited for extractive summarization; therefore, oracle scores and word overlap are relatively high. In contrast, the How2 corpus consists of relatively casual speech, which leads to much lower oracle scores and word overlap. The TED summarization corpus consists of relatively formal speech and its oracle extractive summarization scores are between those of CNNDM and How2.

Finally, we compare the performance of abstractive text summarization on the ground-truth transcriptions. The results indicate an upper-bound value for the speech summarization systems we investigated. Table 3 shows the ROUGE scores obtained with BERTSum models for the three tasks. We found that the difficulty of abstractive summarization is highly dependent on the length of the input and the compression rate, which makes the proposed TED summary task also challenging for abstractive summarization.

Comparing the results of Table 2 and 3 shows that ideal extractive summarization’s scores on TED talk, in particular ROUGE-2 and -L scores, are higher than abstractive summarization ones. We will investigate extractive summarization on the TED corpus and comparing it with abstractive summarization in future works.

Our baseline consists of a cascade of ASR and TS systems trained separately. We built Transformer-based ASR models using the Espnet toolkit³³3https://github.com/espnet/espnet, by following the published recipes for the TEDLIUM2 [43] and How2 [40] tasks, except that we varied the BPE size. For the TS model, we built a BERTSum model using a pre-trained BERT model as an encoder and a Transformer decoder in the same way as [38]. We used the pre-trained BERT model provided by huggingface⁴⁴4https://huggingface.co/transformers/. We pre-trained the BERTSum model using CNNDM data, and fine-tuned it on each summarization task.

We consider three baseline systems, “(1) BERTSum (oracle),” which uses ground-truth ASR transcriptions as input to the BERTsum model, “(2) BERTSum (ASR-BPE),” which uses transcriptions obtained with an ASR system trained with the optimal BPE definition for ASR, and “(3) BERTSum (BERT-BPE),” which uses an ASR system trained with the BPE definition of the pre-trained BERT model. The first baseline illustrates the upper-bound performance on the tasks. The second baseline is optimal for ASR, but it requires re-tokenizing the recognized text to match the BPE of the BERTSum model, and it cannot be used to pass ASR information such as confidence or posteriors to the summarization back-end. The third baseline uses the same BPE definition as our proposed methods.

In addition to the above systems, we created a baseline by retraining (3) BERTSum (BERT-BPE) on the transcriptions generated with ASR (“(4) BERTSum retrain”). We also implemented a system similar to [4] that uses BPE confidence scores as an auxiliary feature at the input of BERTSum (“(5) BERTSum confidence”) as described in Section 2.3.

The proposed posterior-based hypotheses fusion used the same ASR and TS system as the baseline systems, i.e. (3) BERTSum (BERT-BPE), except that the BERTSum system was fine-tuned on input embeddings obtained with Eq. (8) using $N{=}10$ as described in Section 3.1. We call this system “(6) BERTSum Pos. fusion”.

For the proposed attention-based fusion, we inserted the attention fusion layer at the fifth layer of the BERT and used four attention heads ($H=4$). We used the same approach to generate the recognition hypotheses as for the posterior fusion described in Section 3.1 except that we used attention fusion described in Section 3.2 instead of Eq. (8). We used five hypotheses ($N{=}5$) for attention fusion due to GPU memory constraints of our experimental environment. We trained all BERTSum models following the original recipe except that we used a learning rate of $0.0002$ and warm-up steps of $20$k when retraining on the ASR outputs (systems (4) to (7)). We compared the summarization performance of each method with ROUGE [42].

ASR and BERT models use both BPE to represent sub-words; however, the BPE unit definitions used for the two systems differ. Typically, ASR systems achieve optimal performance at a smaller BPE size than that of the BERT model. However, since our proposed method requires that the BPE unit definition of the ASR system must match that of the BERT model, we expect to have some ASR degradation due to too large BPE sizes. Thus, we first investigate the effect of BPE size on ASR performance.

Table 4 shows WER as a function of different BPE sizes for the How2 and TED corpora. We observe a relative WER increase of 12% for How2 and by more than 20% for the TED corpus when adopting the BPE definition used by BERT. Although this is a significant WER increase, we discuss its impact on summarization in the following subsection.

Table 5 shows the ROUGE scores for the baseline systems (systems (1) to (5)) and the proposed method with (6) posterior and (7) attention-based fusion⁵⁵5Note that we confirmed the validity of our implementation of BERTSum as it achieved a similar level of performance on the How2 corpus [29], which reported ROUGE-1 and ROUGE-L scores of 48.3 and 44.0, respectively, although the systems cannot be directly compared because of differences in the training data and ASR front-end..

We observe a large performance gap in summarization performance when using ASR transcriptions (system (2)) instead of ground-truth transcription (system (1)). As we discussed in section 5.2 using BERT’s BPE definition for ASR (system (3)) induces more recognition errors, which clearly degrades summarization performance on both tasks compared to using the BPE optimal for ASR (system(2)). However, this performance degradation can be mostly recovered by fine-tuning on the ASR hypotheses (system (4)). Using confidence scores (system (5)) [29] improves ROUGE scores on the How2 corpus, but degrades ROUGE-1 on the TED corpus compared to simply retraining on the ASR hypotheses.

The proposed posterior (system (6)) and attention-based fusion (system (7)) approaches both achieve equivalent or superior ROUGE-1 scores for both corpora than the baseline systems with retraining on ASR hypotheses (system (4)). In particular, the proposed attention-based fusion improves ROUGE-1 by 2 points on the How2 corpus. This result indicates that the proposed system can better mitigate ASR errors by using multiple hypotheses with the BERTSum model.

Figure 2 shows the ROUGE-1 score difference between systems (4) and (5)-(7) as a function of the WER of the spoken documents. These results show that our proposed attention fusion system achieves higher robustness for ASR error than systems (5) and (6). We hypothesize that this is due to the fact that the proposed attention fusion approach models explicitly multiple ASR hypotheses, and can thus choose alternative word candidates within ASR hypotheses to generate the summary. This may make the attention fusion-based system robust to ASR errors if the correct words are included in the hypotheses. In addition to the ROUGE score, Table 6 provides a couple of summaries generated by our proposed system for the How2 corpus ⁶⁶6We provide more examples, as well as examples on the TED corpus on our webpage https://github.com/takatomokano/ted_summary. We include summaries obtained with BERTSum fine-tuned on the ASR hypotheses as a comparison. We confirm that both systems achieve highly readable summaries that are close to the reference.

The experiments with the proposed attention fusion used aligned hypotheses of the same duration. However, the proposed method can also handle hypotheses of different lengths thanks to the alignment mechanism of Eq. (9). We also tested the attention fusion using hypotheses generated by five different ASR systems each with a different BPE size (the systems used in the experiments of Section 5.2). In this case, the hypotheses were unaligned and of different lengths. The proposed attention fusion achieved a ROUGE-1 score of 48.0 on the HOW2 task, which shows some improvement over the baseline systems (2)-(4). Although this result is behind our best performing system, it shows that the proposed method could handle hypotheses with different lengths. We plan to further investigate such a combination by using more diverse ASR systems to generate the hypotheses in future work.