VADOI: Voice-Activity-Detection Overlapping Inference for End-to-end Long-form Speech Recognition

For OI, decoded results from each pair of overlapped consecutive windows are aligned through dynamic programming. A series of word pairs are concatenated through the tie-breaking algorithm [10]. The target of alignment is to minimize word error rate (WER) between decoded results of consecutive segments, where previous sentence behaves as groundtruth and next sentence as prediction. There is no surprise that the non-overlapped region will introduce external insertion and deletion errors. This problem doesn’t affect alignment performance when the overlapping percentage is high because enough common words can provide sufficient matching reward for good alignment. However, performance degrades dramatically, or even fails when the overlapping percentage is low.

To relax the constraint of high overlapping percentage, POI is proposed to suppress external insertion and deletion costs. Compared with OI, marginal deletion cost and insertion cost are set to be 0. To prevent concatenating two overlapped sequences end-to-end, POI sets negative matching reward to encourage overlapping alignment. Aside from manipulating marginal conditions, POI can choose to do alignment in char-level. However, even though computation cost reduces when using lower overlapping percentage, WER degrades monotonically. For algorithm details, please refer to [15].

As mentioned in Sec. 2, for POI, there is a zero-sum trade-off between computational cost and ASR performance through overlapping inference. Error analysis shows that, the cause of WER degradation is lacking shared words across segments. Alignment only works well when there are enough matching rewards from common words between consecutive segments. This condition can be achieved under large overlapping percentage. Although some words are recognized differently in different segments, the number of common words still prevail. While, for low overlapping percentage, fewer words can provide matching reward. If some are recognized differently due to boundary distortion, alignment will be severely affected.

Therefore, VADOI is proposed to prevent chopping in the middle of a word. The diagram of chopping stage of VADOI is shown in Fig. 1. After a first-stage segment is generated with fixed segment length and overlapping percentage, a VAD is applied on the segment. The starting frame and the end frame are shifted separately to the middle of the closest long-pause with a desired length. To ensure the existence of an overlapped region, we restrict shifting distance to be shorter than half of the overlapping region length. If a long-pause is not found within this range. The desired length will be cut in half. Searching and shifting will be repeated until a sufficient long-pause is detected.

For shifting direction, end frame is always shifted to the left to enforce causality. For starting frame, cases are considered separately. When we always shift starting frame to the long-pause on the left, it’s possible that some words will appear in three segments where overlapping percentage exceeds 40%, which will deteriorate alignment. Therefore, starting frame is shifted to the right when overlapping percentage is over 40%, to prevent triple word-pair, and shifted to the left as shown in Fig. 1 when overlapping length is below 40%, to maintain segment length.

Since VAD can be embedded into the model, frame-level VAD results and RNN-T hypothesis can be obtained simultaneously, computation cost for VAD can be ignored.

For OI and POI alignment, the operation cost related to substitution and matching is obtained using the function:

$$\textstyle e_{sub}=\left\{\begin{matrix}w_{match}&if\>d_{i}\left(j\right)=d_{i+1}\left(k\right)\\ w_{sub}&else\end{matrix}\right.$$

(1)

where $d_{i}\left(j\right)$ denotes the $j$-th word in the $i$-th segment, $w_{sub}$ and $w_{match}$ are substitution cost and matching reward, respectively, and $e_{sub}$ is the operation cost [15]. A substitution error is omitted no matter how similar two words are. Since two words omitted from the same acoustic feature are expected be aligned, Equation (1)’s condition is too strong. To relax the constraint, Soft-Match function is defined as:

$$\textstyle e_{sub}=CER(d_{i}(j),d_{i+1}(k))\cdot(w_{sub}-w_{match})+w_{match}$$

(2)

The Character Error Rate (CER) between words $d_{i}\left(j\right)$ and $d_{i+1}\left(k\right)$ is projected into a continuous number within range between $w_{sub}$ and $w_{match}$. Then similar word-pair can contribute smaller operation cost to encourage correct alignment.

Experiments are conducted on two long-form datasets, MSLT-long and Lib-long, which are simulated from MSLT test set [16] and Librispeech test-clean set [17] respectively. For MSLT, we first sort all utterances by length. The long-form test set is simulated by choosing one utterance with duration longer than the median and one shorter than the median repeatedly. Concatenation stops once the length of the concatenated utterance exceeds 120 seconds. For Librispeech, we just simply concatenate utterances from the same speaker. Concatenation stops at 120 seconds. The average length of MSLT-long dataset is 121 seconds and standard deviation as 5 seconds, and 120 seconds and 3.8 seconds for Lib-long.

Performance of Baseline, Naive-approach, OI and POI under various configurations on MSLT-long dataset are reported in Table.1. For the Baseline system, long-form utterances are decoded directly. For the Naive-approach, long-form utterances are chopped into 12 seconds segments without overlapping. Long-form results are obtained by concatenating segments results in order. Naive-approach is marked as 0% in following tables to represent that there is no overlap. For OI, $w_{del}$, $w_{ins}$, $w_{sub}$, $w_{match}$ are empirically set to 1, 1, 1, 0. For POI, $w_{del}$, $w_{ins}$, $w_{sub}$, $w_{match}$ are set to 2, 2, 1, -2.

Aside from WER comparison, decoding time and overlapping inference time is also reported. Decoding time represents how many folds is needed to decode compared with Baseline (1T represents the baseline runtime). It also represents the ratio of the number of generated segments compared with Naive-approach (T). Overlapping inference time represents absolute duration for OI-based algorithm to do alignment and concatenation in sec/utt. Results from Table. 1 show that by incorporating OI and POI, performances outperform Baseline and Naive-approach for most cases. Regarding POI and OI, POI always performs better than OI, especially for 30% and 15% cases. Since POI sets better margin conditions, non-overlapped regions are handled well. It also shows that word-level alignment almost always outperforms char-level one. Since for char-level alignment, the omitted word might not be in the vocabulary, which introduces extra substitution errors. It’s worth noting, OI fails when applying char-level alignment. Since char-level alignment introduces many more non-overlapped instances than word-level one, OI is therefore not compatible with char-level alignment by nature. It can also be observed that, even POI yields good improvement over the Baseline and the Naive-approach when overlapping percentage is low, there is still monotonic performance degradation as overlapping percentage decreases. Regarding decoding time and overlapping inference duration, intuitively, setting larger overlapping percentage will generates more segments which will increase the decoding time by 1.87$\times$ (50%), 1.37$\times$ (30%) and 1.16$\times$ (15%) compared with the Baseline and the Naive-Approach. Char-level alignment takes significant amount of extra time to do alignment and concatenation. Because when we switch alignment unit from word to char, the dynamic graph size scales up exponentially. Searching a extremely large dynamic graph is not feasible in real-world use-case. Based on our findings here, word-level POI is used as the default configuration in the following experiments. We observed similar outcome on Lib-long dataset, and therefore dropped the result in the interest of saving article space.

We employed a statistical-based VAD [25]. The initial duration of long-pause is empirically set to 0.1 second. Results of VADOI of MSLT-long are reported in Table.2 and Lib-long in Table.3. Because end frame is always shifted to the left, incorporating VAD will generate slightly more segments under the same overlapping percentages.

For 50% overlapping percentage, performance of VADOI is slightly worse on both datasets. We believe it is because 50% overlapping percentage already introduces sufficient common words for good alignment. More segments might not improve alignment performance but rather increase the risk of choosing wrong words. For 30% overlapping percentage, performance is improved from 13.27% to 13.02% on MSLT-long and 6.79% to 6.58% on Lib-long, which is equivalent to performance of 50% on MSLT-long, and only 1.3% relative WER degradation on Lib-long, but with 19.79% and 20% less computation cost respectively. Other computation costs are ignored, including dynamic programming and concatenation, since their computational loads is irrelevant to inferring the RNN-T model. It turns out that for 30% overlapping percentage, VADOI improves WER by chopping segments more intelligently and cause less confusion for alignment when common words do not prevail. For 15%, VADOI still outperforms POI. But VADOI on 15% doesn’t achieve equivalent results as using 50% due to extremely lacking of common words. We also tried 7% and lower overlapping percentage. But results show that 7% POI starts to perform worse than applying VAD in Naive-approach.

Table.4 reports results when applying Soft-Match on VADOI. The table shows that by applying Soft-Match in alignment, consistent yet limited improvement is observed for all trials on two datasets. We suspect the reason for lacking strong outcome might be that the problem expected to be solved by Soft-Match does not prevail in our case. However, Soft-Match do not introduce side effect to the performance and can actually handle mis-aligned similar words problem efficiently.