TRANSFORMER-BASED STREAMING ASR WITH CUMULATIVE ATTENTION

In recent years, the ASR community has witnessed a surge in developing end-to-end (E2E) systems, which have exhibited less computational complexity and competitive performance when compared with the conventional hidden Markov model (HMM) based systems. For E2E ASR, the acoustic model (AM), lexicon and language model (LM) are integrated into a holistic network, and could be optimised without any prior knowledge. The E2E techniques can be generally categorised into three classes, known as connectionist temporal classification (CTC) [1, 2], recurrent neural network transducer (RNN-T) [3] and attention-based encoder-decoder framework [4, 5, 6]. As an epitome of the last class, Transformer [7] has been introduced to ASR following its overwhelming success in the field of natural language processing (NLP). Consequently, state-of-the-art performance is reported on a number of standard ASR tasks [8]. The self-attention mechanism that underpins the Transformer architecture has soon replaced the dominant role of RNN in terms of both acoustic and language modelling, leading to the advent of Speech Transformer [9], Transformer Transducer [10, 11] and Transformer-XL [12]. Unlike RNN structures that process the inputs in the temporal order, the self-attention module captures the correlation between the elements of any distance, which efficiently lifts the restriction imposed by long-term dependency.

Although Transformer has shown prominent advantage over the RNN based systems, it faces major difficulties in online ASR. Both the encoder and the decoder require access to the full speech utterance, leading to large latency and thus limiting the use for practical scenarios. In this paper, we focus on the streaming strategies in the decoder to address the latency in recognition. Previously, the following methods have been proposed and were shown to be effective for streaming Transformer architecture: (1) hard monotonic attention mechanisms, represented by the monotonic chunkwise attention (MoChA) [13, 14, 15] and its simplified variant, monotonic truncated attention (MTA) [16, 17]; (2) Motivated by the streaming characteristic of CTC technique, triggered attention (TA) [18] takes the spikes of CTC score as the boundaries for computing attention weights; (3) blockwise synchronous inference [19] performs standard attention independently on each input chunk, with the broken decoder states carried over to the succeeding chunks; (4) Continuous integrate-and-fire (CIF) [20], along with the decoder-end adaptive computation steps (DACS) [21] and its head-synchronous version (HS-DACS) [22] interpret the online decoding as an accumulation process of attending confidence, which is halted when the accumulator exceeds a predefined threshold.

Among the aforementioned streaming methods, MoChA has been widely acknowledged, where it considers the ASR decoding as a monotonic attending process. The output is emitted around its acoustic endpoint. However, the output decision in MoChA is merely dependent on a single frame, adding possibilities for accidental triggering in complex speech conditions. Apart from this, in the context of multi-head Transformer, some of the heads might not capture valid attentions, and fail to truncate the decoding before the end of speech is reached. HS-DACS circumvents the problems of MoChA by: (i) transforming the decisions based on single-frame-activation into an accumulation of attending confidence; (ii) merging the halting probabilities to produce a same halting position for all attention heads in a decoder layer. Nonetheless, it is still not guaranteed that there is enough confidence to trigger the output, and an additional stopping criterion in the form of maximum look-ahead steps was introduced.

To overcome the above issues encountered by MoChA and HS-DACS, the paper proposes the cumulative attention (CA) algorithm. The CA-based attention heads collectively accumulate the acoustic information along each encoding timestep, and the ASR output is triggered once the accumulated information is deemed sufficient by a trainable device. The proposed algorithm not only involves multiple frames into the acoustic-aware decision making, but also achieves a unified halting positions for all the heads in the decoder layer at each decoding step.

The rest of the paper is organised as follows: Section 2 presents the architecture of Transformer ASR and some online attention approaches. Section 3 elaborates the workflow of the proposed CA algorithm. Experimental results are demonstrated in Section 4. Finally, conclusions are drawn in Section 5.

A typical Transformer-based ASR system consists of three components: front-end, encoder and decoder. The front-end is commonly implemented as convolutional neural networks (CNNs), which aims to enhance the acoustic feature extraction and conduct frame-rate reduction. The output of the CNN front-end enters into a stack of encoder layers, with each comprising of two sub-layers as a multi-head self-attention module and a pointwise feed-forward network (FFN). The decoder also stacks several layers and has a similar architecture as that of the encoder, except for an additional cross-attention sub-layer that interacts with the encoder states.

The attention modules in the Transformer architecture adopts the dot-product attention mechanism to model inter-speech and speech-to-text dependencies:

where $\mathbf{Q,K,V}\in\mathbb{R}^{T/L\times d_{k}}$ denote the encoder/decoder states in the self-attention sub-layers, or $\mathbf{Q}\in\mathbb{R}^{L\times d_{k}}$ denotes the decoder states, and $\mathbf{K,V}\in\mathbb{R}^{T\times d_{k}}$ denote the encoder states in the cross-attention sub-layers, given $d_{k}$ as the attention dimension and $T$, $L$ as the length of the encoder and decoder states, respectively.

The power of the Transformer architecture also arises from the utilisation of multi-head attention at each sub-layer, where the heads project raw signals into various feature spaces, so as to capture the sequence dependencies from various aspects:

where $\mathbf{W}^{Q,K,V}_{h}\in\mathbb{R}^{d_{k}\times d_{k}}$ and $\mathbf{W}^{O}\in\mathbb{R}^{d_{m}\times d_{m}}$ represent the projection matrices, and $H$ is the number of attention heads given $d_{m}=H\times d_{k}$.

With regard to streaming ASR systems, the main challenge for Transformer is that it requires the full speech to perform the attention computations. At the encoder side, a relatively straightforward approach to overcome this issue is to splice the input features into overlapping chunks, and feed them to the system sequentially [17]. However, such a strategy cannot be applied to the decoder side, because the attention boundaries of the ASR outputs might not be restricted within a single chunk. As a result, several online attention mechanisms have been proposed. Monotonic chunkwise attention (MoChA) was one of the first methods to address the problem. During decoding, the attending decision is made monotonically for each encoder state. Once a certain timestep is attended, a second-pass soft attention is conducted within a small chunk to serve as the layer outcome. Head-synchronous decoder-end adaptive computation steps (HS-DACS) was later proposed to take advantage of the history frames and better handle the distinct performance of the attention heads. The halting probabilities were produced and accumulated along the encoder states across all heads, and the output is triggered when the accumulation reaches a threshold. In the next section, we’ll present the cumulative attention (CA) algorithm, which incorporates both the acoustic-aware decision of MoChA, and the head-synchronisation ability of HS-DACS.

According to the investigation presented in [15], the lower Transformer decoder layers tend to capture noisy and invalid attentions, which does harm to the general decoding especially for streaming scenarios. Similarly, we adopt the layer-drop strategy by applying the proposed CA algorithm just to the cross-attention module of the top decoder layer. Meanwhile, the rest of the layers are only equipped with the self-attention module and perform language modelling. The workflow of the algorithm for a certain decoding step $i$ is described as follows.

Like all other online attention mechanisms, at head $h$, for each encoding timestep $j$, an attention energy $e^{h}_{i,j}$ is computed given the last decoder state $\bm{q}^{h}_{i-1}$ and the encoder state $\bm{k}^{h}_{j}$:

The energy is immediately fed to a sigmoid unit to produce a monotonic attention weight:

The sigmoid unit is regarded as an effective alternative to the softmax function in the streaming case, which scales the energy to the range (0, 1) without accessing the entire input sequence. As opposed to MoChA and HS-DACS where the outcome of eq. (5) is directly interpreted as the attending/halting probability that dictates the output triggering, CA interprets it as the relevance of the encoder state to the current decoding step. This is the same as the standard attention mechanism in the offline system or the second-pass attention performed in MoChA.

Next, an interim context vector is generated at $j$ in an autoregressive manner:

which carries all the processed acoustic information accumulated at the current timestep (the term $\bm{c}^{h}_{i,j-1}$ is discarded when $j=1$). Though HS-DACS performs accumulation at each decoding step in a similar way, it is with respect to the halting probabilities instead of acoustic information.

Inspired by HS-DACS, in order to force all the attention heads to halt at the same position, the interim context vectors produced by different heads are concatenated into a comprehensive one:

We introduce a trainable device, referred to as Halting Selector, to determine whether to trigger the ASR output at each timestep, which is implemented as a single/multi-layer deep neural network (DNN) with the output dimension of one in our system. Then $\bm{c}_{i,j}$ is input to the DNN to calculate a halting probability:

which represents the likelihood of halting the $i^{th}$ decoding step at timestep $j$, provided the acoustic features accumulated so far by all the attention heads. The parameters $r$ denotes a bias term that is initialised to -4, and $\epsilon$ is an additive Gaussian noise applied only to training in order to encourage the discreetness of $p_{i,j}$.

Here, one may notice the major difference between CA and other streaming methods. In CA algorithm, the interim context vectors are computed first, based on which the ASR outputs will be activated. Whereas for MoChA and HS-DACS, the attending decision is taken first and then the final context vector is produced. If the decision is made inappropriately, then the corresponding context vector can contain invalid acoustic information for the decoder.

Since the halting selector assigns hard decisions, making the system parameters non-differentiable, the training requires to take the expected value of $c_{i}$ by marginalising all possible halting positions. Hence, a distribution of the halting hypotheses is given as:

which is a simplified version of the counterpart calculated in MoChA [13]. Finally, the expected context vector for the decoding step $i$ is computed as:

It is important to note that HS-DACS doesn’t calculate such an expectation in training, as the accumulation is truncated by a preset threshold and only the one-best context vector is derived.

During CA inference, $p_{i,j}$ is monotonically computed at each timestep from $j=1$, and the decoding step $i$ would be halted at the earliest $j$ where $p_{i,j}\geqslant 0.5$. The corresponding interim context vector $\bm{c}_{i,j}$ is selected and sent to the output layer to predict the ASR output. The pseudocodes of the above inference process are presented in algorithm 1.

One should be aware that although the CA algorithm adopts the Bernoulli sample process at single timesteps similar to what MoChA does, the halting decision is based on the whole encoding history. Unlike MoChA where individual heads detect separate halting positions, in CA all the attention heads simultaneously contribute to $\bm{c}_{i,j}$ and have a unified halting position. This makes it less vulnerable to the disturbance in single frames. Also, CA rules out the use of arbitrary accumulation threshold as does in HS-DACS, allowing the decoding to be halted flexibly in the weak attention scenarios.

The paper presented a novel online attention mechanism, known as cumulative attention (CA) for streaming Transformer ASR. Combining the advantages of the acoustic-aware method (MoChA) and the accumulation based approach (HS-DACS), the CA algorithm utilised an trainable device called halting selector to determine robust halting positions to trigger the ASR output. The attention heads in the CA layer were synchronised to produce a unified halting positing in the decoder layer. By doing so, all the heads simultaneously contribute to the potential ASR output, so that the issues caused by the distinct behaviours of the heads were effectively eased off. Experiments on AIShell-1 and Librispeech showed that the proposed CA approach achieved similar or better ASR performance compared to existing algorithms in literature with significantly noticeable gains in latency during inference.