Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition ^††thanks: ${*}$ Corresponding Author: Jianzong Wang, [email protected]

Recent works [23, 24, 25] based on the attention function propose to use sparse normalizing such as sparsemax [19]. The sparsemax can omit items with small probability and thus can distribute the probability mass on outputs with higher probabilities. It improves the performance and interpretability to a certain extent [20]. However, each head of the multi-head attention mechanism used in Transformer is designed for capturing characteristics from a different point of view. For example, while the heads at the first layer of encoder focus on feature extraction, the heads in the last layer of decoder focus on classification of results. Intuitively the attention function of the two scenarios should be different. So the method of replacing all attention mechanisms with sparsemax may not suit for Transformer [26]. In the field of NLP, there are many breakthroughs in Transformer regarding sparsity. From the perspective of Tsallis entropy [27], [28] unify all transformation functions into the form of $\alpha$-entmax. When $\alpha$ = 1, $\alpha$-entmax is a softmax function. For any $\alpha$ $>$ 1, $\alpha$-entmax is the sparsemax function. Moreover, in order to make the $\alpha$-entmax fit different heads for different tasks, [21] introduces an adaptive method to enable the parameter $\alpha$ to be updated.

Monotonic attention technique [29] aims to restrict the transition of attention in a past-to-now pipeline. Generally speaking, monotonic attention is adopted to reduce the complexity of decoding. In strong context-related tasks as natural language understanding, a fully attention function is commonly applied to both input and output sequences, because the word-order or the part-of-speech is less important. But it may not work well for ASR tasks because the speech waveforms and target text sequence are required to be temporally monotonic. For enhancing the alignment ability between the waveform and text sequence, [30] incorporate the monotonic attention together with the connectionist temporal classification objective in a multi-task learning diagram by using the same encoder together. This recipe becomes the standard optimizing pipeline for most attention-based models [31, 32, 33]. In [34], a monotonic chunkwise attention (MoChA) approach was introduced for a streaming attention. It first splitted the encoder output latents into rather short-length chunks and then applied the soft attention function on these small chunks. Furthermore, an adaptive-length chunk approach was proposed in [35]. Specifically, monotonic infinite lookback (MILK) attention [36] was applied to consider the overall acoustic features preceding to learn an adaptive latency-quality trade-off schedule jointly.

Although attention-based architectures[37, 4, 5] have achieved remarkable success in offline end-to-end ASR systems, they cannot easily be applied for the online or streaming ASR systems. Several optimization paradigms are proposed to overcome this limit. Monotonic multi-head attention[38] extends monotonicity to the multi-head to extract useful representations and complex alignment. For Monotonic multi-head attention [38], HeadDrop[39] is proposed to eliminate the redundant head. MoChA [34] bridges the gap between monotonic attention and soft attention. As the online speech recognizing becoming more and more important, it also brings a drawback of additional designings in the network and training criterias, and precision degradations. As the Transformer’s size is large, a compressed structure [40] is proposed. Conformer [41] is a convolution-augmented Transformer for speech recognition. In triggered attention [42], a CTC task is trained to learn the alignment that triggers the attention decoding. The work in [43] use chunk-hopping for a latency-control end-to-end ASR.

As shown in Figure 1, our network architecture is based on a Transformer-based ASR model, based on an encoder and a decoder. Let us denote the input sequence as $x=\{x_{1},...,x_{T}\}$ with T being the length of the frame sequence. The corresponding encoder states are denoted as $h=\{h_{1},...,h_{S}\}$. Let the output sequence as $y=\{y_{1},...,y_{U}\}$ with $U$ being the length of the character sequence. The encoder first processes the input with a down-sampling convolution layer and stacks with several encoder blocks. The encoder processes sequential speech features, which are 80-dimensional log-mel spectral energies. The down-sampling convolution layer uses a stride of 2, kernels of $3\times 3$, followed by the ReLU activation function. The down-sampling convolution is able to extract useful encodings, shorten acoustic representations’ length, and promote faster alignments in the decoding. Main modules of Transformer contain position encoding, multi-head self-attention calculation, linear, and softmax, point-wise feed-forward network, residual connections, layer normalization. Here, we emphasize the concept of self-attention and multi-head attention. Self-attention is designed to learn the internal dependencies by computing the representation of a single sequence. The Transformer employs the scaled dot-product self-attention formulated as:

Here, $h$ means the number of heads of multi-head attention. As the dimension of $X$ in Transformer has the same dimension as $d_{model}$, the project matrices are $W_{i}^{q}\in\mathbb{R}^{{d_{model}}\times d_{q}}$, $W_{i}^{k}\in\mathbb{R}^{d_{model}\times d_{k}}$, $W_{i}^{v}\in\mathbb{R}^{d_{model}\times d_{v}}$. Note that $d_{q}=d_{k}=d_{v}=d_{model}/h$. The output of self-attention $O$ is concatenated by multi-head mechanism, which is formulated as follows

$W^{O}\in\mathbb{R}^{hd_{v}\times d_{model}}$ and $concat(\cdot)$ means concatenation. Besides, it deploys the sinusoidal positional encoding scheme, which achieves the attention calculation to adapt well on different speech length.

As shown in Figure 2, we introduce several sparse functions in self-attention to prevent the disadvantage of softmax. The first idea is to substitute softmax with sparsemax for every independent head of the Transformer, which can totally omit the attention of low probability and distribute probability mass only on high-interest elements. It is achieved by computing the Euclidean projection of the input vector z onto the probability simplex p:

where $\Delta^{J}={\textbf{p}\in R^{J}|\textbf{p}\leq 0,\sum\limits_{j}^{J}p_{j}=1}$ is J-dimension simplex. To overcome the potential issue of softmax, we propose to treat $\alpha$ values of each attention head differently, where some heads may train to be informative sparser, and others may become similar its original form. We also propose to treat the $\alpha$ values as learnable variables, optimized via back-propagation calculations along with other network parameters. In [19], the sparsity of sparsemax is proved and a closed-form solution is given:

where $[\cdot]_{+}$ denotes function $max(\cdot,0)$, and $\tau$ is the Lagrange multiplier in the context of the constraint $\sum_{j}(p_{j})=1$. Here $\tau$ can also be interpreted as a threshold under which the corresponding probability will be set to zero. Next, we extend the concept of sparse attention in the multi-head architecture. The core change of this architecture is we use $\alpha\text{-entmax}$ [28] to introduce sparsity into attention mechanism of sparse self-attention. For the three mentioned transformations (softmax, sparsemax, $\alpha$-entmax), the relationship among them can be analyzed from the perspective of entropy:

$H_{\alpha}^{T}(\textbf{p})$ is the Tsallis $\alpha$-entropy family definition with the following definition:

$\alpha$ here is a hyperparameter. For the cases $\alpha=1$ and $\alpha=2$, the corresponding $\alpha$-entropies are Shannon and Gini entropy while the corresponding $\alpha$-entmaxes are equal to softmax and sparsemax respectively. Thus it establishes the connection between softmax and sparsemax while also throwing lights on a medium model between these two transformations. [28] proves the uniqueness of $\tau$ and proposed an accurate method for calculating $\alpha$-entmax when $\alpha$ = 1.5 (also called 1.5-entmax). [44] proposed an extended iterative bisection approach that is applicable for $\alpha$-entmax. It is rational to set the $\alpha$ to different values in Transformer for better adaptation to different heads in different layers.

Inspired by [21], we apply an adaptive $\alpha$-entmax attention method in the multi-head attention mechanism for ASR tasks. $\alpha$-entmax is a convex optimization problem. When $\alpha$ is set as a variable but not a hyperparameter, we have $p^{*}=\alpha-entmax(\textbf{z},\alpha)$. It is not trivial to derivate $\frac{\partial\alpha-entmax(\textbf{z},\alpha)}{\partial\alpha}$ from the Lagrangian and optimality conditions by taking the derivative with respect to z in [19]. Due to current deep neural network frameworks can not take the derivatives for this optimization problem automatically. Thus, we tackle this problem from the perspective of the solution. It is trivial that for $j\notin S$, $p^{*}_{j}=0$ which is a constant thus $\frac{\partial p^{*}_{j}}{\partial\alpha}=0$. To simplify the gradient for $i\in S$, we use $\overline{p}^{*}$ and $\overline{\textbf{z}}$ to denote the corresponding vectors whose indices are in the support $S$. Finally we can solve the component of the Jacobian.

For end-to-end ASR, another fundamental challenge is the alignment between the input sequence and output sequence. CTC [1] and RNN-Transducer [2] are monotonic, so that the alignment track is able to be marginal. But the attention approach results in non-monotonic alignment, which may diminish the quality of speech recognition. The hard monotonic attention mechanism was introduced used for streaming time-sync decoding with attention architecture ASR. For the hard monotonic attention mechanism, the process is shown as follows. For the decoding time-step i, the attention method begins inspecting entries starting at the preceding output time-step, referred to as $t_{i-1}$. It then computes an energy scalar $e_{i,j}$ based on a monotonic energy function $MonotonicEnergy(\cdot)$. This function refers the $(i-1)_{th}$ output state and encoding features $h_{j}$ as inputs, where $j=t_{i-1},t_{i-1}+1,...$. The energy scalar is computed as follows:

The detail of monotonic energy function can refer to [38]. Then the energy values $e_{i,j}$ are passed into a sigmoid function $\sigma(\cdot)$ to produce selection probability $p_{i,j}$ formulated as

Then, the selection probability $p_{i,j}$ is used to produce a discrete decision variable $z_{i,j}$, which is formulated as

Whenever $p_{i,}$ is satisfied, the $z_{i,j}$ is stimulated (i.e., value approaching 1). Once $z_{i,j}=1$, the model stops and set $t_{i}=j$ and $c_{i}=h_{t_{i}}$.

In contrast to a single attention head in the recurrent sequence-to-sequence model, multi-head mechanism is capable of learning contexts with diverse paces between input and output sequence. Besides, the multi-head can learn complex alignment. In specific, the results of each head can be regarded as corresponding to some specific frames in the speech, and different heads are related to each other. While some heads can detect boundaries, other heads can capture strong intrinsic speech structures.

Dense attention mechanisms can fail to capture or utilize the strong intrinsic structures present in speech. By taking advantage of the dependencies between specific frames that different heads focus on, we can emphasize the key information in the speech and highlight the characteristic representation of these speech frames. However, there is no gurantee that multiple heads are all contributive to the overall context learning. Besides, the monotonic multi-head alignment can prevent delayed state decoding caused by multi-heads for boundary detection. Every monotonic attention head must learn alignments properly. It is necessary to prune some redundant monotonic attention heads. Thus, to enhance accordance among multi-heads on token boundary discovery, we use L1 regularization to eliminate information-redundant heads in shallow layers.

The results are shown in Table 1. When only sparse transforms are added to the Transformer encoder and decoder, both have better performance. Among them, adding to both the encoder and decoder have achieved better results. We speculate that this is because the sparse transform enhances the discrimination of the acoustic features, and the discrimination of the text information is less important than the comparison. In addition, the model that completely replaces the sparse transform achieves the best results. The method of sparsity is significantly improved compared to the method of changing the softmax temperature. Low softmax temperatures can make the distribution of softmax results steeper, thereby highlighting the expression of some information, but the non-negative output of softmax is still not changed, which still leads to waste of attention. It is worth noting that a too low temperature will cause the model to focus more on a certain part of the information and ignore other important information, leading to a decline in the results.

In self-attention, replacing the softmax function with adaptive $\alpha$-entmax all reduces WER, which shows that adding sparseness is beneficial to the performance of the model. However, sparsemax caused a decrease in performance, which shows that excessive sparseness can cause loss of information. This demonstrates that excessive sparseness cannot highlight the key information in the speech, and only a properly designed adaptive sparse can effectively highlight the key information. We combine the adaptive $\alpha$-entmax in self-attention and report the best result in Table II, III.

In this section, we retain the monotonic attention and study the effect of different sparse transformations in the sparse self-attention. In Table 1, we keep the self-attention module in its most original state (softmax) and introduce it into the cross-head attention block for experiments. The third to sixth rows in Table 1 represent different max functions in the cross-head attention block. Due to the small number of attention heads, too sparse max function (sparsemax) will lead to the loss of information. The max function with insufficient sparsity does not play a sparsity role. It is worth mentioning that the adaptive $\alpha$-entmax function tends to $\alpha$=1. To further investigate the proposed method on more challenging data sets, we also conduct ablation studies on English LibriSpeech dataset in Table IV. In Figure 3, we pose an example with different sparse transformations in sparse self-attention. Compared with softmax, the sparsemax function can produce competitive sparse output, but it will miss some output information. 1.5-entmax can produce more complete output results, thus achieving better performance. $\alpha$-entmax enables each self-attention module to use a specific $\alpha$ according to different emphases, thus obtaining the best performance. We can see that $\alpha$-entmax does not only avoid the waste of attention scores, but also effectively highlights some important frames in speech. The deepening of the color means that it has a higher attention score.

In Figure 4(a), we tracked the change of the $\alpha$ parameter. It can be seen that at the beginning of the training, the $\alpha$ parameter has decreased to a certain degree, which shows that it is desirable to use a dense output similar to softmax in the early stage of training. Especially for the encoder part, the closer to the feature input, the closer the value of $\alpha$ is to 1, which shows that the softmax function can effectively enhance the distinguishability of features. After a certain amount of training, except for the first layer of the encoder, the other parts move closer to the sparse direction, which shows that the method of initializing $\alpha$ cannot determine the degree of sparseness in advance, and to some extent, it shows the method of fixing $\alpha$ as a learnable parameter is suitable for multi-head attention. In Figure 3(b), we visualized the final output attention score of the cross-head attention, and we can see that softmax generates a lot of low score attention due to lack of sparsity. The structure we proposed can produce more high scores to avoid the waste of attention scores.

In Table 1, comparing with the sparse attention alone, the preformance of sparsemax methods is improved by averaging -0.17% with the help of monotonic attention. Furthermore, the monotonic attention is also complementary to the sparsemax and entmax methods. Test results are enhanced by -0.13% CER on AISHELL and -0.07% WER on WSJ, respectively. Results show that the monotonic attention can cooperate with the sparse attention to improve the performance further.

We further experiment the L1 regularization on the monotonic attention. As shown in Table 5, with the help of the regularization, all max function methods gain improvements both on AISHELL-1 and WSJ. The gap between w/o reg. and the proposed can be reduced from 9.29% to 9.00% on AISHELL-1 and 9.43% to 9.36% on WSJ for original softmax version. We applied adaptive sparse where $\alpha$ leading to 1.0, the monotonic multi-head alignment can prevent delayed token generation caused by such heads for boundary detection.