END-TO-END CONTEXTUAL ASR BASED ON POSTERIOR DISTRIBUTION ADAPTATION FOR HYBRID CTC/ATTENTION SYSTEM

We use the hybrid CTC/attention ASR model as our baseline which utilizes both benefits of CTC and attention decoder during the training and decoding steps [4]. The multi-tasks learning framework is employed to improve robustness and achieve fast convergence in training stage. The monotonic alignment characteristic of CTC relieves the burden of estimating the desired alignment by attention decoder. The loss function is described as

$$L_{mtl}=\lambda\mathrm{log}p_{ctc}\{y|X\}+(1-\lambda)\mathrm{log}p_{attn}\{y|X\}$$

(1)

where $X=\{x_{1},...,x_{T_{src}}\}$ and $y=\{y_{1},...,y_{T_{tgt}}\}$ denote the feature sequence and target sequence respectively. $\lambda$ is a tunable parameter, which satisfies $0\leq\lambda\leq 1$.

Because the attention decoder performs decoding in label steps while CTC performs it in frame steps, the model computes the probability of each partial hypothesis using CTC and attention model respectively and incorporates both probability scores in beam searching. The attention decoder is supposed to decode from starting symbol, ¡$sos$¿ and top K units in posterior probabilities are selected to construct partial hypotheses. Then the CTC prefix probabilities can be calculated for a given partial hypothesis $h$ as:

$$p_{ctc}(h,...|X)=\sum_{v}p_{ctc}(h\cdot v|X)$$

(2)

where $v$ denotes all possible label sequences except the empty string. Then the CTC prefix scores is combined with $p_{attn}(h|X)$ like in (1).

An effective method that integrates contextual information into the E2E modeling is called CLAS [14], which is an all-neural structure based on Listen, Attend and Spell (LAS) model. A list of additional bias phrases is given, denoted as $z=\{z_{1},...,z_{N}\}$, and a bias-encoder module is designed to embed each phrase into a fixed dimensional representation $h^{z}=\{h^{z}_{0},...,h^{z}_{N}\}$. We include $h^{z}_{0}=h^{z}_{nobias}$ here which corresponds to the no-bias option. And then a bias attention is computed over $h^{z}$ where the decoder hidden state $d_{t}$ is utilized as attention-query like the one used in audio attention. Given audio and previous labels, CLAS explicitly models the probability of seeing particular phrases:

$$\alpha_{t}^{z_{i}}=P(z_{i}|X;y_{<t})$$

(3)

where $t$ denote the decoding step, $i$ denote the index of bias phrases.

A concatenation of audio context vector and bias context vector, $c_{t}=[c_{t}^{x};c_{t}^{z}]$, is then fed into the LAS decoder as usual. By this way, the bias context information can influence the output predictions:

$$P(y_{t}|y_{<t};X;z)=\mathrm{softmax}(\mathbf{W_{s}}[c_{t}^{x};c_{t}^{z};d_{t}]+b_{s})$$

(4)

And it is up to the model to determine which bias phrase might be relevant in decoding steps and the target distribution would be modified.

The overall structure of the proposed model is shown in Fig.1. We add a bias attention component compared to the standard conformer-transformer structure. The bias encoder consists of a single lstm module which use the last state of the lstm as the fixed dimensional embedding of the entire phrase. Unlike [14], we employ a simple scaled dot-product attention [17] between query and key where query denotes the context vector of source attention in the last layer of decoder and key denotes the output of the bias encoder:

$$\alpha_{t}^{z_{i}}=\mathrm{softmax}((\mathbf{W_{q}}c^{N_{d}-1}_{t})(\mathbf{W_{k}}h^{z}_{i})/\mathrm{sqrt}(d_{model}))$$

(5)

where $\mathbf{W_{q}}$ and $\mathbf{W_{k}}$ denotes a linear transformation of query and key respectively, $d_{model}$ denotes the dimension of key and $N_{d}$ denotes the num of layers in decoder. Bias attention is supposed to extract the potential bias phrase from a bias list in each label step according to the attention probability calculated. In our method, we generate a bias label in each label step and then the bias loss is calculated on the attention distribution. We discuss this in the next subsection.

In the training stage, we first need to create a bias list $z$ for each training batch. As our purpose is to improve the recognition accuracy of the proper nouns, we can naturally employ the named entity recognition annotator to label entities for reference transcripts associated with the training batch. In the following experiments, we utilize the stanford coreNLP pipeline [12] as the extractor for NER which run several named entity recognizers and combine their results. However, only part of the reference transcripts contain named entities, and we process the rest reference transcripts by randomly selecting $n$-grams like the way used in [14]. Therefore, a bias list $z$ will contain both the real named entities and the randomly selected word $n$-grams. Specifically, we explain the bias list creation process as follows. For a given training batch, we apply NER on the reference transcripts, $r_{1},...,r_{Nbatch}$ and then split the references into two parts, one part has entity results and another one has not. As for the references without entities, $k$ word $n$-grams are randomly selected from each reference, where $k$ is selected uniformly from $[1,N\mathrm{phrases}]$ and $n$ is selected uniformly from $[1,N\mathrm{order}]$. Herein, $N\mathrm{phrases}$ denotes the maximal num of phrases can be selected in a reference and $N\mathrm{order}$ denotes the maximal order of a selected phrase. Take a reference with 10 words as example, where $ref=[w_{1},w_{2},...,w_{10}]$. When we set $N\mathrm{phrases}=2$ and $N\mathrm{order}=3$, we may select 2 bias phrases, one phrase $z_{1}=[w_{2}]$ and the other one $z_{2}=[w_{5},w_{6},w_{7}]$ with 1 order and 3 orders respectively. The total number of bias phrases in a training batch is a variable which depends on the randomly selecting process and can differ from other batches. Next, we need to create the bias label for each training reference which will be utilized for bias loss calculation. For a reference with bias phrases, we split the word sequence into word pieces sequence through BPE. For example, the first reference in a training batch ”call hanna phone” with bias phrase ”hanna” may be split into ”call han @@na phone”, and we label it by ”0 1 1 0”, where ”0” means id of nobias and ”1” means id of bias phrase in a bias list. Then, the second reference ”I will go to shanghai” with bias phrase ”shanghai” may be split into ”I will go to shang @@hai”, and we label it by ”0 0 0 0 2 2”. The rest references in the batch will be processed in the same way. The label id will indexed from ”1” to the length of the bias list. After bias attention is performed at each label step, we use the attention distribution and the bias label to calculate the loss of bias attention:

$$L_{bias}=-\sum_{t}\mathrm{log}\alpha_{t}^{z_{t}}$$

(6)

where $z_{t}$ denotes the bias label at time step $t$. Furthermore, we combine the standard CTC/attention loss and the bias loss as our optimization object in the training process:

$$L_{all}=-L_{mtl}+\beta L_{bias}$$

(7)

where $\beta$ denotes a tunable parameter which satisfies $0\leq\beta\leq 1$.

In the inference stage, we prepare the contextual phrases $z$ and send them to the bias encoder which output $h^{z}$ before decoding starts. During beam search, the CBA module calculate the correlation between $c^{N_{d}-1}_{t}$ and $h^{z}$ in each decoding step, and the biasing process is performed based on the bias phrase distribution as in Eq. (5). The phrase corresponding to the biggest probability index (except $h^{z}_{0}$ which means no bias index) will be biased. We perform boosting on both posteriors of attention decoder and CTC according to the selected biased phrase. First, the biasing score $bias\_score$ will be added on the linear output corresponding to the word piece units of the attended phrase before softmax applied to attention decoder, resulting in a boosted probabilities on these word piece units.

Next, we bias CTC posteriors corresponds to each encoded audio feature according to the source attention distribution in the last layer of attention decoder:

$$\alpha_{t}^{h^{x}_{i}}=P(h^{x}_{i}|X;y_{<t})$$

(8)

where $h^{x}$ denotes the encoded audio features and $i$ denotes the index of $h^{x}$. Distinguished from the bias mechanism of attention decoder, the bias score $bias\_score$ will be multiplied by the attention probability $\alpha_{t}^{h^{x}_{i}}$ before adding to the related word piece units of CTC linear output. In default, the bias score $bias\_score$ of attention decoder is consistent with the one in CTC. Consequently, both posteriors of CTC and attention decoder are biased toward the potential contextual phrases before pruning performed on the corresponding word piece units, which improves the recognition accuracy on bias phrases.

We evaluate our proposed method on GigaSpeech [18], an multi-domain English speech recognition corpus with 10,000 hours of high quality labeled audio. A variety of topics such as arts, science, sports are collected in the dataset. We apply NER of stanford coreNLP on the whole training transcripts and obtain approximate one million utterances with NER results, about one-eighth of the dataset. As for the training data without NER results, we randomly select word n-grams as described in Section 3.2. We set $N_{phrases}=2$ and $N_{order}=3$ which means $1\sim 2$ bias phrases each has $1\sim 3$ words can be selected in a specific utterance.

The evaluation sets in GigaSpeech consist of a dev set with 5715 utterances and a test set with 19930 utterances. We keep the dev and test set as general sets and do not apply NER to extract bias phrases. Instead, we construct bias test sets based on commonvoice training set [19] which is contributed by volunteers who record their voices online and is publicly available. After all training corpus of commonvoice is labeled by NER annotator, we select the NER results by four entity classes: person names, location, organization, and other, and the filtered utterances are collected as bias test sets,denoted by PER, LOC, ORG and OTHER. A summary of the four test sets is shown in Table 1.

In the experiments, we find some training batch with fewer bias phrases than others and we suggests adding some extra distractors to them. To examine the effect of distractors in bias training, we supplement extra n-grams when the extracted bias phrases num is less than 200. We count the words frequency in the training transcripts and exclude the top $20\%$ words and randomly select the remainder words as distractors.

The total number of utterances in commonvoice training set is around 564,000 while only less than 20,000 utterances contain entities in the above four classes and ”person names” occupies the highest proportion.

The model in our experiments consists of a conformer encoder with 12 blocks and a transformer decoder with 6 blocks. The full structure details can be found in the recipe of $\mathrm{"espnet/egs2/gigaspeech"}$ in espnet ASR toolkit [20]. However, we use 80-dimensional filter-bank features computed on 25ms window with 10ms shift instead of the raw-wav in the espent recipe. The bias encoder is a single LSTM layer with 512 nodes. The output dimension of trainable parameter matrices in the bias attention is equal to $d_{model}$. As our system belongs to the multi-task learning framework, we set $\lambda=0.3$ in (1) and $\beta=0.5$ in (7). During decoding, we exclude the influence of language model and only evaluate the performance of joint decoding on CTC and attention decoder.

We first train the baseline hybrid CTC/attention model based on conformer-transformer structure. Then, we use the baseline model as our initial model, and add our bias encoder and bias attention components on it and carry on bias training. We compare the WER of baseline model and our bias models in decoding with an empty list of bias phrases on a variety of test sets, from general test sets of GigaSpeech to bias test sets of commonvoice. To better evaluate the biasing performance, we utilize the recall rate of bias phrases as our measurement, which means we care about the percentage of recognized bias phrases in the test set. We show the WER and recall rate in Table 2. We find similar performance in three models when tested in bias-free mode whereas the WER on organization set degrades in bias model trained with distractors. This may be caused by the small scale of organization set as it only has 823 utterances. As for the general test sets of GigaSpeech, we only show the WER and find little performance degradation compared to baseline model.

Next, we study the performance of our bias models on bias test sets. We load the full bias list of each bias set and carry out bias testing except for the ”person names” set. We divide the ”person names” test set to five subsets, each loaded by the respective bias list. Because it may incur inconsistent phenomenon compared to the other three if we load all the 10524 bias phrases once in decoding. We tune the $bias\_score$ in biasing process and find similar performance, so we simply set $bias\_score=5$ in the following testing. We obtain consistent improvement on recall rate of bias phrases on three test sets, ranging from $15\%$ to $28\%$. As for the results of the five subsets of ”person names”, we find the recall rate improvement shrinks compared to the other three as the num of bias phrases is $2\sim 3$ times larger. And the bias model with distractors training always performs better than the model without distractors training. The results are shown in Table 3.

We proceed the experiment by increasing the nums of bias phrases to 1,000 of each bias set which are selected from the ”person names” bias set. Only the bias model with distractors training is examined. We achieve approximate results as the previous one, which means the bias model can adapt to a large number of phrases even the distractors are imported. The results are shown in Table 4.

Finally, we show the performance of false triggering of the bias model. To prove the strong anti-context ability, we load the bias phrases into the bias encoder and perform biasing when we decode general test sets. The loaded number of bias phrases ranges from 100 to 2000 which are picked up from ”person names” bias set who has the largest number of bias phrases. All the activation parameters are consistent with the previous experiments. The WER is shown in Table 5.

Although we observe gradual degradation in WER as a function of number of bias phrases, the performance is still acceptable when loaded num increases to 2000, with only $1.7\%$ relative WER reduction in giga_dev set.