Contextualized Automatic Speech Recognition
with Dynamic Vocabulary

The audio encoder comprises two convolutional layers, a linear projection layer, and $M_{\text{a}}$ conformer blocks [6]. The convolutional layers subsample an audio feature sequence $\bm{X}$, and the conformer blocks then transform the subsampled feature sequence to a $T$-length hidden state vector sequence $\bm{H}=[\bm{h}_{1},\cdots,\bm{h}_{T}]\in\mathbb{R}^{d\times T}$ where $d$ represents the dimension:

$$\bm{H}=\mathrm{AudioEnc}(\bm{X}).$$

(1)

Each Conformer block has two feedforward layers, a multiheaded self-attention layer, a convolution layer, and a layer-normalization layer with residual connections.

Given $\bm{H}$ generated by the audio encoder in Eq. (1) and the previously estimated token sequence $y_{0:i-1}=[y_{0},\cdots,y_{i-1}]$, the decoder with an embedding and output layer estimates the next token $y_{i}$ recursively as follows:

$$P(y_{i}|y_{0:i-1},\bm{X})=\mathrm{Decoder}(y_{0:i-1},\bm{H}),$$

(2)

where $y_{i}$ is the $i$-th subword-level token in the pre-defined static vocabulary $\mathcal{V}^{\text{n}}$ of size $K$ ($y_{i}\in\mathcal{V}^{\text{n}}$). Note that $\mathcal{V}^{\text{n}}$ contains the blank token $\phi$ for RNN-T-based systems.

Specifically, the decoder comprises an embedding layer, a main decoder block (e.g., transformer blocks for attention-based systems and prediction/joint network for RNN-T-based systems), and an output layer. First, the embedding layer with positional encoding converts the input non-blank token sequence $y_{0:i-1}$ to an embedding vector sequence $\bm{E}_{0:i-1}=[\bm{e}_{0},\cdots,\bm{e}_{i-1}]\in\mathbb{R}^{d\times i}$ as follows:

$$\bm{E}_{0:i-1}=\mathrm{Embedding}(y_{0:i-1}).$$

(3)

Thereafter, $\bm{E}_{0:i-1}$ is input to the main decoder block with the hidden state vectors $\bm{H}$ in Eq. (1) to generate a hidden state vector $\bm{u}_{i}\in\mathbb{R}^{d}$ as follows:

$$\bm{u}_{i}=\mathrm{MainBlock}(\bm{H},\bm{E}_{0:i-1}).$$

(4)

For example, attention-based systems employ the transformer blocks, while RNN-T-based systems comprise a prediction network and a joint network for the main decoder block, respectively. Subsequently, the output layer calculates the token-wise score $\bm{\alpha}^{\text{n}}=[\alpha^{\text{n}}_{1},\cdots,\alpha^{\text{n}}_{K}]^{T}$ and the corresponding probability as follows:

$$\bm{\alpha}^{\text{n}}=\mathrm{Linear}(\bm{u}_{i}),$$

(5)

$$P\left(y_{i}\mid y_{0:i-1},\bm{X}\right)=\mathrm{Softmax}(\bm{\alpha}^{\text{n}}).$$

(6)

Here, the vocabulary size $K$ is pre-defined by a static token list. By repeating these processes recursively, the posterior probability of the token sequence is formulated as follows:

$$P(Y\mid\bm{X})=\begin{dcases}\prod_{i=1}^{S}P\left(y_{i}\mid y_{0:i-1},\bm{X}\right)&(\text{attention}),\\ \sum_{Z\in\mathcal{B^{\text{-1}}}(Y)}P(Z\mid\bm{X})&(\text{RNN-T}),\end{dcases}$$

(7)

$$P(Z\mid\bm{X})=\prod_{i=1}^{T+S}P\left(y_{i}\mid y_{0:i-1},\bm{X}\right).$$

(8)

Here, the attention decoder directly outputs the $S$-length non-blank token sequence $Y=(y_{i}\mid i=1,...,S)$, while the RNN-T decoder outputs the ($T+S$)-length alignment sequence $Z=(y_{i}\mid i=1,...,T+S)$. $\mathcal{B^{\text{-1}}}(Y)$ in the RNN-T-based systems is a set of all possible alignment sequences of $Y$. The model parameters are optimized by minimizing the negative log-likelihood described as follows:

$$L=-\log P(Y\mid\bm{X}).$$

(9)

The embedding and output layers in Eqs. (3) and (6) are expanded to the proposed DB method in Section 3.2.

Similar to [24], the bias encoder comprises an embedding layer with a positional encoding layer, $M_{\text{b}}$ transformer blocks, a mean pooling layer, and a bias list $B=\{b_{1},\cdots,b_{N}$}, where $b_{n}\in\mathcal{V}^{\text{n}}$ is the $I_{n}$-lengnth subword token sequence of the $n$-th bias phrase (e.g., [“N”, “el”, “ly”]). After converting the bias list $B$ into a matrix $\bm{B}\in\mathbb{R}^{I_{\text{max}}\times N}$ through zero padding based on the maximum token length $I_{\text{max}}$ in $B$, the embedding layer and the $M_{\text{b}}$ transformer blocks extract the high level representation $\bm{G}\in\mathbb{R}^{d\times I_{\text{max}}\times N}$ as follows:

$$\bm{G}=\mathrm{TransformerEnc}(\mathrm{Embedding}(\bm{B})).$$

(10)

Then, a mean pooling layer extracts phrase-level embedding vectors $\bm{V}=[\bm{v}_{1},\cdots,\bm{v}_{N}]\in\mathbb{R}^{d\times N}$ as follows:

$$\bm{V}=\mathrm{MeanPool}(\bm{G}).$$

(11)

To avoid the complexity associated with learning dependencies within the bias phrases, we introduce a dynamic vocabulary $\mathcal{V}^{\text{b}}=\{<b_{1}>,\cdots,<b_{N}>\}$ where the phrase-level bias tokens represent the bias phrases with the single entities for the $N$ bias phrases in the bias list $\bm{B}$. Unlike Eq. (2), the expanded decoder estimates the next token $y_{i}^{\prime}$ from expanded vocabulary $\mathcal{V}^{\text{n}}\cup\mathcal{V}^{\text{b}}$, i.e., $y_{i}^{\prime}\in\mathcal{V}^{\text{n}}\cup\mathcal{V}^{\text{b}}$ given $\bm{H}$, $\bm{V}$ in Eqs. (1) and (11), and $y^{\prime}_{0:i-1}$ as follows:

$$P(y^{\prime}_{i}|y^{\prime}_{0:i-1},\bm{X},\bm{B})=\mathrm{ExDecoder}(y^{\prime}_{0:i-1},\bm{H},\bm{V}),$$

(12)

where $y^{\prime}_{0:i-1}=[y^{\prime}_{0},\cdots,y^{\prime}_{i-1}]$ represents the expanded token sequence. For example, if a bias phrase “Nelly” exists in the bias list, the expanded decoder outputs the corresponding bias token [$<$Nelly$>$] rather than the decomposed normal token sequence [“N”, “el”, “ly”].

Similar to the conventional decoder described in Section 2.2, the decoder comprises an expanded embedding layer, a main decoder block, and an expanded output layer. First, the input token sequence $y^{\prime}_{0:i-1}$ is converted into the embedding vector sequence $\bm{E}^{\prime}_{0:i-1}=[\bm{e}^{\prime}_{0},\cdots,\bm{e}^{\prime}_{i-1}]\in\mathbb{R}^{d\times i}$. Unlike Eq. (3), if the input token $y^{\prime}_{i-1}$ is a bias token, the corresponding bias embedding $\bm{v}_{n}$ is extracted from $\bm{V}$ (Figure 1(b)); otherwise, the normal embedding layer is used with a linear layer as follows:

$$\bm{e}^{\prime}_{i-1}=\begin{cases}\mathrm{Linear}(\mathrm{Embedding}(y^{\prime}_{i-1}))&(y^{\prime}_{i-1}\in\mathcal{V}^{\text{n}})\\ \mathrm{Linear}(\mathrm{Extract}(\bm{V},y^{\prime}_{i-1}))&(y^{\prime}_{i-1}\in\mathcal{V}^{\text{b}}).\end{cases}$$

(13)

Subsequently, the main decoder block converts $\bm{E}^{\prime}_{0:s-1}$ into the hidden state vector $\bm{u}^{\prime}_{s}$ as in Eq. (4). In addition to the normal token score $\bm{\alpha}^{\text{n}}=[\alpha^{\text{n}}_{1},\cdots,\alpha^{\text{n}}_{K}]^{T}$ in Eq. (5), the bias token score $\bm{\alpha}^{\text{b}}=[\alpha^{\text{b}}_{1},\cdots,\alpha^{\text{b}}_{N}]^{T}$ is calculated using an inner product with two linear layers (Figure 1(c)) as follows:

$\displaystyle\bm{\alpha}^{\text{b}}=\frac{\mathrm{Linear}(\bm{u}^{\prime}_{i})\mathrm{Linear}(\bm{V}^{T})}{\sqrt{d}}.$

(14)

By concatenating the normal token score $\bm{\alpha}^{\text{n}}$ with the bias token score $\bm{\alpha}^{\text{b}}$, which results in $\bm{\alpha}=[\alpha^{\text{n}}_{1},\cdots,\alpha^{\text{n}}_{K},\alpha^{\text{b}}_{1},\cdots,\alpha^{\text{b}}_{N}]^{T}$, Eq. (6) can be expanded as follows:

$$P\left(y^{\prime}_{i}\mid y^{\prime}_{0:i-1},\bm{X},\bm{B}\right)=\mathrm{Softmax}(\mathrm{Concat}(\bm{\alpha}^{\text{n}},\bm{\alpha}^{\text{b}})).$$

(15)

Similar to Eqs. (7) – (9), the posterior probability and the loss function are formulated as follows:

$$P(Y^{\prime}\mid\bm{X},\bm{B})=\begin{dcases}\prod_{i=1}^{S^{\prime}}P\left(y^{\prime}_{i}\mid y^{\prime}_{0:i-1},\bm{X},\bm{B}\right)&(\text{attention}),\\ \sum_{Z^{\prime}\in\mathcal{B^{\text{-1}}}(Y^{\prime})}P(Z^{\prime}\mid\bm{X},\bm{B})&(\text{RNN-T}),\end{dcases}$$

(16)

$$P(Z^{\prime}\mid\bm{X},\bm{B})=\prod_{i=1}^{T+S^{\prime}}P\left(y^{\prime}_{i}\mid y^{\prime}_{0:i-1},\bm{X},\bm{B}\right),$$

(17)

$$L^{\prime}=-\log P(Y^{\prime}\mid\bm{X},\bm{B}),$$

(18)

where $Y^{\prime}$ and $Z^{\prime}$ represent the $S^{\prime}$-length non-blank token sequence and ($T+S^{\prime}$)-length alignment sequence based on the proposed dynamic vocabulary, respectively. Note that Eqs. (14) and (15) do not hold learnable parameters depending on the bias list size $N$, and can replace the bias list dynamically during inference. Also, the proposed method is optimized only with Eq. (18) without the auxiliary loss.

The proposed method can be easily applied to various E2E-ASR architectures (e.g., CTC, RNN-T, and attention), including streaming systems and multilingual systems [4, 5, 9, 34] without major modifications, because it only expands the embedding and output layers in addition to the bias encoder (Figure 2). Note that since CTC does not have the embedding layer and the main decoder block, only the output layer is expanded as described in Eqs. (14) and (15) using the hidden state vector $\bm{h}_{t}$ instead of $\bm{u}_{i}$ (Figure 2a).

Given the simplicity of the proposed method, the proposed method can also be applied to hybrid systems, such as [10, 35, 11, 12, 36, 37], by expanding the output layer of each branch. In this paper, the attention-based and RNN-T-based dynamic vocabulary models described in Sections 3.2 are trained with an auxiliary CTC loss, which is also based on the dynamic vocabulary, with the training weight $\lambda$ as follows:

$$L^{\prime}_{\text{joint}}=(1-\lambda)L^{\prime}+\lambda L^{\prime}_{\text{ctc}},$$

(19)

where $L^{\prime}_{\text{joint}}$ and $L^{\prime}_{\text{ctc}}$ represent loss functions for the joint model and auxiliary CTC decoder, respectively.

Moreover, the flexibility of the proposed method is preserved in joint decoding with multiple decoders [10, 11, 12, 38, 39, 40]. We adopt the joint decoding algorithms similar to [10, 12]. Specifically, the primary decoder (i.e., attention or RNN-T) generate the hypotheses and the scores of the hypotheses are augmented by the CTC decoder with the decoding weight $\gamma$ as follows:

$\displaystyle\begin{split}\beta_{\text{joint}}=(1-\gamma)\beta_{\text{}}+\gamma\beta_{\text{ctc}},\end{split}$

(20)

$$\begin{cases}\beta_{\text{}}=\text{log}P(Y^{\prime}\mid\bm{X},\bm{B})&(\text{attention/RNN-T})\\ \beta_{\text{ctc}}=\text{log}P_{\text{ctc}}(Y^{\prime}\mid\bm{X},\bm{B})&(\text{CTC}),\end{cases}$$

(21)

where $\beta_{\text{joint}}$, $\beta_{\text{}}$, and $\beta_{\text{ctc}}$ represent the scores of joint decoding, primary decoder, and CTC decoder, respectively.

During training, a bias list $\bm{B}$ is created randomly from the reference transcriptions for each batch, where $N_{\text{utt}}$ bias phrases are selected per utterance, each having a token length of $I$. This process yields a total of $N$ bias phrases calculated as $N_{\text{utt}}\times$ batch size. Once the bias list $\bm{B}$ is defined, the corresponding reference transcription $y_{\text{gt}}$ is modified to $y^{\prime}_{\text{gt}}$ based on the dynamic vocabulary. For example, if the phrase “N”, “el”, “ly” ($N_{\text{utt}}=1,I=3$) is extracted as a bias phrase from the reference transcription $y_{\text{gt}}$ = [“Hi”, “N”, “el”, “ly”], the reference transcription is modified to $y^{\prime}_{\text{gt}}$ = [“Hi”, $<$Nelly$>$].

Considering practicality, we introduce a bias weight to Eq. (15) to avoid over/under-biasing during inference:

$$\mathrm{WeightSoftmax}_{j}(\bm{\alpha},\bm{w})=\frac{w_{j}\mathrm{exp}(\alpha_{j})}{\sum_{l=1}^{(K+N)}w_{l}\mathrm{exp}(\alpha_{l})},$$

(22)

where $\bm{w}=[w_{1}.\cdots,w_{(K+N)}]^{T}$ and $j$ represent a weight vector and its index for $\bm{\alpha}=[\alpha^{\text{n}}_{1},\cdots,\alpha^{\text{n}}_{K},\alpha^{\text{b}}_{1},\cdots,\alpha^{\text{b}}_{N}]^{T}$, respectively. The same bias weight $\mu$ is applied to the bias tokens as follows:

$$w_{j}=\begin{cases}1.0&(j\leqq K)\\ \mu&(j>K),\end{cases}$$

(23)

if $\mu<1.0$, the bias tokens are underweighted compared to the normal tokens; otherwise, the bias tokens are overweighted compared to the normal tokens.

The input features are 80-dimensional Mel filterbanks with a window size of 512 samples and a hop length of 160 samples. Subsequently, SpecAugment is applied. The audio encoder comprises two convolutional layers with a stride of two and a 256-dimensional linear projection layer followed by 12 conformer layers with 1024 linear units and layer normalization. For the streaming RNN-T model, the audio encoder is processed blockwisely [41] with block size and look ahead of 800 and 320 ms, respectively. The bias encoder has six transformer blocks with 1024 linear units. Regarding the expanded decoder, the offline CTC/attention model has six transformer blocks with 2048 linear units, and the streaming RNN-T model has a single long short-term memory layer with a hidden size of 256 and a linear layer of 320 joint sizes for prediction and joint networks. The attention layers in the audio encoder, bias encoder, and expanded decoder are four multihead attentions with a dimension $d$ of 256.

The offline CTC/attention and streaming RNN-T models have 40.58 M and 31.38 M parameters, respectively, including the bias encoders. The training weight $\lambda$ in Eq. (19) is 0.3 for the CTC/attention and RNN-T models. The decoding weight $\gamma$ in Eq. (20) is 0.3 and 0.1 for the CTC/attention and RNN-T models, respectively. The bias weight $\mu$ in Eq. (23) is set to 0.8 and 0.01 for the CTC/attention and RNN-T models, respectively (this is discussed further in Section 4.4). During training, a bias list $\bm{B}$ is created randomly for each batch with $N_{\text{utt}}$ = [2 - 10] and $I$ = [2 - 10] (Section 3.4). The proposed models are trained for 150 epochs at a learning rate of 0.0025 and 0.002 for the CTC/attention-based and RNN-T-based systems, respectively.

The Librispeech-960 corpus [42] is employed to evaluate the proposed method using ESPnet toolkit [43]. The proposed method is evaluated in terms of the word error rate (WER), bias phrase WER (B-WER), and unbiased phrase WER (U-WER) as in [26]. The static vocabulary size $K$ is 5000, while the dynamic vocabulary size $N$ ranges from 0 to 2000.

Table 1 shows the results of the offline CTC/attention-based systems obtained on the Librispeech-960 dataset for different bias list sizes $N$. With a bias list size of $N>0$, the proposed method improves the B-WER considerably despite a slight increase in the U-WER, resulting in a substantial improvement in the overall WER. While the B-WER and U-WER tend to deteriorate with larger $N$, the proposed method remains superior to other DB techniques across all bias list sizes. In addition, the proposed method shows significant B-WER improvement for unseen words in the training data. Specifically, the baseline B-WER for unseen words in the test-other set is 73.5%, whereas the proposed method improves the B-WER to 19.0% when the bias list size is $N=1000$.

Figure 3 shows an example of the cumulative log probability described in Eq. (16), where the blue and red lines indicate the results obtained with and without the bias tokens. Without using the bias tokens, the model struggles to capture the subword dependencies, resulting in significantly lower scores for each subword. Conversely, the proposed method assigns a high score to the bias token ($<$Nelly$>$), improving the B-WER (Table 1). Interestingly, the log probabilities before and after the bias token (“fresh” and “is”) remain stable, even though the bias tokens are created dynamically during inference. This indicates that the proposed method preserves the context with non-bias tokens while eliminating the need to learn subword dependencies within the bias phrases.

Figure 4 shows the effect of the bias weights $\mu$ (Section 3.5) on the WER, U-WER, and B-WER results for $N$ = 2000. Increasing the bias weights $\mu$ improves the B-WER but deteriorates U-WER due to the tendency for overbiasing. Under this experimental condition, there is a slight tendency for overbiasing when no bias weights are introduced; thus, when $\mu=0.8$, the overbiasing can be suppressed. The degree to which the model is biased depends on the target user domain; thus, we believe that this mechanism adjusting the bias weights easily during inference is effective.

We validate the proposed method using our in-house Japanese dataset, comprising the Corpus of Spontaneous Japanese (581 h) [44], 181 h of Japanese speech from the database developed by the Advanced Telecommunications Research Institute International [45], and 93 h of our in-house Japanese speech data. The CTC/attention-based system described in Section 4.1 is used in this experiment. Table 2 shows the results in terms of character error rate (CER), B-CER, and U-CER, with the bias list provided by our end users containing $N$ = 203 technical terms. The proposed method significantly improves the B-CER with a slight degradation in U-CER, thereby resulting in the best overall CER.

Figure 5 shows the typical inference results, where the characters in boldface, red, and blue represent the bias phrases, incorrectly, and correctly recognized characters, respectively. As discussed in Section 4.3, the conventional DB method [24] struggles to capture the subword dependencies, especially in Japanese ASR, which operates at the character level, leading to longer subword sequences for bias phrases. In contrast, the proposed method avoids this problem by introducing the dynamic vocabulary where the bias token represents an entire bias phrase within a single token.

Table 3 shows the results of the streaming RNN-T-based systems with a bias list of size $N$ = 100, and 1000. The asterisk (*) indicates the use of external text data for model training (B1-B3). We apply LM shallow fusion to the proposed method for fair comparison. Note that bias tokens are decomposed into static subword token sequences before shallow fusion because the LM is not based on the dynamic vocabulary. B1 and B2 incorporate the DB-based neural LM and unified speech-to-text representation (USTR), respectively [26, 27].

Consistent with the results from the offline CTC/attention-based system, the proposed method significantly improves the B-WER without relying on additional information, such as phonemes, with better overall WER than conventional DB methods (A1-2 vs. A3). The conventional DB methods [26, 27] considerably improve the B-WER by learning subword dependencies within the bias phrases using external text data (A1-2 vs. B1-2). In contrast, the proposed method eliminates this need by introducing the bias tokens. In addition, the proposed method with the external LM performed comparably to conventional DB methods (B3 vs. B1-2), although its main advantage is simplicity and high DB performance (B-WER) without relying on external text data.