IMPROVEMENTS TO EMBEDDING-MATCHING ACOUSTIC-TO-WORD ASR
USING MULTIPLE-HYPOTHESIS PRONUNCIATION-BASED EMBEDDINGS

We begin with a description of our baseline embedding-matching A2W system that is similar to a previously-reported “frozen prediction layer” system [5] where a matrix of embedding vectors obtained from an external text encoder is used to compute the pre-softmax scores of a word CTC model.

First, we show in Figure 1 how embedding-matching A2W can be interpreted. The system can be understood as a continuous-speech extension of the embedding-matching isolated word recognition experiments done in previous work [10]. We imagine a “word segmenter” model scanning a speech utterance to identify possible word segments in the audio. For each hypothesized segment, an audio encoder (usually denoted by $f(\cdot)$ [10, 11]) transforms the audio segment into an embedding vector $\mathbf{f}$ that represents the segment’s phonetic content. Separately, a text encoder (usually denoted by $g(\cdot)$) transforms a dictionary of words into a set of embedding vectors. The audio and text encoder are trained in tandem [10, 11] such that matching audio and text will map to the same embedding vector.¹¹1Note that the encoders must also be trained such that distances between embeddings reflect phonetic confusability as in [10]. A few easy examples can show us that random projections, for example, would not lead to proper recognition. Hence, if the hypothesized segment contains the audio for “weather”, its $\mathbf{f}$ vector will most closely match the column in $G$ that represents “weather”. Each word posterior is modeled as the similarity between ${\mathbf{f}}$ and the word’s embedding vector, so $\hat{P}(weather|X)$ gets a high score at this instance in time. A decoder would choose the high-scoring words to output a recognition result. In effect, the ASR is based on nearest-neighbor searches between the ${\mathbf{f}}$ vectors produced at different instances in time and the words in $G$.

When interpreted this way, the system is similar to embedding-matching whole-word segmental ASR [16], with one key difference: Instead of generating all embeddings for a range of different segments every time, the neural network makes “educated guesses” about prospective word segments and exposes only those guesses.

It is also apparent that $G$ can be arbitrarily changed to almost any vocabulary we wish. For instance, user-dependent contact names can simply be added as new columns to a general-vocabulary $G$. This is an extremely simple way of extending the ASR’s vocabulary, and also scalable because brute-force vector nearest-neighbor search is highly parallelizable and efficient in modern hardware [17].

In the actual system shown in Figure 2, no such “word segmenter” explicitly exists, nor is an $f(\cdot)$ audio segment encoder even needed during training. Only a $g(\cdot)$ text encoder is used to populate the $G$ matrix, and the underlying network (a conformer [18] in this work) is trained via a CTC criterion. During training, the conformer is forced to output embeddings that match the appropriate columns in $G$, effectively learning to play both the roles of word segmenter and audio encoder as in Figure 1.

In the baseline system in Figure 2, at every time $t$ the conformer outputs a scalar $b_{t}$ and an acoustic embedding vector ${\mathbf{f}}_{t}$, which result in posterior scores for a word vocabulary $\{w_{0},\cdots,w_{n}\}$, where $w_{0}$ denotes the conventional “blank” label [19]:

$$p_{t,i}=\hat{P}(w_{i}|X,t)\ \ \ (i=0,\cdots,n).$$

(1)

We use acoustic neighbor embeddings [10] – which were shown to match the accuracy of FST-based DNN-HMM ASR in isolated word matching – for our $G$ matrix. Since the embeddings are trained based on the Euclidean distance rather than cosine distance, we use the negative square of the $\mathcal{L}_{2}$ distance, rather than the inner product in previous studies [5, 11], between the audio embedding ${\mathbf{f}}_{t}$ and each word embedding $\mathbf{g}_{i}$ to represent their phonetic similarity. The similarity is maximized if and only if there is an exact match between the audio embedding and the text embedding, i.e., ${\mathbf{f}}_{t}=\mathbf{g}_{i}$. ²²2Note that, to achieve the same effect when using inner product, the embeddings must be constrained such that $||{\mathbf{f}}_{t}||=||\mathbf{g}_{i}||=c$ for some constant $c$. However, it is not clear whether the previous studies [5, 6] did this.

The pre-softmax output $\mathbf{s}_{t}$ stores the similarity scores at time $t$:

$$\textbf{s}_{t}=s({\mathbf{f}}_{t},G)=2G^{T}{\mathbf{f}}_{t}-{\mathbf{g}}^{\prime}-f_{t}^{\prime}\textbf{1},$$

(2)

where $s(\cdot,\cdot)$ is the negative $\mathcal{L}_{2}^{2}$ distance between a vector and every column of a matrix, ${\mathbf{g}}^{\prime}$ is a constant vector where every $i$’th element is ${\mathbf{g}}_{i}^{T}{\mathbf{g}}_{i}$, $f_{t}^{\prime}={\mathbf{f}}_{t}^{T}{\mathbf{f}}_{t}$, and $\mathbf{1}$ is a vector of 1s.

Previous studies [5] defined an arbitrary embedding for the “blank” CTC symbol. In our study, we avoid this by directly using one of the outputs $b_{t}$ of the underlying neural network as the “blank” score. Since each score in ${\mathbf{s}}_{t}$ is $\leq 0$ and essentially proportional to the square of each element of ${\mathbf{f}}_{t}$, we impose a similar restriction and “scaling” on $b_{t}$ by taking its negative square.

A crucial limitation to embedding-matching A2W is apparent in Figure 1, which is that the system can only hypothesize on one segment of the audio at any given point in time. For example, it is impossible for the system to simultaneously give high posteriors for both “weather” and “wet.” The reason is because “wet” has a shorter audio segment and different pronunciation compared to “weather”, and therefore its audio embedding will be different, but the audio encoder can only produce one embedding at a time.

To address this limitation, we propose a method of producing multiple embeddings at every point in time, and combining the resulting posteriors such that two words with different lengths and pronunciations may still simultaneously receive high scores. This system is shown in Figure 3. The final projection layer of the conformer is widened (the sizes of all other components remain the same) to produce two embeddings ${\mathbf{f}}_{t,1}$ and ${\mathbf{f}}_{t,2}$ at every time $t$. The scores against $G$ are individually computed, and then summed, simulating a “logical OR” operation where both ${\mathbf{f}}_{t,1}$’s matches against $G$ and ${\mathbf{f}}_{t,2}$’s matches against $G$ can be outputted by the system. One may also interpret this as a simplified mixture density network [20].

An example is shown in Figure 4, for the input audio “call Koussevitzky.”³³3We randomly chose the name of this musician as an example. In the single-hypothesis system in Figure 4(a), we can see that the A2W has essentially limited itself to one fixed sequence of three audio segments corresponding to “call”, “Coosa”, and “Visky”. Other words like “Kuscevic,” which overlaps with both “Coosa” and “Visky” only have weak scores. The faint score for “Kuscevic” is likely just a by-product from “Kuscevic” having a similar starting sound (and therefore a somewhat-close embedding) as “Coosa.”

In the multiple-hypothesis system (3 embeddings) in Figure 4(b), we see a more pronounced scores for a diverse set of segments. “Kuscevic” has a strong score, likely because the full length of its audio segment was actually hypothesized by the model. The same reasoning can be applied for the correct word “Koussevitzky.”

Even though both the single-hypothesis and multiple-hypothesis networks have almost the same number of parameters (see Table 1) and are trained on the same data, we can get an improvement in versatility by internally restructuring the network in this manner.

In a previous study [10], it was shown that word pronunciation embeddings are generally more accurate than word orthography embeddings in isolated name recognition tasks. The former is obtained from grapheme (or character) sequences like [r, e, a, d], whereas the latter is obtained from phoneme sequences like [/r/, /iy/, /d/]. Because some words, like “read” or “live”, can have more than one pronunciation, representing them with single embedding vectors can result in inaccurate matching. By using separate embeddings for their individual pronunciations, we can avoid such ambiguities.

Hence, in this study we also propose using pronunciation (or “pron”) embeddings, where the model produces posterior scores for a vocabulary of word pronunciations instead of orthographies.

The model can be trained the same way as the baseline, but instead of providing a reference transcription of words, we provide a reference transcription of word prons. Such a transcription can be obtained by force-aligning the audio to the reference transcriptions using a conventional hybrid DNN-HMM ASR with a pronunciation lexicon and finding the best pronunciation for each word.

When testing, the output pron posteriors $\hat{P}(r_{0}|X,t),\cdots,$
$\hat{P}(r_{m}|X,t)$, assuming a pron vocabulary $\{r_{0},\cdots,r_{m}\}$, is converted to word posteriors for a word vocabulary of size $n$ by setting

$$\hat{P}(w_{i}|X,t)=\max_{r\in R_{i}}\hat{P}(r|X,t),$$

(3)

where $R_{i}$ is the set of prons that word $w_{i}$ can have, e.g. if $w_{1}$=“live”, then $R_{1}$ = { “/l/ /ih/ /v/”, “/l/ /ay/ /v/” }. The “blank” posterior is used as-is, i.e., $\hat{P}(w_{0}|X,t)=\hat{P}(r_{0}|X,t)$.

In the case of homophones, like “stair” and “stare,” the posterior for “/s/ /t/ /eh/ /r/” becomes the posteriors for both words, and we leave it to the language model to choose the correct word.