Serialized Output Training for End-to-End Overlapped Speech Recognition

The AED-based ASR consists of encoder, attention, and decoder modules as shown in Fig. 1 (a). Given input $X=\{x_{1},...,x_{T}\}$, the AED produces the output sequence $Y=\{y_{1},...,y_{n},...\}$ as follows. Firstly, the encoder converts the input sequence $X$ into a sequence, $H^{enc}$, of embeddings, i.e.,

$\displaystyle H^{enc}$

$\displaystyle=\{h^{enc}_{1},...,h^{enc}_{T}\}={\rm Encoder}(X).$

(1)

Then, for every decoder step $n$, the attention module outputs context vector $c_{n}$ with attention weight $\alpha_{n}$ given decoder state vector $q_{n}$, the previous attention weight $\alpha_{n-1}$, and the encoder embeddings $H^{enc}$ as

$\displaystyle c_{n},\alpha_{n}$

$\displaystyle={\rm Attention}(q_{n},\alpha_{n-1},H^{enc}).$

(2)

Finally, the output distribution $y_{n}$ is estimated given the context vector $c_{n}$ and the decoder state vector $q_{n}$ as follows:

	$\displaystyle q_{n}$	$\displaystyle={\rm DecoderRNN}(y_{n-1},c_{n-1},q_{n-1}),$		(3)
	$\displaystyle y_{n}$	$\displaystyle={\rm DecoderOut}(c_{n},q_{n}).$		(4)

Here, ${\rm DecoderRNN}$ consists of multiple RNN layers while ${\rm DecoderOut}$ consists of an affine transform with a softmax output layer. The model is trained to minimize the cross entropy loss given $Y$ and reference label $R=\{r_{1},...,r_{N},r_{N+1}=\langle eos\rangle\}$. Specifically, the loss function is defined as

$\displaystyle\mathcal{L}^{CE}=\sum_{n=1}^{N+1}{\rm CE}(y_{n},r_{n}),$

(5)

where ${\rm CE()}$ represents the cross entropy given the output distribution and the reference label, and $N$ is the number of symbols in the reference $R$. $\langle eos\rangle$ is the special symbol that represents the end of the sentence.

With the conventional multi-speaker ASR with PIT, the model has multiple output branches as shown in Fig. 1 (b). Thus, we have

	$\displaystyle H^{enc_{s}}$	$\displaystyle={\rm Encoder}_{s}(H^{enc})$		(6)
	$\displaystyle c^{s}_{n},\alpha^{s}_{n}$	$\displaystyle={\rm Attention}(q^{s}_{n},\alpha^{s}_{n-1},H^{enc_{s}})$		(7)
	$\displaystyle q^{s}_{n}$	$\displaystyle={\rm DecoderRNN}(y^{s}_{n-1},c^{s}_{n-1},q^{s}_{n-1}),$		(8)
	$\displaystyle y^{s}_{n}$	$\displaystyle={\rm DecoderOut}(c^{s}_{n},q^{s}_{n}).$		(9)

Here, $s$ is the index of each output branch, where $1\leq s\leq S$ with $S$ being the number of speakers. Parameters for the attention and decoder modules are shared across $s$. Given the set of the outputs, $\{Y^{1},\cdots,Y^{S}\}$, and the set of the references, $\{R^{1},\cdots,R^{S}\}$, where $R^{s}$ denotes the $s$th reference defined as $R^{s}=\{r^{s}_{1},..,r^{s}_{N^{s}},r^{s}_{N^{s}+1}=\langle eos\rangle\}$, the PIT-CE loss is calculated by considering all possible speaker permutations as

$\displaystyle\mathcal{L}^{PIT}=\min_{\phi\in{\Phi}(1,...,S)}\sum_{s=1}^{S}\sum_{n=1}^{N^{s}+1}{\rm CE}(y^{\phi[s]}_{n},r^{s}_{n}).$

(10)

Here, ${\Phi()}$ is the function that generates all possible permutations of a given sequence.

There are three theoretical limitations in PIT. Firstly, the number of the output layers in the model constrains the maximum number of speakers that the model can handle. Secondly, it cannot represent the dependency between the utterances of multiple speakers because each output $Y^{s}$ has no direct dependency on the other outputs. Because of this, it might be possible that the duplicated hypotheses are generated from each output layer. Thirdly, even with the Hungarian algorithm [26], it requires a training cost of $O(S^{3})$ which hinders its application to a large number of speakers.

Instead of having multiple output layers as with PIT, we propose to use the original form of AED (Eq. (1)-(4)), which has only one output branch, for the multi-speaker ASR. To recognize multiple utterances, we serialize multiple references into a single token sequence. Specifically, we introduce a special symbol $\langle sc\rangle$ to represent the speaker change and simply concatenate the reference labels by inserting $\langle sc\rangle$ between utterances. For example, for a three-speaker case, the reference label will be given as $R=\{r^{1}_{1},..,r^{1}_{N^{1}},\langle sc\rangle,r^{2}_{1},..,r^{2}_{N^{2}},\langle sc\rangle,r^{3}_{1},..,r^{3}_{N^{3}},\langle eos\rangle\}$. Note that $\langle eos\rangle$ is used only at the end of the entire sequence. We call our proposed approach serialized output training (SOT).

Because there are multiple permutations in the order of reference labels to form $R$, some trick is needed to calculate the loss between the output $Y$ and the concatenated reference label $R$. For example, in the case of two speakers, the reference label can be either $R=\{r^{1}_{1},..,r^{1}_{N^{1}},\langle sc\rangle,r^{2}_{1},..,r^{2}_{N^{2}},\langle eos\rangle\}$ or $R=\{r^{2}_{1},..,r^{2}_{N^{2}},\langle sc\rangle,r^{1}_{1},..,r^{1}_{N^{1}},\langle eos\rangle\}$. One possible way to determine the order is to calculate the loss for all possible concatenation patterns to form $R$ and select the best one, similarly to PIT, as (Fig. 1 (c))

$\displaystyle\mathcal{L}^{SOT-1}=\min_{\phi\in{\Phi}(1,...,S)}\sum_{n=1}^{N^{sot}}{\rm CE}(y_{n},r^{\phi}_{n}),$

(11)

where $N^{sot}=\sum_{s=1}^{S}\{N^{s}+1\}$, and $r^{\phi}_{n}$ is the $n$-th token in the concatenated reference given permutation $\phi$. We call this method as SOT with minimum-loss speaker order. This method has the problem that requires $O(S!)$ training cost.

To reduce the computational cost to $O(S)$, we alternatively propose to sort the reference labels by their start times (Fig. 1 (d)) as follows:

$\displaystyle\mathcal{L}^{SOT-2}=\sum_{n=1}^{N^{sot}}{\rm CE}(y_{n},r^{\Psi(1,...,S)}_{n}).$

(12)

Here, $\Psi$ is the function that outputs the sorted index of $\{1,...,S\}$ according to the start time of each speaker. The term $r^{\Psi(1,...,S)}_{n}$ is the $n$-th token in the concatenated reference given $\Psi(1,...,S)$. As a result, the AED is trained to recognize the utterances of multiple speakers in the order of their start times, separated by a special symbol $\langle sc\rangle$. We call this method as SOT based on first-in, first-out order¹¹1A similar idea to use the start times was proposed in [27] during this paper was reviewed.. The only assumption to perform this first-in, first-out training is that the two utterances do not start at exactly the same time. If that is the case, we randomly determine the utterance order. That said, it rarely happens in real recordings, and thus its impact should be marginal.

With PIT, speech separation is explicitly modeled by the multiple encoder branches. In the proposed SOT framework, the attention module operates on the encoder embeddings that could be contaminated by overlapped speech. Thus, the context vector generated by the attention module may also be contaminated, resulting in potential degracation in accuracy.

We found that simply inserting one unidirectional LSTM layer in DecoderOut() in Eq (4) could solve the problem effectively. Unlike PIT, where the speech separation is performed before the attention module, this additional LSTM works as a separation module, taking place after the attention module. In our experiment, we removed one encoder layer when we applied this “Separation after Attention (SAA)” method for the sake of fairness in terms of the model size.

There are two key advantages of using single output branch instead of multiple branches as with PIT. Firstly, there is no longer a limitation on the maximum number of speakers that the model can handle. Secondly, the proposed model can represent the dependency among multiple utterances, which prevents duplicated hypotheses from being generated.

Furthermore, the proposed model can even predict the number of speakers if the model is trained on a data set including various numbers of speakers. At the inference time, the decoder module will not stop until $\langle eos\rangle$ is predicted. Therefore, it can automatically count the number of speakers in the recording just by counting the occurrences of $\langle sc\rangle$ and $\langle eos\rangle$.