Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR

The E2E SA-ASR model [22] uses acoustic feature sequence $X\in\mathbb{R}^{f^{a}\times l^{a}}$ and a set of speaker profiles $D=\{d_{k}\in\mathbb{R}^{f^{d}}|k=1,...,K\}$ as input. Here, $f^{a}$ and $l^{a}$ are the feature dimension and length of the feature sequence, respectively. Variable $K$ is the total number of profiles, $d_{k}$ is the speaker embedding (e.g., d-vector [27]) of the $k$-th speaker, and $f^{d}$ is the dimension of the speaker embedding. We assume $D$ includes the profiles of all the speakers present in the observed audio. $K$ can be greater than the actual number of the speakers in the observed audio.

Given $X$ and $D$, the E2E SA-ASR model estimates a multi-talker transcription, i.e., word sequence $Y=(y_{n}\in\{1,...,|\mathcal{V}|\}|n=1,...,N)$ accompanied by the speaker identity of each token $S=(s_{n}\in\{1,...,K\}|n=1,...,N)$. Here, $|\mathcal{V}|$ is the size of the vocabulary $\mathcal{V}$, $y_{n}$ is the word index for the $n$-th token, and $s_{n}$ is the speaker index for the $n$-th token. Following the serialized output training (SOT) framework [28], a multi-talker transcription is represented as a single sequence $Y$ by concatenating the word sequences of the individual speakers with a special “speaker change” symbol $\langle sc\rangle$. For example, the reference token sequence to $Y$ for the three-speaker case is given as $R=\{r^{1}_{1},..,r^{1}_{N^{1}},\langle sc\rangle,r^{2}_{1},..,r^{2}_{N^{2}},\langle sc\rangle,r^{3}_{1},..,r^{3}_{N^{3}},\langle eos\rangle\}$, where $r^{j}_{i}$ represents the $i$-th token of the $j$-th speaker. A special symbol $\langle eos\rangle$ is inserted at the end of all transcriptions to determine the termination of inference. Note that this representation can be used for overlapping speech of any number of speakers.

The E2E SA-ASR model consists of two attention-based encoder-decoders (AEDs), i.e. an AED for ASR and an AED for speaker identification. The two AEDs depend on each other, and jointly estimate $Y$ and $S$ from $X$ and $D$.

The AED for ASR is represented as,

	$\displaystyle H^{\rm asr}$	$\displaystyle={\rm AsrEncoder}(X),$		(1)
	$\displaystyle o_{n}$	$\displaystyle={\rm AsrDecoder}(y_{[1:n-1]},H^{\rm asr},\bar{d}_{n}).$		(2)

The AsrEncoder module converts the acoustic feature $X$ into a sequence of hidden embeddings $H^{\rm asr}\in\mathbb{R}^{f^{h}\times l^{h}}$ for ASR (Eq. (1)), where $f^{h}$ and $l^{h}$ are the embedding dimension and the length of the embedding sequence, respectively. The AsrDecoder module then iteratively estimates the output distribution $o_{n}\in\mathbb{R}^{|\mathcal{V}|}$ for $n=1,...,N$ given previous token estimates $y_{[1:n-1]}$, $H^{\rm asr}$, and the weighted speaker profile $\bar{d}_{n}$ (Eq. (2)). Here, $\bar{d}_{n}$ is calculated in the AED for speaker identification, which will be explained later. The posterior probability of token $i$ (i.e. the $i$-th token in $\mathcal{V}$) at the $n$-th decoder step is represented as

$\displaystyle Pr(y_{n}=i|y_{[1:n-1]},s_{[1:n]},X,D)=o_{n,i},$

(3)

where $o_{n,i}$ represents the $i$-th element of $o_{n}$.

The AED for speaker identification is represented as

	$\displaystyle H^{spk}$	$\displaystyle={\rm SpeakerEncoder}(X),$		(4)
	$\displaystyle q_{n}$	$\displaystyle={\rm SpeakerDecoder}(y_{[1:n-1]},H^{\rm spk},H^{\rm asr}),$		(5)
	$\displaystyle\beta_{n,k}$	$\displaystyle=\frac{\exp(\cos(q_{n},d_{k}))}{\sum_{j}^{K}\exp(\cos(q_{n},d_{j}))},$		(6)
	$\displaystyle\bar{d}_{n}$	$\displaystyle=\sum_{k=1}^{K}\beta_{n,k}d_{k}.$		(7)

The SpeakerEncoder module converts $X$ into a speaker embedding sequence $H^{\rm spk}\in\mathbb{R}^{f^{h}\times l^{h}}$ that represents the speaker characteristic of $X$ (Eq. (4)). The SpeakerDecoder module then iteratively estimates speaker query $q_{n}\in\mathbb{R}^{f^{d}}$ for $n=1,...,N$ given $y_{[1:n-1]}$, $H^{\rm spk}$ and $H^{\rm asr}$ (Eq. (5)). A cosine similarity-based attention weight $\beta_{n,k}\in\mathbb{R}$ is then calculated for all profiles $d_{k}$ in $D$ given the speaker query $q_{n}$ (Eq. (6)). A posterior probability of person $k$ speaking the $n$-th token is represented by $\beta_{n,k}$ as

$\displaystyle Pr(s_{n}=k|y_{[1:n-1]},s_{[1:n-1]},X,D)=\beta_{n,k}.$

(8)

Finally, a weighted average of the speaker profiles is calculated as $\bar{d}_{n}\in\mathbb{R}^{f^{d}}$ (Eq. (7)) to be fed into the AED for ASR (Eq. (2)).

The joint posterior probability $Pr(Y,S|X,D)$ can be represented based on Eqs. (3) and (8) (see [22]). The model parameters are optimized by maximizing $\log Pr(Y,S|X,D)$ over training data.

Following [29], a transformer-based network architecture is used for the AsrEncoder, AsrDecoder, and SpeakerDecoder modules. The SpeakerEncoder module is based on Res2Net [30]. Here, we describe only the AsrDecoder because it is necessary to explain the proposed method. Refer to [29] for the details of the other modules.

Our AsrDecoder is almost the same as a conventional transformer-based decoder [31] except for the addition of the weighted speaker profile $\bar{d}_{n}$ at the first layer. The AsrDecoder is represented as

	$\displaystyle z_{[1:n-1],0}^{\rm asr}=\mathrm{PosEnc}(\mathrm{Embed}(y_{[1:n-1]})),$		(9)
	$\displaystyle\bar{z}^{\rm asr}_{n-1,l}=z^{\rm asr}_{n-1,l-1}$
	$\displaystyle\hskip 8.53581pt+\mathrm{MHA}_{\rm l}^{\rm asr{\text{-}}self}(z^{\rm asr}_{n-1,l-1},z_{[1:n-1],l-1}^{\rm asr},z_{[1:n-1],l-1}^{\rm asr}),$		(10)
	$\displaystyle\bar{\bar{z}}^{\rm asr}_{n-1,l}=\bar{z}^{\rm asr}_{n-1,l}+\mathrm{MHA}_{\rm l}^{\rm asr{\text{-}}src}(\bar{z}^{\rm asr}_{n-1,l},H^{\rm asr},H^{\rm asr}),$		(11)
	$\displaystyle z^{\rm asr}_{n-1,l}=\left\{\begin{array}[]{ll}\bar{\bar{z}}^{\rm asr}_{n-1,l}+\mathrm{FF}_{l}^{\rm asr}(\bar{\bar{z}}^{\rm asr}_{n-1,l}+W^{\rm spk}\cdot\bar{d}_{n})&(l=1)\\ \bar{\bar{z}}^{\rm asr}_{n-1,l}+\mathrm{FF}_{l}^{\rm asr}(\bar{\bar{z}}^{\rm asr}_{n-1,l})&(l>1)\end{array}\right.$		(14)
	$\displaystyle o_{n}=\mathrm{SoftMax}(W^{o}\cdot z_{n-1,L^{\rm asr}}^{\rm asr}+b^{o}).$		(15)

Here, $\mathrm{Embed}()$ and $\mathrm{PosEnc}()$ are the embedding function and absolute positional encoding function [31], respectively. $\mathrm{MHA}^{*}_{l}(Q,K,V)$ represents the multi-head attention of the $l$-th layer [31] with query $Q$, key $K$, and value $V$ matrices. $\rm FF_{l}^{\rm asr}()$ is a position-wise feed forward network in the $l$-th layer.

A token sequence $y_{[1:n-1]}$ is first converted into a sequence of embedding $z_{[1:n-1],0}^{\rm asr}\in\mathbb{R}^{f^{h}\times(n-1)}$ (Eq. (9)). For each layer $l$, the self-attention operation (Eq. (10)) and source-target attention operation (Eq. (11)) are applied. Finally, the position-wise feed forward layer is applied to calculate the input to the next layer $z_{n,l+1}^{\rm asr}$ (Eq. (14)). Here, $\bar{d}_{n}$ is added after being multiplied by the weight $W^{spk}\in\mathbb{R}^{f^{h}\times f^{d}}$ in the first layer. Finally, $o_{n}$ is calculated by applying SoftMax function on the final $L^{\rm asr}$-th layer’s output with weight $W^{o}\in\mathbb{R}^{|\mathcal{V}|\times f^{h}}$ and bias $b^{o}\in\mathbb{R}^{|\mathcal{V}|}$ application (Eq. (15)).

The overview of the proposed procedure is shown in Fig. 1. VAD is first applied to the long-form audio to detect silence regions. Then, speaker embeddings are extracted from uniformly segmented audio with a sliding window. A conventional clustering algorithm (in our experiment, spectral clustering) is then applied to obtain the cluster centroids. Finally, the E2E SA-ASR is applied to each VAD-segmented audio with the cluster centroids as the speaker profiles. In this work, the E2E SA-ASR model is extended to generate not only a speaker-attributed transcription but also the start and end times of each token, which can be directly translated to the speaker diarization result. In the evaluation, detected regions for temporarily close tokens (i.e. tokens apart from each other with less than $M$ sec) with the same speaker identities are merged to form a single speaker activity region. We also exclude abnormal estimations where (i) end_time - start_time $\geq$ $N$ sec or (ii) end_time $<$ start_time for a single token. We set $M=2.0$ and $N=2.0$ in our experiment according to the preliminary results.

In this study, we propose to estimate start and end times of $n$-th estimated token from the query $\bar{z}_{n-1,l}^{\rm asr}$ and key $H^{\rm asr}$, which are used in the source-target attention (Eq. (11)), with a small number of learnable parameters. Note that, although there are several prior works that conducted the analysis on the source-target attention, we are not aware of any prior works that directly estimate the start and end times of each token with learnable parameters. It should also be noted that we can not rely on a conventional force-alignment tool (e.g. [32]) because the input audio may be including overlapping speech.

With the proposed method, the probability distribution of start time frame of the $n$-th token over the length of $H^{\rm asr}$ is estimated as

$\displaystyle\alpha_{n}^{\rm start}=\mathrm{Softmax}(\sum_{l}\frac{(W_{l}^{\rm s,q}\bar{z}_{n-1,l}^{asr})^{\mathsf{T}}(W_{l}^{\rm s,k}H^{\rm asr})}{\sqrt{f^{\rm se}}}).$

(16)

Here, $f^{\rm se}$ is the dimension of the subspace to estimate the start time frame of each token. The terms $W_{l}^{\rm s,q}\in\mathbb{R}^{f^{\rm se}\times f^{h}}$ and $W_{l}^{\rm s,k}\in\mathbb{R}^{f^{\rm se}\times f^{h}}$ are the affine transforms to map the query and key to the subspace, respectively. The resultant $\alpha_{n}^{\rm start}\in\mathbb{R}^{l^{h}}$ is the scaled dot-product attention accumulated for all layers, and it can be regarded as the probability distribution of the start time frame of $n$-th token over the length of embeddings $H^{\rm asr}$. Similarly, the probability distribution of the end time frame of the $n$-th token, represented by $\alpha_{n}^{\rm end}\in\mathbb{R}^{l^{h}}$, is estimated by replacing $W_{l}^{\rm s,q}$ and $W_{l}^{\rm s,k}$ of Eq (16) with $W_{l}^{\rm e,q}\in\mathbb{R}^{f^{\rm se}\times f^{h}}$ and $W_{l}^{\rm e,k}\in\mathbb{R}^{f^{\rm se}\times f^{h}}$, respectively.

The parameters $W_{l}^{\rm s,q}$, $W_{l}^{\rm s,k}$, $W_{l}^{\rm e,q}$, and $W_{l}^{\rm e,k}$ are learned from training data that includes the reference start and end time indices on the embedding length of $H^{\rm asr}$. In this paper, we apply a cross entropy (CE) objective function on the estimation of $\alpha_{n}^{\rm start}$ and $\alpha_{n}^{\rm end}$ on every token except special tokens $\langle sc\rangle$ and $\langle eos\rangle$. We perform the multi-task training with the objective function of the original E2E SA-ASR model and the objective function of the start-end time estimation with an equal weight to each objective function. In the inference, frames with the maximum value on $\alpha_{n}^{\rm start}$ and $\alpha_{n}^{\rm end}$ are selected as the start and end frames for the $n$-th token, respectively.