Multi-stage Speaker Extraction with Utterance and Frame-Level Reference Signals

SpEx+ system is a complete time-domain speaker extraction solution, that accepts a time-domain mixture speech and reference speech as inputs to extract the target speech. SpEx+ mainly consists of four modules: twin speech encoder, speaker encoder, speaker extractor, and speech decoder. The twin speech encoder shares the network structure and parameters, projecting the mixture speech and the reference speech of the target speaker into a common latent space. The speaker encoder produces the discriminative utterance-level speaker embedding from the reference speech. The speaker extractor collects the utterance-level speaker embedding from speaker encoder and transformed features from twin speech encoder to estimate masks in various scales, and then goes through the speech encoder to extract the target signals in various scales.

SpEx+ network is trained with multi-task learning objective, that measures both the target speaker prediction and signal reconstruction. The former minimizes the Cross-Entropy (CE) loss between predicted target speaker and actual speaker label. The latter minimizes reconstruction error (multi-scale SI-SDR) between the extracted signals in different scales and the clean target speech.

SpEx+ processes speech in multiple scales. However, at run-time inference, the extracted target signal with smallest scale (highest temporal resolution) is chosen as the final output [15]. As a result, the decoded signals from other scales don’t contribute to target speech generation. In addition, the weights to balance the multi-scale SI-SDR loss between the extracted signals in different scales and the clean target speech are tuned on the development set, which adds complexity to the training process.

To take advantage of the complementary decoded signals from multiple scales during training and inference, we propose a signal fusion strategy before the loss calculation. First, we combine multi-scale decoded signals with automatically learned weights, and then calculate the reconstruction error (SI-SDR) between the fused signal and clean signal.

Let $\hat{s}_{1}^{(k)}$, $\hat{s}_{2}^{(k)}$, and $\hat{s}_{3}^{(k)}$ denote the multi-scale decoded signals in stage $k$, we can use several learnable weights ($w_{1}$, $w_{2}$, and $w_{3}$) to obtain the fused signal as follow:

$$\hat{s}^{(k)}=w_{1}*\hat{s}_{1}^{(k)}+w_{2}*\hat{s}_{2}^{(k)}+w_{3}*\hat{s}_{3}^{(k)},\quad k=1,2,3\vspace{-8pt}$$

(1)

It was reported [14] that the decoded signal with the highest temporal resolution achieves the best performance, and the weights to balance the multi-scale SI-SDR loss are tuned to be $0.8$, $0.1$, $0.1$ for high, middle, and low temporal resolutions [14, 15]. We follow the same weights to combine the signals of 3 temporal resolutions first, then calculate the SI-SDR between the fused signal and the clean signal as,

	$$\displaystyle\mathcal{L}_{\text{SI-SDR}}^{(k)}=-\rho(\hat{s}^{(k)},s),$$		(2)
	$$\displaystyle\rho(\hat{s},s)=20\log_{10}\frac{||(\hat{s}^{T}s/s^{T}s)\cdot s||}{||(\hat{s}^{T}s/s^{T}s)\cdot s-\hat{s}||}$$		(3)

where $\hat{s}$ and $s$ are the estimated signal and the target clean signal, respectively.

The idea of multi-stage architecture is to pipeline several single-stage speaker extraction modules sequentially such that a later module operates on the extracted target speech of an earlier one. The effect of such composition is an incremental refinement of the reference speech from the target speaker. As shown in Fig. 1, each stage takes the extracted target voice from the previous stage as an extra input to refine the reference speech of the target speaker. We implement two mechanisms in SpEx++ to do so.

One is to combine the extracted target voice $\hat{s}^{(k-1)}(t)$ with the original, short reference utterance $x(t)$ to extract the utterance-level speaker embedding $v_{\text{utt}}^{(k)}$. Such speaker embedding is expected to represent characteristics of the target speaker, that is referred to as the first reference signal,

$$v_{\text{utt}}^{(k)}=\text{Enc}_{\text{speaker}}(\text{Enc}_{\text{speech}}(\text{Concat}(x,\hat{s}^{(k-1)}))),$$

(4)

where $k=2,3$, $\text{Enc}_{\text{speech}}(\cdot)$ and $\text{Enc}_{\text{speaker}}(\cdot)$ represent the speech encoder and speaker encoder, respectively.

Another is to pass the extracted speech $s^{(k-1)}(t)$ through the speech encoder to obtain a frame-level speech embedding. As $s^{(k-1)}(t)$ is obtained from the mixture speech $y(t)$, and aligned with the mixture speech frame-by-frame. Such frame-level embedding is different from the utterance-level speaker embedding, but provides a strong signal about the target speaker and his/her speech content. For the first time, we introduce the frame-level speech embedding $v_{\text{frame}}^{(k)}(t)$ as a second reference signal in speaker extraction study,

$$v_{\text{frame}}^{(k)}(t)=\text{Enc}_{\text{speech}}(\hat{s}^{(k-1)}(t)),\quad k=2,3$$

(5)

Now we formulate the way to use two reference signals for speaker extraction,

	$$\displaystyle M_{i}^{(k)}=\text{Ext}(v_{\text{utt}}^{(k)},\text{Concat}(v_{\text{frame}}^{(k)},\text{Enc}_{\text{speech}}(y))),$$		(6)
	$$\displaystyle\hat{s}_{i}^{(k)}=\text{Dec}(M_{i}^{(k)}\otimes\text{Enc}_{\text{speech}}(y))$$		(7)

where $i=1,2,3$ represents the three different scales, and $\otimes$ is an operation for element-wise multiplication. The $\text{Ext}(\cdot)$ and $\text{Dec}(\cdot)$ represent the speaker extractor and speech decoder, respectively. Finally, the reconstruction error can be calculated by applying the operations in Section 2.2.

We trained all systems for 100 epochs on the mixture segments with 4-second and their corresponding reference utterances as in [15], which has an average of 7.3 seconds. The learning rate was initialized to $1e^{-3}$ and decays by 0.5 if the accuracy of validation set was not improved in 2 consecutive epochs. Early stopping was applied if no best model is found in the validation set for 6 consecutive epochs. Adam was used as the optimizer. The network structure follows SpEx+ [15].

The speech encoders at each stage shared the network structure and weights. The filter lengths of convolutions in speech encoder and decoder were 2.5ms, 10ms, and 20ms for the 3 time-scales with speech of 8kHz sampling rate, respectively. The speaker extractor repeated a stack of 8 temporal convolution network (TCN) blocks for 4 times. We included 3 ResNet blocks in speaker encoder, with 256, 256, and 512 filters respectively. The utterance-level speaker embedding dimension was set to 256 in practice. For the signal fusion configuration, the weights are initialized as $w_{1}=0.8$, $w_{2}=0.1$, $w_{3}=0.1$, and then learned automatically during training.

We firstly compare SpEx++ with previous baseline SpEx+ system on WSJ0-2mix in terms of SDRi, SI-SDRi and PESQ with both 7.5 seconds and 2 seconds reference speech.

From Table 1, we conclude: 1) Multi-stage framework significantly outperforms single-stage framework. The improvements mainly come from the augmented frame-level speech embedding through a multi-stage pipeline. As an evidence, the 2-stage SpEx++ outperforms 1-stage counterpart substantially benefiting from the extra frame-level speaker embedding. However, the 3-stage SpEx++ doesn’t improve much further because the 3-stage and 2-stage solutions use similar speaker information overall (i.e., Utt + Fm). 2) The signal fusion strategy in SpEx++ (1-stage) achieves 0.3dB performance gain over SpEx+ baseline in terms of SDRi and SI-SDRi. The results validate the effectiveness of the proposed signal fusion strategy. Besides, the learned weights for $\omega_{1},\omega_{2},\omega_{3}$ were 1.18, 0.32, and 0.11, respectively. This suggests that the decoded signal with higher temporal resolution contribute more. Note that we don’t constrain that the sum of weights is equal to one. 3) The experiments with 2-second reference speech further show that SpEx++ system works well with much shorter reference speech, achieving 1.0dB SI-SDR improvement over SpEx+ baseline system.

We further verify the robustness of SpEx++ system with other baselines under noisy conditions, as reported in Table 2 and Table 3. Compared with the results under noise-free condition in Table 1, we observe that the additive noise and reverberation in the mixture significantly degrades the performance of both blind speech separation and speaker extraction systems. This fact shows that extracting target speaker’s voice from noisy mixture is a challenging task. Despite this, our SpEx++ system still shows effective performance under noisy conditions. Specifically, SpEx++ system achieves about 0.9dB and 0.5dB absolute improvement over SpEx+ baseline in terms of SI-SDRi on WHAM! and WHAMR!, respectively.

We finally study an universal SpEx++ system that is trained on four mixture conditions (clean, noisy only, reverberation only, and noisy and reverberation). The performance of the universal system is further evaluated on each noisy mixture condition individually, as shown in Table 4. Compared with the results in Table 3 and Table 4, we observe that the universal SpEx++ trained on four conditions achieves better performance than that trained on a single condition.