Speakerfilter-Pro: an improved target speaker extractor combines the time domain and frequency domain

The purpose of speaker extraction is to extract the target speech from an observed mixture signal. The work takes anchor speech as auxiliary information. The mixture signal can be written as:

$$y=s_{t}+s_{i},$$

(1)

where $s_{t}$ and $s_{i}$ are the target speech signal and the interfering signal, respectively. The interfering signal may comprise the speech signal from other speakers and background noise. In this paper we only consider the interfering speech. The speaker extractor can be defined as:

$$\hat{s}_{t}=f(y,s_{a}),$$

(2)

where $f$ is the transformation carried out by Speakerfilter-Pro, $\hat{s}_{t}$ is the estimated target speech, and $s_{a}$ is the anchor speech.

The proposed method is based on our previous work [9], and here we give a brief introduction of the Speakerfilter. The Speakerfilter system takes two speech signals as input: noisy speech and anchor speech. A bi-direction gated recurrent unit (BGRU) is used to encode the anchor speech and average the outputs of BGRU in the time dimension to produce the anchor-vector. This anchor-vector can be viewed as the identity vector of the target speaker. Then, it is stacked to each frame of the mixture magnitude spectrum. We use a gated convolutional network (GCNN) to encode the anchor vector into each layer of the speech separator built by a GCRN. The GCRN is used to extract the target speech from noisy speech, with the help of the anchor-vector.

Recently, performs speech separation in the time domain is very attractive. Researchers are also proven that the speech separation performs better in the time domain than in the frequency domain [10].

The WaveUNet [11] is the most commonly used model in the time-domain. The structure is shown in Fig. 1. It computes an increasing number of higher-level features on coarser time scales using downsampling (DS) blocks. These features are combined with the earlier computed local, high-resolution features using upsampling (US) blocks, yielding multi-scale features to make predictions.

Inspired by the WaveUNet’s powerful speech separation capabilities, we propose an improved target speaker extractor, referred to as Speakerfilter-Pro, based on our previous work [9]. Different from the Speakerfilter, the Speakerfilter-Pro add a WaveUNet module in the beginning and the ending, respectively. Second, in order to encode the target speaker better, the complex spectrum instead of the magnitude spectrum is used as the input feature for the GCRN module.

The structure of the Speakerfilter-Pro is shown in Fig. 2. First, the time-domain mixture signal is fed into the beginning WaveUNet. After obtaining the output of the WaveUNet, the short-time Fourier transform (STFT) is used to transfer the time domain signal to the frequency domain. The real and the image part of the spectrum (RI) then stack with the anchor-vector as the input to the GCRN, where the target speaker-vector is obtained by the speaker feature extractor. With an encoder-decoder architecture, the CRN encodes the input features into a higher-dimensional latent space and then models the latent feature vectors’ sequence via two bi-direction long short-term memory (Bi-LSTM) layers. The output sequence of the Bi-LSTM layers is subsequently converted back to the original input size by the decoder. To improve the flow of information and gradients throughout the network, skip connections are used to concatenate each encoder layer’s output to the input of the corresponding decoder layer. After GCRN, the Deepfilter [12] is utilized to recover the target speech better. Deepfilter obtains enhanced speech $\hat{X}$ from mixture speech $X$ by applying a complex filter:

$$\hat{X}(n,k)=\sum_{i=-I}^{I}\sum_{l=-L}^{L}H^{*}_{n,k}(l+L,i+I)\cdot X(n-l,k-i)$$

(3)

where $2\cdot L+1$ is the filter dimension in time-frame direction, $2\cdot I+1$ in frequency direction and $H^{*}$ is the complex conjugated 2D filter of TF bin (n, k). In our experiments, we set $L$ = 2 and $I$ = 1 resulting in a filter dimension of (5, 3). Finally, the WaveUNet is used in the last part to improve the performance further.

For speaker feature extractor, the magnitude spectrum of anchor speech in frequency-domain is used as the inputs. We firstly encode the anchor speech into 129-dimensional features by using BGRU and average the outputs of BGRU in the time dimension. This 129-dimensional anchor vector can be viewed as the identity vector of the target speaker. Then, it is stacked to each frame of the mixture feature. To utilize the speaker information effectively, we propose to use CNN to encode the anchor vector into each layer of CRN, as shown in Fig. 2.

The whole modules are training jointly. The training loss is defined in the frequency domain. The loss function is mean squared error (MSE), which is given by:

$$L_{mse}=\frac{1}{N*K}\sum_{n=1}^{N}\sum_{k=1}^{K}\|X(n,k)-\hat{X}(n,k)\|^{2}$$

(4)

where $N$ is the number of TF unit, $X$ and $\hat{X}$ represent the short-time Fourier transform (STFT) of clean and estimated target speech, respectively.

The proposed system is evaluated by the WSJ0-2mix datasets¹¹1Available at: http://www.merl.com/demos/deep-clustering. In WSJ0-2mix, each sentence contains two speakers. The WSJ0-2mix dataset introduced in [2] is derived from the WSJ0 corpus [13]. The 30h training set and the 10h validation set contain two-speaker mixtures generated by randomly selecting from 49 male and 51 female speakers from si_tr_s. The Signal-to-Noise Ratios (SNRs) are uniformly chosen between 0 dB and 5 dB. The 5h test set is generated similarly by using utterances from 16 speakers from si_et_05, which do not appear in the training and validation sets. The test set includes 1603 F&M sentences, 867 M&M sentences, and 530 F&F sentences.

For each mixture, we randomly choose an anchor utterance from the target speaker (different from the utterance in the mixture).

We compare the proposed algorithm with two baselines: SpeakerBeam+DC[7] and Speakerfilter [9].

Compared with the Speakerfilter, the proposed method has two improvements: 1) the complex spectrum, not the magnitude spectrum, is used in the GCNN part, which can characterize the target speaker better. 2), The time-domain WaveUNet is used in the beginning and ending, which can separate the target speaker better. The proposed method also can be referred to as Speakerfilter+RI+WaveUNet. Based on this, we set up two comparison methods to illustrate the improvement’s impact on the separation effect: Speakerfilter+RI and Speakerfilter+WaveUNet.

The performance is evaluated with two objective metrics: perceptual evaluation of speech quality (PESQ) [14] and Signal-to-Distortion Ratio (SDR) [15]. For both of the PESQ and SDR metrics, the higher number indicates better performance.

Table 1 presents the comprehensive evaluation for different approaches on SDR. It can be seen that the proposed method obtains the best performance. The baseline models are using the T-F domain only for target speaker extraction. However, the proposed method leverages the time-domain modeling ability of the WaveUNet and the frequency-domain modeling ability of the GCRN, simultaneously, that can obtain a better performance. Compared with the unprocessed, the proposed method improves the SDR nearly 12.33 dB.

As shown in Table 2 and 3, most of the modifications have positive effect on the baseline. On average, the Speaker+RI+WaveUNet model obtains the best results.

The WavUNet has a large improvement for Speakerfilter and Speakerfilter+RI models. The average SDR improves 0.37 (from 13.89 to 14.26) and 1.09 (from 13.86 to 14.95) dB, respectively. The average PESQ improves 0.05 (from 3.12 to 3.17) and 0.15 (from 3.09 to 3.24) dB, respectively. We should note that, after adding the WaveUNet module, the RI-based model ($\triangle SDR=1.09,\triangle PESQ=0.15$) is more effective than the amplitude spectrum-based model ($\triangle SDR=0.37,\triangle PESQ=0.05$).

For the Speakerfilter system, after changing the magnitude spectrum to the complex spectrum, both the SDR and the PESQ are reduced by 0.03 dB. However, in the Speakerfilter+WaveUNet system, after changing the magnitude spectrum to the complex spectrum, the SDR and the PESQ are increased by 0.69 and 0.07 dB, respectively, which means that the complex spectrum feature can boost the speaker extractor performance with the participation of WaveUNet.

As mentioned above, the complex spectrum feature and the WaveUNet can boost each other.

Fig. 3 illustrates the spectrogram of separated speech using Speakerfilter and Speakerfilter-Pro, respectively. Our method has better performance on target speaker separation. As shown in the yellow rectangle, the proposed method can suppress the interfering speaker better than Speakerfilter.