A comparison of handcrafted, parameterized, and learnable features for speech separation

Given $C$ speech sources $\left\{\mathbf{s}_{c}(t)\right\}_{c=1}^{C}$ with $t$ as the index of time samples, their mixed signal is

$$\mathbf{x}(t)=\sum_{c=1}^{C}\mathbf{s}_{c}(t)$$

(1)

The problem of speech separation can be described as producing an accurate estimate $\hat{\mathbf{s}}_{c}(t)$ for $\mathbf{s}_{c}(t)$ from $\mathbf{x}(t)$.

The framework in this study is Conv-Tasnet [7]. As shown in Fig. 1, it consists of three main parts—an encoder, a separation network, and a decoder. It uses a small frame size in the encoder and decoder to reduce the time delay significantly. The encoder and decoder are learnable 1D-conv filters, which perform like transforms between the time-domain signal and time-frequency features. The separation network is a fully-convolutional separation module that consists of stacked one-dimensional dilated convolutional blocks [13, 14]. It is optimized with the scale-invariant soure-to-noise ratio (SI-SNR) loss [5]. It produces a mask for each speech source.

The comparison uses handcrafted transforms, parameterized transforms, and learnable filters as the encoder and decoder. The encoder can be thought of as s set of $N$ filters of length $L$. The output of the encoder is a time-frequency representation produced from the convolution of the mixed speech input signal with the filter:

$$\mathbf{X}(n,i)=\mathcal{H}(\sum_{l=0}^{L-1}\mathbf{x}(iD+l)\mathbf{h}_{n}^{\mathrm{Enc}}(L-l))$$

(2)

where $n$ is the filter index, $i$ is the frame index, $D$ is the frame shift, $\mathbf{h}_{n}^{Enc}(\cdot)$ is the $n$-th filter of the filterbanks, $l$ denotes the sample index in a frame, and $\mathcal{H}(\cdot)$ is the rectified linear unit (ReLU) to ensure that the representation is non-negative. In the comparison, $\mathbf{h}_{n}^{\mathrm{Enc}}(\cdot)$ can be any of the three kinds of feature transforms.

The decoder reconstructs the time-domain signal of the $c$-th speaker $\hat{\mathbf{s}}_{c}\in\mathbb{R}^{T}$. The output of the decoder is:

$$\hat{\mathbf{s}}_{c}(k,i)=\sum_{n=0}^{N-1}\hat{\mathbf{S}}_{c}(n,i)\mathbf{h}_{N-n}^{\mathrm{Dec}}(k)$$

(3)

where $\hat{\mathbf{S}}_{c}(n,i)$ is the output of the separation network for the $c$-th speaker, $k$ is the index of the filter weight, $\mathbf{h}_{n}^{\rm Dec}(\cdot)$ is the $n$-th filter of the decoder, and $\hat{\mathbf{s}}_{c}(k,i)$ is the estimate of the $c$-th speech source at the $i$-th frame. To decode the frame-shift operation between speech frames, the decoder further calculates $\hat{\mathbf{s}}_{c}(t)=\sum_{i=-\infty}^{\infty}\hat{\mathbf{s}}_{c}(t-iD,i)$.

The comparison uses STFT, MPGTF, ParaMPGTF, and learnable filters as $\mathbf{h}_{n}^{\mathrm{Enc}}(\cdot)$ with their inverse transforms as $\mathbf{h}_{N-n}^{\mathrm{Dec}}(\cdot)$, where the proposed ParaMPGTF is presented in the next subsection.

Gammatone filterbank, which mimics the masking effect of the human auditory system, are good features for speech separation [15]. The impulse response function $\gamma(t)$ of a gammatone filter is

$$\gamma(t)=\alpha t^{n-1}\exp(-2\pi bt)\cos(2\pi f_{c}t+\phi)$$

(4)

where $n$ is the order, $b$ is a bandwidth parameter, $f_{c}$ is the centre frequency of the filter, $t\textgreater 0$ is the time in seconds, $\alpha$ is the amplitude, and $\phi$ is the phase shift. Ditter and Gerkmann [10] extended the classical gammatone filterbank to MPGTF. The extension has the following three three aspects: First, the length of the filters is set to $2$ms, which keeps the system a low latency. Second, for each filter $h_{n}^{\rm Enc}(\cdot)$, MPGTF introduces $-h_{n}^{\rm Enc}(\cdot)$ to ensure that, for each centre frequency, at least one filter contains energy. Third, the phase shift $\phi$ varies at the same centre frequency. The details of MPGTF can be found in [10].

From (4), we observe that the bandwidth parameter $b$ and filter centre frequency $f_{c}$ are two important parameters. They are determined by the equivalent rectangular bandwidth (ERB) [16] using a rectangular band-pass filter:

	$\displaystyle\mathrm{ERB}(f_{c},c_{1},c_{2})=c_{1}+\frac{f_{c}}{c_{2}}$		(5)
	$\displaystyle{\color[rgb]{0.00,0.07,1.00}f_{c}=c_{2}(\mathrm{ERB}-c_{1})}$		(6)
	$\displaystyle b=\frac{\mathrm{ERB}\sqrt{(n-1)!}}{\pi\left((2n-2)!\right)2^{2-2n}}$		(7)

where $c_{1}$ and $c_{2}$ are two parameters. Traditionally, the parameters $c_{1}$ and $c_{2}$ are set to $24.7$ and $9.265$ respectively in experience [16]. This empirical setting may not be accurate enough, which may lead to suboptimal performance.

To overcome this issue, we propose ParaMPGTF which trains the filterbank parameters $c_{1}$ and $c_{2}$ in MPGTF jointly with the network. For each iteration, we update the parameter $b$ by (7) and the centre frequencies $f_{c_{1}},f_{c_{2}},\dots,f_{c_{M}}$ by:

$$f_{c_{j}}=\mathrm{ERB}_{\rm scale}^{-1}(\mathrm{ERB}_{\rm scale}(f_{c_{j-1}})+1)$$

(8)

according to the updated $c_{1}$ and $c_{2}$, where $f_{c_{j}}$ denotes the centre frequency of the $j$-th filter, $M$ is the number of filters in the filterbank, $\mathrm{ERB}_{\rm scale}$ denotes the ERB scale calculated by integrating $1/\mathrm{ERB}(f_{c})$ across frequency, and $\mathrm{ERB}^{-1}_{\rm scale}$ is the inverse of $\mathrm{ERB}_{\rm scale}$. In practice, $\mathrm{ERB}_{\rm scale}$ and $\mathrm{ERB}_{\rm scale}^{-1}$ are calculated by:

	$\displaystyle\mathrm{ERB}_{\rm scale}(f_{\mathrm{Hz}})=c_{2}\log(1+\frac{f_{\mathrm{Hz}}}{c_{1}c_{2}})$		(9)
	$\displaystyle\mathrm{ERB}^{-1}_{\rm scale}(\mathrm{ERB}_{\rm scale})=c_{1}c_{2}(\mathrm{e^{{\frac{\mathrm{ERB}_{\rm scale}}{c_{2}}}}}-1)$		(10)

where $f_{\mathrm{Hz}}$ denotes a frequency variable. After obtaining $f_{c_{1}},\dots,f_{c_{M}}$ and $b$, we obtain the updated filterbank according to (4). To make ParaMPGTF a meaningful filterbank, $f_{c_{1}},f_{c_{2}},\dots,f_{c_{M}}$ should be constrained between $100$ Hz to $4000$ Hz. To satisfy this constraint, we fix $f_{c_{1}}$ to $100$ Hz in the entire training process.

To summarize, ParaMPGTF combines the data-driven scheme with MPGTF [10]. It inherits the changes of MPGTF.

We conducted the comparison on two-speaker speech separation using the WSJ0-2mix dataset [2]. It contains $30$ hours training data, $10$ hours development data, and $5$ hours test data. The mixtures in WSJ0-2mix were generated by first randomly selecting different speakers and utterances in the Wall Street Journal (WSJ0) training set si_tr_s, and then mixing them at a random signal-to-noise ratio (SNR) level between -$5$ dB and $5$ dB [7]. The utterances in the test set were from $16$ unseen speakers in the si_dt_05 and si_et_05 directories of the WSJ0 dataset. All waveforms were resampled to $8$ kHz.

The network was trained for $200$ epochs on $4$-second long segments. Adam was used as the optimizer with an initial learning rate of $0.001$. The learning rate was halved if the performance of development set was not improved in $5$ consecutive epochs. The network training procedure was early stopped when the performance on the development set has not been improved within the last $10$ epochs. The hyperparameters of the network followed the setting in [10], where the number of filters $N$ is 512. The mask activation function of TCN was set to sigmoid function and rectified linear unit (ReLU) respectively.

For ParaMPGTF, we set the order $n$ and amplitude $\alpha$ to 2 and 1 respectively. We initialize $c_{1}$ and $c_{2}$ to their empirical values, i.e. $c_{1}=24.7$ and $c_{2}=9.265$.

We used SI-SNR as the evaluation metric [5]. We reported the average results over all $3000$ test mixtures.

We first conducted a comparison between STFT, MPGTF, ParaMPGTF, and learnable features when the decoders were set to the learnable features. The comparison results are listed in Table 1. From the table, we observe that the four features do not yield fundamentally different performance. If we look at the details, we find that STFT reaches the highest SI-SNR in both the development set and the test set. MPGTF and ParaMPGTF show competitive performance, where ParaMPGTF performs slightly better than MPGTF on the development set, and slightly worse than the latter on the test set.

Fig. 2 shows the magnitude spectrograms of the MPGTF, ParaMPGTF, and STFT encoders with their corresponding learnable decoders, where we only plot the STFT bins with indices from $1$ to $256$ [17, 18] since that the real and imaginary parts share similar patterns. The filters are uniformly distributed in the frequency range from $0$ Hz to $4000$ Hz. From the figure, we see that the magnitude spectrograms of ParaMPGTF and MPGTF are similar. This phenomenon not only accounts for their similar performance, but also demonstrates that the parameterized feature is able to be optimized successfully. As a byproduct, it shows that (i) MPGTF is a well-designed handcrafted feature; (ii) the learnable decoders are able to learn effective inverse transforms of their encoders.

Table 2 lists the comparison between the handcrafted $c_{1}$ and $c_{2}$ in MPGTF and the optimized $c_{1}$ and $c_{2}$ in ParaMPGTF. From the table, we see that the two groups of the parameters are similar, which further accounts for the similar performance of MPGTF and ParaMPGTF.

In this experiment, we set the encoder to STFT, MPGTF, and ParaMPGTF respectively, and set the decoder to their inverse transforms accordingly.

Table 3 lists the performance of MPGTF, ParaMPGTF, and STFT with their (pseudo) inverse transforms. From the table, we see that the performance of the three comparison methods is similar in general. If we look into the details, we see that the proposed ParaMPGTF reaches the best performance among the comparison methods on both the development set and the test set, which demonstrates the potential of the parameterized training strategy in improving conventional handcrafted features.

Fig. 3 shows the convergence curves of the deep models on the development set when the decoders are set to the (pseudo) inverse transforms of their encoders. From the figure, we find that the learnable feature converges faster than the handcrafted and parameterized features. Although the handcrafted features and ParaMPGTF converge in a similar rate at the early training stage, ParaMPGTF converges faster at the late training stage.