All-neural beamformer for continuous speech separation

Most of the existing speech separation algorithms operate on pre-segmented mixtures by assuming an ideal overlap detector to be available. However, in real scenarios, speech separation systems need to deal with a continuous audio stream consisting of multiple speakers, which can be hours long and include both overlapped and non-overlapped utterances. More recently, the CSS scheme [22] has been proposed, which is defined as the process of generating a limited number of overlap-free signals from the continuous audio stream. To deal with the long input signals, we adopt the chunk-wise CSS scheme proposed in [23]. As illustrated in Fig. 1, a sliding window is used which contains three sub-windows, including the history sub-window ($N_{h}$ frames), current sub-window ($N_{c}$ frames), and future sub-window ($N_{f}$ frames). For each time step, the window is moved forward by $N_{c}$ frames. During test time evaluation, the speech separation algorithm takes in the entire chunk of information (i.e., $N=N_{h}+N_{c}+N_{f}$ frames) to estimate $K(=2)$ overlap-free signals for the current $N_{c}$ frames. Estimated signals for each chunk are later aligned via block stitching [19].

The MVDR filter aims to preserve the information from the target direction while minimizing the power of interfering sources. Specifically, the MVDR filter is defined as

$$\mathbf{h_{\text{MVDR}}}=\underset{\mathbf{h}}{\arg\min\mathbf{h}}^{\mathrm{H}}\mathbf{\Phi}_{\mathrm{VV}}\mathbf{h}\quad\bf{\text{s.t.}}\quad\mathbf{h}^{\mathrm{H}}\mathbf{\boldsymbol{v}}=\mathbf{1},$$

(1)

where $\mathbf{h_{\text{MVDR}}}$ is the MVDR filtering weights, superscript ^H denotes the Hermitian transpose, and $\mathbf{\Phi}_{\mathrm{VV}}$ is the covariance matrix of interfering sources (background noise and/or interfering speakers). Variable $\mathbf{\boldsymbol{v}}$ represents the steering vector which can be approximated by extracting the principal eigenvector of the speech covariance matrix [24], i.e., $\mathbf{\boldsymbol{v}}=\mathcal{P}\{\mathbf{\Phi}_{\mathrm{SS}}\}$. By solving Eq. (1), we have [25, 26]

$$\mathbf{h_{\text{MVDR}}}=\frac{\mathbf{\Phi}_{\mathrm{VV}}^{-1}\boldsymbol{v}}{\mathbf{\boldsymbol{v}^{\mathrm{H}}}\mathbf{\Phi}_{\mathrm{VV}}^{-1}\mathbf{\boldsymbol{v}}}.$$

(2)

Most existing studies estimate the covariance matrices in a chunk-wise fashion. However, this chunk-wise processing also incurs less flexibility and adaptability of the MVDR filter.

ADL-MVDR has been recently proposed as a fully neural network-based beamformer that has demonstrated superior performance in a target speech extraction task [15, 16]. The core idea of ADL-MVDR is utilizing two separate GRU-based networks (denoted as GRU-Nets) to replace the matrix inversion and principal eigenvector extraction involved in the conventional MVDR solution. Each of the GRU-Nets takes in the speech/noise covariance matrix and estimates the steering vector or the matrix inverse on a per-frame basis. Note that, unlike the conventional MVDR systems, which take sums or expectations over the entire chunk to derive the covariance matrices, the temporal information in the covariance matrices can be leveraged in the ADL-MVDR network.

In the CSS scheme, at most $K$ speech sources are assumed to be active within each chunk. Here, we describe the proposed method with $K=2$. The normalized time-varying input covariance matrices for the speaker and interfering noise sources can be derived as

	$\displaystyle\mathbf{\Phi}_{\mathrm{SS}}^{(k)}(t,f)$	$\displaystyle=\frac{\mathbf{\hat{S}_{\text{mask}}}^{(k)}(t,f)\mathbf{\hat{S}_{\text{mask}}}^{\mathrm{H}\>(k)}(t,f)}{\sum_{t=1}^{T}\mathbf{M}^{(k)2}_{\mathrm{S}}(t,f)},$		(3)
	$\displaystyle\mathbf{\Phi}_{\mathrm{VV}}^{(k)}(t,f)$	$\displaystyle=\frac{\mathbf{\hat{V}_{\text{mask}}}^{(k)}(t,f)\mathbf{\hat{V}_{\text{mask}}}^{\mathrm{H}\>(k)}(t,f)}{\sum_{t=1}^{T}\mathbf{M}^{(k)2}_{\mathrm{V}}(t,f)},$

where $\mathbf{\Phi}_{\mathrm{SS}}^{(k)}$ is the instantaneous speech covariance matrix for speaker source $k\in\{0,1\}$, and $(t,f)$ represent the time and frequency indices. $\mathbf{\hat{S}_{\text{mask}}}^{(k)}=\mathbf{M}^{(k)}_{\mathrm{S}}\mathbf{Y}$ is the masked speech for source $k$, where $\mathbf{Y}$ is the multi-channel noisy speech and $\mathbf{M}_{\mathrm{S}}^{(k)}$ denotes the T-F speech mask for source $k$. $T$ is the total number of frames. Similarly, $\mathbf{\Phi}_{\mathrm{VV}}^{(k)}$ is the interfering source covariance matrix for source $k$. The interfering source consists of two parts (i.e., ambient noise and interfering speaker) as $\mathbf{\hat{V}_{\text{mask}}}^{(k)}=\mathbf{M}_{\mathrm{N}}\mathbf{Y}+\mathbf{\hat{S}_{\text{mask}}}^{(1-k)}$, where $\mathbf{M}_{\mathrm{N}}$ is the isotropic noise mask. Similarly we have $\mathbf{M}_{\mathrm{V}}^{(k)}=\mathbf{M}_{\mathrm{N}}+\mathbf{M}_{\mathrm{S}}^{(1-k)}$. Next, the time-varying variables corresponding to the steering vector and inverse noise covariance matrix are estimated using two separate GRU-Nets as

	$\displaystyle\mathbf{\boldsymbol{\hat{\boldsymbol{v}}}}^{(k)}(t,f)$	$\displaystyle=\mathbf{GRU{\text{-}}Net}_{\upsilon}(\mathbf{\Phi}_{\mathrm{SS}}^{(k)}(t,f)),$		(4)
	$\displaystyle{\mathbf{\hat{\Phi}}_{\mathrm{VV}}}^{-1\>(k)}(t,f)$	$\displaystyle=\mathbf{GRU{\text{-}}Net}_{\mathrm{VV}}(\mathbf{\Phi}_{\mathrm{VV}}^{(k)}(t,f)).$

Once these time-varying coefficients are obtained, the beamforming weights $\mathbf{h}_{\text{ADL-MVDR}}\in\mathbb{C}^{F\times T\times C}$ ($F$ and $C$ represent the frequency and channel dimensions, respectively) can be derived on a per-frame basis by plugging these estimated terms into Eq. (2). Finally, the ADL-MVDR filtered speech for source $k$ can be obtained as

$$\mathbf{\hat{S}}^{(k)}_{\text{ADL-MVDR}}(t,f)=\mathbf{\hat{h}}^{\mathrm{H}\>(k)}_{\text{ADL-MVDR}}(t,f)\mathbf{Y}(t,f).$$

(5)

To further enhance the performance of the ADL-MVDR in the CSS scheme, we propose several techniques. Firstly, we introduce constraints on the estimated steering vector and the inverse of the covariance matrix. Specifically, we apply normalization on the estimated time-varying steering vector, where the normalized steering vector is derived as $\bar{\boldsymbol{v}}=\hat{\boldsymbol{v}}/|\hat{\boldsymbol{v}}|$. A positive semi-definite constraint is also imposed on the estimated inverse of the interfering noise covariance matrix. This is done by modifying the GRU-Net to estimate an upper triangular matrix $U$, by which the inverse matrix is calculated as $\mathbf{\hat{\Phi}}^{-1}_{\mathrm{VV}}=UU^{H}$.

Secondly, as we mentioned at the beginning of this section, we introduce the VAD network to control the gain of the output.

	$\displaystyle\hat{\mathbf{w}}_{\text{VAD}}^{(k)}(t)$	$\displaystyle=\mathbf{GRU{\text{-}}Net}_{\text{VAD}}(\mathbf{M}_{\mathrm{S}}^{(k)}(t,f)),$		(6)
	$\displaystyle\hat{\mathbf{S}}_{\text{VAD}}^{(k)}(t,f)$	$\displaystyle=\hat{\mathbf{S}}_{\text{ADL-MVDR}}^{(k)}(t,f)\hat{\mathbf{w}}_{\text{VAD}}^{(k)}(t),$

where $\hat{\mathbf{S}}_{\text{VAD}}^{(k)}$ is the VAD filtered speech for source $k$, $\hat{\mathbf{w}}_{\text{VAD}}^{(k)}\in\mathbb{R}$ denotes the estimated frame-level VAD weights. Finally, the separated speech $\hat{\mathbf{S}}^{(k)}$ can be obtained with residual connection as

$$\hat{\mathbf{S}}^{(k)}(t,f)=\hat{\mathbf{S}}_{\text{VAD}}^{(k)}(t,f)+\alpha\cdot\mathbf{\hat{S}_{\text{mask}}}^{(k)}(t,f),$$

(7)

where $\alpha$ is a weighting factor, and we use the first channel of the masked speech for the residual connection.

We use a T-F mask estimation model proposed in [28]. The multi-channel input speech stream is first encoded by a shared encoding block (based on conformer layers [29, 7]) to extract the intra-channel feature independently, followed by a stack of geometry agnostic modules. Within each geometry agnostic module, there is a transform-average-concatenate (TAC) block [6] and another shared conformer to alternately encode the inter- and intra-channel information. Finally, the encoded features are pooled out by taking the average across the channel dimension and fed into another conformer block to estimate the three T-F masks (i.e., two speakers and one noise [23]).

Each GRU-Net consists of two unidirectional GRU layers, followed by another feed forward layer (FFL) as illustrated in Fig. 3. For frame-wise coefficients estimation, the GRU-Net takes in the covariance matrix that is derived following Eq. (3). Note that the real and imaginary parts are concatenated as input. We use the same architecture for both the ADL-MVDR and VAD networks except the difference in the number of units in each layer.

Our training set contains 219 hours of randomly mixed and reverberated utterances from WSJ1 SI-284 [30]. We simulated the multi-channel mixtures by randomly picking the audio of one or two speakers and convolving it with a 7-channel room impulse response, which was simulated with the image method [31]. Then, the reverberated signals were mixed with a source energy ratio between -5 and 5 dB. Simulated isotropic noise was then added with a 0-10 dB signal-to-noise ratio (SNR). Noise samples from MUSAN [32] were also reverberated and added at an SNR between -5 and 10 dB. The average overlap ratio of the training set was about 50%.

LibriCSS [19] was used for the first experiment, which consists of 10 hours of 7-channel recordings. Sound sources were generated by mixing LibriSpeech utterances [20]. They were played back in a real meeting room and recorded by a 7-channel microphone array. Several overlap ratios were included, ranging from 0 to 40%. There were two subsets for the 0% overlap condition: one with short (S) inter-utterance silence and one with long (L) inter-utterance silence.

To evaluate the performance of the proposed neural beamformer in more realistic and challenging environments, we also carried out experiments using real meeting datasets; namely, the AMI corpus [21] and Microsoft internal meeting dataset, dubbed as MS. The AMI recordings were made with an 8-channel circular microphone array, while the MS data collection used a 7-channel array used in [19]. MS contained 60 sessions in total with various numbers of speakers per session. In order to get the transcriptions, we adopted a modified version of the conversation transcription system described in [3] with a hybrid ASR model.

We considered two different geometry agnostic T-F mask estimation models as the baseline spectral masking systems: one predicting real-valued masks using sigmoid activation and the other estimating complex-valued masks. We used the mask estimation model from our recent paper [28]. The model consists of nine consecutive conformer layers that were interleaved by two TAC layers, followed by mean pooling and another stack of six conformer layers. Each conformer layer used four heads with 64 dimensions and 33 convolution kernels. See [28] for further details¹¹1Note that our model setting is different from the one used in [28], and thus the numbers cannot be directly compared..

As regards the ADL-MVDR component, we used 200 and 100 units in the two recurrent layers of $\mathbf{GRU{\text{-}}Net}_{\upsilon}$, followed by another 14-unit FFL with linear activation. For the $\mathbf{GRU{\text{-}}Net}_{\text{VV}}$, there were 200 units in both GRU layers with another 98-unit linear FFL. For VAD weights estimation, the corresponding GRU-Net featured a two-layer GRU with 200 units each, followed by a 1-unit (i.e., frequency-independent) FFL with ReLU activation.

Two loss functions were examined; namely, the mean squared error (MSE) of the magnitude spectra [27] and the MSE of the log-mel features [33] obtained with 80 mel filter banks. We did not use the scale-invariant signal-to-distortion ratio (Si-SDR) loss which was used before for target speech extraction [15, 16]. This is because the Si-SDR loss was found to be unstable when one source was silent as with the CSS task. $\alpha$ was set to 0.5 for residual connection. The AdamW optimizer [34] was used with a weight decay of $1e^{-2}$. A warm-up learning schedule was used with the peak learning rate of $1e^{-3}$ followed by an exponential decay. The baseline systems were trained for 50 epochs, and the ADL-MVDR systems were trained jointly with the baseline systems for another 100 epochs. For CSS chunk-wise processing, $N_{h},N_{c},N_{f}$ were set to 1.2 s, 0.8s and 0.4 s, respectively.

The experimental results with different system configurations are shown in Table 1. Among the systems implementing some or all of the proposed improvements, the best performing approach was system 10 with a real-valued mask estimator. The proposed system achieved an approximately 9% relative WER improvement on average (i.e., 10.1 % vs. 11.1%) compared with the baseline system (i.e., system 0 with real-valued mask). The proposed end-to-end neural beamforming was more advantageous for more challenging conditions, yielding a 15% relative gain for the 30% overlap condition (i.e., 12.5% vs. 14.7%).

It is noteworthy that system 3 significantly outperformed system 2 (11.3 % vs. 13.7% for real-valued masks), and the latter even underperformed the baseline system. This indicates that the beamformer cannot fully eliminate the speech signals even when the output T-F masks are almost zero, especially in reverberant conditions [5]. The use of the VAD network helped the neural beamformer deal with the case of the output source being occasionally zero, as in the CSS task.

The results of Table 1 also show the impact of the normalized steering vector and applying the positive semi-definite constraint. Normalizing the estimated steering vector improved the ASR accuracy in most conditions. In particular, if we compare systems 3 and 4, we can see 3.5% (i.e., 11.3% vs. 10.9%) and 3.6% (i.e., 11.1% vs. 10.7%) relative WER gains for the real- and complex-valued mask configurations, respectively. Similar trends were observed for systems 7 and 8. Also, the positive semi-definite constraint on the estimate of $\mathbf{\Phi}^{-1}_{\mathrm{VV}}$ was found to preserve the performance with the reduced number of parameters.

For the proposed end-to-end neural beamformers, the log-mel scale magnitude loss consistently yielded better ASR accuracy. For instance, between systems 10 and 6, there were 5.6% and 6.4% relative improvements for the real- and complex-valued mask configurations, respectively.

For the real meeting experiments, we used a larger training set consisting of 438 hours of mixtures. Table 2 compares the proposed all-neural beamformer (i.e., system 10 with real-valued masks) with two baseline systems: one used the conventional chunk-wise MVDR as with [28]. We also show the result with a super-directive beamformer with real-time beam steering, which only removes ambient noise and does not perform speech separation (denoted as BF).

For the MS dataset, the proposed ADL-MVDR achieved the best ASR accuracy (i.e., 16.3%), which is about 8.4% and 1.8% relative improvements over BF and the conventional MVDR systems. While comparable performance was observed on the AMI development set for two MVDR systems, the proposed ADL-MVDR slightly outperformed the conventional MVDR by relative 2.4% (i.e., 20.6% vs. 21.1%) in the AMI evaluation set. These results here suggest that the proposed neural beamformer achieved comparable ASR accuracy with a state-of-the-art MVDR-based system while also enabling frame-wise beamforming.