a unified multichannel far-field speech recognition system: combining neural beamforming with attention based end-to-end model

Illustrated as in Fig.2, the neural beamforming block contains two parts - the spatial filtering and spectral filtering components. The spatial filtering part is similar as to form different look directions, while spectral filtering is analog to find the most informative direction and extract features for recognition task.

Specifically, the input of this block is the multichannel waveforms, which are then transformed into frequency domain by short-time Fourier transform (STFT) with window size as $L$. The outcome is denoted as $X_{c}[t]\in\mathbb{C}^{K}$, where $t$ and $c$ are time and channel indices, $K=N/2+1$, where $N$ is the number of fast Fourier transform (FFT). Suppose $P$ look directions are applied, then $P$ spatial filter $H_{c}^{p}\in\mathbb{C}^{K}$ will be applied to beamform the frequency domain signals as

$$Y_{p}[t]=\sum_{c=1}^{C}X_{c}[t]\cdot H_{c}^{p}$$

(5)

where $\cdot$ denotes the matrices element-wise product.

As for the spectral filtering part, we use the factored Complex Linear Projection (fCLP) [21] as it reduces computational complexity and results in similar performance compared to a single channel system [22]. This is accomplished by using complex filterbank $S_{f}^{p}\in\mathbb{C}^{K}$ to spatial filter’s outcome as

$$Z_{f}^{p}[t]=Y_{p}[t]\cdot S_{f}^{p}$$

(6)

The outputs of the filterbank is then summed and log compressed for each direction and spectral filter as

$$O_{f}^{p}[t]=\log|\sum_{k=1}^{K}Z_{f}^{p}[t,k]|$$

(7)

The last step of the neural beamforming block serves as feature extraction when compared with the conventional way, e.g. filter-bank acoustic features. As can be seen in (7), although elements in one spectral filter has been summed, there are $P$ directions that may be redundant. In the conventional signal processing module, there are usually $P_{dsp}$ (e.g. $P_{dsp}=5$) look directions in the beamforming that is formed by algorithms like GSC. DOA is then used to choose the best matching direction of the beamforming as the output. Similarly, pooling layer over the look direction is then applied on $O_{f}^{p}[t]$ before fed into the encoder. In the proposed framework, we tried three different pooling strategies, as shown in Fig.3. (1) Max-pool: the most significant features from different look directions were integrated into a single feature vector. (2) Projection: the features from all look directions were concatenated, and a linear layer was then used to project the concatenated feature into 40 dimension. (3) Attention: features from all directions were integrated according to their importance to recognition, the importance weight was computed by a learnable matrix $W$ and the output is the weighted summation as

$$O[t]=\sum_{p}^{P}O_{p}[t]\cdot\mathrm{softmax}(WO_{p}[t]).$$

(8)

The rest of system is the same as described in section 2. As the multiplication operation in (6-7) and the norm can be represented by real matrix multiplication [22], gradients are represented in real value and back-propagation can be used to train the entire model.

When compared to the conventional method, another advantage of this unified framework also comes from the computational complexity. In the conventional signal processing module, beamforming algorithms may require matrix inversion to get the solution. As there are $P_{dsp}$ (e.g. $P_{dsp}=5$) look directions, multi-time matrix inversion needs to be processed, which is computational intensive. In the proposed framework, computations are mainly from (6-7), but the matrices multiplication can be well parallelized by modern processors.

As latest research indicates, incorporating source direction in beamforming may improve the speech separation performance [19]. The source direction as prior is usually available in scenario like multi-modality, in which source direction can be more accurately estimated by computer vision. In this work, the source direction information is integrated in aforementioned neural beamforming as illustrated in figure 4. Specifically, two different methods are explored. In both methods, source direction is the angle of source, which is then represented by angle embedding. Angle embeddings are learnable variables. First method augments neural beamforming as a direction aware module which concatenates angle embedding with outputs of each spatial filter in neural beamforming before projection as illustrated by figure 4.a. Second method applies a direction attentive module as illustrated by 4.b. This mechanism is similar as CLAS [23]. The output of spatial filter is

$$Y[t]=\sum_{p}^{P}\alpha_{p}(t)Y_{p}[t]$$

(9)

where

$$u_{p}(t)=v^{T}\mathrm{tanh}(W_{p}^{0}Y_{p}+W_{p}^{e}E)$$

(10)

$$\alpha_{p}(t)=\mathrm{softmax}(u_{p}(t))$$

(11)

and $W_{p}^{0}$, $W_{p}^{e}$, $v$ are learnable variables. The idea is to use source direction in order to select the most informative direction.

To fully test the effectiveness of the proposed framework, two different databases were used. The first one was collected in a near-field cellphone setting, which includes around 3000 hours data. The second is an in-house database which containing 2400 hours online audio data.

The near-field waveforms in the original databases were artificially corrupted by a room simulator, and then processed by adding different levels of noise and reverberation. Room simulator described in [24] was used. We randomly chose different room dimensions and the position of microphone array from a pools of samples. Room configurations with RT60s range from 50 to 500 ms. In this simulation task, we used two microphone arrays to generate two-channel recordings. The spacing of the microphone array was set as 4cm for training set. When adding noise, the level of noise is varying from utterance to another, with SNR obey to normal distribution from 0 to 20 dB. The distance between speech source and microphones varies in range of 0.5 to 7m. To test the robustness against the varying angle between speech source and interference, azimuths of speech and noise were randomly chosen from range of $\pm 180$ degrees and elevation of speech and noise was constrained to be between [0.6m, 2.0m] and [0.4m, 3.0m]. We composed a evaluation set by randomly selecting utterances from the full-sized database. It roughly contains 15K utterance and is around 15h hours. Same simulation process was used as the training set.

For the in-house real data, online audio data are anonymiz-ed and extracted from on-line service. Compared to simulated data, real data is usually recorded in a less complicated environment with fewer noise sources and smaller room size which results in a higher SNR and lower reverberation. On the other hand, real data is more in-domain and has less generalization.

The baseline system is a transformer based CTC/Attention speech recognition system with conventional beamforming. GSC with five look directions and DOA was used to obtain the enhanced speech signal. Filter-bank features with 40 dimensions were then extracted before sent to the encoder. The inputs were optionally processed by offline weighted prediction error (WPE) for de-reverberation [25]. The number of look direction $P$ in neural beamforming was chosen as 10 and the number of filters in spectral filtering part is 40.

In the encode-decoder configuration, 7 transformer blocks and 2 transformer block were used for encoder and decoder. $\lambda=0.1$ in (4) was used in the entire experiments. Adam optimizer was used to train the networks with initial learning rate as 0.001. The learning rate is decayed as in [26]. Similarly, we also use warm up step which is set as 4000. The entire networks were trained with 30 epochs for both conventional and proposed framework. When decoding, beam search algorithm was used. An external LSTM-based language model was used to rescore the decoding paths in the manner of shallow fusion [27].

First experiment was conducted to evaluate the proposed neural beamforming end-to-end ASR system (NBE2E) and compared it with the baseline single-channel system (DSPE2E). Word error rate (WER) of both systems on the evaluation set with and without WPE processing were reported. To evaluate different pooling methods in neural beamforming, we also reported the performance of different pooling strategies (max-pool/proj/attn) with or without WPE processing (w/o WPE and w/ WPE).

The results are presented in table 1. Firstly, the proposed system achived WER of 24.94%, while the one of baseline is 30.89%, which accounts for 19.26% relative improvement with the unified system. This supports the idea that extending end-to-end ASR system to unify neural beamforming is beneficial and suggests a better option for far-field scenarios. Furthermore, amongst the three different pooling approaches, the projection is the best and outperformed the attention method by relative 6.31%. This is perhaps because the projection approach preserves more information. It is also observed that, for both systems, when cascaded with WPE, better performances can be obtained. Relative 5.10% and 2.31% improvements are obtained for proposed system and baseline. This suggests that de-reverberation is beneficial for both systems.

Further experiments on 2400h in-house data are used to test the effectiveness of the proposed unified ASR system in real world scenario and the results are presented in table 1. As negligible reverberation can be observed from real data, experiments on WPE were not conducted. The WER of the conventional DSP with end-to-end ASR system is 5.89%, while the unified ASR system achieved 5.40%, accounting for 8.32% relative improvement.

To evaluate how the proposed system and the baseline system designed for a specific microphone array geometry perform when generalizing to other microphone array geometries, the effects of different microphone array spacing was also evaluated in this section. In this experiment, models are the same as those of the previous section. The difference is that the array geometry in the evaluation set is categorized as 4cm, 6cm, 8cm and 10cm. In particular, we used the projection pooling method in the proposed method as it performs the best. The results are presented in table 3.

From table 3, it can be seen that the proposed system obtained better performance in 4cm category. This is reasonable as the models are trained with data from microphone array with 4cm. But it is also observed that the baseline system degrades more sharply and fluctuates more severely than the one of the unified end-to-end system. For example, when the distance of microphone array is 10cm, the baseline degraded to 32.03%.

This may be because that algorithms in conventional DSP module rely more on the prior knowledge and neural beamforming has better generalization ability for different microphone array geometry.

To test the effectiveness of integrating source direction as prior in the proposed network, in this experiment, the 2400 hours real data was simulated as randomly choosing the source direction in RIR. The source direction was recorded as prior for the model training phase and evaluation phase. Note that we did not add artificial noise in this setting. As usually the DOA algorithms are not that accurate, we randomly add a perturbation in the simulated source direction. It uniformly distributed between $-10$ to $10$ degrees.

The results are presented in table 4.5. The baseline is the proposed vanilla unified ASR system (without source direction), which achieved 10.90% WER in this task. NBE2E Dir-aware and NBE2E Dir-atten represent the proposed system augmented with a direction aware module and a direction attentive module as described in Section 3.3 respectively. 6.90% and 6.86% WER were achieved by the augmented models, accounting for 37.06% relative improvement. It can be seen that both two integrating methods outperformed the vanilla unified neural beamforming ASR system, indicating the superiority of using source direction as prior in the proposed unified ASR system. This finding is useful especially for multi-modality scenario.

The last experiment was conducted to compare the proposed unified multichannel end-to-end system with the baseline in a real-world scenario. It has been reported that the conventional beamformer and DOA may results in incorrect direction estimation of speech and interference source, as the resolution is limited [28]. As for the neural beamforming method, there is less reliance on the module functioning as estimating angle. The angle information of multichannel recordings is jointly learned and exploited by spatial and spectral filtering block.

Fig. 5 gives the experimental results as the rate of incorrect direction estimation enlarging. We artificially controlled the incorrect rate as to give an idea how systems reacts to this factor. To simulate the resolution limitation, the gpuRIR toolkit is used to obtain evaluation data [29], which makes it different with the one of previous sections. As can be seen that the overall trend for conventional beamforming technique tends to degrade from 37.92% to 39.25% when the incorrect rate of angle estimate increases from 0 to 100%. But since the proposed unified multichannel end-to-end system does not require angle estimation, the performance is flat, which indicates the proposed method could achieve more stable performance in the real-world scenario.