Multi-Channel Automatic Speech Recognition Using Deep Complex Unet

DCUnet [28] is an extension of the Unet structure proposed in image segmentation using complex-valued convolutional encoders and decoders. The encoder and decoder are stacked by several blocks which contain 2D (de)convolution layers, batch normalization (BN) and leaky rectified linear units (leaky-ReLU). The multi-channel version we used in this paper takes the short-time Fourier transform (STFT) of multi-channel signals as the input features. The structure of the input layer is shown in Fig.1, where the STFT of multi-channel signal is divided into real and imaginary parts and convolves with the corresponding filters. For each channel $c$, the complex-valued convolutional operation can be interpreted as two real-valued ones, following the formula:

$$\mathbf{W}\circledast\mathbf{X}_{c}=\begin{bmatrix}\mathbf{W}_{i}\\ \mathbf{W}_{r}\end{bmatrix}\circledast\begin{bmatrix}\mathbf{R}_{c}\\ \mathbf{I}_{c}\end{bmatrix}=\begin{bmatrix}\mathbf{W}_{i}*\mathbf{R}_{c}-\mathbf{W}_{r}*\mathbf{I}_{c}\\ \mathbf{W}_{i}*\mathbf{I}_{c}+\mathbf{W}_{r}*\mathbf{R}_{c}\end{bmatrix}$$

(1)

where $\circledast$ and $*$ denote complex-valued and real-valued convolution, respectively. $\mathbf{W}=[\mathbf{W}_{r},\mathbf{W}_{i}]^{T}$ are complex-valued convolution filters and $\mathbf{X}_{c}=[\mathbf{R}_{c},\mathbf{I}_{c}]^{T},c\in[1,2,\text{ref}]$ are STFT of the signal at channel $c$. Batch normalization is applied on real and image channels independently, excluding the final layer of the decoders.

Different from the original structure designed for single-channel speech enhancement, the multi-channel DCUnet does not estimate complex ratio mask (cRM) [30] but predicts the complex spectrogram directly because the training data is collected in real-world environments and lack of ground-truth signals. The audio after the signal processing steps are used as the supervision of the DCUnet and the scale issues may exist. Thus the mask-based loss function can not be optimized stably. We use the mean square error (MSE) between the magnitude of the supervision $\mathbf{M}_{\text{sup}}$ and the prediction output $\mathbf{O}_{\text{DCUnet}}$ as the objective function as shown in Eq. (2), considering that the phase is always neglected during the feature extraction stage in the recognition task.

$$\mathcal{L}_{\text{enh}}=\|\mathbf{M}_{\text{sup}}-|\mathbf{O}_{\text{DCUnet}}|\|_{F}^{2}$$

(2)

One straightforward way to joint the multi-channel DCUnet with the acoustic model is to compose them in a cascaded structure, as shown in Fig. 2. We expect the DCUnet to learn the spectrogram representation that benefits speech recognition without signal supervision and fully depends on the optimization of the back-end. At the beginning, STFT is applied to the input signals (1,2 and reference channels) to obtain complex spectrogram. Then the DCUnet predicts the complex spectrogram of each channel. After that, the power spectrogram is calculated and the feature extraction module computes the log mel-filter bank (log-FBank) features. The acoustic model used in the back-end contains a 2-layer CNNs and a 12-layer time delay neural networks (TDNN). The log-softmax layer is applied in the last to obtain the log-probabilities.

Along with the enhancement-recognition cascade structure, we can also use a multi-task learning (MTL) framework, where the main task is speech recognition and the auxiliary task is speech enhancement. We make the encoders shared by enhancement decoders and recognition network in order to extract the representation of the input signals that can benefit both tasks. The framework is shown in Fig.3, where TCNN is the abbreviation of transposed convolution layers.

It is worth noting that the features generated by DCUnet encoder are still in complex values, while the first layer of the back-end is a CNN of real-value. In order to join the complex CNN and the real-valued one, real and imaginary parts of the complex CNN outputs are regarded as two different channels in order to be compatible with the real-valued CNNs. For example, the output of DCUnet encoder consists of 8 complex feature maps, and we send it to the real-valued CNN as 16 real-valued feature maps.

The structure of the DCUnet in cascaded and MTL framework are same, where the encoder contains 4 complex convolutional layers and the decoders consists of 4 complex deconvolutional layers. Specifically, the number of the output channels for the first three convolutional layers are 16, 32, 64 with the kernel size $7\times 5$, respectively. And the last layer contains 64 filters with the kernel size $5\times 3$.

In DCUnet-MTL structure, the gradients back-propagated from the two outputs are weighted by $\beta$ and $1-\beta$ for enhancement and recognition tasks respectively. Specifically, the objective function for the DCUnet-MTL network can be written as:

$$\mathcal{L}=\begin{cases}(1-\beta)\mathcal{L}_{\text{asr}}+\beta\mathcal{L}_{\text{enh}}&\text{$t\leqslant T_{\text{enh}}$},\\ \mathcal{L}_{\text{asr}}&\text{$t>T_{\text{enh}}$}\end{cases}$$

(3)

where $\mathcal{L}_{\text{asr}}$ and $\mathcal{L}_{\text{enh}}$ stands for recognition and enhancement loss functions, respectively. $\beta$ is a weight parameter between 0 and 1. $T_{\text{enh}}$ represents the number of enhancement task iterations and $t$ indicates the current iteration number in the training progress. The enhancement task will only iterates for the first $T_{\text{enh}}$ times, and after that the recognition criterion will take over the training. The end-to-end lattice-free MMI criterion [31, 1] is served as the objective function on the recognition task in our paper.

The training of the enhancement branch requires the noisy-clean data pairs. As explained in Section 2.1, we use the data generated by signal pre-processing steps, which consists of the Wiener filter [12] based automatic echo cancellation (AEC) followed by delay and sum (DS) beamforming^†††https://github.com/xanguera/BeamformIt, as the supervision. Specifically, we performed AEC on the first two channels (channel 1, 2) using the third channel signal as the echo reference. Then a simple DS beamformer is adopted to form the final enhanced signal, which is treated as the “target signal” for training of the DCUnet. Obviously it’s not the ideal supervision, so we only enable $\mathcal{L}_{\text{enh}}$ for several iterations.

In order to further improve the performance, we use a well-trained DCUnet as initialization. Specifically, the full-structured DCUnet for the initialization purpose is firstly trained on the same dataset, and the DCUnet part in the two structures (cascaded and MTL) is initialized with this well-trained DCUnet. We find that such initialization benefits the performance for both structures. Besides, extra gains can be found by inserting a dropout layer before the back-end acoustic model.

All experiments are performed on the real-world Mandarin data from XiaoMi smart speaker. The device is equipped with two receivers in horizontal dimension, and the distance between the two receivers is about 7cm. A build-in loudspeaker is located in the middle of the two receivers. Before speech communications triggered by a hot word, the smart speaker may be playing music or speech responses, or it keeps silence. Once waken up, it will turn the speaker volume down rather than off; at the same time, it will give a speech response, such as “I’m here” in Chinese. The interactions with the smart speaker are unconstrained. The 16kHz training and testing data amounts to approximately 1000 hours and 35 hours, respectively.

In order to be comparable with the traditional technique solutions, we use the audio data processed by AEC followed by the DS beamforming to train the acoustic model, which is served as the single channel baseline. The 80-dimensional log-FBanks are extracted as the input features, with 25ms window size and 10ms frame shift. The acoustic model consists of 2-layer real-valued CNN followed by 12-layer TDNN. Specifically, the first convolutional layer with 64 filters of size $5\times 3$ is used and the second one is of $3\times 3$ with 128 filters. The first 11 TDNN layers, each has 1024 hidden nodes and followed by a batch normalization layer and the rectified linear units (ReLU). The last layer has 2888 units with log-softmax applied to the output.

Besides, we try a typical neural fixed beamforming (NNFB) method similar to [8, 14] as the multi-channel acoustic model baseline. The architecture contains a fixed beamformer layer, which aims to enhance source signals on different look directions. As suggested in [14], we initialize the filter coefficients using the SuperDirective (SD) beamformers. For beam selection, we employ a TDNN layer to predict the weight on each direction and the sum of the weighted directional log-FBank features is used as the input of the acoustic model, as shown in Fig. 4. In our experiments, we use 8 different directions for the fixed beamformer layers and the two-channel inputs are processed after the AEC steps, considering that the neural beamforming method can not handle echo issues well.

The comparison of the four systems – single-channel ASR, NNFB, DCUnet-MTL and DCUnet-CAS, is displayed in Table 1. It is clear that the NNFB provide higher accuracy than single-channel ASR because the trainable beamforming filters can be optimized jointly with the input features and acoustic network, which is more likely to achieve the optimal solution for recognition task. For the DCUnet-MTL structure, we test two values for the iteration number $T_{\text{enh}}$ of the enhancement task. Experiments show that the result of training without the regularization of the enhancement task ($T_{\text{enh}}=0$) is better and we observe that the ASR branch converges faster than the enhancement branch during training progress. In order to achieve better performance, we train the DCUnet front-end separately and get those DCUnet based structures initialized with the well-trained model, which will be discussed in the next section.

As is demonstrated in Table 2, recognition accuracy improves after better initialization strategy is used and the DCUnet-MTL structure can get higher accuracy with the regularization of the enhancement branch. Considering the supervision we used is not ideal for enhancement task, moderate guidance of enhancement can be a better choice. We find that iterating the enhancement and recognition branch for 3 and 6 epochs shows the best CER 14.03% on this task. DCUnet-CAS model can also benefit from the better initialization, improving the result from 16.51% to 14.09%.

Another conclusion could be drawn from the table is that the extra gains would be got if we insert a dropout layer before acoustic model in our proposed structure, especially after getting DCUnet better initialized. And the best CER is pushed to 13.48% with the usage of the dropout. However, single channel network and NNFB cannot get gains from that.

We further tune the weights of recognition and enhancement tasks, while fixing the iteration times of the enhancement branch with the usage of the dropout layer and model initialization. The result is shown in Table 3, where we can see that the structure can benefit from the enhancement task with higher weight in the first 3 iterations. Finally the CER reaches the lowest 13.23% on our test set.

In order to observe the results closely, we further divided the test set into four subsets, one with echos in the reference channel and the others are divided according to the signal-to-noise ratio (SNR). All the tested utterances start with the speech response of the smart speaker, which last around 0.8 second. Roughly speaking, in test set, data with echos takes up 23.5%. And for the rest non-echo part, data with SNR lower than 5dB occupies 14%, and data with SNR between 5dB and 15 dB accounts for 32%. The reset 30.5% of test set is the data whose SNR is higher than 15dB.

The comparison of proposed and baseline systems on different subsets is shown in Table 4. DCUnet-MTL structure gives best performance on each subset. Compared with the NNFB method and baseline system, it shows 9.8% and 12.2% relative CER reduction totally. DCUnet-CAS brings sub-optimal results, but still shows advantages. Another conclusion drawn from the table is that the recognition rate of low SNR is still worse than that of high SNR. And even if there is a relatively small echo, it still has an impact on recognition accuracy.