Scaling sparsemax based channel selection for speech recognition with
ad-hoc microphone arrays

Fig. 1 shows the single-channel system. It is trained with clean speech. Specifically, given the input acoustic feature of an utterance $\bm{X}\in\mathbb{R}^{T\times D_{x}}$ and its target output $\bm{O}\in\mathbb{R}^{L\times D_{v}}$, where $T$ and $D_{x}$ is the length and dimension of $\bm{X}$ respectively, and $D_{v}$ is the vocabulary size. First, $\bm{X}$ is processed by a convolutional downsampling layer which results in $\bm{\tilde{X}}\in\mathbb{R}^{\tilde{T}\times D_{x}}$. Then, $\bm{\tilde{X}}$ passes through an encoder $\rm{Enc(\cdot)}$ and a decoder ${\rm Dec}(\cdot)$:

which produces a context vector $\bm{c}_{l}\in\mathbb{R}^{D_{h}}$ at each decoding time step $l$ given the decoding output of the previous time steps $\bm{y}_{1:l-1}\in\mathbb{R}^{l-1\times D_{v}}$, where $\bm{H}\in\mathbb{R}^{\tilde{T}\times D_{h}}$ is a high level representation extracted from the encoder. Finally, $\bm{c}_{l}$ is transformed to an output vector $\bm{y}_{l}$ by a linear transform. The objective function of the conformer is to maximize:

where $\bm{o}_{l}$ is the $l$-th time step of $\bm{O}$.

Multi-head attention (MHA) mechanism is used in both the encoder and the decoder, which is the key difference between our conformer architecture and the model in [13]. Each scaled dot-product attention head is defined as:

where $\bm{Q_{i}}\in\mathbb{R}^{T_{1}\times D_{k}},\bm{K_{i}},\bm{V_{i}}\in\mathbb{R}^{T_{2}\times D_{k}}$ are called the query matrix, key matrix and value matrix, respectively, $n$ is the number of the heads, $D_{k}=D_{h}/n$ is the dimension of the feature vector for each head. Then, MHA is defined as:

where ${\rm Concat}(\cdot)$ is the concatenation operator of matrices,

and $\bm{W}^{O}\in\mathbb{R}^{D_{h}\times D_{h}}$ is a learnable projection matrix.

Fig. 1 shows the multichannel system. In the figure, (i) the modules marked in blue are pre-trained by the single-channel ASR system, and shared by all channels with their parameters fixed during the training of the multichannel ASR system; (ii) the module marked in green is the stream attention, which is trained in the second stage with the noisy data from all channels.

The architecture of the multichannel system is described as follows. Given the input acoustic feature of an utterance from the $k$-th channel $\bm{X}_{k}\in\mathbb{R}^{T\times D_{x}},\ k=1,\cdots,C$ where $C$ represents the total number of channels, we extract high level representations $\bm{H}_{k}$ from each channels:

Then, we concatenate the context vectors of all channels:

where

At the same time, we extract a guide vector $\bm{g}_{l}\in\mathbb{R}^{D_{h}}$ from the output of the decoder at all previous time steps by:

where $\bm{W}^{Y_{1}},\bm{W}^{Y_{2}},\bm{W}^{Y_{3}}\in\mathbb{R}^{D_{v}\times D_{h}}$ denote learnable projection matrices. The guide vector $\bm{g}_{l}\in\mathbb{R}^{D_{h}}$ is used as the input of the stream attention which will be introduced in Section 3.

As shown in the fusion layer in Fig. 1, the stream attention takes the output of a MHA layer as its input. The MHA extracts high-level context vector by:

where $\bm{W}^{C},\bm{W}^{H_{1}},\bm{W}^{H_{2}}\in\mathbb{R}^{D_{h}\times D_{h}}$ denote learnable projection matrices.

Then, the stream attention calculates

with $\bm{Q}=\bm{g}^{T}_{l}\bm{W}^{G}$, $\bm{K}=\bm{\hat{C}}_{l}\bm{W}^{\hat{C}_{1}}$ and $\bm{V}=\bm{\hat{C}}_{l}\bm{W}^{\hat{C}_{2}}$, where $\bm{g}_{l}$ is the guide vector defined in (10),

and $\bm{W}^{G},\bm{W}^{\hat{C}_{1}},\bm{W}^{\hat{C}_{2}}\in\mathbb{R}^{D_{h}\times D_{h}}$ are learnable projection matrices. Finally, we get the output vector $\bm{y}_{l}$ of the decoder through an output layer from $\bm{r}_{l}$.

Fig. 2 shows the architecture of the Softmax-based stream attention that has been used in [13].

The common Softmax has a limitation for ad-hoc microphone arrays that its output ${\rm Softmax}_{i}(z)\neq 0$ for any $z$ and $i$, which can not be used for channel selection. To address this problem, we propose Sparsemax stream attention as shown in Fig. 2, where the Sparsemax [15] is defined as:

where $\Delta^{K-1}=\left\{\bm{p}\in\mathbb{R}^{K}\ |\sum_{i=1}^{K}p_{i}=1,p_{i}\geq 0\right\}$ represents a $(K-1)$-dimensional simplex. Sparsemax will return the Euclidean projection of the input vector $\bm{z}$ onto the simplex, which is a sparse vector. Its solution has the following closed-form:

where $\tau:\mathbb{R}^{K}\rightarrow\mathbb{R}$ is a function to find a soft threshold, which will be described in detail in the next section.

From section 3.2, we see that the output of Sparsemax is related to the input vector and the dimension of the simplex. However, in our task, the values of the input vector vary in a large range caused by the random locations of the microphones. The dimension of the simplex is related to the number of the channels which is also a variable. Therefore, Sparsemax may not generalize well in some cases.

To address this problem, we propose Scaling Sparsemax as shown in Fig. 2. It rescales Sparsemax by a trainable scaling factor $s$ which is obtained by:

where $\|\bm{z}\|$ is the L2 norm of the input vector, and $\rm{Linear}(\cdot)$ is a $1\times 2$-dimensional learnable linear transform.

Algorithm 1 describes the Scaling Sparsemax operator. When $s=1$, Scaling Sparsemax becomes equivalent to Sparsemax.

Our experiments use three data sets, which are the Librispeech ASR corpus [16], Librispeech simulated with ad-hoc microphone arrays (Libri-adhoc-simu), and Librispeech played back in real-world scenarios with 40 distributed microphone receivers (Libri-adhoc40) [17]. Each node of the ad-hoc microphone arrays of Libri-adhoc-simu and Libri-adhoc40 has only one microphone. Therefore, a channel refers to a node in the remaining of the paper. Librispeech contains more than $1000$ hours of read English speech from $2484$ speakers. In our experiments, we selected $960$ hours of data to train single channel ASR systems, and selected $10$ hours of data for development.

Libri-adhoc-simu uses 100 hours ‘train-clean-100’ subset of the Librispeech data as the training data. It uses ‘dev-clean’ and ‘dev-other’ subsets as development data, which contain 10 hours of data in total. It takes ’test-clean’ and ’test-other’ subsets as two separate test sets, which contain 5 hours of test data respectively. For each utterance, we simulated a room. The length and width of the room were selected randomly from a range of $[5,25]$ meters. The height was selected randomly from $[2.7,4]$ meters. Multiple microphones and one speaker source were placed randomly in the room. We constrained the distance between the source and the walls to be greater than $0.2$ meters, and the distance between the source and the microphones to be at least $0.3$ meters. We used an image-source model¹¹1https://github.com/ehabets/RIR-Generator to simulate a reverberant environment and selected T60 from a range of $[0.2,0.4]$ second. A diffuse noise generator²²2https://github.com/ehabets/ANF-Generator was used to simulate uncorrelated diffuse noise. The noise source for training and development is a large-scale noise library containing over $20000$ noise segments [18], and the noise source for test is the noise segments from CHiME-3 dataset [19] and NOISEX-92 corpus [20]. We randomly generated $16$ channels for training and development, and $16$ and $30$ channels respectively for test.

Libri-adhoc40 was collected by playing back the ‘train-clean-100’, ‘dev-clean’, and ‘test-clean’ corpora of Librispeech in a large room [17]. The recording environment is a real office room with one loudspeaker and $40$ microphones. It has strong reverberation with little additive noise. The positions of the loudspeaker and microphones are different in the training and test set, where the loudspeaker was placed in 9, 4, and 4 positions in the training, development, and test sets respectively. The distances between the loudspeaker and microphones are in a range of $[0.8,7.4]$ meters. We randomly selected $20$ channels for each training and development utterances, and $20$ and $30$ channels for each test utterance which corresponds to two test scenarios.

The feature and model structure are described in Table 1. In the training phase, we first trained the single channel conformer-based ASR model with the clean Librispeech data. When the model was trained, the parameters were fixed and sent to the multichannel conformer-based ASR model. Finally, we trained the stream attention with the multichannel noisy data. In the testing phase, we used greedy decoding without language model. WER was used as the evaluation metric.

We compared the proposed Sparsemax and Scaling Sparsemax with the Softmax stream attention. Moreover, we constructed an oracle one-best baseline, which picks the channel that is physically closest to the sound source as the input of the single channel conformer-based ASR model. Note that the keyword “oracle” means that the distances between the speaker and the microphones are known beforehand.

Table 2 lists the performance of the comparison methods on Libri-adhoc-simu. From the table, we see that (i) all three stream attention methods perform good in both test scenarios. Particularly, the generalization performance in the mismatched $30$-channel test environment is even better than the performance in the matched $16$-channel environment. It also demonstrates the advantage of adding channels to ad-hoc microphone arrays. (ii) Both Sparsemax and Scaling Sparsemax achieves significant performance improvement over Softmax. For example, Scaling Sparsemax stream attention achieves a relative WER reduction of $33.90$% over Softmax on the ‘test-clean’ set and $26.0$% on the ‘test-other’ set of the 30-channel test scenario.

Table 3 shows the results on the Libri-adhoc40 semi-real data. From the table, one can see that the proposed Scaling Sparsemax performs well. It achieves a relative WER reduction of $17.4$% over the “oracle one best baseline” on the 20-channel test scenario, and $14.2$% on the mismatched 30-channel test scenario.

Fig. 3 shows a visualization of the channel selection effect on an utterance of Libri-adhoc40. From the figure, we see that (i) Softmax only considers channel reweighting without channel selection; (ii) Although Sparsemax conducts channel selection, its channel selection method punishes the weights of the channels too heavy. (iii) Scaling Sparsemax only sets the weights of very noisy channels to zero, which results in the best performance.