Neural Ambisonic Encoding For Multi-Speaker Scenarios Using A Circular Microphone Array
^††thanks: *This work was conducted during Yue Qiao’s internship at Tencent AI Lab, Bellevue, USA.

In this stage, the DNN uses the extracted features to predict complex-valued ratio filters (cRFs) for estimating the virtual loudspeaker signals corresponding to sound sources captured by the microphone array. This process is conceptually similar to plane wave decomposition of the captured sound field [20]. The extracted features are first processed through a one-layer long short-term memory (LSTM) module, a multi-head self-attention (MHSA) [21] module, and a linear layer for feature aggregation. The aggregated features are then passed through a series of four 1-D time-domain convolutional layers (T-Conv) to estimate the cRF masks, $\mathrm{cRF}_{\text{LS}}\in\mathbb{C}^{L\times M}$, for generating $L$ virtual loudspeaker signals, $\mathrm{VLS}(t,f)\in\mathbb{C}^{L}$:

where $i$ represents the taps of the cRFs.

In this stage, the DNN uses the estimated virtual loudspeaker signals and a concatenated single-channel microphone signal (see Fig. 2) to estimate a nonlinear spatial transformation for generating Ambisonic signals. Incorporating the microphone signal helps the DNN accurately generate the zeroth-order (omnidirectional) Ambisonic signal, similar to how residual connections in deep learning preserve key information and stabilize training. The concatenated signals are passed through linear layers for dimension reduction and gated recurrent unit (GRU) layers for feature aggregation, followed by four narrow-band blocks from SpatialNet [22] to estimate another set of cRFs. Each block includes a MHSA module and a time-convolutional feedforward module (T-ConvFFN), which perform spatial clustering and temporal smoothing/filtering, respectively. The MHSA also helps adapt the DNN weights to each loudspeaker’s spatial position. For more details on the narrow-band block, see [22]. The Ambisonic signals, $\hat{\mathbf{B}}(t,f)$, are generated by filtering the concatentaed signals with the estimated cRFs, $\mathrm{cRF}_{\text{SH}}\in\mathbb{C}^{(N+1)^{2}\times(L+1)}$:

Next, energy normalization is employed on the estimated Ambisonic signals to ensure the energy of the estimated zeroth-order Ambisonics matches that of the microphone signal. Finally, $\hat{\mathbf{B}}(t,f)$ is transformed to the time domain, $\hat{\mathtt{b}}$, with the inverse-STFT operation.

We use four loss functions to train the proposed DNN. The first three, Magnitude $l_{1}$-norm loss, signal-invariant signal-to-noise ratio (SI-SNR, [23]) loss, and coherence loss [11], aim to minimize the channel-wise errors between the estimated and ground truth Ambisonics. They are defined as

where $\hat{\mathtt{b}}_{n}$ and $\mathtt{b}_{n}$ are the estimated and ground truth time-domain Ambisonic signal of channel $n$, respectively, and

The fourth loss function is based on the spatial power map [24] derived from Ambisonic signals. The power map, $\Gamma(\theta,\phi)$, is computed using fixed beamformer weights, $\mathbf{w}(\theta,\phi)\in\mathbb{C}^{(N{+}1)^{2}}$, corresponding to directions, $\{\theta_{i},\phi_{i}\}_{i=1}^{Q}$:

In this work, we choose $\mathbf{w}(\theta,\phi)$ to be the maximum directivity index (max-DI) beamformer weights [25] for 1296 directions (spherical design of degree 50 [26]). The power map loss minimizes the difference between the estimated power map, $\hat{\Gamma}(\theta,\phi)$, and the ground truth power map, $\Gamma(\theta,\phi)$:

where $\mathcal{L}_{\text{KL}}$ is the Kullaback-Leibler divergence between the power map distributions, and $\alpha,\beta$ are weighting parameters ($\alpha{=}1,\beta{=}100$ used in training). The loss definition is inspired from [27] where audio localization maps are considered. The power map loss helps regularize the cross-channel correlation and therefore enhances the prediction of spatial information. During training, these four loss functions are equally weighted and combined.

When using a horizontal microphone array to encode Ambisonics, the vertical sound field components cannot be accurately encoded due to the lack of ability to distinguish between the sound sources “mirrored” in the horizontal plane. However, if we assume that all the sound sources are located in the same half-space, and this positioning is known or can be assumed, such ambiguity can be addressed by permuting the vertical Ambisonic channels. This assumption is often valid in scenarios like teleconferencing, where most speakers are either above or below the microphone array. Specifically, we permute the vertical channels (e.g., for second-order Ambisonics (SOA), $y_{1,0},y_{2,-1},y_{2,0},y_{2,1}$) by multiplying them with -1 (i.e., inverting the phase, equivalent to vertically flipping the sound sources) at model inference when the sources are in the lower half-space. This ensures a one-to-one mapping from the microphone input to the estimated sound field.

We evaluate two aspects of Ambisonic encoding performance using the speech and noise datasets: audio quality (timbral and spatial) and source localization accuracy. For timbral audio quality, we choose SI-SNR (Eq. 7) and envelope distance (ENV, [12]) between the estimated ($\hat{\mathtt{b}}$) and ground truth ($\mathtt{b}$) Ambisonic signals after Hilbert transform to measure temporal quality. Log spectral distance (LSD, [30]) and channel-wise coherence (Eq. 8) between $\hat{\mathbf{b}}(f)$ and $\mathbf{b}(f)$ are used for spectral quality. For spatial quality, we calculate the root mean square error (RMSE) between the power map $\hat{\Gamma}(\theta,\phi)$ and $\Gamma(\theta,\phi)$ in the SH domain; in the binaural domain, following [4], we decode Ambisonics into binaural signals using head-related transfer functions (HRTFs) from the KEMAR dataset [31]), then compute RMSE for inter-aural level difference (ILD) and inter-aural coherence (IC). For source localization accuracy, using the noise dataset, we estimate the direction of arrival by finding the maximum RMS value of the spatial power map, $\hat{\Gamma}(\theta,\phi)$, from the estimated Ambisonics and then calculate the azimuth and elevation errors relative to the ground truth. For simplicity, all metrics are computed for SOA and averaged across time and frequency for all evaluation samples.

We implement four baseline encoding methods (two SP-based and two DL-based) for performance comparison with the proposed method. The first SP-based method uses a delay-and-sum beamformer (DSB) to generate virtual loudspeaker signals for 160 fixed directions and synthesizes Ambisonics by filtering each loudspeaker signal with corresponding SH functions. The second SP-based method, similar to the conventional baselines in [6, 4, 11], filters the microphone signals using an encoding matrix, $\mathbf{E}\in\mathbb{C}^{(N{+}1)^{2}\times M}$, which linearly transforms the signals as

where $\hat{\mathbf{b}}(f)$ is the estimated Ambisonic signals. A closed-form solution for $\mathbf{E}(f)$ can be obtained in a least-squares sense, by minimizing the expectation of the $l_{2}$-norm difference between $\hat{\mathbf{b}}(f)$ and $\mathbf{b}(f)$[3, 32]:

where $\lambda(f)$ is a regularization parameter set to 0.01 in our experiment. The steering matrix $\mathbf{V}$ is sampled at 5^∘ resolution. To address the ambiguity issue (Sec. III-D), when it is unknown whether the sound sources are above or below the horizontal plane, the encoding matrix is generated using only the steering vectors above the horizontal plane. When such information is known, separate encoding matrices are generated for both cases using the corresponding steering vectors. The third method adopts the U-net-based architecture [11] with unchanged hyperparameters and loss functions, while the fourth method adopts the “SpatialNet-big” preset from SpatialNet [22], employing all the loss functions from Sec. III-C except the spatial power map loss. No channel permutation is applied to these DNN models.

Table I shows the performance of the four baseline methods and the proposed method (including two in ablation settings) in terms of audio quality and source localization accuracy. The second and third rows refer to single LS-based filtering (I) using steering vectors above the horizontal plane, and two filters (II) generated separately for above and below, respectively.

Regarding audio quality, the DSB-based method performs worst in both timbral and spatial quality, likely due to the frequency-dependent energy roll-off (which leads to signal coloration) and low spatial resolution outside the horizontal plane. LS-based filtering improves audio quality, especially in phase-related metrics such as SI-SNR, with further gains when separate filters are used. The DL-based baselines outperform SP-based ones as discretization artifacts are avoided, and SpatialNet outperforms U-net, possibly due to its narrow-band processing. Our proposed model performs similarly or better in timbral quality and SH-domain spatial quality, with significantly better binaural spatial quality, likely due to the addition of the power map loss. For source localization accuracy, SP-based baselines, despite having lower audio quality, achieve better azimuth prediction than DL-based ones. This indicates that DNNs trained with only channel-wise loss functions may not fully preserve spatial information. In addition, using separate LS-based filters reduces the elevation error, confirming the ambiguity between upper and lower half-spaces. In comparison, our proposed method yields the lowest localization errors for both azimuth and elevation angles. The impact of adding channel permutation and the power map loss is analyzed by comparing the last three rows, where we see significant improvements in spatial quality and localization accuracy, with slight degradation in timbral quality. Compared to SpatialNet, our model without permutation and the power map loss still yields better ENV, LSD, and binaural quality, with similar performance in other metrics.

Fig. 3 shows the spatial power maps computed from the ground truth and estimated Ambisonics, for a segment of a two-speaker scenario. The proposed method demonstrates better preservation of the spatial information in the captured sound field, while the two baseline methods introduce artifacts, such as “ghost” source images and shifts in the source positions.