Multiple scattering ambisonics: three-dimensional sound field estimation using interacting spheres

Audio is indispensable in immersive technologies such as mixed reality (MR) and virtual reality (VR), which are receiving much attention. For these applications, it is essential to develop technologies to capture, process, and render spatial sound fields with high precision for the presentation of truly realistic and immersive MR/VR experiences. Ambisonics [1] as well as higher-order ambisonics (HOA) [2], which are established spatial audio frameworks to capture, process and reproduce spatial sound fields based on its representation in the spherical harmonics domain, are receiving much attention due to the popularization of MR/VR platforms [3, 4], and its high compatibility with first-person view MR/VR. Ambisonics spatial audio capturing and processing consists of a microphone array and signal processing that is used to encode the raw microphone array signal to the spherical harmonics-domain spatial description format, which is referred to the ambisonics signal. This ambisonics signal is decoded to the signal which is fed to loudspeaker arrays to render the spatial sound field. Such loudspeaker arrays are often virtualized by means of binaural technologies [5, 6, 7] and played back via headphones. Hence the high compatibility of ambisonics with MR/VR applications that usually use headphones for audio playback. Due to its formulation in the spherical harmonics-domain, a typical implementation of an ambisonics recording device is employing a spherical microphone array (SMA) [1, 8, 2, 9, 10]. Often, SMAs are mounted on sound-hard spherical scatterers in order to avoid the instability arising in encoding filters for hollow microphone arrays due to singularities originating from the roots of the spherical Bessel function [2], and for its mechanical stability as hardware. This form of a SMA is referred to as a rigid SMA (RSMA). Despite its success in first-person immersive audio with only three degrees-of-freedom (DoF) which are associated with the rotation of the listener, ambisonics suffers from the diminishing size of the accurate reconstruction area, referred to as the sweet-spot, as the frequency increases, hence limiting its efficacy in higher DoF spatial audio reproduction allowing translation of the listener. This is visualized in Fig. 1 (left), showing the resulting reconstruction sweet-spots for incident plane waves with various frequencies. Here, the sweet-spot is defined as the region where the signal-to-distortion ratio (SDR) of the estimated field with respect to the ground truth incident field is above 30 dB.

In order to expand the sweet-spot of ambisonics reproduction, the simplest way is to develop RSMAs with larger number of microphones. Although this is an effective approach, it comes with a significant development and device cost. An alternative approach is to combine multiple existing RSMAs and integrate the captured information. However, this is not a trivial task due to the inter-array interference. Here, multiple scattering higher-order ambisonics (MS-HOA), a three-dimensional (3D) sound field capturing scheme using multiple RSMAs with fully considering inter-array interaction due to multiple scattering [11] is proposed. Numerical experiments show that MS-HOA successfully creates sound field representations with expanded sweet-spots even when the RSMAs are densely arranged with small spacing, which is not achieved without the consideration of inter-array interaction. An example sound field recording and reproduction setup allowing translation of the listener is illustrated in Fig. 1 (right).

The conventional framework of ambisonics encoding using a single RSMA is briefly reviewed. Ambisonics encoding and decoding can be performed by either relying on solving a linear system using least squares [2] or relying on spherical harmonic transformation using numerical integration [12]. Since the first approach allows more flexibility of the microphone array configuration, this approach is adopted here. In the present work, all formulations are presented in the frequency-domain, which can be converted to time-domain representations by inverse Fourier transform. All individual microphone capsules are assumed to be omnidirectional. The spherical harmonics used are defined as

$$Y_{n}^{m}(\theta,\varphi)\equiv\sqrt{\frac{(2n+1)}{4\pi}\frac{(n-m)!}{(n+m)!}}P_{n}^{m}(\cos\theta)e^{im\varphi},$$

(1)

with $\theta$ and $\varphi$ the polar and azimuthal angle, respectively, and $P_{n}^{m}(x)$ and $P_{n}(x)$ respectively the associated and regular Legendre polynomials:

$$P_{n}^{m}(x)\equiv(-1)^{m}\left(1-x^{2}\right)^{m/2}\frac{d^{m}}{dx^{m}}\left(P_{n}(x)\right),\quad P_{n}(x)\equiv\frac{1}{2^{n}n!}\frac{d^{n}}{dx^{n}}\left(x^{2}-1\right)^{n}.$$

(2)

The above definition of spherical harmonics provides an orthonormal basis:

$$\int_{\theta=0}^{\pi}\int_{\varphi=0}^{2\pi}Y_{n}^{m}(\theta,\varphi)Y_{n^{\prime}}^{m^{\prime}}(\theta,\varphi)^{*}d\Omega=\delta_{nn^{\prime}}\delta_{mm^{\prime}},$$

(3)

with $\delta_{ij}$ the Kronecker delta.

The process of obtaining the ambisonics signal $A_{n}^{m}(k)$, the weights of the spherical basis functions of the three dimensional sound field representing an arbitrary incident field of wavenumber $k$, from the signal captured by the microphone array is referred to as ambisonics encoding. An arbitrary incident field can be expanded in terms of the regular spherical basis functions $j_{n}(kr)Y_{n}^{m}(\theta,\varphi)$ of the three-dimensional Helmholtz equation in the spherical coordinate system $(r,\theta,\varphi)$:

$\displaystyle p_{\mathrm{in}}$

$\displaystyle=\sum_{n=0}^{\infty}\sum_{m=-n}^{n}A_{n}^{m}(k)j_{n}(kr)Y_{n}^{m}(\theta,\varphi),$

(4)

with $j_{n}(x)$ the spherical Bessel function of degree $n$. The total field $p_{\mathrm{tot}}$, which is the sum of the incident field and the field scattered by a rigid sphere with radius $R$ located at $O$, the origin, is given by:

$\displaystyle p_{\mathrm{tot}}$

$\displaystyle=\sum_{n=0}^{\infty}\sum_{m=-n}^{n}A_{n}^{m}(k)\left\{j_{n}(kr)-h_{n}(kr)\frac{j_{n}^{\prime}(kR)}{h_{n}^{\prime}(kR)}\right\}Y_{n}^{m}(\theta,\varphi),$

(5)

with $h_{n}(x)$ the spherical Hankel function of the first kind with degree $n$ [13]. On the surface of the rigid sphere, i.e. $r=R$, this total field is evaluated as:

$\displaystyle p_{\mathrm{tot}}|_{r=R}$

$\displaystyle=\sum_{n=0}^{\infty}\sum_{m=-n}^{n}A_{n}^{m}(k)\frac{i}{(kR)^{2}h_{n}^{\prime}(kR)}Y_{n}^{m}(\theta,\varphi).$

(6)

The total field captured by the $q$-th microphone on the surface of the RSMA located at $(R,\theta_{q},\varphi_{q})$ is therefore given by:

$\displaystyle p_{\mathrm{tot}}^{(q)}$

$\displaystyle=\sum_{n=0}^{\infty}\sum_{m=-n}^{n}\frac{i}{(kR)^{2}h_{n}^{\prime}(kR)}Y_{n}^{m}(\theta_{q},\varphi_{q})A_{n}^{m}(k).$

(7)

By truncating the infinite series with $n=N_{\mathrm{c}}$, this result can be represented in the following vector form:

$\displaystyle\mathbf{p}_{\mathrm{tot}}$

$\displaystyle=\Lambda\mathbf{A},$

(8)

where $\mathbf{p}_{\mathrm{tot}}$ is a vector holding $p_{\mathrm{tot}}^{(q)}$ in its $q$-th entry, $\mathbf{A}$ is a vector holding $A_{n}^{m}(k)$ in its $(n^{2}+n+m+1)$-th entry, and $\Lambda$ is the matrix holding $\frac{i}{(kR)^{2}h_{n}^{\prime}(kR)}Y_{n}^{m}(\theta_{q},\varphi_{q})$ in its $(q,n^{2}+n+m+1)$ entry. The goal of ambisonics encoding is to obtain $A_{n}^{m}(k)$ for all $n$s and $m$s up to the truncation degree $n=N_{\mathrm{c}}^{\mathrm{(in)}}$, i.e. $0\leq n\leq N_{\mathrm{c}}$ and $|m|\leq n$, from the observation $\mathbf{p}_{\mathrm{tot}}$. This problem can be solved by regularized least squares with a minimization objective:

$\displaystyle L_{\mathrm{enc}}$

$\displaystyle=||\mathbf{p}_{\mathrm{tot}}-\Lambda\mathbf{A}||_{2}^{2}+\sigma||\mathbf{A}||_{2}^{2},$

(9)

with $\sigma$ a regularization parameter, and the solution given by:

$\displaystyle\mathbf{A}^{\mathrm{(est)}}$

$\displaystyle=\operatorname*{argmin}_{\mathbf{A}}L_{\mathrm{enc}}=(\Lambda^{H}\Lambda+\sigma I)^{-1}\Lambda^{H}\mathbf{p}_{\mathrm{tot}}=E\mathbf{p}_{\mathrm{tot}},$

(10)

where $E\equiv(\Lambda^{H}\Lambda+\sigma I)^{-1}\Lambda^{H}$ is the regularized encoding matrix.

In the proposed method, a grid of multiple RSMAs is used to estimate $p_{\mathrm{in}}$ 4. The goal of ambisonics encoding in MS-HOA is to estimate $A_{n}^{m}(k)$ 4, for $0\leq n\leq N_{\mathrm{c}}$ and $|m|\leq n$ from observations of the sound pressure at discrete microphone capsule positions mounted on the surfaces of multiple RSMAs. In the following, a system of $N_{\mathrm{S}}\geq 2$ RSMAs where each RSMA has a radius $a_{s}$ is considered. Here, $s$ is the index of the RSMA. Hereafter, the argument $k$ is omitted from $A_{n}^{m}(k)$.

MS-HOA recording and encoding into HOA coefficients was validated by numerical experiments. Grids of RSMAs where each individual RSMA is a 252-channel SMA mounted on a rigid spherical scatterer with a radius of 8 cm are considered. The spherical Fibonacci grid [17, 10] of 252 points was used for the microphone capsule positions. A real-world implementation of a 252-channel RSMA with a similar size has been demonstrated in the past [9]. As the RSMA grid, a linear grid of 6 RSMAs and a regular Cartesian grid of 9 RSMAs was used in the experiments. The spacing between the nearest neighbour RSMA was set to 25 cm. The sound field generated by a monopole source located at $\mathbf{r}_{\mathrm{s}}=(10\mathrm{m},10\mathrm{m},10\mathrm{m})$ was used as the incident field. The signal captured by the grid of RSMAs was encoded into the HOA coefficients $\mathbf{A}^{\mathrm{(est)}}$ with the proposed method (MS-HOA). While prior works on the forward problem report heuristics for choosing the parameter $N_{\mathrm{c}}^{\mathrm{(fwd)}}$, e.g. $N_{\mathrm{c}}^{\mathrm{(fwd)}}=\left\lfloor eka\right\rfloor$ [14], here $N_{\mathrm{c}}^{\mathrm{(fwd)}}$ was treated as a free hyper-parameter. The case where inter-sphere interaction is switched off, i.e. the method which only considers single scattering (Single), and the case of conventional HOA encoding using only one building block RSMA (HOA) are also computed as baselines. The analytical reconstruction of the estimated incident field was computed by 4 and was compared to the ground truth incident field $p_{\mathrm{in}}$ in terms of the SDR and the size of the reconstruction sweet-spot area (SSA) measured in the $xy$-plane or the $yz$-plane depending on the configuration of the RSMA grid. The SSA is defined here as the total area where the SDR surpasses 30 dB, which is measured using the regular Cartesian grid points on a plane which correspond to the pixels in Fig. 3-Fig. 4. The regularization hyperparameter was optimized by grid search independently for both the Single baseline and the proposed MS-HOA. Regularization was not applied to the single sphere HOA baseline due to its minor effect for this case, while the truncation number $N_{\mathrm{c}}$ was chosen as the one providing the largest SSA for the given RSMA. The results for the linear 6-sphere RSMA grid with a incident field frequency of 4kHz are shown in Fig. 3. The sweet-spot of reconstruction is successfully expanded with MS-HOA while the SSA and SDR is significantly degraded if only single scattering is considered. The results for the regular Cartesian 9-sphere grid is shown in Fig. 4, demonstrating planar expansion of the sweet-spot.

Multiple scattering ambisonics, a method to capture 3D sound fields using multiple acoustically interacting RSMAs, was proposed. MS-HOA allows to integrate the information captured by multiple densely arranged RSMAs and can be used to expand the reconstruction sweet-spots in 3D sound field reproduction. The numerical experiments demonstrated that the proposed method successfully captures spatial sound fields with expanded reconstruction sweet-spots which was not possible without the consideration of inter-array interaction due to multiple scattering.

A related method using the translation of multipoles was introduced in [18]. This method was based on the assumption that the SMAs do not physically interact with each other, i.e. the SMAs do not cause scattering that affect other SMAs. This assumption is violated if the SMAs are densely arranged RSMAs, which scatter the incident field and interact with each other by multiple scattering. As shown in the numerical experiments, the approach without considering inter-array interaction becomes inaccurate if the RSMAs are arranged with small spacing. Recently, the consideration of inter-array multiple scattering has been demonstrated to improve the reconstruction accuracy in a two-dimensional sound field reconstruction problem using multiple cylindrical microphone arrays [19]. Two-dimensional modeling, however, is insufficient for modern spatial audio applications such as MR/VR where 3D audio representation and rendering is essential. Our work enables the use of interacting rigid microphone arrays for 3D spatial audio.

The expanded reconstruction sweet-spots with linear or planar spreads realized by the proposed method could be useful in applications including sound field reproduction in theaters or in meeting rooms where the sweet-spot should cover multiple listeners sitting next to each other, or higher DoF MR/VR where the translation of the listener needs to be supported. Developing techniques to reduce the cost of MS-HOA recording in terms of hardware, computation, and bandwidth is important for practical applications and are subjects of future research.

Shoken Kaneko thanks to Japan Student Services Organization and Watanabe Foundation for support via scholarships.