DPLM: A Deep Perceptual Spatial-Audio Localization Metric

We begin by building a source-localization model that predicts the DOA of a given sound source.  Given a binaural input, the DOA model processes both the magnitude and phase spectrograms of the two channels, and systematically outputs a framewise location estimate of the source. We evaluate two variants to ensure generalization of the proposed framework for realistic sound sources: a Static-Source model for recordings with a fixed source in the scene, and a Moving-Source model where the source can move smoothly in the scene. Note that the moving-source model would result in a finer resolution estimate of DOA.

DPLM comprises three components (Fig 1a): a feature-extraction block, a temporal aggregation block and a task-specific localization block. Both DOA variants have the same model architecture for fair comparison.  The feature-extraction block maps the input features to an embedding maintaining the temporal structure of the signal. This embedding is then processed by temporal aggregation models to learn long-term dependencies.  The resulting hidden representations are then fed to a task-specific localization head that outputs a location embedding per frame.

For the feature-extraction block, we evaluated a variety of different convolutional network building blocks, including a basic conv-batchnorm-maxpool block, ResNet, Squeeze-and-Excitation and Inception [14]. Inception resulted in best features, and so, we do not discuss the other structures in this paper. The 6-block Inception network consists of 64 conv filters, followed by 1x1, 3x3 and 5x5 filters, leading to 3x3 max-pooling, and a 1x2 maxpool along the frequency dimension. For the temporal aggregation block, we evaluated both Long Short-Term Memory networks (LSTMs) and Temporal Convolutional Networks (TCNs). LSTMs generalized better to unseen rooms and subjects. We used 2 bi-directional LSTM layers that output an embedding of size 64 for each time-frame.

We pose the localization task as a simple classification problem. We divide the azimuth plane ($\in(-180^{\circ},180^{\circ})$) into 50 equally spaced bins, and the elevation plane ($\in(-90^{\circ},90^{\circ})$) into 25 equally spaced bins. However, we mainly focus on azimuthal plane localization since elevation cues are highly individualized and datasets are sparse (see also Sec 4.2). The task-specific localization head then maps the outputs from temporal aggregation to a one-hot class encoding. Note that all the layers in the blocks use BatchNorm and LeakyReLU as the activation function.

The hidden embeddings resulting from the proposed framework are then aggregated to compute a deep-feature distance (Fig 1b). Although the network is trained to predict the source location, we claim that the subspace of hidden layers contains additional useful information about estimating localization similarity. Accumulating these hidden features to compute a deep-feature distance has been shown to correlate with human perceptual judgements in some recent studies, both in machine vision [13] and audio [8, 15]. They have also been shown to be effective for representation learning without the need for prior expert knowledge [13]. Given a $L$-layered network, we denote the $l^{th}$ hidden layer activations as ${{F}}_{l}(x)\in\mathds{R}^{T_{l}\times B_{l}\times C_{l}}$, where $T_{l}$, $B_{l}$, and $C_{l}$ are the time resolution, frequency bands, and number of channels respectively. The distance between two audio recordings is then given by,

$$D(x_{\mathtt{1}},x_{\mathtt{2}})=\sum_{l}\frac{1}{T_{l}C_{l}B_{l}}||{{F}}_{l}(x_{\mathtt{1}})-{{F}}_{l}(x_{\mathtt{2}})||_{1}.$$

(1)

We train the static-source model on temporally averaged predictions, while the moving-source model uses finer, frame-level predictions. Hence its reasonable to expect that the latter model captures finer estimates of DOA. For loss function, we use the average of label-smoothed cross-entropy loss [16] and haversine distance [17].

Cross-entropy is the standard loss for classification learning with neural networks. However, label-smoothing encourages small logit gaps, which prevents model over-fitting and overconfident predictions. If $y_{k}$ and $p_{k}$ denote the target and prediction respectively ($y_{k}=1$ for the correct class, $0$ otherwise), and $\alpha$ denotes the smoothing parameters, the loss is given by,

$$H(\textbf{y},\textbf{p})=\sum_{k=1}^{K}{-(y_{k}(1-\alpha)+\frac{\alpha}{K})\log(p_{k})}$$

(2)

On the other hand, haversine distance [17] emphasizes a smooth, continuous space for DOA predictions whereas cross-entropy focuses on a discrete space. It captures the great-circle distance between any two points on a sphere and serves as a good proxy to compute distance between the predicted and ground-truth source location. Let $P=(\theta_{1},\phi_{1})$ and $Q=(\theta_{2},\phi_{2})$ be the predicted and ground truth source location from our DOA model respectively, where $\theta_{1}$ and $\theta_{2}$ are azimuth angles, and $\phi_{1}$ and $\phi_{2}$ are the elevation angles (all in radians), then the distance between $P$ and $Q$ is:

$L_{(P,Q)}=2\arcsin{[\sqrt{\sin^{2}(\frac{\phi_{1}-\phi_{2}}{2})+\cos{(\phi_{1})}\cos{(\phi_{2})}\sin^{2}{(\frac{\theta_{1}-\theta_{2}}{2})}}]}$

(3)

The predicted source locations from the DOA model are obtained by computing the softmax-weighted sum of the model predictions for each frame.

Speech recordings from the TIMIT dataset [28] are used as the source for anechoic recordings. The static-source experiments are carried out using a pool of 11 Binaural Room Impulse Response (BRIR) databases, listed in Table 1. The resulting pool contained approximately 125k BRIR pairs from 36 different rooms. For learning moving-source models, we used the Hearsay binaural audio dataset [29] which contains a total of 2 hours of paired mono and echoic binaural audio from 8 different speakers. Participants were asked to walk around a mannequin and talk (no script was used), and their position and orientation were tracked. In addition, we also used ambisonic audio data from the DCASE 2021 Challenge [30], which consists of 600 1-min long sound recordings of multiple sources with annotations. We convert ambisonic formats to binaural using the measurements from subject 2 of the ARI HRTF dataset [31].

For all cases, 3-sec audio excerpts are used for training. Phase and magnitude spectra of a $512$-point DFT spectrogram are extracted from these excerpts (at $16$kHz sampling rate).  To ensure robustness of the metric against noise, we train DPLM  with added background noise using samples from the DNS Challenge [32], spatialized using the BRIR datasets in Table 1. For learning, we use the Adam optimizer with a learning rate of $10^{-4}$ and batch size of $32$. The label-smoothing parameter $\alpha$ (from Eq. 2) is $0.25$. For all cases, $80\%$ of the data is used for training and the remainder for testing.

We compare our approach to BAMQ , a binaural audio quality metric proposed by Fleßner et al. [2]. BAMQ estimates the various binaural cues (ILD, ITD, and IACC) at frame-level and combines them using a set of learned weights to output an overall quality metric between two recordings. Further, observe that any learning model that is trained using binaural signals as inputs can, in principle, be used as a surrogate to compute a distance metric. For instance, one can compute a deep-feature distance (similar to Sec 2.3) between hidden layers of a pretrained deep learning model.  Hence, we use two state-of-the-art binaural speech separation models - TasNet[33] and SAGRNN[34] to obtain such auxiliary localization metrics.  For both these models, we compute the average of deep-feature distances across all layers except the final decoder block as alternate baselines to BAMQ.

We first evaluate the static-source and moving-source models for localization errors on a held-out set of sound sources from TIMIT, spatialized using a held-out set of BRIRs. The best performing static-source model produced a root mean square error (RMSE) of 13.2^∘ in azimuth front-back folded, i.e., reflected about the coronal plane to discount front-back confusions). The moving-source model produced an RMSE of 8.4^∘ confirming that it leads to more accurate DOA estimates.

Fig 2 shows an example of a framewise comparison between static-source and moving-source models. We observe that the framewise predictions from the moving-source model (blue curve) closely follow the actual trajectory of the source (black curve).  As expected, the static-source model (orange dotted curve) is not accurate at the frame level for tracking moving objects. On the other hand, the prediction improves (red curve) when localization is computed independently for each interval, after splitting the moving trajectory into various intervals of constant DOA (shown by the three intervals in Fig 2). All these observations are expected, and overall, the results show that the continuous temporal tracking information available to the moving-source model helps improve the frame-level predictions, leading to fewer localization errors in general.

To verify DPLM’s sensitivity to increasing angular distance, Fig 3 shows the metric’s distance between a fixed reference, and a moving test source for four different source positions. We see that the absolute distance values generally increase with increasing angular distance across all four source positions, indicating that DPLM  obeys the general trend quite well. To quantify this trend, Table 3 shows the Spearman’s rank order correlation (SC) between the output of DPLM  and angular distance between two sources across subjects and (anechoic/echoic) listening conditions. We see that both our models (static-source and moving-source) outperform all baselines. Surprisingly, even the pre-trained models have a non-trivial correlation with angular distance, suggesting that deep-feature distances across these models serve as a good proxy for assessing localization differences between recordings.

We now use previously published diverse third-party studies to verify that our trained metric correlates well with subjective ratings of their tasks. We compute the correlation between the proposed model’s predicted distance with the publicly available subjective ratings. These correlation scores are evaluated per condition (averaging samples per condition). The datasets used are:

1. 1.

   Bilateral Ambisonics [35] (P1 and P1'): This compares the standard and bilateral spatial audio reproduction methods across various spherical harmonic orders to assess overall quality. It uses various stimuli including speech, castanets and guitar. There are two versions (denoted by P1 and P1'), each with different subjects, training sessions and different spherical-harmonic orders.

2. 2.

   Spherical Microphone Array [36] (P2): This is designed to compare audio quality improvements across algorithms for binaural rendering of spherical microphone array signals using music as stimuli. It provides an overall quality rating, with 96 variations of test signals per subject. The pairs of recordings to be compared are not time-aligned and can be of different lengths.

3. 3.

   HpEQ [37] (P3): This data is from headphone equalization (HpEQ) study across generic and individualized BRIRs, with individualized, generic or no headphone equalization using speech, pink noise and guitar sounds for stimuli. We have an overall quality rating, and pairs of recordings may contain very subtle differences (recordings are also not time-aligned).

4. 4.

   Bitrate Compr. Ambis. [38] (P4): This comes from assessment of the degree of timbral distortions introduced by compression at different ambisonic orders (1st, 3rd and 5th) across various modalities including simple scenes (vocals, castanets, glockenspiel, pink noise) and complex scenes (EM: electronic music and AM: acoustic music). Similar to P3, we have overall quality ratings, and recordings are not time-aligned.

Results for the correlations with subjective ratings are in Table 2. Overall, our proposed metric achieves the best performance across all datasets. Firstly, DPLM’s correlation is stable with changes in stimuli (shown by P1, P1' and P4). This shows the generalization power of deep-feature distance metrics, and their ability to capture attributes across speech, music, noise etc. Furthermore, the two deep network baselines (TasNet  and SAGRNN ) trained on an unrelated task (binaural source separation) outperform BAMQ  on most datasets. This suggests that deep-feature distances transfer well even across unrelated tasks, and are able to model low-level perceptual similarity well. However, absolute correlation values are lower for P3 showing that the metric is not robust enough to capture subtle differences driven by headphone equalization (some of which are very close to JNDs). Secondly, the moving-source model performs better than the static-source model on most tasks, following a similar trend as shown in Table 3 earlier. Hence, frame-wise optimization for localization also seems to improve subjective ratings. Third, the trends with P2, P3 and P4 suggest that DPLM  (and the two deep network pre-trained baselines) are robust to non time-aligned data.  Lastly, since DPLM  is trained and tested under realistic, echoic, conditions, we can clearly see its better generalization power compared to BAMQ  (which is learned under anechoic environments).

Recall that one can characterize azimuth localization by utilizing binaural cues such as ITD and ILDs. However, elevation localization is quite challenging because of the subject-specific influence of monaural spectral cues. Further, lack of a wide range of elevation angles in publicly available BRIR datasets also limits building and evaluating robust models. We also observed similar trends in our analysis (not shown), with high error for elevation localization. The proposed metric performed almost the same as a simple spectral subtraction, suggesting that it does not capture elevation cues well.