Spatial data augmentation with simulated room impulse responses
for sound event localization and detection

The primary problem in the SELD task is how to associate sound event detection (SED) predictions with DOA predictions (or vice versa) when multiple sound events overlap, which is called the data association problem [1]. To solve this problem, supervised approaches using deep neural network such as the convolutional recurrent neural network (CRNN) are typically used.

Spectral information, e.g., multi-channel spectrogram, log-mel spectrogram, and spatial information, e.g., GCC-PHAT, inter-channel phase differences (IPDs), are combined and used as input features. These features are fed into several two-dimensional convolutional layers followed by recurrent layers such as a gated recurrent unit (GRU). For instance, the assigning of a densely connected dilated DenseNet (D3Net) [14] to the convolutional layers achieved state-of-the-art performance in previous DCASE challenges [8, 15]. Finally, the output of the recurrent layers is transformed by fully-connected layers into an output representation. While two-branch representation, e.g., SELDnet [1], uses two branches for two targets, an SED target and a DOA target, activity-coupled Cartesian DOA (ACCDOA) representation assigns a sound event activity to the length of a corresponding Cartesian DOA vector [8]. The ACCDOA representation enables us to handle a SELD task as a single task with a single network.

Some data augmentation techniques have been widely used for the SELD task. EMDA [16, 17] mixes audio events with random amplitudes, delays, and the modulation of frequency characteristics, i.e., equalization. The rotation augmentation method [10] rotates an observed signal represented in the first-order Ambisonics (FOA) format and enables us to increase the number of DOA labels without losing the physical relationships between steering vectors and observations. SpecAugment [18], which was originally proposed for the speech recognition task and recently also showed efficacy on the SELD task [8], applies frequency masking to input features.

In addition to these augmentation methods, the MCS is also effective for creating new combinations of spectral information and spatial information [9]. Fig. 1(a) describes the workflow of the MCS. First, non-overlapping and static, i.e., not moving, events are extracted from an original dataset on the basis of label information. Then, the complex Gaussian mixture model (CGMM)-based beamformer [19] block takes the extracted events and outputs enhanced signals and covariance matrices. The covariance matrices are calculated with the spectrograms enhanced by CGMM-based masking. The enhanced signals are stored as spectral information, and the covariance matrices are stored as spatial information. They are independently and randomly sampled to create new combinations of spectral information and spatial information. The multi-channel simulator block finally takes them and simulates multi-channel input signals.

The MCS actually contributed to improving the performance on the TAU-NIGENS Spatial Sound Events 2020 dataset [3, 9], which does not include directional interference events. However, it assumes no directional interference events and highly relies on non-overlapping events (i.e., clean events) in the original dataset. Therefore, using the MCS for datasets including directional interference events such as the TAU-NIGENS Spatial Sound Events 2021 dataset could lead to performance degradation.

The interference elimination block is composed of two blocks: the detection-based elimination block and eigenvalue-based elimination block. The motivation for using two types of elimination block is to eliminate events with different ranges of signal-to-interference ratio (SIR).

First, the detection-based elimination block uses a model pre-trained without the IRS, whose performance is shown as ID 0 in Table 1 . The non-overlapping static events extracted by the label-based extraction block are processed with the pre-trained model. Events that the pre-trained model cannot detect are regarded as events overlapped with interference events, which are eliminated.

Then, the eigenvalue-based elimination block first applies a short-time Fourier transform (STFT) to the events extracted by the detection-based elimination block and computes a spatial covariance matrix from the obtained spectrogram. Its eigenvalues are then computed and normalized such that the maximum eigenvalue becomes 1. These eigenvalues can be considered to reflect how many sound sources the observation signals include. Let $\gamma_{c}(c=1,\dots,C)$ be the normalized eigenvalues, where $C$ is the number of input channels. We introduce two types of thresholds, $\alpha$ and $\beta$. We define the frequency bin where the number of eigenvalues that satisfy $\gamma_{c}>\alpha$ is more than 1 as an overlapped bin. Then, focusing on the limited range from the minimum frequency $f_{\text{min}}$ to maximum frequency $f_{\text{max}}$, let $K_{\text{focus}}$ be the number of total focused bins and $K_{\text{overlap}}$ be the number of overlapped bins. Extracted events that satisfy $(K_{\text{overlap}}/K_{\text{focus}})>\beta$ are regarded as events overlapped by interference events, which are also eliminated.

Fig. 2 illustrates our assumption regarding the relationship between the interference elimination block and SIR, where we divide the SIR range into three levels: “range 1 (low),” “range 2 (middle),” and “range 3 (high).” We assume that the events in range 1 shown in Fig. 2 can be eliminated by the detection-based elimination block because the events here are heavily overlapped by interference events; thus, the pre-trained model cannot detect such events. The events in range 2 are expected to be eliminated by the eigenvalue-based elimination block because the power of a target sound event and that of an interference event are almost the same and $K_{\text{overlap}}/K_{\text{focus}}$ tends to be higher. The events in range 3 are fed into the CGMM-based beamformer block, and the SIRs are improved by the beamformer. As a result, the enhanced signals are clean enough to be used as spectral information.

The RIR simulation block is composed of two blocks, the microphone-array RIR simulation block and format converter block, by which spatial aliasing caused by the discrete sampling of a sound field can be simulated to reflect the actual recording condition.

In the microphone-array RIR simulation block, an image source method [11] is utilized to define the positions for all image sources depending on given reverberant conditions. RIRs are obtained as linear combinations of the frequency responses $H$ for all image sources. We assume that the array can be regarded as a rigid spherical array [20, 21]. By considering the waves from all image sources as plane waves, the frequency response of $h$-th microphone with a wave number of $k$ on a rigid baffle of radius $R$ for $l$-th image source is obtained as

$$H_{hl}(k,\psi_{hl})=g_{hl}e^{-\mathrm{i}kd_{hl}}\sum_{n=0}^{\infty}\mathrm{i}^{n}(2n+1)b_{n}\left(kR\right)P_{n}\left(\cos\psi_{hl}\right),$$

(1)

where $\psi_{hl}$ denotes the angle between the DOA of the $l$-th plane wave and the orientation of the $h$-th microphone, $P_{n}(\cdot)$ denotes the Legendre polynomial [22, 20], $g_{hl}$ and $d_{hl}$ denote attenuation factor and delay time respectively, both of which are caused by the number of reflections and distance. The imaginary unit is denoted by $\mathrm{i}$. The $b_{n}$ is a radial function for a rigid baffle array written as

$$b_{n}(kR)=\frac{\mathrm{i}}{(kR)^{2}h_{n}^{(1)^{\prime}}(kR)},$$

(2)

where $h^{(1)^{\prime}}_{n}(\cdot)$ denotes the derivative of the $n$-th-order spherical Hankel function of the first kind. Computing the linear combinations of $H_{hl}(k,\psi_{hl})$ in terms of all image sources, the simulated RIR of $h$-th microphone can be obtained as

$$x_{h}(k)=\sum_{l}H_{hl}(k,\psi_{hl}).$$

(3)

Then, in the format converter block, the simulated RIRs are converted to the intended format, e.g., higher-order Ambisonics (HOA). The $n$-th-order and $m$-th-degree spherical harmonic function is defined with the angle $\Omega=\{\theta,\phi\}$ [22] as

$$Y_{nm}(\Omega)\equiv\sqrt{\frac{2n+1}{4\pi}\frac{(n-m)!}{(n+m)!}}P_{nm}(\cos\theta)e^{\mathrm{i}m\phi},$$

(4)

where $\theta$, $\phi$, and $P_{nm}(\cdot)$ denote the elevation, azimuth, and associated Legendre function, respectively. The spherical-harmonic representation of the RIR can be computed by using the following encoding process [12, 13]:

$$\mathbf{a}(k)=\mathbf{B}(k)^{-1}\mathbf{Y}^{{\dagger}}\mathbf{x}(k),$$

(5)

with

$$\mathbf{B}(k)=\left(\begin{array}[]{cccccc}b_{0}&0&0&0&\ldots&0\\ 0&b_{1}&0&0&\ldots&0\\ 0&0&b_{1}&0&\ldots&0\\ 0&0&0&b_{1}&\ldots&0\\ \vdots&\vdots&\vdots&\vdots&\ddots&\vdots\\ 0&0&0&0&\ldots&b_{N}\end{array}\right),$$

(6)

$$\mathbf{Y}=\begin{bmatrix}\mathbf{y}(\Omega_{1})&\mathbf{y}(\Omega_{2})&\cdots&\mathbf{y}(\Omega_{M})\end{bmatrix}^{T},$$

(7)

where $\mathbf{y}(\Omega)\in\mathbb{C}^{(N+1)^{2}}$ is a column vector containing $Y_{nm}(\Omega)$, $N$ is the spherical-harmonic order, e.g., $N=1$ corresponds to FOA, $M$ is the number of microphones, $\mathbf{x}(k)\in\mathbb{C}^{M}$ is a column vector containing $x_{h}(k)$, and $(\cdot)^{{\dagger}}$ represents the Moore–Penrose pseudoinverse.

We evaluated our approach on the development set of the TAU-NIGENS Spatial Sound Events 2021 dataset using the suggested setup [5]. The dataset was composed of 6 folds, which contained 600 one-minute Eigenmike recordings with the FOA format: 400 for training (folds 1 to 4), 100 for validation (fold 5), and 100 for testing (fold 6). The sound event samples were from the NIGENS general sound events database [23], which consists of 12 event classes such as footsteps and barking dog. Each event had an equal probability of being either static or moving. The signal-to-noise ratios ranged from 6 dB to 30 dB. The sampling frequency was 24 kHz.

We prepared two types of CRNNs: the conventional CRNN used in [2] and RD3Net [14]. Both networks have four convolutional blocks followed by a GRU. They take frame-wise multi-channel amplitude spectrograms and inter-channel phase differences (IPDs) as frame-wise features and output frame-wise ACCDOA. The STFT was applied with a configuration having a 20-ms frame length and 10-ms frame hop.

The IRS was applied to the training data by adding new training folds created by the IRS. In this experiment, we added 2 folds, which increased the amount of training data by 50%. The thresholds for our interference elimination block, $\alpha$ and $\beta$, were 0.3 and 0.4 respectively. The $f_{\text{min}}$ and $f_{\text{max}}$ were 100 Hz and 4 kHz, respectively. A minimum variance distortionless response (MVDR) beamformer [24] was applied to the CGMM-based beamformer block. In the RIR simulation block, the reverberation time (RT60) was randomly set to be within 100 to 500 ms. We only simulated static impulse responses. The SN3D normalization scheme of Ambisonics was used in the format converter block [3]. For comparison, the MCS was also applied by using covariance matrices for the spatial information instead of RIRs. In addition, we investigated the efficiency of combining the IRS and other augmentation techniques, that is, EMDA, rotation augmentation methods, and a multi-channel version of SpecAugment, whose efficacy were already shown in [15]. These augmentation techniques were applied on-the-fly [25].

The frame length of the network input during training was 128 frames except for the experiment to show the performance of RD3Net. The batch size for the training was 32. The learning rate was linearly increased from 0.0 to 0.001 with 50,000 iterations [26]. After the warm-up, the learning rate was decreased by 10% if the SELD score of the validation did not improve in 40,000 consecutive iterations. We used the Adam optimizer with a weight decay of $10^{-6}$. We validated and saved model weights every 10,000 iterations up to 400,000 iterations. Finally, we averaged the model weights from the last 5 models as in [27].

Four metrics were used for the evaluation [28]: $\text{LE}_{\text{CD}}$, $\text{LR}_{\text{CD}}$, $\text{ER}_{20^{\circ}}$, and $\text{F}_{20^{\circ}}$. $\text{LE}_{\text{CD}}$ is a localization error that indicates the average angular distance between predictions and references of the same class. $\text{LR}_{\text{CD}}$ is a simple localization recall metric that expresses the true positive rate of how many of these localization predictions are correctly detected in a class out of the total number of class instances. $\text{ER}_{20^{\circ}}$ and $\text{F}_{20^{\circ}}$ are the location-dependent error rate and F-score, where predictions are considered as true positives only when the distance from the reference is less than $20^{\circ}$. To evaluate the overall performance, we adopted $\text{SELD}_{\text{score}}$, which is defined as

$$\text{SELD}_{\text{score}}=[\text{ER}_{20^{\circ}}+(1-\text{F}_{20^{\circ}})+\text{LE}_{\text{CD}}/\pi+(1-\text{LR}_{\text{CD}})]/4.$$

(8)

Table 1 shows the experimental results for the IRS using CRNN. ID 0 represents the training method without any data augmentation techniques, and ID 7 represents that with other data augmentation techniques (i.e., EMDA, rotation augmentation method, multi-channel version of SpecAugment). IDs 1 and 8 are equivalent to the MCS, and IDs 2 and 9 are the MCS with the interference elimination block.

The $\text{SELD}_{\text{score}}$ of ID 0, 3, 4, 5, and 6 shows that the IRS consistently improved the $\text{SELD}_{\text{score}}$ and that using both the eigenvalue-based elimination block and detection-based elimination block was the most effective. In this experiment, the number of events extracted by the label-based extraction was 1243, the number of events eliminated by the eigenvalue-based elimination block was 244, and the number of events eliminated by the detection-based elimination block was 16. This result suggests that using the events from range 1 in Fig. 2 for the spectral information degrades the performance even if the number of events is relatively small. This is because such events have an extremely low SIR as shown in Fig. 2. Comparing IDs 1, 2, 3, and 6 (i.e., comparing the IRS with the MCS), it is shown that the RIR was more suitable for the spatial information than the covariance matrix calculated from the extracted events. This is because the events from range 3 in Fig. 2 still had interference events with low energy, which disturbed the spatial information of the target sound events. The results for IDs 7 to 13 show the same tendency as the results for IDs 0 to 6, which suggests that the IRS can be used with other augmentation techniques.

Table 2 shows the performance of RD3Net. We trained RD3Net with other augmentation techniques and 1024 input frames. It is shown that the IRS contributed to further improving the performance of the state-of-the-art method (i.e., RD3Net).