IMPROVING TRAJECTORY LOCALIZATION ACCURACY VIA DIRECTION-OF-ARRIVAL DERIVATIVE ESTIMATION

The SALSA-Lite was introduced as an efficient computational version of the Spatial Cue-Augmented Log-Spectrogram (SALSA) feature for MIC (audio format) data [22, 23]. For $M$-channel audio recording, SALSA-Lite is a ($2M-1$) channel feature consisting $M$ log-power spectrogram with ($M-1$) frequency-normalized interchannel phase differences (NIPDs). The NIPD ($\Lambda$) approximating the relative distance of arrival (RDOA) can be written as

	$\displaystyle\Lambda(t,f)$	$\displaystyle\approx-\frac{c}{2\pi f}\,\text{arg}~{}|\boldsymbol{H}_{1}^{*}(t,f)\boldsymbol{H}_{2:M}(t,f)|,$		(1)
		$\displaystyle\approx[d_{12}(t)\ldots d_{1M}(t)],$		(2)

where $\boldsymbol{H}_{m}(t,f)=e^{\frac{j2\pi fd_{1m}(t)}{c}}$ is the array response for any arbitrary array structure under the far-field assumption and $d_{1m}(t)$ is the RDOA between the first (reference) and $m^{\text{th}}$ mic. The SALSA-Lite provides the exact time-frequency positioning between the spectrogram and the NIPD resulting in the model being able to localize multiple overlapping sources.

Figure 1 shows the neural network architecture designed to predict both the DOAs and their derivatives simultaneously. Here derivative represents the change in DOAs between two time frames. The SALSA-lite is fed to the CRNN network, which consists of one convolutional layer followed by four ResNet22 blocks [25] in the network body [22, 23].

The output of ResNet block is fed into a two-layer bidirectional Gated Recurrent Unit (GRU) followed by two distinct regression heads for predicting DOAs and their derivatives in Cartesian coordinates (X, Y, and Z), respectively. Unlike the SELD network in [23], we focus only on the localization task and replace the detection head with the regression head, which predicts the derivatives of DOAs at different timestamps as shown in Fig. 1. Along with predicting the intermediate DOAs, the additional derivative information helps to obtain better overall DOA estimates. Once the network predicts the DOAs and their derivatives, the final DOAs are obtained using the following update equation

$\displaystyle\hat{y}_{N}^{\text{final}}~{}=~{}\frac{\hat{y}_{N}~{}+~{}(\hat{y}_{N-1}~{}+~{}\hat{y}^{\prime}_{N})}{2},$

(3)

where $\hat{y}^{\prime}_{N}$ is an estimate of the DOA derivative $(y_{N}-y_{N-1})$ at $N^{\text{th}}$ time and $\hat{y}^{\prime}_{0}=0$ is the first derivative assumed to be zero. We expect that by additionally incorporating derivative predictions, DOA of moving targets can be estimated more accurately. For both the regression heads (predicting DOAs and derivatives), the number of active sources are assumed to be known. The ground truth is used to compute the losses for both the prediction heads. The mean squared error (MSE) loss is minimized while training both the network heads and can be written as

$$\text{LOSS}_{\text{total}}=\sum_{n}(y-\hat{y})^{2}+\sum_{n}(y^{\prime}-\hat{y}^{\prime})^{2},$$

(4)

where $y~{}\text{and}~{}y^{\prime}$ are true DOAs and their derivatives at different timestamps, and $\hat{y}$ and $\hat{y}^{\prime}$ represent the predicted DOAs and the estimated derivatives, respectively.

The TAU-NIGENS Spatial Sound Events 2021 dataset was used for the performance analysis of the proposed models. The dataset consists of 600 recordings (1 minute long and four channels each) with 24 kHz sampling frequency in MIC data format. In this paper, 400 recordings were used for training, and 100 recordings were used each for validation and testing. The audio files consist of various static and moving sources from 12 unique classes. The angular range for azimuth and elevation angles are $[-180~{}180)^{\circ}$ and $[-45~{}45]^{\circ}$, respectively.

For feature extraction, we followed the same setup as in [23], and processed frequencies from 50Hz to 2kHz to avoid the aliasing. While training, Adam optimizer [26] was used, with the initial learning rate as $3\text{x}10^{-4}$ which linearly decreased to $10^{-4}$ over last 15 epochs. A total of 70 epochs with $32$ batch size were processed. The validation set was used for model selection, whereas the test data was used for the performance analysis. While generating the ground truth for DOA derivatives, the derivative is considered zero if the source appears for the first time or reappears after 20 or more frames. In all other cases the derivative is the difference between the previous and current DOA.

Following the framework of baseline method [23], the detection ground truth was used to compute the error only for the frames where sources were present. For this study, the error was computed as the DOA error $\Delta\sigma$ which corresponds to the spatial angular distance between the predicted and true positions, [27, 14].

$$\Delta\sigma=\arccos{({\mathbf{n}}_{\text{true}}.{\mathbf{n}}_{\text{pred}})}\,.\,\frac{180}{\pi},$$

(5)

where ${\mathbf{n}}_{\text{true}}$ and ${\mathbf{n}}_{\text{pred}}$ are unit norm vectors of [X, Y, Z] coordinates corresponding to the true and predicted positions, respectively. A source is considered to be localized only when the average (averaged across the active frames) spatial distance between the predicted and true positions is less than $20^{\circ}$; we call these cases as true positive (TP). The false negative (FN) counts the number of incidences when the averaged spatial distance is more than $20^{\circ}$. The probability of detection ($P_{d}$) is the fraction of frames where the distance between the predicted and true positions is less than $20^{\circ}$. Note that as the network is designed to provide only one prediction per class, the error was not calculated for the cases where multiple sources were present from the same class in the frame.

In this subsection, the effect of combining the predicted derivatives with predicted DOAs is demonstrated. Fig. 2 shows the predicted source trajectories from both Model1 (with derivative estimation) and Model2 (without derivative estimation) along with classwise mean absolute error (MAE) computed for the correctly detected sources for one of the recordings. The cross $(\times)$ in MAE plot denotes the cases when the source has not been detected by the models. It can be seen that Model1 detects more number of sources, hence resulting in higher MAE for some classes compared to Model2. It can be observed that by combining the derivatives via the proposed update rule (3), the final DOA exhibits a smoother trajectory since the outliers are eliminated. The update rule is helpful even for static sources. We observed that Model2 gives more erroneous DOAs for static sources than Model1. Overall, Model1’s estimates are closer to true trajectories resulting in higher $P_{d}$. For this recording, the average $P_{d}$ for static and moving sources for Model1 and Model2 are, $P_{ds}=64$, $P_{dm}=78$, $P_{ds}=53$, and $P_{dm}=52.4$, respectively. The total $P_{d}$ averaged over 100 recordings from the test data is reported in Table 1.

From Table 1, it is evident that both Model1 and Model2 show similar performance for the clean dataset. We observed that Model1’s performance degrades when the network predicts the erroneous DOAs; hence combining them with equal weights leads to incorrect estimates. As a correction step, the current DOA prediction with derivative and the past prediction can be weighted depending on the threshold. A choice must be made depending on confidence in the present and past predictions.

To speed up the training process and reduce the risk of overfitting, we repeated the experiments using the pre-trained CRNN weights from an existing SELD model using SALSA-Lite, where the best model was obtained at the 47th epoch [23]. The dataset, architecture, and framework for the pre-trained model detailed in [23] are same as the CRNN body used in this paper. By keeping the CRNN body’s weights fixed using the pre-trained SELD model, the overall $P_{d}$ is increased by 10 % for both Model1 and Model2 as shown in Table 2. From Fig. 3, it can be observed that Model1 outperforms Model2 with higher $P_{d}$ and lower DOA error.

Although the dataset consists of unknown interference and noise, to analyze the robustness of both the models, we synthetically added additive white gaussian noise to the recordings. From Table 1, it is observed that as the SNR level decreases, the $P_{d}$ of both the models degrades drastically. However, it can also be seen that Model1 outperforms Model2 in noisy scenarios as the estimated derivatives help improve the final DOAs. Fig. 4 shows the source trajectories obtained from both Model1 and Model2 for comparison. We observe that the estimates obtained from Model1 are more reliable than those from Model2.