SoundDet: Polyphonic Moving Sound Event Detection and Localization from Raw Waveform

We have a multi-channel sound recording (aka sound waveform) $W_{c}$ recorded by one microphone array station or multiple microphone array stations at a fixed sampling rate, $W_{c}$ contains a sound event set ¹¹1In this paper, a sound event is restricted to be a temporally and spatially continuous and semantically meaningful. Two events with exactly the same temporal location belong to different class. $E=\{e_{i}=(t_{s,i},t_{e,i},l_{i},c_{i})\}_{i=1}^{N}$, where $t_{s,i}$, $t_{e,i}$ are the $i$-th sound event’s start time and end time, respectively, $c_{i}$ indicates the sound event’s semantic information (i.e. category, baby cry, phone ring). $l_{i}$ indicates the sound event’s spatial location, it can either be a fixed location if the sound event is stationary or a spatial trajectory if it undergoes a motion. The task is to learn a model $\mathcal{F}$ to accurately recover these sound events directly from raw sound waveform,

These sound events are polyphonic, which means they may overlap in the time dimension.

Sound waveform usually has high temporal resolution, a typical 1 minute recording of 30kHz has 1.8 million data points. This leads to unbearable computation burden (FLOPs) for standard 1D/2D convolution. To counteract the large data point size issue and further to enforce the neural network to learn meaningful sound event spatio-temporal representation, we propose to learn a set of rectangular band-pass filters in frequency domain, each of which contains a lower learnable frequency cuttoff $f_{1}$ and a higher learnable frequency cutoff $f_{2}$. After converting the rectangular band-pass filter to time domain, we obtain $sinc$ convolution kernels $k$,

where $sinc(x)=sin(x)/x$. Note that the band-pass filter is a parametric filter that the learnable parameter number is independent of the kernel length. Moreover, the kernel length is usually much larger than standard 1D/2D convolution kernel length. A typical band-pass filter kernel length can be set as a large number (in our case 481), whereas the standard 1D/2D convolution kernel size is usually 3 or 5. These two properties guarantee these band-pass filters are highly efficient at both FLOPs computation and model parameter number. The SincNet filter has been successfully applied for speech recognition(Ravanelli & Bengio, 2018).

To further enforce the above designed filter banks to be able to dynamically decouple the sound event’s temporal information and spatial information. We further expand the sinc filter to multiple channels to model the internal correlation between different channels. We draw inspiration from canonical time difference of arrival (TDoA) estimation to explicitly encode sound events’ geometrical information recorded by microphone array. TDoA is a well-established technique for sound source geolocalization. It exploits the arrival time difference among microphones to recover the sound event spatial information (i.e. DoA or physical location) by computing the maximum time delay in either frequency domain or time domain(Van Den Broeck et al., 2012). We involve TDoA estimation into our band-pass filter bank design and propose a new filter bank $g(\cdot)$ that is capable of learning the correlation between different channels. Learning such filter bank helps the neural network to dynamically learn how to integrate between-channel correlation and event frequency band pass to best recover sound events, we thus call it MaxCorr filter bank,

where $t_{i}$ is the $i$-th channel time shift parameter, $f_{1}$, $f_{2}$, $t_{1},\cdots,t_{c}$ are learnable parameters and in our case $c=4$ as there are four channels. To make the time shift parameter to be differentiable, we adopt sigmoid like soft truncation to approximate the round operation, like (Fu et al., 2017) do. Please see Fig.2 for MaxCorr filter illustration and its comparison with standard convolution.

Following the MaxCorr filters, we add an encoder-decoder like 1D convolution network to learn framewise representations. In the encoder, we gradually reduce the temporal length but the increase the filter number, resulting in a compressed representation. In the decoder, it goes the opposite way and finally forms a “bottleneck”-like backbone neural network. To mitigate the model training degradation dilemma, we add skip connection between the encoder and decoder. The MaxCorr filter bank and encoder-decoder like neural network together constitute the Backbone neural network, which learns a framewise representation $[f_{1},f_{2},\cdots,f_{N}]$, where each representation’s temporal duration equals to ground truth labelling resolution, like 100ms.

Given the representation learned by the backbone, the next step is to generate sound event proposal. Potential sound events freely span in the temporal and spatial space, which means sound events are largely varying in their temporal length and spatial location. An efficient sound event proposal generation module thus should be: 1. able to handle sound events with large varying temporal length and spatial location. 2. computationally efficient enough (i.e. generating potential event proposals at parallel). To this end, we propose a dense sound event proposal generation module, which organizes all potential sound events into a compact organization and all the potential events can be generated in parallelization.

Specifically, we compactly organize all potential sound events into a matrix-like representation $\mathcal{M}$, where the row indicates the sound event’s temporal length and the column represents its start time. Each cell $C_{i,j}$ in $\mathcal{M}$ corresponds to a sound event with certain start time $j$ and end time $i+j$. A careful selection of the size of matrix $\mathcal{M}$ guarantees all potential sound events have their unique “cell” in $\mathcal{M}$. The matrix-like compact organization enables fast computation due to matrix parallelization computation supported by most GPU/CPU engines. Similar idea has been used for video-based activity recognition(Lin et al., 2018).

Accompanying the dense proposal map are two score maps of the same size: $\mathcal{M}_{t}$ measuring a sound event proposal’s temporal overlap with ground truth, and $\mathcal{M}_{s}$ sound event proposal motion smoothness map. The two score maps jointly help to measure sound event proposals’ confidence to be a true positive sound event. Specifically, the potential proposal indicated by cell $C_{i,j}$ is associated with a event wise feature representation $f_{ij}$, a temporal overlap confidence score $s_{i,j}^{t}$ from $\mathcal{M}_{t}$ and a motion smoothness score $s_{i,j}^{s}$ from $\mathcal{M}_{s}$. $f_{i,j}$ derives from the averaging features learned by Backbone network spanning its temporal duration $[t_{j},t_{i+j}]$, $f_{i,j}=\frac{1}{i}\sum_{k=j}^{i+j}f_{k}$.

Temporal overlap score map $\mathcal{M}_{t}$ measures two sound events’ temporal overlap degree. Here we adopt temporal intersection over union (tIoU). Given two sound events $s_{1}=[t_{s1},t_{e1}]$ at cell $\mathcal{M}_{i,j}$ and its ground truth temporal location $s_{2}=[t_{s2},t_{e2}]$, their tIoU at $\mathcal{M}_{t}^{i,j}$ is defined as,

The higher of tIoU, the more likely the proposal at $C_{i,j}$ is a true positive proposal. Note that $\mathcal{M}_{t}$ is class-agnostic and its responsibility is to generate all potential event proposals (also can be thought as eventness proposal). In training, we train a $\mathcal{M}_{t}$ map regressor to learn the mapping from event wise feature representation to tIoU score. During test, we partially use the regressor to generate raw sound event proposals by choosing the large scores. To further enhance to regressor to fast localize a sound event, we propose a boundary sensitive tIoU $tIoU_{bs}$ which explicitly penalizes even a small temporal location alteration by multiplying an exponential term,

where $w$ is a decay weight controlling tIoU decay rate, which we set 2. Our observation is that as a sound event temporal length increases, slight temporal alteration results in small tIoU score change. For example, given the ground truth sound event resides in frames [20,80] and the proposal is [22,82], the original tIoU will be very high (0.94 in this case) as there is a high degree of overlap. This does not give sufficient steering to the network to correctly align the event boundaries. Under our new metric, however, $tIoU_{bs}=0.83$, the misalignment is prominently magnified so that the network receives enough clue to improve its localization capability.

An independent sound event’s spatial trajectory should be consistent and smooth. In other words, there should be no abrupt “jump” along the sound event’s trajectory, regardless of its motion status. We call this property as motion smoothness. Specifically, we model motion smoothness as the maximum neighboring location displacement along the sound event’s motion path. Mathematically, for the sound event $C_{i,j}$’s sequential spatial location $\{l_{0},\cdots,l_{i}\}$, the motion smoothness is defined as the maximum neighboring spatial location displacement $d$,

$d(\cdot)$ is modelled as squared Euclidean distance. Small motion smoothness score serves as a strong indicator of the existence of a potential complete sound event because an inaccurate sound event localization inevitably introduces large “motion jump”. We hereby jointly use $\mathcal{M}_{s}$ and $\mathcal{M}_{t}$ to efficiently generate proposals. The introduction of motion smoothness map has two main advantages. First, it integrates temporal/spatial head so that the whole framework can be trained in a unified manner. Second, it explicitly adds temporal regularization to spatial location head, because spatial location head alone is a framewise regressor and no temporal priors are involved. Please see Fig.3 for dense event proposal illustration and its connection with $\mathcal{M}_{t}$ and $\mathcal{M}_{s}$.

The final component of SoundDet is the spatial location head. As the the sound source physical location or DoA can change over the duration of an event, the spatial location is computed in a framewise manner, using the framewise features extracted by the backbone network. We regress per-class spatial location for each frame and all inactive sound classes’ spatial location is set 0.

The overall SoundDet pipeline is shown in Fig.1. The multi-channel raw waveform input is fed to the backbone neural network to efficiently learn framewise representation. The spatial location recovery head and dense event proposal generation head directly build on the learned framewise representation to learn framewise per-class sound event spatial locations and densely generate event proposals (per-event representation), respectively. The densely generated event proposals are fed to temporal recovery head to learn proposals’ category label.

In practice, the temporal location head builds on event wise feature. It consists of three sub-heads: a multi-label classification head deciding the proposal’s category label ($head1$); A binary classification head deciding whether a proposal is a foreground or background event (we can call it eventness prediction for better understanding). A tIoU regression head to learn to regress tIoU score based on the event wise feature input. The three heads consist of several full connection layers (FC). For multi-label classification and binary classification, we adopt the standard cross entropy loss. For the tIoU regression head, we adopt mean squared error (MSE) loss. The computation of $\mathcal{M}_{s}$ requires no specific learning process because it directly derives from spatial location head, MSE loss is adopted again to reinforce event’s smoothness.

Data Imbalance The dense sound event proposal map presented above covers all possible events, it inevitably leads to the data imbalance problem because most pre-generated proposals are negative and just very few proposals are positive. We adopt two strategies to mitigate this problem, 1) increase positive proposal number, in which we set a tIoU threshold $t_{d}$ and any event with the tIoU larger than the threshold is treated as positive proposal. 2) negative proposal random dropout, with the remaining negative samples, we randomly drop part of them with a probability $p_{d}$. By carefully setting $t_{d}$ and $p_{d}$ value, we can roughly keep positive and negative ratio to be 1:1.

During inference, the calculated $\mathcal{M}_{s}$ and $\mathcal{M}_{t}$ are jointly used to generate sound event proposals. The sound event proposal at cell $C_{i,j}$ is treated as positive proposal only if it passes two test: 1) $\mathcal{M}_{s}^{i,j}>d_{s}$ so that it satisfies motion smoothness rule; 2) $\mathcal{M}_{t}^{i,j}>d_{t}$ so that it resides in the designated temporal location $(t_{j},t_{i+1})$ with a high confidence. $d_{s}$ and $d_{t}$ are predefined spatial motion smoothness and temporal overlapness threshold. Once it passes the two tests, the final event detection score derives from the multiplication of its motion smoothness score, temporal overlapness score and the corresponding multilabel classification score (($head1$) discussed earlier). The class label comes from the same multilabel classification head and the spatial location derives from spatial location head.

Event Refinement The raw event candidates obtained above overlap in the temporal dimension. A post-processing is thus required to remove the redundancy (see Fig.4). The post-processing consists of two main parts: intra-class temporal non-maximum suppression (tNMS) and max-event clip. In intra-class tNMS, we compare within the same event class, for two events of the same class with temporal overlap (tNMS) above a predefined threshold, we delete the event with smaller confidence score. In max-event clip, we restrain the maximum number events that can happen at the same time. Max-event clip is a class-agnostic operation, for a set of events that happen at the same time, we sort them according the confidence score $s_{i}$ in descending order and only keep the top-M events. $M$ is a predefined max-event threshold. The post-processing is an iterative process, after which we get the final $K$ detected sound events from $N$ rw proposals $K\leqslant N$.

Segment-based Metric is the standard evaluation metric for existing methods (Grondin et al., 2019),(Tho et al., 2014),(Mesaros et al., 2019). It compares prediction and ground truth within non-overlapping short temporal segments (one sec.) and evaluates each frame’s prediction result by jointly considering its predicted class and spatial location. For event detection evaluation, it calculates $F1$-score ($F_{\leq T^{\circ}}$) and Error Rate ($ER_{\leq T^{\circ}}$), where a true positive prediction has to be within a spatial distance threshold with the corresponding ground truth spatial location (typically with an angular threshold $T=20^{\circ}$). For event localization evaluation, we compute class dependent localization error $LE_{CD}$ which measures the average location distance between prediction and ground truth, and localization recall $LR_{CD}$ measures the ratio of how many such localizations were estimated within a class. For details, please refer to (Mesaros et al., 2019).

Event-based Metric draws inspiration from evaluation on 2D image object detection. It treats a sound event as an independent instance with specific start time, end time, framewise spatial location and confidence score belonging to one class (see Sec. 3.8). Rather than simply binarizing a detection as false or true detection with an arbitrary threshold, event-based metric comprehensively evaluates performance under various event confidence score and tIoU thresholds, calculating average precision (AP) and average recall (AR) for each class separately. Finally, mean average precision (mAP) and mean average recall (mAR) is accumulated by averaging AP and AR throughout classes respectively.

Specifically, given $K$ detected sound events $E_{p}=\{t_{s,i},t_{e,i},l_{i},c_{i},s_{i}\}_{i=1}^{K}$ and $N$ ground truth sound events $E_{gt}=\{t_{s,i},t_{e,i},l_{i},c_{i}\}_{i=1}^{N}$ of the same class, we compute the average precision and average recall under tIoU in range $[0.1,0.05,1.0]$, where $0.05$ is the stepsize. For a particular tIoU, the AP and AP is computed by averaging precision scores and recall scores obtained under various confidence score in range $[0.1,0.05,1.0]$. Please note that we don’t take any predefined tIoU and confidence score threshold because arbitrary chosen thresholds inevitably introduces human bias e.g. the choice of an entirely arbitrary angular threshold $T=20^{\circ}$ in the segment based approach. Our proposed event-based metric instead provides a more objective and comprehensive metric.

We compare SoundDet with two existing methods: SELDNet(Grondin et al., 2019) and EIN(Cao et al., 2020). Since we focus on polyphonic and moving scenario, other relevant methods(Tho Nguyen et al., 2020)(Cao et al., 2019) that merely work on stationary and monophonic sound events are not discussed here. SELDNet(Grondin et al., 2019) jointly trains sound event detection and localization with CRNN network. It extracts Log-mel, GCC-PHAT, Intensity features as neural network input. Bidirectional GRU network is involved to model temporal dependency. SELDNet is treated as the baseline. EIN(Cao et al., 2020) is a very recent work which models sound event detection and localization with two identical but separate neural networks as a multi-task learning format. It uses a log-mel spectrogram as input for event detection, and a GCC-PHAT approach for DoA estimation. The two parallel networks are independent but with soft parameter sharing. In addition, multi-head self-attention(Vaswani et al., 2017) is applied. EIN has large parameter size and is currently the state of the art approach under segment-based evaluation metric.

The above two methods generate framewise predictions, in which each frame is associated with class classification score and predicted DoA value. In order to generate a score that metrics event classification score and DoA closeness between predicted location and ground truth location, we propose to integrate mean classification score $s_{c}$ and mean DoA Euclidean distance $d_{DoA}$ together in the formula $s=s_{c}\cdot e^{-d_{DoA}}$. For fair comparison, we don’t involve any data augmentation methods. We report the result for both FOA and MIC data format if possible.

Within SoundDet, we want to answer four main questions: 1. If our proposed MaxCorr band-pass filters is capable of learning useful representation for sound event spatio-temporal information recovery? 2. Event length impact on SoundDet and other methods. 3. Efficiency comparison in terms of parameters and inference time. 4. Quantitative comparison between SoundDet and other methods. To this end, we test four SoundDet variants: backbone only SoundDet with MaxCorr filter (SoundDet_backbone) and without MaxCorr filter (SoundDet_nomaxcorr). SoundDet without motion smoothness (SoundDet_nomots). Moreover, in addition to report the overall metric, we further divide the events into Small (0-2s), Medium (2s-7s) and Large ($>$7s), three categories and report result for them separately.

SoundDet network architecture is shown Table 4. Please note that the network might need to be slightly changed to process waveform with different input length, sample rate and label interval length. $H$ and $W$ indicate dense proposal map height and width, respectively, in our experiment $H=60$ and $W=60$, they can be directly modified to fit other input waveform temporal length or ground truth labelling resolution.

Training Detail We adopt multi-stage training strategy. First, we train the SoundDet backbone network in a framewise DoA regression and multilabel classification manner, like SELDNet(Grondin et al., 2019) and EIN(Cao et al., 2020) do. Training SoundDet backbone neural network at first provides two advantages: 1) it helps to test if our proposed MaxCorr filter bank helps to learn essential representation for sound event temporal detection and spatial localization (as we compare in the experiment), 2) it guarantees to learn a reasonable framewise representation first so that the later joint training (combine the backbone and two heads) converges much faster. For the backbone training, we use SGD optimizer with an initial learning rate 0.5, the learning rate decays every 30 epochs with decay rate 0.7. With the pretrained Backbone network, we continue to train the whole SoundDet network with the same optimizer.

The segment-based evaluation result is shown in Table 1. We can see that, by involving MaxCorr filter banks, our backbone-only SoundDet achieves comparable performance comparing with SELDNet(Grondin et al., 2019). SELDNet extracts Log-mel, GCC-PHAT and Intensity features (see Table 2) as its neural network input, replacing these hand-crafted features with learnable MaxCorr filter bank helps to achieve similar performance, even with 1D convolution. If disabling SoundDet to learn between-channel correlation (SoundDet_nomaxcorr), we can witness a huge performance drop in both FOA and MIC format, especially the high event detection error rate and large DoA error angle. This shows that our proposed MaxCorr filter bank is capable of learning essential representations for recovering temporal and spatial information. Actually, we observed a fast convergence of the between-channel time shift parameters (see Eqn.3), SoundDet quickly updates time shift parameters in the very first couple of training iterations and swiftly achieves a relatively stable state. Two learned MaxCorr filters are shown in Fig.5, from which we can observe that different MaxCorr filters have learned different time shifts and frequency cutoffs. These learned frequency-selective and phase-sensitive MaxCorr filter bank is of vital importance for sound event representation learning.

Both SoundDet and SELDNet work better for FOA format than MIC format. Framewise based learning (both SoundDet, SELDNet and EIN) shows no obvious difference in estimating events with different temporal length, which is reasonable because of their framewise processing property. The whole SoundDet (combine backbone and two heads) far outperforms SELDNet baseline and achieves comparable performance with EIN with much less parameter number. Specifically, SoundDet shows advantage on longer sound event estimation. It thus attests the necessity of directly modelling “sound-object” when it comes to events with longer temporal length or more complex motion trajectory. Excluding motion smoothness largely reduces the performance, we observed involving motion smoothness map greatly helps the neural network to accurate localize sound events.

The event-based evaluation result is shown in Table 2, from which we can observe that SoundDet outperforms all existing methods in both the mAP and mAR metric. This shows that the segment-based metrics in current use do not comprehensively reflect an algorithm’s performance. Rather than focusing on a local segment evaluation, the event-based metric highlights an event’s completeness and continuity and SoundDet is naturally designed to meet these requirements. This advantage is echoed by the qualitative comparison in Fig. 6. We can clearly see that SELDNet and EIN inevitably contain many mixed sound events and frequently cut complete events into small disconnected segments. The situation becomes more serious when faced with overlapping situation. However, SoundDet better avoids this dilemma because by design it treats events as complete instances, rather than discrete segments.

We further report the average inference time of different methods to process a one-minute long audio in Table. 3, showing that it is almost twice as fast as EIN, and comparable to SELDNet. In summary, SoundDet is capable of learning a sound event’s spatial, temporal and class information from raw waveforms, achieving competing performance under existing segment-based metrics and leading results on the proposed event-based metrics.