Voice activity detection in the wild via weakly supervised sound event detection

Voice activity detection (VAD), whose main objective is to detect voiced speech segments and distinguish those from unvoiced ones, is a crucial component for tasks such as speech recognition, speaker recognition, and speaker verification. Deep learning approaches have been successfully applied to VAD [1, 2, 3, 4]. For VAD in complex environments, neural networks (NN) have been successful. Deep neural networks (DNN) and specifically convolutional neural networks (CNN) offer improved modeling capabilities compared to traditional methods [2], while recurrent- (RNN) and long short-term memory (LSTM) networks can better model long-term dependencies between sequential inputs [1, 5, 6]. However, despite the application of deep learning methods, NN-based VAD training still requires frame labels. Thus training data utilized is usually under controlled environment with or without additional synthetic noise [7]. This inevitably prevents VAD from real-world applications, where speech in the wild is often accompanied by countless unseen noises with different features.

Therefore, this paper intends to propose a method to detect speech beyond clean and controlled noisy environment. It should be noted that frame-level labels are quite unlikely to come with real-world recordings since manual labeling is costly, and label predictions from a Hidden Markov model need prior knowledge about the language being spoken [8]. A task to detect speech components while enabling noisy data training, is related to weakly-supervised sound event detection (WSSED), which detects and localizes different sounds, including speech via clip-level supervision. Since WSSED systems are reported [9] to be robust to noise and only require clip-level labels, this work integrates WSSED methods in scaling VAD to speech in-the-wild scenarios and relaxing its dependence on frame labeling. Specifically, we investigate two questions: 1) Are current, multi-class WSSED models comparable in performance to DNN-based VAD; 2) Is utterance-level training a viable alternative compared to frame-level? We thus introduce our framework, a general-purpose training framework for VAD (GPVAD, see Figure 1). By general purpose, we refer to two distinct aspects: First, the framework is noise-robust and capable of being deployed in wild, real-world scenarios; Secondly, the framework can be trained on unconstrained data, thus enabling learning from big webly data like noisy online-videos.

The paper is structured as follows: In Section 2, we briefly review the related work on WSSED and how it can be transferred for VAD in the wild. In Section 3, the GPVAD approach is introduced. Moreover, in Section 4 we introduce our experimental setup and provide implementation details. In Section 5 the results are presented and finally in Section 6 a conclusion is provided.

Since WSSED can work well in detecting speech in a noisy environment without frame-level labeling, we borrow this idea to realize VAD in the wild. Here we present related work on sound event detection (SED), which aims to classify (audio tagging) and possibly localize multiple co-occurring sound events from a given audio clip. In this work, we mainly focus on weakly-supervised SED (WSSED), a semi-supervised task, which has only access to clip-level labels during training, yet needs to classify and localize a specific event during evaluation. This weakly-supervised fashion enables training on noisy data with lower requirements for the labeling method. Recent advances in weakly supervised sound event detection, in particular, the detection and classification of acoustic scenes and events (DCASE) challenges [10], led to large improvements for predicting accurate sound event boundaries as well as event-labels [11, 12, 13, 14, 15, 16]. In particular, recent work [9] has shown promising performance regarding short, sporadic events such as speech.

Traditionally, VAD for noisy scenarios is modeled as in Equation 1. The assumption is that additive noise $\mathbf{u}$ can be filtered out from an observed speech signal $\mathbf{x}$ to obtain clean speech $\mathbf{s}$.

However, directly modeling $\mathbf{u}$ is rather tricky, since each type of noise has its individual traits. Therefore, we aim at learning the properties of $\mathbf{s}$ by observing it with potentially $L$ different non-speech events $\left(\mathbf{u}_{1}\ldots,\mathbf{u}_{L}\right)$. Those events are not restricted to being background/foreground noises and can have distinct real-world sounds (e.g., Cat, Music).

Our approach stems from multiple instance learning (MIL), meaning that training set knowledge about specific labels is incomplete (e.g., Speech never directly observed). Here, we model our observed speech data $\mathcal{X}$ as a “bag”, containing all co-occurrences of Speech in conjunction with another, possibly noisy background/foreground event label $l\in\{1,\ldots,L\}$ from a set of all possible event labels $L<E$ (Equation 2). So to speak, our approach aims to refine a model’s belief about the speech signal $\mathbf{s}$, within complex environmental scenarios. The advantage of this modeling method is that it can be applied for both frame- and clip-level training. Our GPVAD, therefore, relaxes these constraints by allowing training on clip/utterance level, where each training clip contains at least one event of interest. We propose two different models: GPV-F, which outputs $E=527$ labels ($L=405$) and the naive GPV-B, $E=2,L=1$. GPV-F can be seen as a full-fledged WSSED approach using maximal label supervision and is, therefore, more costly than GPV-B, which only requires knowledge about a clip containing Speech. However, GPV-F should be capable of modeling each individual noise-event instead of clustering all noise into a single class (GPV-B), thus possibly enhancing performance in heavy noise scenarios. The two models are compared against a model trained on frame-level, further referred to as VAD-C.

All models share a common backbone convolutional recurrent neural network (CRNN) [9] approach used in WSSED, which is shown to be robust towards short, sporadic events such as Speech. The following modification to [9] have been done: 1. Add an upsampling operation, such that the models’ time-resolution remains constant. 2. Use $L^{p}$ pooling as our default with $p=4$, as it has been seen to be beneficial for duration invariant estimates. Different from VAD-C training, where frame-level labels are available, our GPVAD framework is split into two distinct stages. During training, only clip/utterance-level labels are accessible. Therefore a temporal pooling function is required (Equation 4). During inference, post-processing needs to be applied (Section 4.3) to convert probability sequences into binary labels (absence/presence of an event) as well as any predicted non-speech label is discarded. The framework is depicted in Figure 1.

In our work, deep neural networks were implemented in PyTorch  [17], front-end feature extraction utilized librosa [18]. Code is available online¹¹1Available at github.com/richermans/gpv.

Our results can be seen in Table 2. Firstly, we show that our VAD-C model is capable of performing on an equal footing to other deep neural network approaches [6]. Comparing VAD-C with GPV-B/F, it can be seen that VAD-C is indeed the best performing model given our metrics for clean and synthetic noise datasets. However, evaluation on the real-world dataset reveals a different picture. Here, VAD-C seems to be struggling against the naive GPV-B approach (AUC 87.87 vs. 89.12, FER 21.92 vs. 19.65), indicating that VAD-C is more likely to misclassify speech in the presence of real-world noise. Moreover, in real-world scenarios, GPV-F is outperforming VAD-C for each proposed metric. Our proposed GPV-F approach can also be seen to be consistently noise-robust since its performance difference between synthetic noise and real-world scenarios is minor.

Even though GPV-B is, on average underperforming against the other two approaches, one should note that it is the least costly system, since labeling data for GPV-B is essentially a binary question whether one heard any speech within a clip, making this approach capable of cheaply scaling to large data. We conclude that the GPVAD models trained with only clip-level labels are capable of competing trained on frame-level labels.

This paper introduces a noise-robust VAD approach by utilizing weakly labeled sound event detection. Two GPVAD systems are investigated: GPV-B, trained on binary speech and non-speech pairs only, as well as GPV-F, which utilizes all 527 AudioSet labels. Our evaluation protocol thoroughly compares our proposed GPVAD approach to traditional VAD utilizing five distinct metrics. Results indicate that GPV-B, even though trained on clip-wise, unconstrained speech, can be used to detect spoken language, without requiring clean, frame-labeled training data. Further, while GPV-B/F both fall short in clean and synthetic noise scenarios against VAD-C, they excel at stable predictions for real-world data. Specifically it can be seen that our proposed approach is robust in its performance across the synthetic and real-world noise datasets. Our best performing model, GPV-F outperforms traditional supervised VAD approaches by a significant margin on real-world data, culminating in an absolute increase of 5.57% F1-macro, 6.45% F1-micro, 3.93% AUC, 6.45% FER and 10.4% Event-F1.

This work has been supported by National Natural Science Foundation of China (No.61901265) and Shanghai Pujiang Program (No.19PJ1406300). Experiments have been carried out on the PI supercomputer at Shanghai Jiao Tong University.