DcaseNet: An integrated pretrained deep neural network for detecting and classifying acoustic scenes and events

Recent advances in deep learning have led to improved performance of acoustic scene classification (ASC) and event related systems in various applications [1, 2, 3, 4, 5, 6]. The detection and classification of acoustic scenes and events (DCASE) community holds annual challenges with public datasets [7, 8, 9]. The DCASE challenge datasets have fostered research on various tasks including ASC, audio tagging (TAG), sound event detection (SED), bird audio detection, and sound localization.

However, these tasks have been independently studied using different deep neural networks (DNNs), despite that their characteristics and required information are highly related. Few studies have explored DNN architectures that use two tasks. Jung et al. [10, 11] applied an attention mechanism for ASC using a pretrained TAG system. The study was inspired by human cognition (e.g., [12]), as humans first perform SED and leverage this information to classify scenes. For instance, perceiving car horns and traffic sounds can be helpful for knowing that he/she is standing in a street. Imoto et al. [13, 14] explored the relation between ASC and SED, proposing DNNs to perform the two tasks simultaneously through a multi-task learning framework [15]. However, the integration of DNNs for related tasks remains in a preliminary stage because only pairs of two related tasks are investigated and motivations on the relationship of these tasks has not been explored. Hence, studies which address motivations for using multiple tasks and analyze the relationship between such tasks requires further investigation.

We explore the integration of three tasks by proposing different DNN architectures (Fig. 1), from which two are inspired by human cognitive processes and the third extends the second architecture. The integrated framework performs two segment-level tasks and one frame-level task using a single DNN that jointly learns and performs ASC, TAG, and SED (three tasks are introduced in details in Section 2). The first architecture resembles the short-term perception of adults for ASC, being similar to the method used by Jung et al. [10]. Instead of using a TAG system to improve an ASC system, we propose an integrated framework that first performs SED followed by ASC and TAG.

The second architecture resembles the long-term learning of babies, which is also a motivation for self-supervised learning studies [16, 17, 18]. In self-supervised learning, pretraining a DNN for relatively abstract tasks and then fine-tuning it for a specific task is an effective approach. It mimics babies first acquiring abstract notions (e.g., the effects of gravity) and then learning specific tasks based on the corresponding perceptions. Likewise, we assume that segment-wise multiclass ASC might require a relatively low abstraction level compared with SED, which requires frame-wise multilabel binary classification. Thus, in the proposed architecture, we consider the abstraction level of each task and perform relatively coarse scene classification followed by specific SED. Comparative experiments demonstrate that the second architecture provides high performance, and thus we extend it using a few separate layers for each task.

The goal of this study is to build a single DNN that integrates ASC, TAG, and SED and can be fine-tuned to emphasize the performance of any of these tasks. The proposed DNNs, called DcaseNets, are intended to establish pretrained models for a wide range of acoustic tasks, like the ImageNet pretrained model [19]. The main contributions of this study are threefold:

1. 1.

   Integrating DNN architectures, DcaseNets, to simultaneously perform ASC, TAG, and SED.

2. 2.

   Developing DNN architectures based on human cognitive processes, namely, scene perception of adults and long-term learning of babies.

3. 3.

   Demonstrating that fine-tuning the integrated DNNs for a specific task improves performance.

The remainder of this paper is organized as follows. In Section 2, we describe ASC, TAG, and SED and discuss early studies on their integration. In Section 3, we explain the proposed DNNs for setting integrated architectures to perform the three tasks. In Section 4, we report the experiments and results to verify the architecture performance. Finally, we draw conclusions in Section 5.

ASC is a multiclass classification task that identifies a segment as being one of predefined scenes (i.e., classes). Acoustic scenes have an abstract (i.e., ambiguous) definition, and thus various characteristics may coincide across different scenes [20, 21]. For example, ‘airport’ and ‘shopping_mall’, which are both predefined scenes in the DCASE ASC challenge, may contain people talking and the acoustic properties of large indoor spaces. These characteristics lead to a large intraclass variability, possibly degrading the classification performance. In addition, recent studies on ASC have addressed the impact of different recording devices on classification [22, 23, 24].

TAG and SED are both multilabel binary classification tasks that perform event detection (i.e., judge the presence of various sounds). TAG conducts segment-level event detection and provides a vector in which each dimension is a real number between 0 and 1, indicating the probability of presence of a sound event throughout an input segment. SED conducts frame-level event detection and provides a matrix in which each row indicates the presence of sound events in a frame. Therefore, SED performs TAG with the onset and offset information from each sound event. Various studies on deep learning have addressed frame-level and the subsequent aggregation into segment-level classification. Likewise, we perform SED before TAG in the three proposed DNN architectures.

Scarce research on the combination of related sound classification tasks has been conducted [13, 14, 10, 11]. Imoto et al. [13] assumed that ASC and SED are related and performed them simultaneously using a multitask learning framework. Jung et al. [10] aimed to mimic the human perception mechanism of leveraging TAG for ASC by applying an attention mechanism using the output of a pretrained TAG network. However, we assume that analysis and architecture designs accounting for the task relations can further improve integrated systems.

We propose and investigate three DNN architectures to jointly perform ASC, TAG, and SED, as illustrated in Fig. 1. The architectures are called as DcaseNet because it jointly detects and classifies acoustic scenes and events. The design choice of the first and second architectures (Fig. 1-(a) and -(b)) is inspired by human cognitive processes. The first architecture resembles the perception mechanism of an adult, expanding the method in [10]. This architecture first performs the event detection (i.e., SED) using a convolutional recurrent neural network (CRNN) to then perform segment-level scene classification (i.e., ASC). From the two event detection tasks, TAG and SED, we perform SED in lower layers (closer to the input layer) in the three proposed architectures by assuming that detecting short events improves the overall detection throughout an audio segment.

The second architecture (Fig. 1-(b)) resembles long-term learning of babies. First, abstract notions (e.g., gravity and dimension) are acquired to then perform specific tasks (e.g., moving things) using these notions. Studies on self-supervised learning [16, 17, 18] are also based on this process to pretrain DNNs with unlabeled data. Similarly, we consider the required abstraction level in each task representation. We assume that ASC requires a relatively lower abstraction level than both TAG and SED. Hence, the second architecture performs ASC after a few convolutional neural network (CNN) blocks and then performs TAG and SED after a gated recurrent unit (GRU) layer.

As the two architectures are inspired by human cognition, we experimentally test their performances and find that the second architecture tends to outperform the first one. Thus, the third architecture (Fig. 1-(c)) extends the second architecture with additional layers before conducting each task. These separate layers are also concatenated and feed-forwarded to subsequent layers. As applied in [25], we aim to maintain an information path (green layers in Fig. 1) while dedicating few layers to solely concentrate on performing an individual task. In addition, instead of performing TAG and SED in parallel, hidden layers perform SED and TAG in sequence, assuming that TAG, a segment-level task, requires a relatively higher abstraction level than SED, a frame-level task. For the three architectures, we first perform single and joint training. The models trained for single tasks establish the baselines for the corresponding architectures. After training, we fine-tune every model by using jointly trained DNNs for each task to determine the final performance on the target task.

Each component of DcaseNet is based on high-performing architectures for each task. The CRNN (two light green boxes in Fig. 1) in the three DcaseNet architectures is the one used by Cao et al. [26], who achieved the second place in task 3 of the DCASE 2019 challenge using this network. The CRNN adopts eight convolutional layers with batch normalization [27], followed by a bidirectional GRU layer. ‘CNN’ at the top of Fig. 1-(a) establishes a residual block and was implemented in [11], which was submitted for task 1-a in the DCASE 2020 challenge. ‘Dense’ block in the three architectures before the TAG output is the one used by Akiyama et al. [28], who won task 2 of the DCASE 2019 challenge. It adopts two fully-connected layers followed by dropout [29] and ReLU non-linear activation. ‘Dense’ block before the SED output in Fig. 1-(b) also comprises two fully-connected layers with dropout and ReLU activation.

We propose three integrated DNN architectures, DcaseNets, inspired by human cognitive processes to simultaneously perform ASC, TAG, and SED. The first architecture resembles the perception mechanism for acoustic scenes of adults, and the second architecture resembles the learning of babies, with the latter providing higher performance. The third architecture that adds a few layers before the output of each intermediate task further improves the performance of the second architecture. Jointly trained models can be further fine-tuned for each task. Experimental results show the high performance of the proposed DcaseNet-v3 for the three tasks after joint training, outperforming all the corresponding baselines with fine-tuning.

However, as this is the first study that integrates the three related tasks, further investigation and improvements should be performed. As we experimentally verified that DcaseNet-v2 outperforms DcaseNet-v1, we will adopt self-supervised learning aiming to improve the task performance. In addition, we will apply knowledge distillation and leverage soft-label training. Soft-labels will enable cross-dataset training by, for example, generating soft-labels for the ASC dataset and using the result to calculate the TAG loss.