A Two-Stage Approach to Device-Robust Acoustic Scene Classification

Figure 1 shows the proposed two-stage ASC system. The overall system consists of two independent classifiers and outputs the class of the input audio scene choosing among ten classes. Different from a more conventional one-stage ASC systems, which directly feed acoustic features into the 10-class classifier and get final scene class, an extra general classifier, namely a 3-class classifier is introduced, which we expect it to enhance the classification process and leverage the over-fitting issue of 10-class classifier.

Three CNN based models are investigated to build our ASC system, namely Resnet, FCNN and fsFCNN. Resnet is a deep residual neural network [20] empowered by residual learning. Residual learning is a major efficient method to avoid gradient vanish issues while increasing the depth of networks to allow high-level feature learning. Our Resnet structure is based on the model in [21], which has no frequency sub-sampling throughout the whole network. Each input feature map is divided into two sub-feature mapping along the frequency dimension. To be specific, if we have $N$ frequency bins, the first $N/2$ and the second half are processed by two parallel stacked residual layers. A global pooling layer and 10-way soft-max are used to get the final utterance level prediction results.

The FCNN (fully convolutional neural network) model is a VGG [22]-like model which is built with 9 stacked convolutional layers with small-size kernel. Each convolutional layer is followed by a batch normalization operation and ReLU activation function. Dropout is also used in order to alleviate over-fitting issues. Before the final global average pooling layer, channel attention [23] is applied to each output channel of the last layer. fsFCNN (frequency sub-sampling FCNN) is an extension of FCNN model which mainly has 2 more convolutional layers and reduces max-pooling size in the frequency axis. In our experiments we notice that it can help to leverage over-fitting issue.

Finally, we build an ensemble using those CNN models and make the final predictions. The output probabilities after softmax are summed by equal weights, and then the scene class with the maximum posterior represents the final classification decision. For the 3-class classifier, only Resnet and FCNN are used; whereas, for the 10-class classifier, all three CNN models are used. The two-stage score fusion is performed after model ensemble of classifiers in each stage.

When training the CNN models, 9 different data augmentation methods are investigated. The training for 3-class models and 10-class models adopt the same data augmentation strategy. They can be categorized into two main classes: generating extra data or not.

Data augmentation strategies, which do not generate any extra data, include: (i) Mixup [24], which randomly mixes data batches with corresponding labels; (ii) Random cropping [21], which randomly crops input features into ones with a smaller length along the time axis; (iii) SpecAugment [25], which randomly masks input features by zero along both time and frequency axes. SpecAugment is performed on batch level, and the parameter of mask is set to 10% of the time and frequency dimensions, respectively.

Strategies that generate extra data include: (i) Spectrum correction [26], which aims at transforming a given input spectrum to that of a reference, possibly ideal, device. Different from the original idea, we newly employ spectrum correction as a data augmentation technique, where we create a reference device spectrum, by averaging the spectrum from all training devices except that from device A, and then correct the spectrum of each training waveform collected with device A to obtain extra data. (ii) Reverberation with Dynamic Range Compression (DRC), where we simulate more audio clips from ‘new devices’ by introducing reverberation and applying DRC - this data-agumentation solution is also new for ASC. Room impulse responses [27] are used to create reverberation into the device A data, and DRC is then applied to dynamically compress the amplitude range. Room impulse responses and DRC settings are randomly chosen to generate new training waveforms. (iii) Pitch shift, where we randomly shift the pitch of each audio clip based on the uniform distribution. (iv) Speed change, where we randomly change the audio speed based on the uniform distribution. If output waveform is longer than the original one, extra samples are dropped from the end, otherwise, padding is applied till attaining the same input length. (v) Random noise, we randomly add Gaussian noise to each training waveform. (vi) Mix audios, where we randomly mix two audios from the same acoustic scene class. It’s device-independent so it may help simulate a ‘new device’. Although the solutions from (iii) to (vi) are not new strictly speaking, their application for ASC device robustness has never been evaluated before.

All ASC systems are evaluated on the DCASE 2020 task1a development data set [3], which consists of $\sim$14K 10-second single-channel train audio clips and $\sim$3K test audio clips recorded by 9 different devices, including real devices A, B, C, and simulated device s1-s6. Only device A, B, C, s1-s3 are in the training set; whereas, devices s4-s6 are not available in the training phase. The greatest amount of training audio clips are recorded with device A, namely over 10K audio clips. In the test set, the number of audio clips from each device is the same. For each input audio clip, a short-time Fourier transform (STFT) with 2048 FFT points is applied, using a window size of 2048 samples and a hop length of 1024 samples. The Librosa [28] library is employed to extract log-mel filter bank (LMFB) features. Log-mel deltas and delta-deltas without padding are also computed, which finally generates an input feature tensor with size of $423\times 128\times 3$. Before feeding the input tensors into the CNN classifier, we perform utterance-level scaling operation to scale LMFB coefficients into [0,1]. All deep models in this work are built with Keras [29]. Stochastic gradient descent (SGD) with a cosine-decay-restart learning rate scheduler is used to train all deep models. Maximum and minimum learning rates are 0.1, and 1e-5, respectively.¹¹1Code available: https://github.com/MihawkHu/DCASE2020_task1

Table 1 shows the accuracy for 3-class classifiers, namely Resnet, FCNN and Ensemble of these two models. The 3-class classification task is relatively easy since an accuracy above 90% is attained by each of the three models. FCNN shows a better accuracy than Resent, and the ensemble of these two models allows a further boost of the classification result. In the following experiments with the two-stage system, we use Ensemble system as the 3-class classifier.

All main experimental results concerning the 10-class ASC task are shown in Table 2. In the training set, device A data accounts for around 75%, and device B, C, s1-s3 accounts for around 5%, respectively. Thus, we can think of device A as a source device, and the remaining ones as target devices. Based on the device information of data, we divide the test set into four different subsets, which represent real source data (device A), real target data (device B & C), target seen-simulated data (device s1-s3), and target unseen-simulated data (device s4-s6). The first row in Table 2 gives ASC accuracy of the official baseline system [3], which uses a two-fully connected layer neural classifier, and OpenL3 [15] to extract input audio embedding. From baseline results, we can argue that good ASC accuracy can be attained on real source data (device A), but a severe performance drop is observed on other devices. Specifically, a severe ASC accuracy drop is reported on unseen devices (s4-s6), where the ASC accuracy is as low as 44.3%. Rows from 6th through 9th in Table 2 display ASC results attained with a 10-class classifier. Resnet attains a significant improvement on all test devices over the baseline system. FCNN and fsFCNN outperform Resnet on all testing scenarios, and FCNN achieves slightly better results than fsFCNN. Finally, the ensemble of these three models attains a 79.4% overall ASC accuracy, which is a meaningful improvement compared to any of the single-stage models reported in Table 2 .

The experimental results of two-stage systems are shown in the last four rows of Table 2. When comparing single-stage 10-class classifier with the two-stage system, a significant improvement can be attained. For example, comparing ‘Resnet’ with ‘2-stage Resnet‘, 3.1% absolute improvement on overall accuracy is observed (74.6% vs. 77.7%). Accuracies on all the test sets have indeed improved, especially on the unseen-simulated test set (device s4-s6), where a 5.4% absolute improvement is obtains. The overall absolute accuracy gains with the proposed two-stage technique are 3.2% for FCNN, 2.8% for fsFCNN, and 2.5% for the ensemble. The best performance is attained by the two-stage of three models’ ensemble, which achieves 81.9% overall ASC accuracy. Although the top performance is still delivered on the source device A (87.9%), the average ASC accuracy on the remaining test devices is above 79%, which compares favourably against the baseline system. When comparing ASC accuracy between seen-simulated test set (device s1-s3) and unseen-simulated test set (device s4-s6), we can argue that the gap in performance is mild (80.4% vs. 79.9%), and they are close to the overall average accuracy (80.4% vs. 79.9% vs. 81.9%). Those results confirm the effectiveness of the proposed two-stage system with respect to improving device robustness.

To better compare our results, the results of top-5 ranking systems (excluding ours) [30, 31, 32, 33] are also listed from 2st to 5th rows in Table 2. The experimental results clearly demonstrated that the proposed two-stage system achieves a competitive performance with all benchmark models.

We now investigate the effectiveness of different data augmentation strategies in combination. We test only the Resnet-based ASC system to keep the experimental setup consistent. Table 3 shows the classification accuracy attained with different data augmentation schemes. It should be noted that mixup and random cropping are always employed in the training phase.

When comparing the first two rows, we can see that specAugment can boost the overall ASC accuracy from 71.0% to 71.6%, but on unseen data the accuracy becomes worse (from 69.5% to 67.7%). Next, we add spectrum correction, which results in an increase in the amount of training data. As described before, device A data is used to generate data from other devices, in which we get the same size extra data with device A data. From the results in Table 3, we can observe that spectrum correction has a beneficial effect on overall classification results, but the accuracy on device A decreases. That outcome is not unexpected since there is now much more training data from other devices than from device A. New extra data is generated by reverberation with DRC, which aims to simulate data from more different devices. From the results we can see when extra data with reverberation and DRC is added in the training set, the overall ASC accuracy is further boosted (72.8%), where most of the gain is imputable to an increse of the accuracy on the unseen test data. Finally, we employ four more data augmentation approaches, including pitch shift, speech change, random noise and mix audios. As shown in the last row of Table 3, performance can be further boosted. Using all strategies, a 74.6% accuracy on Resnet is attained²²2We also increased amount of Resnet parameter, i.e, the channel number, when all data-augmentation schemes are used..

Neural saliency methods [18, 34] aim to provide interpretable analyses on the weight distribution over hidden neurons of a well-trained artificial neural network model. Benchmark saliency methods, such as class activation mapping (CAM) [18], have shown a correlation between model performance and ability in spotting physical pattern in DNN-based acoustic models [34].

In this work, we use CAM [18] to gain a better understanding about what sound patterns are found by our CNNs to accomplish the ASC task. Indeed, CAM can highlight the class-specific discriminative regions in the input feature map. In other words, CAM helps generating a two-dimension activation map that can be used to better interpret the prediction decision made by deep architectures. In our case, where an audio clip is transformed into a time-frequency representation, CAM reveals whether the ASC decision is triggered by CNN’s attention posed on time and frequency regions of input audio clip having a meaningful semantic content with respect to the target acoustic scene. Examples of some analysis results obtained with CAM are shown in Figures 2 and 3, where the spectrogram of the input audio clip (top panel), and the CAM for the 3-class classifier (center panel), and the 10-class classifier (bottom panel) are shown. Regions in CAM with deep red color imply that the CNN pays more attention to them, and those regions have a higher class-specific discriminative power. As the size of FCNN and fsFCNN’s output feature map is too small to map back to the input, only Resnet model is used for neural saliency analysis.

In Figure 2, a 10-second audio clip referring to a metro station scene is visualized; the audio clip contains brake and horn sounds from 0s to around 5s. After 5s, only reverberation remains in the audio. The CAM result indeed reveals that the CNN pays more attention to the segment between 0s and 5s. Moreover, 3-class classifier and 10-class classifier have similar activation regions, and the 3-class classifier seems to give more weight to the information in that region. Indeed, if we check the confidence in making the prediction, i.e., scaled class posterior probability from the CNN, the proposed two-stage fusion can increase it from 70.7% to 85.0%. The same pattern can be observed in the results of Figure 3, in which it contains a 10-second audio from a bus station. This audio has brake sound between around 2s and 5s, and human speech starts around 5s till 7s. From the CAM results, we can see that CNN pays more attention to the segment from 2s to 5s, which contains a brake sound. However, the CNN is not interested in the human speech although it is clearly audible and related to the bus arriving announcement. This result is indeed interesting: Human beings would use the bus-station announcement to make ASC decision; however, the CNN cannot perform speech recognition, so it may simply classify that region as belonging to an ideal human speech class, which however appears in many scenes and has therefore a low discriminative power.

We can thus argue that CNNs use acoustic events to classify the input scene audios. Specifically, CNNs pay more attention to audio segments containing acoustic events, such as birds sound from park, or car horn sound from street traffic. It should be noted that our conclusion with CAM is different from that in [19], where the authors claim that CNNs use background sound rather than acoustic events. By comparing the experimental setups, the main difference is that each audio clip is chunked into 10 1-second segments in their setup; in contrast, we use the whole 10-second audio clip as the input. Since acoustic events usually last for several seconds, it’s difficult for CNNs to capture them when inputs are very short.