Domain Generalization on Efficient Acoustic Scene Classification using Residual Normalization

To design a low-complexity network in terms of the number of parameters, we use a Broadcasting-residual network (BC-ResNet) [12] which uses 1D and 2D CNN features together for better efficiency. While the BC-ResNet targets human voice, we aim to classify the audio scenes. To adapt to the differences in input domains, we make two modifications to the network, i.e., limit the receptive field and use max-pool instead of dilation.

The proposed architecture, BC-ResNet-ASC, is shown in Table 1. The model has 5x5 convolution on the front with a (2, 2) stride for downsampling followed by BC-ResBlocks [12]. In [8], they show that the size of the receptive field can regularize CNN-based ASC models. We change the depth of the network and use max-pool to control the size of the receptive field. With a total of 9 BC-ResBlocks and two max-pool layers, the receptive field size is 109x109. We also do the last 1x1 convolution before global average pooling that the model classifies each receptive field separately and ensembles them by averaging. Original BC-ResNets use dilation in temporal dimension to obtain a larger receptive field while maintaining temporal resolution across the network. We observe that time resolution does not need to be fully kept in the audio scene domain, and instead of dilation, we insert max-pool layers in the middle of the network.

In this work, we use BC-ResNet-ASC-1 and BC-ResNet-ASC-8 whose base numbers of channels $c$ are 10 and 80, respectively, in Table 1. Table 2 compares our BC-ResNet-ASC-8 with two baselines: CP-ResNet [8] which is a residual network-based ASC model with limited receptive field size; and original BC-ResNet-8 with the number of Subspectral Normalization [14] groups of 4. As shown in Table 2, BC-ResNet-ASC-8 records Top-1 test accuracy 69.5% with only one-third number of parameters compared to CP-ResNet showing 67.8% accuracy. Moreover, BC-ResNet-ASC-8 outperforms the original BC-ResNet-8 by a 1% margin with the modifications.

Instance normalization (IN) [15] is a representative approach to reducing unnecessary domain gaps for better domain generalization [16] or domain style transfer [17, 18] in the image domain. While domain difference can be captured by channel mean and variance in the image domain, we observe that differences between audio devices are revealed along frequency dimension as shown in Figure 1. To get audio device generalized features, we use instance normalization by frequency (FreqIN) as below.

$$FreqIN(x)=\frac{x-\mu_{nf}}{\sqrt{\sigma_{nf}^{2}+\epsilon}},$$

(1)

where,

	$\displaystyle\mu_{nf}$	$\displaystyle=\frac{1}{CT}\sum_{c=1}^{C}\sum_{t=1}^{T}x_{ncft},$
	$\displaystyle\sigma^{2}_{nf}$	$\displaystyle=\frac{1}{CT}\sum_{c=1}^{C}\sum_{t=1}^{T}{(x_{ncft}-\mu_{nf})^{2}}.$		(2)

Here, $\mu_{nf}$, $\sigma_{nf}\in\mathbb{R}^{N\times F}$ are mean and standard deviation of the input feature $x\in\mathbb{R}^{N\times C\times F\times T}$, where $N$, $C$, $F$, $T$ denote batch size, number of channel, frequency dimension, and time dimension respectively. $\epsilon$ is a small number added to avoid division by zero.

Direct use of IN can result in loss of useful information for classification contained in domain information. To compensate for information loss due to FreqIN, we add an identity shortcut path multiplied by a hyperparameter $\lambda$. We suggest a normalization method, named Residual Normalization (ResNorm) which is

$$\textit{ResNorm}(x)=\lambda\cdot x+\textit{FreqIN}(x).$$

(3)

We apply ResNorm for input features and after the end of every stage in Table 1. There are a total of five ResNorm modules in the network.

To compress the proposed model, we utilize three model compression schemes: pruning, quantization, and knowledge distillation.

We do the experiments using BC-ResNet-ASC-1 and BC-ResNet-ASC-8, and the overall results are on Table 3. The task has multi-device inputs which are imbalanced with dominant device A. As a result, the baseline, BC-ResNet-ASC-1, shows that the accuracy of the device A is relatively higher than other seen devices, B, C, S1, S2, and S3. Furthermore, the accuracy on unseen devices, S4, S5, and S6 are even lower, and these results imply that the model is not generalized well to multiple devices, especially for unseen devices. When we use global normalization by frequency dimension, the result shows 60.0% accuracy which is 1% improvements compared to the baseline, but still we can observe poor domain generalization. We also try PCEN, a normalized feature instead of log Mel spectrogram. PCEN shows improvements for unseen devices, but we also observe that the performance of device A degrades due to its normalization. The proposed ResNorm uses FreqIN to get domain invariant features while not loosing the useful class-discriminative information through identity shortcut connection. The ‘BC-ResNet-ASC-1 + ResNorm’ shows a large improvement, 6% compared to baseline and records 65.8% test accuracy. The ResNorm shows performance improvements not just for unseen devices but also for all seen devices.

We do some ablation studies for the component of ResNorm. First, we use the ResNorm module as the preprocessing module, and do not use the module in the middle of the network; ‘w/o ResNorm in Network’. For the small model, BC-ResNet-ASC-1, ‘w/o ResNorm in Network’ shows better performance, 67.1%, and for the larger model, BC-ResNet-ASC-8, it shows a performance degradation of 1%. Due to ResNorm’s regularization effect, it was expected that this module could degrade the performance of a small network. We expect that the module can control the normalization power by the hyperparameter $\lambda$ in Equation 3 to adapt to various size of networks. In this work, we use fixed $\lambda=0.1$, and leave the automatic update of the $\lambda$ as a future work. Second, ‘w/o shortcut’ shows the result when $\lambda=0$ in ResNorm which equals to FreqIN in Equation 1. Our design motivation is that the shortcut path will keep the useful information for classification. The results show that FreqIN records relatively lower accuracy for seen devices compared to ResNorm. Especially, the margins on device A are 8.2% and 3.0% on BC-ResNet-ASC-1 and BC-ResNet-ASC-8, respectively.

Simultaneously, we distill the knowledge of the pre-trained teacher network (‘Vanilla’ model) into the compressed model for enhancing the performance and achieve the 0.2% improvement in test accuracy. In detail, we prune the convolution layers of the model with 89% pruning ratios compared to vanilla and quantize all convolution layers in a compressed model as an 8-bit. Other layers are quantized as a 16-bit. The resulting ‘Compressed’ model has 33K 8-bit nonzero for convolution layers and 15K 16-bit parameters for normalization, resulting in 61.5kB and shows 75.3% test accuracy which is 1% lower than Vanilla model. We use the ensemble of two compressed model in the DCASE 2021 challenge, task 1A.