AudioRepInceptionNeXt: A lightweight single-stream architecture for efficient audio recognition

Single-stream convolution neural network (CNN) is widely used in audio recognition tasks including sound event classification, speech recognition, and music classification [40, 2, 41, 42, 43]. There are two widely used approaches in single-stream CNN-based audio recognition systems. The first approach treats the raw audio waveform as input and utilizes a single-stream CNN to extract the feature for classification [43, 18, 44, 5]. Wav2vec [43] is a representation work in this regard that takes the 1D audio signal as input and trains a 1D CNN in an unsupervised contrastive manner to learn the intermediate audio representation for speech recognition. The second approach first preprocesses the 1D audio signal into a 2D time-frequency Mel-spectrogram followed by feeding it to the 2D CNN [45, 3, 21, 46].

On the other hand, multi-stream architectures simultaneously work with either single or multimodal data such as raw audio and spectral features of audio [23, 47, 1]. Slow-Fast model [1] is a representative work that uses two-stream network architecture to capture both frequency and fine-grained temporal information from two different resolutions of the Log-Mel-Spectrogram independently. Their results demonstrated that the performance of the two-stream network outperforms the state-of-the-art single-stream network.

Although the above-mentioned single-stream and two-stream networks demonstrate promising results in audio recognition, they incur high computational and memory footprints during training and inference, which hinder their deployment on low-resource edge devices, such as smartphones. Therefore, it is imperative to explore a lightweight and parameter-efficient single-stream network that can achieve comparable results with the single-stream and two-stream networks while achieving low computational and memory footprints. In this work, we propose a parallel multi-scale convolutional kernel block to enrich the temporal and frequency feature representation during the training. This results in a single-stream CNN-based network architecture with lower parameters and computational footprints. The proposed design further enables the use of model reparameterization by combining multiple kernels into a single kernel during inference without any loss of performance.

Model Reparameterization [33, 34, 38] approaches simplify complex multi-branch network architecture into a single-branch network structure during the model inference stage, without sacrificing performance. These approaches enable the use of the complex network architecture for learning efficient feature representations during the training stage and a simplified and parameters-efficient network during the inference stage. For instance, RepVGG [34] proposed extra $1\times 1$ convolutional layers parallel to the $3\times 3$ convolution layers in the original VGG network to learn richer feature representations during the training. After the training, the additional $1\times 1$ layers are merged with the $3\times 3$ layers via linear transformations of convolution, resulting in a VGG-like model with no extra parameters or computational cost during inference. RepVGG has been shown to outperform the original VGG model without adding any inference-time cost. Similarly, the Diverse Branch Block (DBB) method proposed in [38] enhances the representation capacity of a single convolution by combining multiple branches of varying complexities to enrich the feature representation. The DBB block includes a sequence of convolutions, multi-scale convolutions, and average pooling during training, which can be converted into a single convolutional layer using the linear transformation properties of convolution. Inspired by the success of RepVGG and DBB, RepLKNet [33] employed a similar technique by using a large kernel and a small kernel in parallel during training. After training, the small kernel is absorbed into the large kernel, enabling the large kernel to capture both global and local information resulting in improved the model’s performance.

Our motivation for model reparameterization differs from these previous works in one important way. We have discovered that the inference speed of the multi-branch architecture designs without reparameterization does not correlate closely with the number of model parameters and GLOPs (see. Table II). For example, the multi-branch architecture designs before reparameterization, such as InceptionNeXt[48] and the proposed AudioRepInceptionNeXt, is slower compared to other models like the multi-stream Slow-Fast model, despite having lower parameters and GFLOPs. We attribute this discrepancy in speed to the increased memory access and synchronization time required by the multi-branch design [35]. Therefore, the reparameterization approach adopted in our work focuses on addressing the issue of optimizing the memory access and synchronization time of multi-branch designs. By optimizing memory access and synchronization time in this way, the reduction in the number of model parameters and GLOPs becomes highly correlated with the increase in inference speed in multi-branch CNN design.

In this work, we use a typical hierarchical architecture design as depicted in Fig. 2 and the details are listed in Table I. The hyperparameters of the model are listed as follows:

• •

  ${S_{i}}$: the stride used in the convolutional layer in the input stem and in the downsampling layers in stage $i$;

• •

  ${K_{i}}$: the kernel size of the convolutional layer in the input stem and in the downsampling in stage $i$;

• •

  ${C_{i}}$: the number of output channels of stage $i$;

• •

  ${E_{i}}$: the channel expansion ratio of inverted bottleneck in stage $i$;

• •

  ${L_{i}}$: the number of blocks in stage $i$;

In this section, we describe the main components of the AudioRepInceptionNeXt block.

We employ the AudioRepInceptionNeXt depicted in Figure 3(a) during the training stage to learn feature representations from audio signals. During the inference stage, we first apply the reparameterization technique to convert the multi-branch AudioRepInceptionNeXt blocks into a single-branch reparametrized block as shown in Figure 3(b).

In this subsection, we describe how to convert the trained multi-branch kernels with varying scales into a single kernel (i.e., $1\times k$ and $k\times 1$) for inference as shown in figure 3(b). Here we use the horizontal $1\times k$ kernels as an example. A similar operation can be applied to the vertical $k\times 1$ kernel. One can see from Figure 3(b) we use three horizontal kernels of size $1\times 21$, $1\times 11$, and $1\times 3$. Specifically, we use $W^{(21)}\in\mathbb{R}^{C_{in}\times C_{out}\times 1\times 21}$ to denote the kernel of a $1\times 21$ convolution layer with $C_{in}$ input channels and $C_{out}$ output channels, $W^{(11)}\in\mathbb{R}^{C_{in}\times C_{out}\times 1\times 11}$ and $W^{(3)}\in\mathbb{R}^{C_{in}\times C_{out}\times 1\times 3}$ for kernel $1\times 11$ and $1\times 3$, respectively. We use $\mu^{(i)}$, $\sigma^{(i)}$, $\alpha^{(i)}$, and $\beta^{(i)}$, where $i\in{3,11,21}$, as the mean, standard deviation, learned scaling factor and bias of the batch normalization (BN) layer following the $1\times 21$, $1\times 11$ and $1\times 3$ convolution layers. Let $F^{(1)}\in\mathbb{R}^{N\times C_{in}\times H_{1}\times W_{1}}$ and $F^{(2)}\in\mathbb{R}^{N\times C_{out}\times H_{2}\times W_{2}}$ be the input and output features of the multi-scale horizontal convolution layers, respectively. $N$, $H$ and $W$ represent the batch size, height and width of the feature map, respectively. We assume $C_{in}=C_{out}$, $H_{1}=H_{2}$, and $W_{1}=W_{2}$ for simplifying the calculation. Before the re-parameterization, the output of the multi-branch horizontal convolution layers can be obtained by using the following equation.

where

Note that the identity branch is ignored in the equation LABEL:eq:before-reparam for simplifying the calculation. In equation LABEL:eq:before-reparam, the $*$ represents the convolution operation. In equation 2, $BN(.)$ and $j$ represent the batch normalization function and output channel index, respectively. $F_{:,j,:,:}$ represents the $j^{th}$ feature map output by the layer preceding the batch normalization layer. Note that the sub-indices of the $BN(.)$ function and F follow the following order: batch size, output channel, height of the feature map, and width of the feature map. We first convert every BN layer and its corresponding horizontal convolution layer into a single convolution layer with a bias term. Let $\bar{W}^{(i)}$ and $\bar{b}^{(i)}$ be the kernel and the bias term after the combination, respectively. The kernel weight and bias can be obtained via the following equation.

Note that the sub-indices of the kernel weight $\bar{W}^{(i)}_{j,:,:,:}$ follow the following order: output channel, input channel, height of the kernel, and width of the kernel. After the combination, the output of each branch can be obtained by the following equation.

After the combination of the BN layer and its corresponding convolution layer, we can obtain three convolutional layers with horizontal kernel and three bias terms. Then we obtain the final $1\times 21$ kernel by adding the $1\times 11$ and $1\times 3$ onto the central point of the $1\times 21$ kernel and the final bias term by adding three bias terms together. The final kernel weight and bias can be obtained by the following equation.

Before adding both the small kernels to the large kernel, we apply zero padding to both small kernels such that they have the same kernel size as $1\times 21$. Note that such transformation requires the $1\times 11$ and $1\times 3$ layer to have the same stride.

Compared to the current state-of-the-art CNN-based Slow-Fast model[1], our new proposed network uses a single-stream architecture instead of two-stream architecture. Our $21\times 1$ and $1\times 21$ large separable kernel can focus on the global-frequency semantic information and long-duration activities inherent in the similar functionality of the Slow model. Meanwhile, the $3\times 1$ and $1\times 3$ separable kernels act as a Fast model that captures the local-frequency semantic information and short-duration activities. The major benefit of using the single-stream network is that the computational complexity and memory footprint can be reduced by 54% and 56%, respectively, compared to the two-stream slow-fast model. In contrast, our network can achieve comparable performance as the slow-fast model. The following section will provide the complexity analysis of the new proposed architecture.

VGG-Sound. VGG-Sound [39] is a large-scale audio dataset extracted from YouTube videos. It contains more than 200K audio clips each with 10 seconds duration sampled at 16KHz. There are a total of 309 classes which include the sound emitted from objects, human actions, and interactions.

EPIC-KITECHENS-100. EPIC-KITECHENS-100 [55] is a large-scale egocentric audio-visual dataset, which captures daily activities in the kitchen. The videos are being recorded in 45 different kitchens and contain 100 hours of data. It includes 90K trimmed action clips and they capture the hand-object activities. The ground-truth labels are formed by a verb and a noun (e.g., move tap, open kettle, and open bin). There are 97 verb classes and 300 noun classes in total. Most of the actions are short-duration (average action length is 2.6 seconds). The audio is sampled at 24kHz.

EPIC-Sound. EPIC-Sound [56] is a large-scale audio event classification dataset that is re-annotated from the EPIC-KITCHENS-100 dataset. Unlike EPIC-KITCHENS-100, the actions in EPIC-Sound can be discriminated purely from the audio, for example, cutting food instead of cutting tomatoes. This dataset includes 75.9k segments of audible events and actions with 44 classes.

Speech Commands V2 (KS2). Speech Commands V2 [57] is an automatic speech commands recognition dataset that contains more than 100K audio clips with 1 second for each of them. It also contains 35 common speech commands for the recognition task.

UrbanSound8K (Urban8K). UrbanSound8K [58] is a urban sound event classification dataset. It contains 8K labeled sound excerpts with less than 4 seconds for each clip and 10 urban sound classes.

NSynth. Nsynth [4] is an audio dataset containing 305,979 musical notes and each of them with a unique pitch, timbre, and envelope. The sounds were collected from 1006 instruments from commercial sample libraries and annotated based on their source, instrument family, and sonic qualities. There are 11 instrument families in total.

We follow the same training strategy as the baseline Slow-Fast model [1], i.e., we pretrain our models on the VGG-SOUND dataset [39] followed by fine-tuning it on the downstream tasks datasets. The input audio signal is first converted into a Log-Mel spectrogram with 128 Mel bands before feeding it to the network. During both the pretraining and fine-tuning stages, we applied the augmentation methods proposed in SpecAugment [17] following the common practice as in [1, 10, 11, 12]. These augmentations include frequency masking, time masking, and time warping. All the models are trained with a batch size of 32 on 4x NVIDIA RTX3090 GPUs during the pretraining and fine-tuning stages.

For the evaluation of the VGG-Sound classification task, we follow the protocol of [39, 2, 1] and report the top-1 accuracy, top-5 accuracy, mean average precision (mAP), area under curve (AUC) and d-prime. For the evaluation of the EPIC-Sounds, KS2, Urban8K, and Nsynth, we report the top-1 and top-5 accuracy. For the EPIC-KITCHENS-100, we follow the evaluation method in [55] and report the top-1 and top-5 accuracy of verb and noun classes. Additionally, we report the top-1 and top-5 accuracy on unseen audio clips in EPIC-KITCHENS-100 to test the generalization ability of the fine-tuned model.

For the measurement of the GPU inference speed (sample/secs), we test all the models on an NVIDIA RTX3090 using a batch size of 32. We first feed 50 batches to warm up the hardware, followed by 50 batches to record the average running time. To measure the CPU inference speed, we conduct the tests on Intel®Xeon®Gold 6226R CPU @ 2.90GHz using a batch size of 1. The tests are performed using a single thread, and we record the average time after the 50 rounds.

To verify the effectiveness of the re-parametrization in our proposed AudioRepInceptionNeXt, we perform the comparison against the state-of-the-art (SOTA) methods. The comparison is performed in terms of the parameter size, GFLOPs, throughput, and top-1 accuracy before and after the re-parametrization. We report the results in Table II. One can see that before reparameterization, AudioRepInceptionNeXt-B1 is 12% and 23% slower than the Slow-Fast [1] and ResNet50 [46] respectively, and 60% faster than RepLKNet [33]. We conjecture the low throughput of AudioRepInceptionNeXt is due to its complicated multi-branch design although it incurs lower parameters and theoretical GFLOPs than SlowFast and ResNet50. As discussed in [35, 34, 33], the large kernel with depthwise convolution increases the memory access costs. Additionally, as mentioned in [35], small operators (i.e., individual convolution (e.g., $1\times 1$) and pooling operations) with multi-branch design are less efficient on GPUs and introduce extra kernel launching and synchronization overhead thus reduces the degree of parallelism on GPU. In our AudioRepInceptionNeXt block, the separable kernels of size $1\times 11$, $1\times 3$, $11\times 1$, and $3\times 1$ are relatively small operators compared to $1\times 21$ and $21\times 1$. To address these issues, we employ the re-parametrization technique, as discussed in Section III-C, that results in a network style similar to ResNet50, as shown in Figure 2b. One can see from Table II that the reparametrized version of AudioRepInectionNeXt achieves $1.28\times$, $1.11\times$, and $3.65\times$ improvement in throughput compared to the Slow-Fast, ResNet50, and RepLKNet respectively while maintaining comparable performance. Notably, the re-parametrization technique does not impact the accuracy, and results in lossless compression of the model. In the rest of the sections, we report the evaluation of the reparametrized version of AudioRepInceptionNeXt.

We compare the performance of AudioRepInceptionNeXt against the CNN-Based baselines that include BCResNet-8 [59], BN-InceptionNet [60], ResNet [46], Slow-Fast model [1], InceptionNeXt [48], RepLKNet [33] and 2D AudioRepInceptionNeXt(without any separable convolution kernel and with branch configurations of $21\times 21$, $11\times 11$ and $3\times 3$ kernels) as shown in Figure 2b (Right). To ensure a fair comparison, we downscaled the original RepLKNet-31B model, with 79M parameters, to the RepLKNet-31T model with 29.3M parameters. This downsizing involved reducing the channel size to 64, 128, 320, and 512 for model stages 1 to 4, respectively. All the evaluation is performed on the VGG-Sound event classification dataset. We report the results in Table II.

Our key findings include: First, the proposed AudioRepInceptionNeXt demonstrates a remarkable reduction in the number of parameters and theoretical GFLOPs while achieving comparable or superior performance compared to other CNN-based methods. For instance, when compared to the multi-stream Slow-Fast model, AudioRepInceptionNeXt-B1 achieves a 56% reduction in parameters and 54% reduction in theoretical GLOPs, with only a slight performance drop of 0.28%. Similarly, in comparison to multi-branch large kernel InceptionNeXt-Tiny, our proposed model achieves a 52% reduction in parameters and a 53% reduction in GFLOPs while attaining a higher accuracy of 1.8%. Furthermore, when compared to 2D large kernel-based RepLKNet-31T, our model achieves comparable accuracy with a difference of only 0.26% while saving 71.05% of parameters and 68% of GFLOPs. Additionally, compared to the 2D version of the proposed AudioRepInceptionNeXt-B1, our model achieves a 0.7% higher accuracy while saving 11% of model parameters and 22% of GFLOPs. This result is consistent with the findings in [1, 32], demonstrating that the model with separable kernels performs better by enabling the extraction of temporal and frequency information independently. Second, our model demonstrates the fastest inference speed on GPU compared to other CNN-based baselines, aligning with the theoretical GFLOPs. The proposed AudioRepInceptionNeXt-B0 obtains 2245 fps (vs. 2031 fps and 492 fps obtained by BN-InceptionNetV1 and BCResNet-8, respectively), while AudioRepIncetpionNeXt-B1 obtains 1019 fps (vs.796 fps obtained by Slow-Fast). Third, the proposed AudioRepInceptionNeXt-B1 achieves a higher inference speed on the CPU compared to the Slow-Fast model, RepLKNet-31T, AudioRepInceptionNeXt-B1 2D, and BN-InceptionNetV1, while maintaining a comparable performance with Slow-Fast and RepLKNet-31T. We also note that the CPU inference speed of AudioRepInceptionNeXt-B1 is comparable to ResNet50 and InceptionNeXt-Tiny while surpassing InceptionNeXt-Tiny in terms of accuracy and achieving competitive accuracy compared to ResNet50. When comparing the inference speeds of ResNet50 and AudioRepInceptionNeXt-B1 on CPU and GPU, we conjecture that the difference in the inference speed among these models can be attributed to the optimized implementation of depthwise convolution on GPU compared to CPU, as discussed in [61]. Additionally, the disparity in arithmetic intensity (the ratio of compute to memory operations), as mentioned in Section IV-E, also contributes to the observed variations in speed on GPU and CPU. These findings highlight the superior performance of AudioRepInceptionNeXt in terms of parameter efficiency, theoretical GFLOPs, and inference speed compared to other CNN-based baselines.

To verify the conclusion from Section IV-F, we conduct transfer learning experiments on multiple datasets: Speech Command V2 [57], UrbanSound8K [62], EPIC-KITCHEN-100[55], NSynth [4] and EPIC-Sound [56]. As shown in Table III and Table IV, our proposed model achieves comparable performance to the multi-stream Slow-Fast model in terms of accuracy on four transfer learning datasets, while saving 56% and 54% of parameters and GFLOPs, respectively. Moreover, when compared to the Urban8K dataset our model outperforms the Slow-Fast model by 1.61% in terms of top-1 accuracy. Additionally, when compared to other CNN-based methods, our methods achieve comparable or superior performance in terms of accuracy, while having the lowest parameters count and computational GFLOPs. These results indicate that our model can learn the rich representations that are applicable to different domains and are robust when transferred to various audio understanding tasks. These findings are also aligned with the results obtained from the pretraining on the VGG Sound dataset, as discussed in section IV-F.

To evaluate the inference speed on the mobile device, we deploy all models on an Android mobile platform. The implementation procedure is summarized in Fig.4. We first convert the PyTorch models to the Open Neural Network Exchange (ONNX) format [63]. It is an open-source machine-independent format compatible with different hardware, drivers, and operating systems. We then utilize the Android development tools, specifically Android Studio and Kotlin, to build an application interface for conducting the speed evaluation. To run the ONNX models on Android mobile devices, we leverage the ONNX Runtime mobile package. It provides a lightweight and optimized runtime specifically designed for mobile platforms. During the evaluation, we run all models on Redmi Note 9 Pro smartphone with Snapdragon 720G SoC with floating point 32. Specifically, we feed an input with a batch size of 1 and record the average inference time by processing 50 batches.

As shown in Table V, our model demonstrates superior performance in terms of inference speed and model size on mobile devices compared to other state-of-the-art CNNs. Notably, the AudioRepInceptionNeXt-B1 model outperforms the classical ResNet50 by reducing the inference time and model size by 25% and 50%, respectively. In comparison to the recent multi-branch large kernel InceptionNeXt-Tiny, our model achieves a 35% faster inference speed and 51% smaller model size. When compared to RepLKNet-31T, a 2D large kernel-based model, our proposed model exhibits substantial improvements, saving 89% of the inference time and reducing the model size by 60%. Moreover, our method surpasses the multi-stream Slow-Fast model, achieving 27% lower inference time and 56% smaller model size. The results clearly demonstrate the efficiency of our model in terms of both inference speed and model size on mobile devices, showcasing its superiority over existing state-of-the-art CNNs.