A Novel Audio Representation using Space Filling Curves

A RSFC is built recursively, where $C_{k+1}$ is obtained by subdividing the cell $(i,j)$ of the curve $C_{k}$ into $b^{2}$ cells and by modifying the curve $C_{k}$ according to a set of rules. A large number of RSFCs exist. In this paper, we focus on the following RSFCs: the Hilbert curve [10], the Z curve (also known as the Z-order [15]), the Gray curve [16], the H curve [17], and a curve, that we will call OptR, proposed by Asano et al. [18]. A representation of these curves with $k=3$ can be found in Figure 1. Consecutive points $C_{k}(i),C_{k}(i+1)$ are connected by a line. Dotted lines represent jumps.

The Hilbert curve shows good performance in locality preservation, which is desirable in the context of audio samples: data points close in time should be close in the image representation in order to exploit the local nature of the convolution layers. Depending on the locality metric, the Hilbert curve outperforms the Z curve and the Gray curve [19], but is also inferior to the Z curve [20]. Nonetheless, the Z curve and the Gray curve include jumps between consecutive indices, which don’t preserve locality. We included both of them to study the impact of this behavior on models.

Niedermeier et al. [17] have shown that the H curve almost reaches the optimal lower bound among closed cyclic curves (i.e. $\|C_{k}(i)-C_{k}(i+1)\|_{\infty}=1$, $\forall i$). In particular, they have shown that $\|C_{k}(i)-C_{k}(j)\|_{p}$, $p=1,2,\infty$ roughly behaves like $\sqrt{a|i-j|+b}$ in the worst case. This bound is smaller than $|i-j|$, which is the lower bound in the one-dimensional case achieved by the identity mapping. It shows the benefit from using two-dimensional inputs over one-dimensional inputs, as the distance between indices is reduced. Note that the OptR curve is particularly complex, which might lead to poor performance, since regular patterns of the curve are harder to identify.

This category groups all curves that cannot be built recursively. We investigate three additional curves: the Scan curve, the Sweep curve, and the Diagonal curve [21]. Figure 2 shows their representation when $k=3$. The Sweep curve is a two-dimensional ordering scheme, where the sequence is cut into equally long intervals and stacked together to obtain an image. The Scan curve is obtained by reversing one interval out of two, which removes all jumps. Finally, the Diagonal curve is a 45 degrees rotated version of the Scan curve restricted to the $N_{k}\times N_{k}$ grid interior. These curves do not have particular locality preservation properties apart from being continuous mappings, i.e. $\|C_{k}(i)-C_{k}(i+1)\|_{\infty}=1$ (except for Sweep). Moreover, $\|C_{k}(i)-C_{k}(j)\|_{\infty}$ scales like $|i-j|$ in the worst case, which is much larger than $\sqrt{|i-j|}$.

We distinguish two pre-processing steps: functions $f\in\mathcal{P}$ that are applied on the raw audio signal and transformations $g\in\mathcal{T}$ that are performed on images. Assuming that the SFC mapping could be implemented in hardware, transformations in $\mathcal{T}$ should be preferred over functions in $\mathcal{P}$.

We might first center audio recordings in the middle of the frame. The procedure consists of first computing the weighted average energy in separate windows of size $w$ using a Gaussian kernel with standard deviation $\sigma$, and aligning the signal that is above the threshold $th$ with the middle of the output sequence. Note that $\texttt{center}\in\mathcal{P}$.

We might also use two additional data augmentations: mixup [22] and shift. Mixup encourages the model to behave linearly in-between two training examples by taking random convex combinations. This operation is parametrized by $\lambda$, which follows a Beta distribution $\lambda\sim\mathcal{B}(\alpha,\alpha)$, where $\alpha>0$ is some hyperparameter. In the audio setting, this method can be applied either to the raw audio waveform or to the image representation. Since the SFC mapping is a linear operator, both approaches are equivalent. We choose to apply it to the image (i.e. $\texttt{mixup}\in\mathcal{T}$). Shift consists of applying random shifts $s_{u}$ from a uniform distribution $s_{u}\sim\mathcal{U}(-L/4,L/4)$, where $L$ is the input length, to reduce the location dependency. This operation requires that the input is already centered in order to lose less information.

We used MFCC, one of the most common audio representations, as a baseline in this study¹¹1Code available at https://github.com/amari97/sfc-audio. We focused on keyword spotting using two different datasets: Google Speech Commands V2 [23] and a dataset based on the 1000 most frequent words of LibriSpeech’s “clean” split [24], for which Beckmann et al. [25] have provided aligned labels²²2Available at https://github.com/bepierre/SpeechVGG. The Speech Commands dataset contains 105829 words classified into 35 classes, where 84843, 9981, 11005 of them were respectively used for training, validation, and test. Among the 1000 most frequent words in LibriSpeech, we have selected 31 words that were appearing between 4000 and 8000 times in order to have a balanced dataset and spoken words of similar duration. We ended up with 164674, 2519, 2400 training/validation/test examples. All audio recordings were sampled at 16 kHz and had a duration of one second.

To properly compare both approaches, chosen models should process inputs of different sizes without changing their structure. This was achieved by introducing an average pooling layer with an area equal to the input size. Several CV networks using this layer were tested: the small MobileNetV3 network (Mo) [26], ShuffleNetV2 (Shuffle) [27], SqueezeNet (Squeeze) [28], MixNet (Mix) [29] and EfficientNet B0 (Eff) [30]. Additionally, the representations were evaluated using Res8 [31], a small network that was specifically designed for keyword spotting tasks with MFCCs. Few modification were needed since the models had different input requirements. Since Mo accepts three channel images (RGB), we added a pointwise convolution to expand the input, followed by a ReLU activation layer and BatchNorm normalization layer [32]. Our implementation of Squeeze added BatchNorms after the squeezing layer and the classifier was replaced by a fully connected layer.³³3We use the version 1.1 as described in the official code https://github.com/forresti/SqueezeNet. Finally, the pooling area of Res8 was extended to $4\times 4$ to increase the RF, and the dilation of the convolutions was set to 1,1,1,2,2,2, following the increasing dilation strategy of the Res15 network [31]. As a result, the RF of Res8 was changed from 54 to 78.

Unless specified otherwise, all models were trained with stochastic gradient descent (SGD) with learning rate 0.5 and batch size 256 on 2 GPUs with distributed data parallel. We used an early stopping rule with 50 waiting epochs and we set to 300 the maximal number of epochs. We set the mixup hyperparameter $\alpha=0.2$, as in [22]. We set $th=0.0001$, such that only signals that were 1.6 higher than a flat audio signal exceeded it. We also set the Gaussian kernel parameters to $w=100$ and $\sigma=25$, such that boundary values had a weight close to zero. We used 40 MFCCs that were computed on frames of 0.025 seconds with overlaps of 0.015 seconds [31, 33]. Finally, we used curves of order $k=7$ to represent one second of audio, since $16000<16384=2^{7}\times 2^{7}$.

Curve comparison As shown in Table 1, the data augmentations improved the accuracy of the model, even if this effect was slightly less evident in the case of LibriSpeech. The benefits of these augmentations were larger for SFCs: the generated images were very different, improving the selection of invariant features. The Z curve yielded the best results in all situations and reached the baseline performance with Data Aug. Non-recursive curves performed equally well, but were inferior to Z and H. Finally, OptR was the worst one.

Ablation study As shown in Table 2, centering the audio clip inside the one second frame improved the accuracy for almost all curves. In particular, the improvement was larger for recursive curves. The Shift step provided a large gain by forcing the model to select features that are shift invariant in time. Mixup still improved the previous accuracy.

Model comparison Table 3 gives the results of the comparison between the Z curve and the MFCC approach. Except for Res8, there was no difference with the baseline. The best results were unexpectedly obtained with the largest network (Eff), but Res8 achieved impressive results in terms of efficiency and accuracy with MFCCs.

Receptive field influence Figure 3 shows the average output probability $\bar{p}(s)$ of the true class as a function of time shifts $s$. The figure revealed that $\bar{p}(s)$ was lower when the audio signal was centered ($s=0$), and when the RF was much smaller than the size of the input image ($78<128$). This effect was alleviated when increasing the RF to 125 by setting the dilation of the Res8 filters to $1,1,2,2,4,4$. In this case, the model accuracy reached 86.5.

Number of parameters Figure 4 shows the model accuracy after changing the number of parameters by shrinking or expanding the width of the network (i.e. the number of channels of each convolution layer) by a factor width_mult. The large gap with the baseline persisted on Res8. Moreover, the difference remained similar when reducing the number of parameters. Due to its prohibitively large computational cost, the standard deviations were only computed for Mo using cross-validation, but they showed that the gap with the baseline was not significant.

Curve Comparison on Res8 Table 4 shows the performance of each curve with Res8. Non-recursive curves yielded the best results, while the $Z$ curve outperformed the other recursive curves. The gap with the baseline was significant.

Despite having jumps between consecutive indices, which could harm the model performance (see Figure 3), the Z curve showed the best performance among other SFCs. Our intuition is that the Z curve guarantees a rather simple relation between the features of the hidden layers, when randomly shifting the input in time. In particular, the Z curve exhibits a shift equivariance under convolution operations due to its regular patterns (Z shape) that are always oriented in the same direction (Figure 1). More precisely,

On the other hand, curves like the Hilbert curve (Figure 1) include some rotation of its elementary block, which destroys the equivariance property. This intuition is further assessed by the fact that the Z curve performed well without imposing some shift invariance with data augmentations (Table 1), which shows that the model has already built a coherent feature extraction.

Table 3 shows that the SFC approach was competitive with the baseline when trained with Data Aug., except for the Res8 network, for which the large jump in the middle of the Z mapping had an influence on the accuracy (Figure 3). Increasing the RF of the network improved the accuracy from 85.3 to 86.5, but remained substantially smaller than the baseline ($86.5\ll 94.0$). The number of parameters was not sufficient to justify the discrepancy observed in Table 3, since the gap persisted on Figure 4. It might be related to the network architecture, which was specifically designed for MFCC inputs and which contained a small number of layers. Indeed Res8 worked best with non-recursive curves (Table 4), which have an image structure similar to the MFCC representation: the image can be decomposed into two perpendicular axes — a time axis and a feature axis — which is not possible for the recursive curves (see Figure 1).