Sampling-Frequency-Independent Audio Source Separation Using Convolution Layer Based on Impulse Invariant Method ^††thanks: This work was supported by JSPS KAKENHI Grant Number JP20K19818.

Conv-TasNet is a recent time-domain DNN for audio source separation that works well for speech [3] and music source separation [17, 5]. Since the architectures of Conv-TasNet are slightly different in those three papers, we adopt the Conv-TasNet architecture for music source separation defined in [17], as illustrated in Fig. 1(a). Conv-TasNet consists of a pair of an encoder and a decoder and $C$ source-specific masking modules, where $C$ denotes the number of sources. The encoder and decoder imitate a traditional time-frequency transform (e.g., the short-time Fourier transform) and its inverse transform. The encoder transforms a monaural time-domain signal into an $N$-channel latent representation by a one-dimensional (1D) convolution layer (with kernel size $L$ and stride $W$) and the rectified linear unit (ReLU). Each masking module estimates a mask for the target source from the latent representation. It comprises $R$ convolution blocks, each of which consists of $X$ 1D dilated convolution layers with an exponentially increasing dilation factor. The details of the convolution block are shown in [3]. The decoder converts the masked latent representations into the separated time-domain signals by a 1D transposed convolution layer with kernel size $L$ and stride $W$.

In [7], the multi-phase gammatone filter (MP-GTF) was introduced to design the weights of the convolution layer of the encoder of Conv-TasNet, which improved the speech separation performance. The impulse response of the MP-GTF is given by

where $a$ denotes the amplitude, $p$ the filter order, $b$ the bandwidth, $f$ the center frequency, and $\phi$ the phase shift. The parameter $b$ is given by $b=\mathrm{ERB}(f)/1.57$, where $\mathrm{ERB}(f)=24.7+f/9.265$. By sampling $L$ points from $g^{\mathrm{(MP\text{-}GTF)}}(t)$ for various $f$ and $\phi$, we obtain $N$ discrete-time impulse responses of length $L$ and concatenate them along the channel axis to form the weights as a $1\times N\times L$ tensor. The convolution layer of the encoder is followed by the ReLU nonlinearity, which blocks the negative values of the MP-GTF output and discards the information of the input signal. To avoid a lack of information, $g^{\mathrm{(MP\text{-}GTF)}}(t)$ is used together with its phase-reversed version, i.e., with the phase shift $\phi+\pi$ [7].

There exist a few methods of training DNNs using audio signals sampled at multiple sampling frequencies [13, 14, 15, 17]. In [13, 14], an automatic speech recognition (ASR) model was trained using audio signals sampled at $8$ and $16$ kHz, where the part of input features corresponding to the missing frequency band was padded by zeros. In [15], to compensate the missing frequency band, an ASR model was jointly trained with a bandwidth expansion model. The music source separation model presented in [17] was constructed by stacking three Conv-TasNets that account for sampling frequencies of $8$, $16$, and $32$ kHz. The Conv-TasNets for $16$ and $32$ kHz estimate the source signals of the target sampling frequencies, referring to the masked latent representations obtained with the lower sampling frequencies.

While these training methods are valid for the trained sampling frequencies, they are not guaranteed to work with unseen sampling frequencies. In contrast, we explicitly define an SFI structure to handle any sampling frequency without retraining as shown later in Section III.

To realize an SFI network, we introduce latent analog filters and analog-to-digital filter conversion into a convolution layer. By interpreting the weights of a convolution layer as a collection of time-reversed digital FIR filters, we consider them from a signal processing viewpoint. The digital filters are inherently sampling frequency dependent, whereas analog filters are SFI owing to their definition in the continuous time domain. Focusing on this fact, we introduce latent analog filters behind a convolution layer so that its weights can be adjusted using the sampling frequency of an input signal.

As shown in Fig. 1(c), the proposed layer consists of the usual 1D convolution layer and impulse responses of $M^{\mathrm{(in)}}M^{\mathrm{(out)}}$ analog filters defined in the continuous time domain, where $M^{\mathrm{(in)}}$ and $M^{\mathrm{(out)}}$ are the input and output channel sizes, respectively. The generating process of the proposed layer consists of three steps. Given the sampling frequency of an input signal, the proposed layer (i) generates a discrete-time impulse response of length $L$ from each analog filter, (ii) stacks the time-reversed versions of these discrete-time impulse responses to form the weights as an $M^{\mathrm{(in)}}\times M^{\mathrm{(out)}}\times L$ tensor, and (iii) works as the usual convolution layer using them. Since steps (i) and (ii) depend only on the sampling frequency and the continuous-time impulse responses, we only need to perform them once (before the features are input) whenever the sampling frequency changes.

For step (i), we employ the impulse invariant method to generate digital FIR filters from their analog counterparts. Note that while this method is originally for the design of infinite impulse response filters, we can use it for the digital FIR filter design. Let us denote the sampling period as $T$, a discrete time index as $l=1,\cdots,L$, and the continuous time as $t\in\mathbb{R}$. The impulse invariant method generates a discrete-time impulse response $h[l]$ from an analog filter $g(t)$ so that the sampled instants coincide:

Changing $T$ yields an impulse response for different sampling frequencies $1/T$. By stacking the generated impulse responses, the weights for the convolution layer is obtained in step (ii). Similarly, an SFI version of a transposed convolution layer (SFI transposed convolution layer) is given by changing the convolution layer in the SFI convolution layer to the transposed convolution layer.

For the analog filter $g(t)$, we can use the MP-GTF in Eq. (1). The continuous-time impulse responses can be different for each channel, and hence, hereafter, a channel subscript $m$ is added to $g^{\mathrm{(MP\text{-}GTF)}}(t)$ and its parameters: $g^{\mathrm{(MP\text{-}GTF)}}_{m}(t),a_{m},p_{m},b_{m},f_{m}$, and $\phi_{m}$. Whereas all parameters of $g^{\mathrm{(MP\text{-}GTF)}}_{m}(t)$ were fixed in [7], we propose to train $f_{m}$ and $\phi_{m}$ jointly with the other DNN components by the commonly-used backpropagation algorithm.

The gradient of $h[l]$ can be computed in the same manner as the usual convolution layer. Since the gradient of $h[l]$ equals that of $g(lT)$ owing to Eq. (2) and $g^{\mathrm{(MP\text{-}GTF)}}_{m}(t)$ is differentiable with $f_{m}$ and $\phi_{m}$, the gradients of the trainable parameters of $g^{\mathrm{(MP\text{-}GTF)}}_{m}(t)$ can be computed by the chain rule. These computations can be easily implemented by defining the forward computation process of the proposed layer owing to the automatic differentiation mechanism installed in modern deep learning frameworks (e.g., PyTorch and TensorFlow).

Since the impulse invariant method simply performs sampling to an analog filter, aliasing occurs in the obtained digital filters. As reported in [18, 19, 6], aliasing causes degradation of the DNN performance, and thus we introduce an aliasing reduction technique. Since the energy of aliased components is distributed above the Nyquist frequency, we propose to set the weights of the $m$th channel to zero whenever the center frequency $f_{m}$ of the corresponding analog filter is above the Nyquist frequency. This aliasing reduction technique is important for the proposed layer when it is utilized with low sampling frequencies, as shown later in Section IV.

As shown in Fig. 1(b), we build an SFI audio source separation model by incorporating the proposed layers into Conv-TasNet [3]. The convolution layer of the encoder and the transposed convolution layer of the decoder are respectively replaced with the SFI convolution and transposed convolution layers. The masking modules are the same as in [3].

For our model, we should modify the kernel size $L$ and stride $W$ in accordance with the sampling frequency during inference. As described in Section II-A, the encoder and decoder can be interpreted as the time-frequency transform and its inverse transform. With this interpretation, $L$ and $W$ correspond to the frame length and the frame shift, respectively. Hence, as the sampling frequency doubles, $L$ and $W$ should double to make the representation consistent for the masking module. For this reason, we determine $L$ and $W$ for each target sampling frequency to keep the frame length and shift unchanged in the continuous time domain. By replacing all convolution layers in the masking modules with the SFI convolution layers, this issue might be resolved. However, we left it as a future work because additional care regarding the combination with other layers (e.g., group normalization [20]) must be considered.

To evaluate the efficacy of the proposed method, we conducted music source separation on the MUSDB18-HQ dataset [21], which consists of $86$ training, $14$ validation, and $50$ test tracks. Each track contains separate recordings of four musical instruments (vocals, bass, drums, and other), i.e., $C=4$. The training and validation tracks were down-sampled to $16$ kHz, and we created the test data by down- and up-sampling the test tracks to several target sampling frequencies, $F_{s}^{\mathrm{(target)}}=8,12,\ldots,48$ kHz. As an evaluation metric, we used the median signal-to-distortion ratios (SDRs) computed with the BSSEval v4 toolkit [22].

We used the same data augmentation techniques as those of [17]: random cropping of the $8$ s training audio segments, random amplification within $[0.75,1.25]$, random selection of the left or the right channel, and random intertrack shuffling of the instruments in half of the minibatch. We also applied standardization (zero mean and unit variance) to the tracks.

We compared the proposed model (Proposed) and that using the aliasing reduction technique (Proposed+) with Conv-TasNet and its variant (T-MP-GTF) whose encoder and decoder instead use the trainable extension of MP-GTF as their weights of the convolution and transposed convolution layers, respectively. T-MP-GTF was included to separately evaluate the trainable extension of the MP-GTF and the sampling frequency adaptation. We applied all models to the audio signals of the unseen sampling frequencies without resampling them at the trained sampling frequency in order to examine the effects of the sampling frequency mismatch and the proposed sampling frequency adaptation. Table I summarizes the features of these models. For Proposed and Proposed+, we determined $L$ and $W$ to be $5.0$ and $2.5$ ms at the sampling frequency of $16$ kHz, respectively, as described in Section III-C, whereas we set $L=80$ and $W=40$ for the other models. For all models, we set $X=6$ and $R=2$. The hyperparameters of the masking modules are shown in Table II, where the symbols correspond to those used in the literature of Conv-TasNet (see Table 1 in [3]).

For $g_{m}^{\mathrm{(MP\text{-}GTF)}}(t)$, we trained $f_{m}$ and $\phi_{m}$ for $m=1,\ldots,220$ jointly with the entire network, and constrained these parameters for the other $m$s so that $f_{m+220}=f_{m}$ and $\phi_{m+220}=\phi_{m}+\pi$, as described in Section III-A. We initialized $f_{m}$ and $\phi_{m}$ as in [7]; let us denote $48$ frequencies distributed uniformly in the equivalent rectangular bandwidth (ERB) scale [23] from $50$ to $8000$ Hz by $f^{\mathrm{(center)}}_{i}$, where $i=1,\cdots,48$ is the center frequency index and $f_{i}<f_{i+1}$ for all $i$. We initialized $f_{m}$ as $f_{m}=f^{\mathrm{(center)}}_{\lfloor m/K\rfloor+1}$ for $m=1,\ldots,140$ ($m=141,\ldots,220$), where $K=5$ ($K=4$, respectively). The phase shifts $\phi_{m}$ of the filters with the same $f^{\mathrm{(center)}}_{i}$ were initialized to be uniformly distributed in $[0,\pi)$. The other parameters were set as $a_{m}=1$, $p_{m}=2$. As in [7], these filters were normalized so that they have the same $l^{2}$ norm.

For training, we used the RAdam optimizer [24] with a weight decay rate of $5.0\times 10^{-4}$ and the Lookahead mechanism [25] with a synchronization period of $6$ and a slow weights step of $0.5$. Gradient clipping with the maximum $L_{2}$-norm of $5$ was applied. The learning rate scheduler presented in [26] was employed with an initial learning rate of $1.0\times 10^{-3}$ and a restart period of $200\,000$ iterations. We trained each model with a batch size of $12$ for $250$ epochs, using the negative scale-invariant source-to-noise ratio as the loss function, and selected the model with the lowest validation loss. We applied the trained models to the left and right channels of the test tracks separately, and scaled the source estimates using instrument-wise factors to minimize the mean squared error between the input mixture and the sum of all instrument estimates, resulting from the scale invariance of the loss function [17].

Fig. 2 shows the separation performances for the test data with sampling frequencies from $8$ to $48$ kHz. At the trained sampling frequency, $16$ kHz, the proposed models including T-MP-GTF achieved higher SDRs than Conv-TasNet for all instruments, showing the effectiveness of the proposed trainable extension of the MP-GTF. Interestingly, the proposed models gave much lower standard errors than Conv-TasNet, which was not reported in [7]. This observation reveals that the use of the trainable MP-GTF makes Conv-TasNet robust to the initialization of the DNN parameters.

As the sampling frequency moved away from $16$ kHz, the SDRs of Conv-TasNet and T-MP-GTF greatly decreased and they failed to separate the sources. By contrast, the proposed models with the sampling frequency adaptation, Proposed and Proposed+, provided similar SDRs for the $12$- to $48$-kHz-sampled data and outperformed the other models by a large margin, particularly at $20$ kHz and higher, even though they were trained only with the $16$-kHz-sampled data. This result clearly shows that the proposed sampling frequency adaptation plays a crucial role in achieving the consistent performance.

Fig. 3 shows the magnitudes of the frequency responses of the filters of the trained SFI convolution layer at sampling frequencies of $8,16,$ and $32$ kHz. We can confirm that the filters of the $16$ and $32$ kHz sampling frequencies exhibited consistent frequency responses. Importantly, the filters of the $32$ kHz sampling frequency blocked the frequency components higher than around $8$ kHz. Nevertheless, the proposed model achieved a consistent performance at the sampling frequencies higher than $8$ kHz, which should be because the dominant frequency components of the music signals were distributed below $8$ kHz.

For the sampling frequency of $8$ kHz, aliasing occurred from $4$ kHz (see Fig. 3(a)). This resulted in the performance degradation of Proposed (the proposed method without the aliasing reduction technique). By contrast, Proposed+ showed a consistent performance at all sampling frequencies, demonstrating the effectiveness of the proposed aliasing reduction technique when the sampling frequency is reduced. For drums and other, Proposed+ gave slightly lower SDRs than Proposed at the sampling frequency of $12$ kHz, which might be because the filters with center frequencies near the Nyquist frequency are helpful for the separation. A further investigation of this observation remains as a future work.