We propose two different kinds of structural modification to reduce the model's overall complexity compared to other NWC models including our prior works [25, 26].

First, both encoder and decoder adopt gated linear units (GLU) [30]. We also define the GLU's convolution with dilation [31] to expand the receptive field in the time domain, which is a scheme that showed promising performance in speech enhancement [32]. Fig. 1 (b) shows our dilated GLU module. It first reduces the channel from 100 to 20 using a unit-width kernel. Then, two separate dilated convolution layers are applied to produce two feature maps, one of which goes through a sigmoid activation. Hence, the Hadamard product of the two feature maps can be seen as a gated version of the linear feature map. It is known that this gating mechanism in the middle boosts the gradient flow thanks to the linear path that does not involve the gradient vanishing issue. The final result is a mixture of the input and the last feature map, turning this block into a residual learning function as proposed in ResNet [29]. This kind of architecture shows superior performance as evidenced in [30, 33].

Our encoder reduces the data rate by using a ``downsampling" convolutional layer, whose ``stride" parameter is set to be 2. As a counterpart, the decoder's ``upsampling" layer makes up the loss. More details about this down and upsampling operations will be discussed in Sec. II-B. In this subsection, we focus on the actual module that performs the upmixing, where we introduced additional reduction in complexity. Out of various choices, we employ the depthwise separable convolution [34] to further save the computational cost (Fig. 1 (c)). For example, to transform a feature map of size $256\times 100$ (features, channels) into its upsampled version of size $512\times 50$, we first perform depthwise convolution using a $c\times 1$ kernel. In our system $c=9$. Since the depthwise convolution applies to each channel separately, we eventually need $100$ such kernels. It is a reduction of model complexity, because a normal convolution requires a kernel of size $c\times 100\times 100$ (features, input channels, output channels), which is 100 time larger. In depthwise separable convolution, another $1\times 1$ convolution follows to add more nonlinearity, for which we need a $1\times 100\times 100$ kernel. In our case, it is easy to show that $c\times 100\times 100>c\times 100+100\times 100$ when the integer $c>1$. For example, when $c=9$, it is a reduction of about 88% of parameters.

Likewise, the proposed NWC is lightweight with only 0.35 million parameters, which is a reduction of 0.1 million parameters compared to our previous works [25][26]. The reduction comes from the streamlined upsampling operation implemented via the depthwise separable convolution. Eventually, the decoder accounts for 0.12 million parameters out of 0.35.

One way to compress the input signal in the proposed encoder architecture is to reduce the data rate. The CNN encoder function takes an input frame, $\bm{x}\in\mathbb{R}^{T}$, and converts it into a feature map $\bm{h}\in\mathbb{R}^{N}$,

which then goes through quantization, transmission, and decoding to recover the input as shown in Fig. 1 (a). During the encoding process, we introduce a downsampling operation, reducing the dimension of the code vector $\bm{h}$. We employ a dedicated downsampling layer by setting up the stride value to be $2$ during its convolution, reducing the data rate by $50\%$, i.e., $N=T/2$. Accordingly, the decoder needs a corresponding upsampling operation to recover the original sampling rate. We use subpixel CNN layer proposed in [35] to recover the original sampling rate. Concretely, the subpixel upsampling involves a feature transformation implemented in depthwise convolution, and a shuffle operation that interlaces features from two channels into a single channel, as shown in Eq. (2), where the input feature of the shuffle operation is shaped as ($N$, $2$) and the output is shaped as ($2N$, $1$).

The dimension-reduced feature map can be further compressed via bit depth reduction. Hence, the floating-point code $\bm{h}$ goes through quantization and entropy coding, which will finalize the bitrate based on the entropy of the code value distribution. Typically, a bit depth reduction procedure lowers the average amount of bits to represent each sample. In our case, we could employ a quantization process that assigns the output of the encoder to one of the pre-defined quantization bins. If there are $2^{5}=32$ quantization bins, for example, a single-precision floating-point value's bit depth reduces from 32 to 5. In addition, various entropy coding techniques, such as Huffman coding, can be further employed to losslessly reduce the bit depth. While the quantization could be done in a traditional way, e.g., using Lloyds-Max quantization [36] after the neural codec is fully trained, we encompass the quantization step as a trainable part of the neural network as proposed in [37]. Consequently, we expect that the codec is aware of the quantization error, which the training procedure tries to reduce it. It is also convenient to control the bitrate by controlling the entropy of the code value distribution, which can be also done as a part of network training.

In NWC, the quantization process is represented as classification on each scalar value of the encoder output. Given a vector with $K$ centroids, $\boldsymbol{\beta}=[\beta_{1},\beta_{2},\cdots,\beta_{K}]^{\top}$, the quantizer's goal is to assign each feature $h_{n}$ to the closest centroid in terms of $\ell_{2}$ distance, which is defined as follows:

where $n$-th row in $\bm{D}$ is a vector of $\ell_{2}$ distance between $n$-th code value $h_{n}$ to all $K$ quantization bins. Then, we employ the softmax function to turn each row of $\bm{D}$ into a $K$-dimensional probabilistic assignment vector:

where we turn the distance into a similarity metric by multiplying a negative number $-\alpha$, such that the shortest distance is converted to the largest probability. In our implementation, $\boldsymbol{\beta}$ is initialized as a vector of $K=32$ uniformly spaced numbers within the interval of $[-1,1]$. As for $\alpha$, we begin with a large enough number $300$. Both $\alpha$ and $\beta$ are trainable parameters to optimize the quantization process.

Note that Eq. (4) yields a soft assignment matrix $\bm{A}^{\text{(soft)}}\in\mathbb{R}^{N\times K}$. In practice, though, the quantization process must perform a hard assignment, so each code value $h_{n}$ is replaced by an integer index to the closest centroids: $z_{n}\in\{1,2,\cdots,K\}$, which is represented by $\lceil\log_{2}K\rceil$ bits as the quantization result. The hard kernel assignment matrix $\bm{A}^{\text{(hard)}}$, where each row is a one-hot vector, can be induced by turning on the maximum element of $\bm{A}^{\text{(soft)}}$ while suppressing the non-maximum:

On the decoder side, $\tilde{\bm{h}}=\bm{A}^{\text{(hard)}}\bm{\beta}$ recovers $\bm{h}$.

Since $\arg\max$ operation in Eq. (5) is not differentiable, a soft-to-hard scheme is proposed in [37], where $\bm{A}^{\text{(hard)}}$ is used only at test time. During backpropagation for training, the soft classification mode is enabled with $\bm{A}^{\text{(soft)}}$ so as not to block the gradient flow. In other words, $\hat{\bm{h}}=\bm{A}^{\text{(soft)}}\bm{\beta}$ represents each encoder output with a linear combination of all quantization bins. The process is summarized in Algorithm 1. Although this soft quantization process is differentiable and desirable during training, the discrepancy between $\bm{A}^{\text{(soft)}}$ and $\bm{A}^{\text{(hard)}}$ creates higher error during the test time, requiring a mechanism to reduce the discrepancy as in the following section.

LPC has been widely used to facilitate speech compression and sysnthesis, where source-filter model ``explains out" the envelope of a speech spectrum, leaving a low-entropy residual signal [39]. Similarly, LPC serves as a pre-processor in our system before its residual signal being compressed by NWC as we will see in Sec. III-B. In this subsection, we redesign the LPC coefficient quantization process as a trainable module. We introduce collaborative quantization (CQ) to jointly optimize the LPC analyzer and NWCs as a residual coder.

To achieve scalable coding performance towards transparency at high bitrates, we propose cross-module residual learning (CMRL) to conduct bit allocation among multiple neural codecs in a cascaded manner. CMRL can be regarded as a natural extension of what is described in Sec. III-A, where the LPC as a codec conducts the first round of coding by only modeling the spectral envelope. It leaves the residual signal for a subsequent NWC to be further compressed. With CMRL, we employ the concept of residual coding to cascade more NWCs. We also present a dual-phase training scheme to effectively train the CMRL model.

CMRL's scalability comes from its residual coding concept that enables a concatenation of multiple autoencoding modules. We define the residual signal recursively: $i$-th codec takes the residual of its predecessor as input, and the $i$-th reconstruction creates another residual for the next round, and so on. Hence, we have

where $\hat{\bm{x}}^{(i)}$ stands for the reconstruction of the $i$-th input using the $i$-th coding module ${\mathcal{F}}^{(i)}(\cdot)$, while the input to the first codec is defined by the raw input frame $\bm{x}$. If we expand the recursion, we arrive at the non-recursive definition of $\bm{x}^{(i)}$,

which means the input to $i$-th model is the residual of the sum of all preceding $i-1$ codecs' decoded signals. It ensures the additivity of the entire system: adding more modules keeps improving the reconstruction quality. Hence, CMRL can scale up to high bitrates at the cost of increased model complexity.

CMRL is optimized in two phases. During Phase-I training, we sequentially train each codec from the first to the last one using a module-specific residual reconstruction goal,

The purpose for Phase-I training is to get parameters for each codec properly initialized. Then, Phase-II finetunes all trainable parameters of the concatenated modules to minimize the global reconstruction loss,

The reconstruction loss measures the waveform discrepancy in both time and frequency domains. Quantization penalty and entropy control are introduced as regularizers. Sec. IV-B details the definition of the training target and ablation tests on how to find the optimal blending weights between the loss terms.

Fig. 3 shows the full system signal flow with $N$ sub-codecs, having an LPC module as the first one. On the transmitter side the LPC analyzer first processes the input frame $\bm{x}$ of 1024 samples and computes 16 coefficients, $\bm{h}^{(1)}$, as well as 512 residual samples $\bm{x}^{(2)}$ at the center of the frame. Then, the residual signal goes through the $N-1$ NWCs in sequentially. Note that the transmission process's primary job is to produce a quantized bitstring $\tilde{\bm{h}}^{(i)}$ from LPC and each NWC. To this end, NWC's decoder part must also run to compute the residual signal and relay it to the next NWC module. The bitstring is generated as a concatenation of all encoder outputs: $\tilde{\bm{h}}=\big{[}{\tilde{\bm{h}}^{(1)}};{\tilde{\bm{h}}^{(2)}};\cdots;{\tilde{\bm{h}}^{(N)}}\big{]}$. Once the bitstring is available on the receiver side, all NWC decoders run to reconstruct the LPC residual signal, i.e., $\hat{\bm{x}}^{(2)}\approx\sum_{i=2}^{N}{\mathcal{F}}_{\text{dec}}^{(i)}(\tilde{\bm{h}}^{(i)})$. Then it is used as input of the LPC synthesizer, along with the LPC coefficients.

The training dataset is created from 300 speakers randomly selected from the TIMIT corpus [48] with no gender preference. Each speaker contributes 10 utterances totaling 2.6 hour-long training set, which is a reasonable size due to our compact design. The same scheme is adopted when creating the validation dataset and test dataset with 50 speakers, respectively. All three datasets are mutually exclusive with the sample rate of $16$ kHz. All neural codecs in this work are trained and tested via the same set of data for a fair comparison. We normalize each utterance to have a unit variance, then divided by the global maximum amplitude, before being framed into segments with the size of 512 samples. On the receiver side, we conduct overlap-and-add after the synthesis of the frames, where a 32-sample Hann window is applied to the overlapping region of the same size.

With the LPC codec, we apply high-pass filtering defined in the z-space, $\mathcal{G}_{\text{hp}}(z)\!=\frac{0.989502-1.979004z^{-1}0.989502z^{-2}}{1-1.978882^{-1}0.979126^{-2}}$, to the normalized waveform. A pre-emphasis filter, $\mathcal{G}_{\text{preemp}}(z)\!=1\!-\!0.68z^{-1}$, follows to boost the high frequencies.

The loss function is defined as

where the first term measures the mean squared error (MSE) between the raw waveform samples and their reconstruction. Ideally, if the model complexity and the bitrate is sufficiently large, an accurate reconstruction is feasible by using MSE as the only loss function. Otherwise, the result is usually sub-optimal due to the lack of bits: coupled with the MSE loss, the decoded signals tend to contain broadband artifact. The second term supplements the MSE loss and helps suppress this kind of artifact. To this end, we follow the common steps to conduct mel-scaled filter bank analysis, which results in a mel spectrum $\bm{y}$ that has a higher resolution in the low frequencies than in the high frequencies. The filter bank size defines the granularity level of the comparison. Following [38], we conduct a coarse-to-fine filter bank analysis by setting four filter bank sizes, $F_{1}=8,F_{2}=16,F_{3}=32,F_{4}=128$ as shown in Fig. 4, which result in four kinds of resolutions for mel spectra $\bm{y}^{(b)}$ indexed by $b\in\{1,2,3,4\}$.

All models are trained on Adam optimizer with default learning rate adaptation rates [49]. The batch size is fixed with $128$ frames . The initial learning rate is $2\times 10^{-3}$ for the first neural codec. With CMRL, the learning rate for the successive neural codecs is $2\times 10^{-4}$. Finetuning of all those models is with a smaller learning rate $2\times 10^{-5}$. All models are sufficiently trained until the validation loss converges after being exposed to about $5\times 10^{5}$ batches. These hyperparameters were chosen based on validation.

The blending weights in the loss function in Eq. (14) are also selected based on the validation performance. Empirically, the ratio between the time-domain loss and mel-scaled frequency loss affects the trade-off between the SNR and perceptual quality of decoded signals. If the time-domain loss dominates the optimization process, the model compresses each sub-band with an equal effort. In that case, the artifact will be audible unless the SNR reaches a rather high level (over $30$ dB) which entails a high bitrate and model complexity. On the other hand, if only the mel-scaled frequency loss is in place, the reconstruction quality in the high frequency will degrade. The impact of these blending weights for these two loss terms is detailed in Sec. IV-F via an ablation analysis.

The weights for the quantization regularizer $\lambda_{\text{Q}}$ and entropy regularizer $\lambda_{\text{ent}}$ are initially set to be $0.5$ and $0.0$, respectively. As for $\lambda_{\text{ent}}$, we alter it after every epoch by $0.015$: if the current model's bitrate is higher than the target bitrate, $\lambda_{\text{ent}}$ increases to penalize the model's entropy more; otherwise, $\lambda_{\text{ent}}$ decreases to boost the entropy. Note that we omit the module index $i$ in Eq. (14), so the meaning of $\hat{x}_{t}$ depends on the context: either the module-specific reconstruction as in Eq. (12) or the sum of all recovered residual signals for Phase-II finetuning as in Eq. (13). Similarly, ${\mathcal{L}}_{\text{Q}}$ and ${\mathcal{H}}_{\bm{\beta}}$ can encompass all modules' quantization and entropy losses including LPC's for Phase-II. We delay the introduction of the quantization and entropy loss until the fifth epoch.

We consider three bitrates, $12$, $20$, and $32$ kbps, to validate models' performance in a range of use cases. We evaluate following different versions of neural speech coding systems:

• •

  Model-I: The NWC baseline (Sec. II).

• •

  Model-II: Another baseline that combines the legacy LPC and an NWC module for residual coding.

• •

  Model-III: A trainable LPC quantization module followed by an NWC and finetuning (Sec. III-A);

• •

  Model-IV: Similar to Model-III but with two NWC modules: the full-capacity CMRL implementation (Sec.III-B). It is tested cover the high bitrate case, $32$ kbps.

Regarding the standard codecs, AMR-WB [5] and Opus [50] are considered for comparison. AMR-WB, as an ITU standard speech codec, operates in nine different modes covering a bitrate range from 6.6 kbps to 23.85 kbps, providing excellent speech quality with a bitrate as low as 12.65 kbps in wideband mode. As a more recent codec, Opus shows the state-of-the-art performance in most bitrates up to 510 kbps for stereo audio coding, except for the very low bitrate range.

We first compare all models with respect to the objective measures, while being aware that they are not consistent with the subjective quality. Hence, we also evaluate these codecs in two rounds of MUSHRA subjective listening tests: the neural codecs are compared in the first round, whose winner is compared with other standard codecs in the second round.

We conduct two rounds of MUSHRA tests: (a) to select the best one out of the proposed models (from Model-I to IV) (b) to compare it with the standard codecs, i.e., AMR-WB and Opus. Each round covers three different bitrate ranges, totaling six MUSHRA sessions. A session consists of ten trials, for which ten gender-balanced test signals are randomly selected. Each trial has one low-pass filtered signal serving as the anchor (with a cutoff frequency at 4kHz), the hidden reference, as well as signals decoded from competing systems. We recruit ten participants who are audio experts with prior experiences in speech/audio quality evaluation. The subjective scores are rendered in Fig. 7 as boxplots. Each box ranges from the $25$ to $75$ percentile with a $95\%$ confidence interval. The mean and median are presented as the green dotted line and pink hard line, respectively. Outliers are represented in circles.

In this section, we perform some ablation analyses to justify our choices that led to CQ and CMRL's superior subjective test results. We investigate how different blending ratios between loss terms can alter the performance. We will also explore the optimal bit allocation strategy among coding modules.

The proposed NWC model is with $0.35$ million parameters, a half of which is for the decoder. Hence, in $32$ kbps with two NWC modules for residual coding, the model size totals 0.7M parameters, with the decoder size of $0.35$M. Even though our decoder is still not as compact as those in traditional codecs, it is $100\times$ smaller than a WaveNet decoder.

Aside from the model size, we investigate the codec's delay and the execution induced latency. The codec will have algorithmic delay if it relies on future samples to predict the current sample. The processing time during the encoding and decoding processes also adds up to the runtime overhead.