DDD: A Perceptually Superior Low-Response-Time DNN-based Declipper

Let $\mathbf{y}\in\mathbf{S}$ be an arbitrary-length speech signal, where $\mathbf{S}=\cup_{t=1}^{\infty}[-1,1]^{t}$. Also let us denote the length of signal $\mathbf{y}$ by $\text{len}(\mathbf{y})$; in other words, $\mathbf{y}\in[-1,1]^{\text{len}(\mathbf{y})}$. Given any threshold $\theta\in[0,1]$, we can define a hard-clip map $f_{\theta}:\mathbf{S}\to\mathbf{S}$:

Where $y_{i}$ is the $i$th element of $\mathbf{y}$. The aim of the SD task is to infer $\mathbf{y}$ from $f_{\theta}(\mathbf{y})$. This is in stark contrast to common formulations of speech enhancement (SE), where the aim is frequently to recover the original signal from its sum with an instance of roughly stochastic environmental noise.

Our setting for training DDD can be seen in Fig. 1. We sought a twofold improvement in perceived speech quality and inference lookahead. To this end, we adopt the Generative Adversarial Network (GAN) [16] framework to introduce adversarial training objectives.

The Generator. We sought a model allowing for low-response-time, real-time inference on consumer-grade CPUs. We found an instance of causal Demucs [14], shown in Fig. 1, to be the most suitable. The five 1D-strided-convolution blocks of the Demucs encoder each downsample the input by a factor of 4. Channels are doubled every block, with the exception of the first block which maps monochannel audio to a 64-channel representation. The decoder reverses these steps using transposed convolutions. Between the encoder and decoder is a causal 2-layer LSTM. To further reduce temporal lookahead, we upsample input and downsample output by a factor of four. Thus, the generator needs no more than 500 samples of lookahead. Although this exact setting was reported to be incapable of real-time streaming inference [14], we explain in section 3.4 a way to accelerate inference at the expense of response time. Notably, faster SE models such as Conv-Tasnet [12] or DPRNN [13] failed to converge in our preliminary experiments.

The Discriminator. We use Multi-Scale Discriminators (MSD) and Multi-Period Discriminators (MPD), which were successfully applied to speech generation [15] and speech enhancement [17]. These discriminators are sub-stacks of strided convolution layers that take different subsets of the time-domain speech samples as input: speech subsampled with a period of {2,3,5,7,11} for the MPD, and {1,2,4}x average-pooled speech for the MSD. The discriminator is trained to label clean speech signals as 1, and restored signals as 0. We use LS-GAN [18] objectives for a successful training. Moreover, for the generator, we also use the feature-matching loss as outlined in [19] and successfully applied in [15, 17].

Training Objective. We used the L1 loss and multi-resolution STFT loss, as follows.

where $\mathbf{y}$ is the ground truth, $\hat{\mathbf{y}}$ is the network output, and

where $||\cdot||_{1}$ and $||\cdot||_{F}$ each refers to the L1 and Frobenius norm. FFT sizes of 512, 1024, and 2048 were used for each $\text{STFT}^{(i)}$. The final training objective for the generator was a sum of the above, the feature matching loss multiplied by four, and the discriminator loss.

The “clean” splits of the Voicebank-DEMAND dataset [20], comprised of a “train” split of length 9.4 hours and a “test” split of 35 mins, were downsampled to 16kHz. In training, the “train” split samples were clipped on-the-fly with a random threshold and aligned with the unprocessed speech. To place more emphasis on heavily clipped speech, we sampled $s$ uniformly from $[-2.0,-0.9]$ and used $\theta=10^{s}$ as the final threshold. The range of $\theta$ covered SNRs of approximately 1dB to 9dB.

The 150 clean “test”-split utterances of Interspeech 2020 DNS-Challenge dataset [21], also downsampled to 16kHz, were used together with the “test” split of Voicebank-DEMAND for evaluation. We clip these sets, henceforth “VBDM-Testset” and “DNS-Testset”, to have SNRs of 1dB, 3dB, 7dB, and 15dB, calculating the SNR as

where $\mathbf{y}$ and $\mathbf{x}$ represent the clean and noisy signals. We denote each of these sets as “{VBDM,DNS}-XdB-Testset” where X is the SNR.

As a DNN baseline, we re-implemented and trained T-UNet [9], the state-of-the-art for the SD task. This architecture employs strided convolutions to downsample input speech by a factor of two every layer, then upsamples with sub-pixel convolutions to generate raw waveforms. We do not use the discriminator for T-UNet. As a non-DNN baseline, we use A-SPADE which has shown competitive performance in AD [8]. We use a MATLAB implementation provided by a past work [4]. Finally, as an ablation, we train the generator without the discriminator. We refer to the trained model as DD.

For the T-UNet, we followed the training and inference procedures given in the original paper [9]. We padded and split all utterances into $2^{14}$-sample-segments to make them compatible with the T-UNet. In inference, the segments were stitched back together. For DD and DDD, 24,000-samples-long utterances were used. Inference was performed without any segmentation.

All models are trained for a total of 75 epochs with the AdamW optimizer with a learning rate of $10^{-4}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, and a weight decay of $10^{-2}$. A batch size of 32 was used for DD and T-UNet, and 2 was used for DDD.

We excluded A-SPADE outputs from the longer 15 DNS-Testset questions to minimize ear fatigue. This exclusion in only one dataset can be justified since A-SPADE is a non-parametric model.

To measure response times and the real-time factors (RTF), we simulated a typical audio streaming application. We fed samples into the model at a rate of 16,000 samples/sec for 100 seconds, then calculated the mean time interval between the feeding and output of every 500${}^{\text{th}}$ sample. Samples were buffered, and then fed into the network once a sufficient number of samples were gathered to yield output. Due to its architecture, the T-UNet needs at least $2^{14}$ samples to generate an output - therefore $2^{14}$ samples were buffered before yielding any output. DD and DDD needs much less, only requiring enough to make one input vector to feed the LSTM. To enable out model to run real-time in streaming settings, we buffered enough samples to produce four input vectors for the LSTM. In this case, the required lookahead is 1,429 samples, due to how we upsample inputs by a factor of four. This evaluation was performed on a Ryzen 7 5700X CPU @ 3.4GHz.

Subjective evaluation results are presented in Fig. 2. Out of twenty-four responses, four were filtered out as described in Section 3.4. To account for each participant’s different scoring criteria, the 20 scores for each samples were averaged before drawing the violin plot. A pairwise post-hoc paired t-test with Bonferroni correction concludes that differences between DDD and all other models for both datasets are statistically significant ($p<0.0001$). While still distinguishable from the hidden reference, DDD may be a significant improvement over the T-UNet or DD. Moreover, comparing DD and DDD suggests that the adversarial training objectives may have helped boost the quality of speech produced by DDD.

The scores for the DNS-Testset are smaller than those for the Voicebank-DEMAND dataset - a natural result, as the models were trained on the latter. Thus, the DNS-Challenge evaluations were in effect zero-shot evaluations on utterances recorded in a different environment. The long ”tail” of score distributions exhibited by T-UNet on this set hints at some catastrophic failures of the T-UNet on unseen recording environments. This result implies that DDD is more robust compared to the T-UNet.

In Fig. 3a, a typical waveform of clean and clipped (1dB) speech is presented with speech declipped by T-UNet, DD, and DDD. Notably, T-UNet and DD are often unable to reconstruct the up-and-down “ruggedness” of the clean-speech waveform as shown in Fig. 3a.

In the regions that were previously saturated, the DD and T-UNet outputs are less “rugged” than the clean signal; indeed, over the entire VBDM-1dB-Testset, the associated clean signals have a total of 1.52M local extrema over the previously saturated regions, about 35-50% more than DD (1.17M extrema) or T-UNet outputs (1.09M extrema). This problem is alleviated by DDD (1.67M extrema). Therefore, we suspect that the inability of DD and T-UNet to achieve a perfect reconstruction of speech is at least partially associated with their incapacity to reproduce the up-and-down “ruggedness” of the clean-speech waveform.

Fig. 3b shows the Fourier transform coefficients of the same region. Across the entire dataset, all model outputs conform to the original spectrum in the sub-1kHz regions. However, the T-UNet frequently fails to reproduce higher-order formants at 3kHz and beyond. In contrast, DDD does seem to model higher-frequency(2-4kHz) regions, sometimes resulting in formants minutely different from the clean speech but resulting in coherent and high-quality speech nonetheless. These observations may explain the higher subjective scores of DDD as presented in Fig. 2.

Table LABEL:tab:objective_results presents objective evaluation results. DD performs comparably to T-UNet in terms of similarity-based metrics, with DDD following close behind, and A-SPADE scoring far lower. However, we cannot conclude that T-UNet or DD provides better reconstructions - subjective evaluations are known to be more reliable [23]. Moreover, similarity-based scores have been previously pointed out to be more unreliable for adversarially-trained speech networks such as DDD [17, 26]. Reflecting on previous analyses, we may conclude that DDD has traded away an acceptable degree of dataset conformance for perceptual quality.

RTF and response time measurement results are presented in the rightmost columns of Table LABEL:tab:objective_results. MACs per sample and the throughput with respect to temporal length were measured as well: 0.19M/sample and 0.15x for T-UNet, and 0.48M/sample and 0.34x for DD or DDD. As noted in past works [4], the RTF for A-SPADE differs with input SNR, but was typically over 20. While both DNN-based approaches have a sub-unity RTF, T-UNet has a mean response time of 0.58 seconds, which is unacceptably high for many applications. In contrast, DD and DDD have sub-decisecond mean response times. Moreover, as discussed in section 3.4 DD and DDD can trade RTF and response time using buffers: it can, for instance, be configured to match the RTF of T-UNet and still yield one third the response time. Therefore, DD and DDD may be better suited to real-time audio processing.