TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement

In the time domain, let $\mathbf{y}$ denote a signal with discrete duration $M$ such that $\mathbf{y}\in\mathbb{R}^{M}$. We define clean speech signal $\mathbf{s}$, noise signal $\mathbf{n}$, and noisy speech signal $\mathbf{x}$ with the following additive relation:

Similarly, in the time-frequency domain, let $\mathbf{Y}\in\mathbb{C}^{T\times F}$ denote a complex spectrogram with $T$ discrete time frames and $F$ discrete frequency bins. $\mathfrak{Re}\{\mathbf{Y}\}\in\mathbb{R}^{T\times F}$ denotes real components and $\mathfrak{Im}\{\mathbf{Y}\}\in\mathbb{R}^{T\times F}$ denotes complex components. Let $Y(t,f)$ be the complex-valued time-frequency bin of $\mathbf{Y}$ at discrete time frame $t\in[0,T)$ and discrete frequency bin $f\in[0,F)$. By the linearity of Fourier transforms, the complex spectrograms for clean speech $\mathbf{S}$, noise $\mathbf{N}$, and noisy speech $\mathbf{X}$ relate with the additive relation:

A speech enhancement model $G$ outputs enhanced signal $\mathbf{\hat{s}}$ such that:

During optimization, $G$ minimizes the divergence between $\mathbf{s}$ and $\mathbf{\hat{s}}$. We denote $\mathbf{\hat{S}}$ to be enhanced complex spectrogram derived from $\mathbf{\hat{s}}$.

Let $\mathbf{A}_{\mathbf{y}}\in\mathbf{R}^{T\times 25}$ represent the 25 temporal acoustic parameters of signal $\mathbf{y}$ with $T$ discrete time frames. We represent $A_{\mathbf{y}}(t,p)$ as the acoustic parameter $p$ at discrete time frame $t$. We standardize the acoustic parameters to have mean 0 and variance 1 across the time dimension. Standardization helps optimization and analysis through consistent units across features. To predict $\mathbf{A}_{\mathbf{y}}$, we define estimator:

$\mathcal{TAP}$ takes a signal input $\mathbf{y}$, derives complex spectrogram $\mathbf{Y}$ with $F=257$ frequency bins, and then passes the complex spectrogram to a recurrent neural network to output the temporal acoustic parameter estimates $\mathbf{\hat{A}}_{\mathbf{y}}$.

For loss calculation, we define total mean absolute error as:

During training, $\mathcal{TAP}$ parameters learn to minimize the divergence of $\text{MAE}(\mathbf{A_{s}},\mathbf{A_{\hat{s}}})$ using Adam optimization.

We developed temporal acoustic parameter loss, $\mathcal{L}_{\text{TAP}}$, to enable divergence minimization between clean and enhanced acoustic parameters. This section expounds the mathematical formulation of $\mathcal{L}_{\text{TAP}}$.

Let magnitude spectrogram $||\hat{S}(t,f)||$ represent the magnitude of complex spectrogram $\mathbf{\hat{S}}$. Using Parseval’s Theorem, the frame energy weights, $\boldsymbol{\omega}$, is derived from the magnitude spectrogram mean across the frequency axis:

Because high energies are perceived more noticeably, we apply sigmoid, $\sigma$, to emulate human hearing with bounded scales, resulting in smoothed energy weights $\sigma(\omega)$.

Finally, we define our temporal acoustic parameter loss, $\mathcal{L}_{\text{TAP}}$, as the mean absolute error between clean and enhanced acoustic parameter estimates with smoothed frame energy-weighting:

Here, ”$\odot$” denotes elementwise multiplication with broadcasting. Note that this loss is end-to-end differentiable and takes only waveform as input. Therefore, this loss enables acoustic optimization of any speech model and task with clean references.

This section describes the workflow with TAPLoss applied to speech enhancement models. To demonstrate that our method generalizes on both time-domain and time-frequency domain models, we apply the TAPLoss, $\mathcal{L}_{\text{TAP}}$ to two competitive SE models, Demucs [25] and FullSubNet [26]. Demucs is a mapping-based time domain model with an encoder-decoder structure that takes a noisy waveform as input and outputs an estimated clean waveform. FullSubNet is a masking-based time-frequency domain fusion model that combines a full-band and a sub-band model. FullSubNet estimates a complex Ideal Ratio Mask (cIRM) from the complex spectrogram of the input signal and multiplies the cIRM with the complex spectrogram of the input to get the complex spectrogram of the enhanced signal. The enhanced complex spectrogram translates to the time-domain through inverse short-time Fourier transform (i-STFT).

Our goal is to fine-tune the two baseline enhancement models with $\mathcal{L}_{\text{TAP}}$ to improve their perceptual quality and intelligibility. During forward propagation, the enhancement model takes a noisy signal as input and outputs an enhanced signal. The TAP estimator predicts temporal acoustic parameters for both clean and enhanced signals. $\mathcal{L}_{\text{TAP}}$ is then computed through the methods discussed in the previous subsection. Demucs and FullSubNet also have their own loss functions. FullSubNet uses mean squared error (MSE) between the estimated cIRM and the true cIRM as loss ($\mathcal{L}_{\text{cIRM}}$). Demucs has two loss functions, L1 waveform loss ($\mathcal{L}_{\text{wave}}$) and multi-resolution STFT loss ($\mathcal{L}_{\text{STFT}}$). The baseline Demucs model pre-trained on the DNS 2020 dataset only uses L1 waveform loss. In order for a fair comparison, we first fine-tune Demucs using L1 waveform loss and $\mathcal{L}_{\text{TAP}}$. However, previous works have shown that Demucs model is prone to generating tonal artifacts [27] and we have observed this phenomenon during fine-tuning with L1 waveform loss and $\mathcal{L}_{\text{TAP}}$. Moreover, we discovered that the multi-resolution STFT loss could alleviate this issue because the error introduced by tonal artifacts is more significant and obvious in the time-frequency domain than in the time domain. Therefore, from the best fine-tuning result, we fine-tune again with L1 waveform loss, $\mathcal{L}_{\text{TAP}}$, and multi-resolution STFT loss to remove the tonal artifacts. The following equations show final loss functions for fine-tuning Demucs and FullSubNet, where $\lambda_{1}$, $\lambda_{2}$ and ${\gamma}$ denote weight hyperparameters:

During backward propagation, TAP estimator parameters are frozen and only enhancement model parameters are optimized.

This study uses 2020 Deep Noise Suppression Challenge (DNS) data [28], which includes clean speech (from Librivox corpus), noise (from Freesound and AudioSet [29]), and noisy speech synthesis methods. We synthesize thirty-second clean-noisy pairs, including 50,000 samples for training and 10,000 samples for development. In our experiments, we use the official synthetic test set with no reverberation, which has 150 ten-second samples.

We fine-tune official pre-trained checkpoints of Demucs ¹¹1https://github.com/facebookresearch/denoiser and FullSubNet ²²2https://github.com/haoxiangsnr/FullSubNet. We fine-tune Demucs for 40 epochs with acoustic weight $\lambda_{1}$ and another 10 epochs with spectrogram weight $\lambda_{2}$. We fine-tune FullSubNet for 100 epochs with acoustic weight $\gamma$. TAP and TAPLoss source code will be available at https://github.com/YunyangZeng/TAPLoss. In our experiments, the estimator architecture is a 3-layer bidirectional recurrent neural network with long short-term memory (LSTM), with 256 hidden units. After 200 epochs, the acoustic parameter estimator converges to a training error of 0.15 and a validation error of 0.15.

As an auxiliary loss, $\mathcal{L}_{\text{TAP}}$ requires ablation to determine an optimal hyperparameter specification. We observe that Demucs benefits from high acoustic weights in the time-domain model. However, we also observed tonal artifacts when listening to the audio and visualizing the spectrogram. Upon investigation, these artifacts were caused by the model’s architecture and are a known issue with some transpose convolution specifications. To address tone issues, we performed another ablation to improve spectrograms, given the acoustic weight. We observed that high spectrogram weights help, but it is not as important as optimizing the acoustics. In the time-frequency domain, 0.03 as an acoustic weight gave the best result on FullSubNet. Notably, a weight of 1 for Demucs and a weight of 0.03 for FullSubNet account for respective scale differences.

Consider a clean speech target $\mathbf{s}$, noisy speech input $\mathbf{x}$, baseline enhanced speech $\mathbf{\hat{s}_{1}}$, and our enhanced speech $\mathbf{\hat{s}_{2}}$. Let $\mathbf{MAE}_{0}$ denote the mean absolute error across time (axis 0), and let $\oslash$ represent element-wise division. We define percent acoustic improvement, $\mathbf{PAI}$, as follows:

Acoustic evaluation involves three components: (1) baseline improvement over noisy speech input, $\mathbf{PAI}(\mathbf{A}_{\mathbf{\hat{s}_{1}}},\mathbf{A}_{\mathbf{x}})$, (2) our improvement over noisy speech input, $\mathbf{PAI}(\mathbf{A}_{\mathbf{\hat{s}_{2}}},\mathbf{A}_{\mathbf{x}})$, and (3) our improvement compared to the baseline, $\mathbf{PAI}(\mathbf{A}_{\mathbf{\hat{s}_{2}}},\mathbf{A}_{\mathbf{\hat{s}_{1}}})$.

Acoustic improvement measures how well enhancement processes noisy inputs into clean-sounding output. 0% improvement means enhancement has not changed noisy acoustics, while 100% is maximum possible improvement with enhanced acoustics identical to clean acoustics. Relative acoustic improvement measures how well enhancement fine-tuning yields a more clean-sounding output. 0% improvement means TAPLoss has not changed enhanced acoustics after fine-tuning. 100% improvement means TAPLoss enhanced acoustics sound identical to clean acoustics.

Figure 1 presents percentage acoustic improvement for Demucs and FullSubNet. On average, Demucs with $\mathcal{L}_{\text{wave}}$, $\mathcal{L}_{\text{TAP}}$ and $\mathcal{L}_{\text{STFT}}$ improved noisy acoustics by 53.9% while the baseline Demucs improved them by 44.9%. FullSubNet with $\mathcal{L}_{\text{cIRM}}$, $\mathcal{L}_{\text{TAP}}$ improved noisy acoustics by 50.3% while the baseline FullSubNet improved them by 42.6%. On average, TAPLoss improved Demucs baseline acoustics by 19.4% and FullSubNet baseline acoustics by 14.5%.

As an analytic tool, acoustics decompose enhancement quality changes – identifying potential architectural or optimization criteria in need of development. For example, Demucs and FullSubNet architectures demonstrate difficulty optimizing formant frequency and bandwidth. As such, these empirical results suggest that future work introducing related digital signal processing mechanisms could enable improved acoustic fidelity optimization capacity. By providing a framework for acoustic analysis and optimization, this paper provides the tools needed to understand and improve acoustics and perceptual quality.

Enhancement evaluation includes relative metrics that compare signals, and absolute metrics that valuate individual signals. Relative metrics include Short-Time Objective Intelligibility (STOI), extended Short-Time Objective Intelligibility (ESTOI), Cepstral Distance (CD), Log-Likelihood Ratio (LLR), Weighted Spectral Slope (WSS), Wide-band (WB) and Narrow-band (NB) Perceptual Evaluation of Speech Quality (PESQ) [30]. Absolute metrics include overall (OVRL), signal (SIG), and background (BAK) from DNSMOS P.835 [31]. Finally, Non-matching Reference based Speech Quality Assessment (NORESQA) includes its absolute (unpaired) and relative (paired) MOS estimates [32].

Many enhancement evaluation metrics benefit from the explicit optimization of acoustic parameters using TAPLoss. Perceptual evaluation of speech quality, both narrow band (PESQ-NB) and wideband (PESQ-WB), improved most significantly. While STOI did not improve much in the time-frequency domain model, it improved modestly in the time-domain model. Improvement occurs in most DNSMOS metrics (OVRL, SIG, BAK) in time-frequency and time domain models; however, enhancement outperforming clean suggests the metric is unreliable. NORESQA saw significant gains in this time-domain application, though explicit spectrogram optimization hurts the metric given a source time-domain model. Based on this empirical analysis, we recommend TAPLoss in situations with perceptual quality improvement objectives. Future works may significantly benefit perceptual quality by weighing acoustics given a specific metric optimization objective.

Figure 2 presents more details that help us analyze the 150 ten-second samples. In order to facilitate pairwise comparison, we rank order by the baseline enhanced speech evaluation. By comparison, our enhanced speech outperforms the baseline enhanced speech on the two relative metrics, NB-PESQ and ESTOI. A similar pattern can be observed while analyzing NORESQA. The NORESQA of our enhanced speech mostly outperforms baseline-enhanced speech.