SAR: Self-Supervised Anti-Distortion Representation for End-To-End Speech Model
^††thanks: *Corresponding author: Xulong Zhang, [email protected]

In common neural vocoder training [8], acoustic features like mel-spectrogram are extracted from ground-truth speech samples and condition neural vocoders for speech generation. For the downstream neural vocoder training, there are two differences compared to the common scheme. Firstly, the recurrent encoder part $E(\cdot)$ of the trained auto-encoder is jointly fine-tuned with the neural vocoder, so that the recurrent encoder can be further adapted directly to the speech samples generation. Secondly, the mel-spectrogram is replaced by the learned representation SAR to condition the neural vocoder $V(\cdot)$.

As presented in Figure 1 (b), the mel-spectrogram of each frame $m(t)$ is firstly obtained by human-crafted signal processing from ground-truth speech samples as in a common scheme. Then, the frames of mel-spectrogram $m(t)$ are fed into the recurrent encoder $E(\cdot)$ to get the learned representation of each frame $z(t)$. The masking strategy in the pre-training stage is kept to maintain the anti-distortion property, which transforms $z(t)$ into the masked representation $z^{\prime}(t)$.

where $E(\cdot)$ is pre-trained in the section III-A3 but fine-tuned in the second stage, ”Neural vocoder training”, of training.

Finally, the frames of masked representation $z^{\prime}(t)$ condition the neural vocoder for downstream training. The masking strategy is only activated during the training to incorporate distortion-aware prior into both the recurrent encoder and the neural vocoder. In the copy-synthesis inference of the downstream neural vocoder, the masking strategy is deactivated. In other words, frames of $z(t)$ instead of $z^{\prime}(t)$ are used to condition the neural vocoder to generate speech samples. Because the fine-tuned encoder will generate self-fixed latent representations, which will replace the role of mel-spectrogram to supervise the neural vocoder training.

Hence, during the training, the conditional features are the masked learned representation $z^{\prime}(t)$. During the copy-synthesis of the neural vocoder, the presentation encoder is fixed to become a feature extractor, which extracts $z(t)$ for each frame to condition the neural vocoder for waveform generation. In the fine-tuning stage, the pre-trained auto-encoder is connected with a pre-trained neural vocoder originally based on mel-spectrogram to further adapt the learned representation directly to the waveform. Finally, the learned latent space features $z(t)$ are used to replace the mel-spectrogram.

The neural vocoder in this training stage basically follows the design of Waveglow shown in the Figure 3, the likelihood function of the model is as follows.

where

where the first component is derived from the log-likelihood of a spherical Gaussian distribution of the variable $z(x)$. $\sigma^{2}$ represents the hypothesized variance of the Gaussian distribution, while the remaining terms are included to accommodate the change of variables. $n_{cp}$ denotes number of coupling layers and $n_{cv}$ denotes number of convolutional layers. $s_{j}$ and $t$ denotes the change of variables when doing affine xform function. $m(t)$ denotes the mel spectrograms. $W_{k}$ denotes the weights used in the 1x1 convolutions. These weights are initialized to be orthonormal, thereby ensuring their invertibility. $\log\det$ denotes the log-determinant of the Jacobian function.

As illustrated in Figure 1 (c), unlike the downstream neural vocoder training stage which jointly trains both the recurrent encoder and the neural vocoder, the downstream acoustic model training only utilizes the recurrent encoder as a feature extractor. Moreover, The masking strategy is deactivated and the acoustic model directly uses $z(t)$ instead of masked features $z^{\prime}(t)$ as the target output for training. There are two conjecture about the advantage of the neural vocoder training stage. Firstly, connecting the auto-encoder with the neural vocoder further adapts the learned representation $z(t)$ directly to waveform generation, which leads to a better match. Secondly, it makes it much easier for training by connecting the trained auto-encoder and the trained neural vocoder for joint modeling, since the output of the autoencoder exactly matches the input acoustic features condition of the neural vocoder. During the inference, only the decoder part of the auto-encoder which is simply a stack of FC layers is kept connected with the neural vocoder to generate speech samples. In other words, the input latent space features $z(t)$ with the decoder part of the auto-encoder as a whole, replacing the conditional acoustic features part.

The whole inference pipeline of the back-end of the downstream-trained TTS system works just as in the common scheme. The acoustic model $A(\cdot)$ follows Tacotron-2 which accepts text information as input $s$ and generates a sequence of predicted acoustic representation $\hat{z}(t)$. Then, the frames of $\hat{z}(t)$ condition the neural vocoder to generate speech samples. The objective function of the model is as follows:

where

During the inference, this whole back-end of the TTS system uses the learned acoustic representation. Firstly the predicted representation $\hat{z}(t)$ is generated by the acoustic model, then the $\hat{z}(t)$ conditions the neural vocoder to generate speech samples.

Thus, for the acoustic model, the only difference is the change of target acoustic features. Although the new predicted acoustic features $\hat{z}$ still have a gap with their ground truth counterpart, they have anti-distortion properties to significantly reduce the effects of distortions.

We evaluate our proposed representation on both English and Mandarin datasets. For English, the self-supervised pre-training is based on VCTK [37] dataset (109 English speakers with 400 sentences each speaker) and the downstream neural vocoder is trained with LJSPEECH [38] dataset which is a single-speaker dataset with 13,100 sentences. The Mandarin dataset used for pre-training is an internal dataset, which is composed of 3 males and 3 females with an average of around 9,000 read-out sentences for each speaker. The downstream Mandarin dataset is a single-speaker CSMSC [39] corpus with 10,000 sentences in total. The above datasets are split into training, validation, and test sets at percentages of $90\%$, $5\%$, and $5\%$ respectively.

In our experiments of the auto-encoder self-supervised pre-training, the encoder of the auto-encoder structure first transforms an 80-dim mel-spectrum through two FC layers with 256 hidden units. To circumvent the issue of the output of the ReLU activation function being entirely zero when all input features are negative, the Parametric Rectified Linear Unit (PReLU) was implemented as a nonlinear activation function. After the FC transform, two BLSTM layers are stacked to encode the whole sequence in a many-to-many scheme, with the number of output states being 256 as well. Finally, an FC layer further transforms the output of BLSTM into $z(t)$ with a Hyperbolic tangent (Tanh) activation function to constrain the representation to be within $[-1,1]$. The decoder is much simpler by stacking a PReLU activated FC layer and a linear FC layer to output the final acoustic representation. The first FC layer contains 128 hidden units, whereas the output FC layer consists of 80 units. As for training, the batch size used is 64, and Adam Optimizer is applied with 1e-4 learning rate. For checkpoint selection, an early stop is applied regarding the validation loss. The $\alpha$ is sampled from a uniform distribution of interval $[0,0.2]$.

For downstream tasks in our experiments, Waveglow is selected as a neural vocoder. We use similar parameters setup to those in [8]. Each audio clip contains 16,000 samples. The batch size used is 8 with Adam Optimizer of learning rate 1e-4. For ground-truth acoustic feature extraction, Fast Fourier Transform (FFT) window size is selected to be 1024 with a hop size equal to 16 milliseconds (256 for 16000 sampling rate, and 128 for 8000 sampling rate). As for the acoustic model, Tacotron2 [7] is utilized to predict acoustic features. The Pre-net dimension is set to 256, and the number of embedding dimensions is selected to be 512 with a dimension of attention recurrent neural network (RNN) equal to 1024. The dimension of decoder RNN is set to 1024 with the dimension of Postnet embedding equal to 512. The rest of the hyper-parameters are the same with [7].

To objectively compare the anti-distortion property of the learned representation under different corruption conditions, we built 3 copy-synthesis systems based on both Mandarin and English datasets:

• •

  Mel-WaveGlow: Waveglow based on mel-spectrogram.

• •

  SAR-WaveGlow: Waveglow based on the learned representation with auto-encoder self-supervised pre-training.

• •

  AR-WaveGlow: Waveglow based on the learned representation without auto-encoder self-supervised pre-training, i.e., the recurrent encoder of the auto-encoder is directly connected to a trained Waveglow for downstream fine-tune training.

We designed two types of distortions to corrupt the extracted ground-truth acoustic features:

• •

  White noise: white noise is simulated by sampling from a normal Gaussian distribution, which is additive noise directly added to the acoustic features with a target Signal-to-Noise Ratio (SNR). In our experiments setup, we tried two different target SNRs: 10 $dB$ and 15 $dB$.

• •

  Masking: Similar to the masking strategy in the auto-encoder pre-training, we randomly masked the acoustic features with different masking ratios $\alpha=0.1$ and $\alpha=0.2$.

The Extended Short-Time Objective Intelligibility (ESTOI) scores are computed for objective evaluation. The ESTOI score is sensitive to both to both incorrect spectral profile reconstructions and inconsistent temporal pattern reconstructions. A higher ESTOI score indicates better voice quality and demonstrates a stronger anti-distortion property. Tables I and II describe the ESTOI evaluation on 100 sentences randomly selected from the test sets of the CSMSC and LJSPEECH corpora, respectively. For the uncorrupted acoustic features, the original mel-spectrogram achieves the best ESTOI score. However, in cases of different distortion conditions, the ESTOI scores of Mel-WaveGlow degraded dramatically. While the ESTOI scores of SAR-WaveGlow in those conditions suffer minor loss compared to the raw condition. This shows the strong anti-distortion property of the learned representation against both types of corruption. Even the noise-adding corruption is not seen during the representation learning, the learned presentation still generalizes well. Also, by comparing the ESTOI scores of SAR-WaveGlow and AR-WaveGlow, the self-supervised pre-training part of our proposal proved to be necessary for anti-distortion representation training. Because for both Table I, every ESTOI score in SAR-WaveGlow is dominantly better than that in AR-WaveGlow.

To evaluate the anti-distortion property of the learned representation in speech synthesis scenarios, we built Mandarin TTS systems for subjective evaluation. According to our presumption, there exists a coherent structure within acoustic features that help recover missing parts. And in real-world scenarios, there are some cases in that we may only be able to retrieve low-sampling-rate speech data of a specific speaker due to limited internet bandwidth or device limits, and yet we still want to build a high-sampling-rate TTS system. To mimic those scenarios, we simply simulate a low-quality speech corpus by firstly downsampling and then upsampling the speech data. Specifically, we downsampled the CSMSC corpus from 16000 $Hz$ to 8000 $Hz$, and then upsampled it back to 16000 $Hz$, so that the high-frequency information is deleted. Based on this corrupted CSMSC corpus, we separately trained two Tacotron2 models, between which the only difference is the acoustic features, i.e., mel-spectrogram $m(t)$ versus the learned representation $z(t)$. The previously trained 16000 $Hz$ Waveglow models were reused for both acoustic features.

We did two Mean Opinion Score (MOS) evaluations based on the same setup: 50 sentences were selected from CSMSC test set and 30 Mandarin speakers participated in score ratings. We first did a MOS test on results from the copy-synthesis of Waveglow models using different acoustic features. As shown in the Figure 4(a), for uncorrupted acoustic features the neural vocoder based on mel-spectrogram achieves the best voice quality. The learned representation introduces minor degradation of the copy-synthesis voice quality, compared to the mel-spectrogram. The other MOS test was conducted to compare the anti-distortion property of different predicted acoustic features from acoustic models based on low-quality corrupted CSMSC corpus. Although the learned representation introduces degradation into the copy-synthesis waveform, according to Figure 4(b), the learned representation shows stronger robustness to the low-quality training data for acoustic modeling. The learned representation together with the jointly trained neural vocoder seems to be capable of recovering some of the missing high-frequency information in the generated speech.