Neural Text-to-Speech with a Modeling-by-Generation Excitation Vocoder

The basic WaveNet framework is an autoregressive network which generates a probability distribution of discrete speech symbols from a fixed number of past samples [16]. The ExcitNet vocoder is an advanced version of this network which takes advantages of both the LP vocoder and the WaveNet structure. In an ExcitNet framework, an LP-based adaptive predictor is used to decouple the spectral formant structure from the input speech signal (Fig. 1a). The WaveNet model is then used to train the distribution of the prediction residuals (i.e. excitation) as follows:

$$p(\mathbf{e}|\mathbf{h})=\prod\limits_{n=1}^{N}{p({{e}_{n}}|{{e}_{1}},...,{{e}_{n-1}},\mathbf{h})},$$

(1)

$${e}_{n}={x}_{n}-\sum_{k=1}^{p}\alpha_{k}x_{n-k},$$

(2)

where $x_{n}$ and $e_{n}$ denote the $n^{th}$ sample of speech and excitation, respectively; $\alpha_{k}$ denotes the $k^{th}$ LP coefficient with the order $p$; $\mathbf{h}$ denotes the conditional inputs composed of acoustic parameters.

In the speech synthesis step (Fig. 1c), the acoustic parameters of the given input text are generated by a pre-trained acoustic model. These parameters are then used as conditional inputs for the WaveNet model to generate the corresponding time sequence of the excitation signal. Finally, the speech signal is reconstructed by passing the generated excitation signal through the LP synthesis filter.

To further improve the quality of the synthesized speech, we propose the incorporation of an MbG structure into the training process of the ExcitNet vocoder. As illustrated in Fig. 1a, conventional vocoding models are trained separately from the acoustic model, even though the generated acoustic parameters, which contain estimation errors, are used as direct conditional inputs (Fig. 1c). This inevitably causes quality degradation of the synthesized speech as the estimation errors from the acoustic model are boosted non-linearly throughout the synthesis process in the vocoder back-end.

Fig. 1b shows the proposed MbG training method which uses closed-loop extraction¹¹1This extraction method has been adopted in analysis-by-synthesis speech coding frameworks [17, 18] where the encoder and decoder share the same quantized filter parameters for minimizing their mismatch conditions. of the excitation signal. To minimize the mismatch between the training and the generation processes, the LP coefficients in the training step are replaced with those generated by the pre-trained acoustic model as follows:

$$\hat{e}_{n}={x}_{n}-\sum_{k=1}^{p}\hat{\alpha}_{k}x_{n-k},$$

(3)

where $\{{\hat{\alpha}_{1}},...,{\hat{\alpha}_{p}}\}$ denotes the generated LP coefficients. By combining equations (2) and (3), the excitation sequence can be represented as follows:

$$\hat{e}_{n}={e}_{n}+{e}^{am}_{n},$$

(4)

where $e^{am}_{n}$ denotes an intermediate prediction defined as follows:

$${e}^{am}_{n}=\sum_{k=1}^{p}(\alpha_{k}-\hat{\alpha}_{k})x_{n-k}.$$

(5)

Using the excitation signal (i.e. $\hat{e}_{n}$) as the training target means that it becomes possible to guide the model to learn the distributions of the true excitation signal (i.e. $e_{n}$) as well as compensate for the acoustic model’s estimation errors (i.e. $e^{am}_{n}$). Furthermore, because the training and synthesis processes share the same LP coefficients, it is also possible to minimize any mismatch.

The merits of the proposed method are presented in Fig. 2 which shows the negative log-likelihood obtained from the training and validation sets. The proposed MbG-ExcitNet model enables a reduction in both training and validation errors as compared to a plain ExcitNet approach. It is therefore expected that the proposed method will provide more accurate training and generation results, to be further discussed in the following section.

To evaluate the perceptual quality of the proposed system, naturalness MOS tests were performed²²2Generated audio samples are available at the following URL:
https://sewplay.github.io/demos/mbg_excitnet by asking 13 native Korean speakers to make quality judgments about the synthesized speech samples using the following five responses: 1 = Bad; 2 = Poor; 3 = Fair; 4 = Good; and 5 = Excellent. In total, 20 utterances were randomly selected from the test set and were synthesized using the different generation models. In particular, the speech samples synthesized by the below conventional vocoding methods were evaluated together to confirm performance differences:

• •

  WaveNet: Plain WaveNet vocoder [3]

• •

  ExcitNet: Plain ExcitNet vocoder [7]

• •

  G-WaveNet: WaveNet vocoder trained with generated acoustic parameters [11]

• •

  G-ExcitNet: ExcitNet vocoder trained with generated acoustic parameters

The G-ExcitNet vocoder was configured similarly to the proposed MbG-ExcitNet, but its target excitation was extracted from the ground-truth spectral parameters.

Table 1 presents the MOS test results for the TTS systems with respect to the different vocoding models, and the analysis can be summarized as follows: First, when training vocoding models using ground-truth acoustic parameters, ExcitNet performed better than WaveNet (Tests 1 and 2). This implies that ExcitNet’s adaptive spectral filter is beneficial to reconstruct a more accurate speech signal [7]. Second, training the model with generated parameters provided better perceptual quality than using the ground-truth approach in WaveNet (Tests 1 and 3), but vice versa in ExcitNet (Tests 2 and 4). This result confirms that target excitation should be replaced by considering the acoustic model’s estimation errors in excitation-based methods. Lastly, the proposed MbG-ExcitNet performed best across the different vocoders (Tests 5 and the others). Because the MbG training strategy guided the vocoding model to compensate for errors from the acoustic model, it was possible to significantly improve synthesis accuracy. Consequently, the TTS system with the proposed MbG-ExcitNet vocoder achieved $4.57$ MOS.

To further verify the effectiveness of the proposed method, we designed additional experiments to refine the initialization of MbG-ExcitNet’s model weights. Since the MbG training process utilizes generated spectral parameters and the corresponding excitation signals as the input conditions and target outputs, respectively, it may be difficult to capture the speech signals’ original characteristics. We therefore adopted a transfer learning method [23] through which the MbG-ExcitNet was initialized by the plain ExcitNet model whose own weights were optimized by ground-truth speech spectra and excitations. All weights were then fine-tuned by the MbG framework. As a result, it was possible to guide the entire training process to learn the characteristics of both the original and the generated speech segments.

Fig. 4 depicts the results of an A/B/X preference test between the proposed MbG-ExcitNet and this initialization-refined version (MbG-ExcitNet^∗). The setup for this test was the same as for the MOS assessment except that listeners were asked to rate the quality preference of the synthesized speech samples. The results confirm that the initialization-refined system provided better perceptual quality than the originally proposed MbG-ExcitNet. This confirms that adopting a transfer learning method is advantageous to generating more natural speech signal in an MbG-structured TTS system.