Embedding a differentiable mel-cepstral synthesis filter
to a neural speech synthesis system

In statistical parametric speech synthesis [1], linear time-variant synthesis filters such as line spectral pairs (LSP) synthesis filter [2], mel-cepstral synthesis filter [3], and WORLD vocoder [4] are traditionally used as a waveform model to bridge the gap between acoustic features and the audio waveform. An important advantage of linear synthesis filters is that they can easily control voice characteristics and the pitch of synthesized speech by modifying input acoustic features. However, the speech quality is often limited due to the linearity of the filters. In 2016, the first successful neural waveform modeling, WaveNet [5], was proposed, enabling various nonlinear synthesis filters [6, 7, 8, 9] to have been intensively developed by researchers. Nonlinear neural synthesis filters outperform linear synthesis filters and can synthesize high-quality speech comparable to human speech. However, their generation of the waveform which is quite different from training data does not work well, resulting in less controllability of speech synthesis systems. The main reason is that the nonlinear synthesis filters ignore the clear relationship between acoustic features and the audio waveform, which is simply described by conventional linear synthesis filters. There have been a number of attempts to solve the problem, especially in terms of pitch [10, 11, 12]. Although they introduced special network structures against neural black-box modeling, the problem has yet to be fully solved.

In this paper, to take the advantage of both linear and nonlinear synthesis filters, we embed a mel-cepstral synthesis filter in a neural speech synthesis system. The main advantages of the proposed system are as follows: 1) Voice characteristics and the pitch of synthesized speech can be easily modified in the same manner as using the conventional mel-cepstral synthesis filter. 2) Since the embedded filter is essentially differentiable, speech waveform can be accurately modeled by the simultaneous optimization of the neural waveform model and the neural acoustic model that predicts acoustic features from text. 3) The neural waveform model is enough to have less trainable parameters because the mel-cepstral synthesis filter captures the rough relationship between acoustic features and the audio waveform. The proposed system may not require a complex training strategy such as using generative adversarial networks (GANs) [7] because the rough relationship is already constructed by the embedded filter without training.

The main concern about the proposed system is how to implement the mel-cepstral synthesis filter as a differentiable parallel processing module for efficient model training. Usually, the mel-cepstral synthesis filter is widely implemented as an infinite impulse response (IIR) filter [3]. This recursive property is not suitable for parallel processing. We solve the problem by formulating the mel-cepstral synthesis filter as cascaded finite impulse response (FIR) filters, i.e., stacked time-variant convolutional layers.

Tokuda and Zen [13, 14] combined a neural acoustic model and a cepstral synthesis filter [15]. The acoustic model was trained by directly maximizing the likelihood of waveform rather than minimizing cepstral distance. The speech quality was still limited due to the linearity of the cepstral synthesis filter. LPCNet [6] combined an all-pole filter with recurrent neural networks for fast waveform generation. ExcitNet [16] also used an all-pole filter as the backend of waveform generation. They did not introduce a specific signal-processing structure into their excitation generation modules.

In 2020, differentiable digital signal processing (DDSP) has been proposed [17]. The main idea is similar to our work: a differentiable linear filter is embedded in a neural waveform model. One of the main differences is that DDSP assumes a sinusoidal model as a linear synthesis filter. In the DDSP framework, an acoustic model for speech synthesis is not unified currently [18]. Nercessian [19] proposed a differentiable WORLD synthesizer for audio style transfer. The synthesizer uses log mel-spectrogram as a compact representation of spectrum and aperiodicity ratio while our work uses a mel-cepstral representation.

Linear synthesis systems assume the following relation between an excitation signal $\bm{e}=[e[0],\ldots,e[T-1]]$ and a speech signal $\bm{x}=[x[0],\ldots,x[T-1]]$:

$$X(z)=H(z)E(z),$$

(1)

where $X(z)$ and $E(z)$ are $z$-transforms of $\bm{x}$ and $\bm{e}$, respectively. The time-variant linear synthesis filter $H(z)$ is parameterized with a compact spectral representation for speech application, e.g., text-to-speech synthesis and speech coding. In the mel-cepstral analysis [20], the spectral envelope $H(z)$ is modeled using $M$-th order mel-cepstral coefficients $\{\tilde{c}(m)\}_{m=0}^{M}$:

$$H(z)=\exp\sum_{m=0}^{M}\tilde{c}(m)\tilde{z}^{-m},$$

(2)

where

$$\tilde{z}^{-1}=\frac{z^{-1}-\alpha}{1-\alpha z^{-1}}$$

(3)

is a first-order all-pass function. Since the scalar parameter $\alpha$ controls the intensity of frequency warping, the voice characteristics of $\bm{x}$ can be easily modified by changing the value of $\alpha$. The paper uses Eq. (2) as a linear synthesis filter to have the benefits from the property.

Usually, a mixed excitation signal is assumed as the excitation $E(z)$. Namely,

$$X(z)=H(z)\{H_{a}(z)E_{\mathrm{noise}}(z)+H_{p}(z)E_{\mathrm{pulse}}(z)\},$$

(4)

where $E_{\mathrm{noise}}(z)$ and $E_{\mathrm{pulse}}(z)$ are $z$-transforms of white Gaussian noise and pulse train computed from fundamental frequency $f_{0}$, respectively. The $H_{a}(z)$ and $H_{p}(z)$ are zero-phase linear filters represented as

	$\displaystyle H_{a}(z)$	$\displaystyle=$	$\displaystyle\cos\,\,\sum_{m=0}^{M_{a}}\,\,\tilde{c}_{a}(m)\tilde{z}^{-m}$		(5)
		$\displaystyle=$	$\displaystyle\exp\!\!\!\sum_{m=-M_{a}}^{M_{a}}\!\!\!\tilde{c}^{\prime}_{a}(m)\tilde{z}^{-m},$		(6)
	$\displaystyle H_{p}(z)$	$\displaystyle=$	$\displaystyle 1-H_{a}(z),$		(7)

where

$\displaystyle\tilde{c}^{\prime}_{a}(m)=\left\{\begin{array}[]{ll}\tilde{c}_{a}(0),&(m=0)\\ \tilde{c}_{a}(|m|)/2,&(m\neq 0)\end{array}\right.$

(10)

and $\{\tilde{c}_{a}(m)\}_{m=0}^{M_{a}}$ is a mel-cepstral representation of aperiodicity ratio [4]. The aperiodicity ratio represents the intensity of the aperiodic components of $\bm{x}$ for each frequency bin and its values are in $[0,1]$. Equation (4) describes a simple relationship between $\bm{x}$ and $\bm{e}$, but there is a limit to accurate modeling of the speech signal $\bm{x}$.

To generate speech waveform with high controllability, our approach models speech waveform by extending the conventional linear synthesis filter as

	$\displaystyle X(z)$	$\displaystyle=$	$\displaystyle H(z)\{P_{a}(H_{a}(z)E_{\mathrm{noise}}(z))$		(11)
			$\displaystyle\hskip 22.76219pt+\,P_{p}(H_{p}(z)E_{\mathrm{pulse}}(z))\},$

where $P_{a}(z)$ and $P_{p}(z)$ are nonlinear filters represented by trainable neural networks called prenets. To capture speech components that cannot be well captured by acoustic features including mel-cepstral coefficients and $f_{0}$, the prenets are conditioned on the two $Q$-dimensional latent variable vectors, $\bm{h}_{a}$ and $\bm{h}_{p}$. The latent variable vectors $\bm{h}_{a}$ and $\bm{h}_{p}$ as well as acoustic features are jointly estimated by the acoustic model from text.

An overview of the proposed speech synthesis system is shown in Fig. 1. Since all linear filters are inherently differentiable, the acoustic model and prenets can be simultaneously optimized by minimizing the objective

$$\mathcal{L}=\mathcal{L}_{\mathrm{feat}}+\lambda\,\mathcal{L}_{\mathrm{wav}},$$

(12)

where $\mathcal{L}_{\mathrm{feat}}$ is the loss between the predicted and ground-truth acoustic features, $\mathcal{L}_{\mathrm{wav}}$ is the loss between predicted and ground-truth speech samples, and $\lambda$ is a hyperparameter to balance the two losses. In this paper, the multi-resolution short-time Fourier transform (STFT) loss [7] is used as $\mathcal{L}_{\mathrm{wav}}$:

$$\mathcal{L}_{\mathrm{wav}}=\frac{1}{2S}\sum_{s=1}^{S}\left(\mathcal{L}_{\mathrm{sc}}^{(s)}(\bm{x},\hat{\bm{x}})+\mathcal{L}_{\mathrm{mag}}^{(s)}(\bm{x},\hat{\bm{x}})\right),$$

(13)

where

	$\displaystyle\mathcal{L}_{\mathrm{sc}}^{(s)}(\bm{x},\hat{\bm{x}})$	$\displaystyle=$	$\displaystyle\frac{\|\,A^{(s)}(\bm{x})-A^{(s)}(\hat{\bm{x}})\,\|_{F}}{\|\,A^{(s)}(\bm{x})\,\|_{F}},$		(14)
	$\displaystyle\mathcal{L}_{\mathrm{mag}}^{(s)}(\bm{x},\hat{\bm{x}})$	$\displaystyle=$	$\displaystyle\frac{\|\,\log A^{(s)}(\bm{x})-\log A^{(s)}(\hat{\bm{x}})\,\|_{1}}{Z^{(s)}},$		(15)

and $\hat{\bm{x}}$ is the predicted speech waveform, $A(\cdot)^{(s)}$ is an amplitude spectrum obtained by STFT under $s$-th analysis condition, $Z^{(s)}$ is a normalization term, $\|\cdot\|_{F}$ is the Frobenius norm, and $\|\cdot\|_{1}$ is the $L_{1}$ norm. Note that during training, ground-truth fundamental frequencies are used to generate the pulse train fed to $H_{p}(z)$.

This paper proposed a neural speech synthesis system with an embedded mel-cepstral synthesis filter. Experimental results showed that the proposed system can synthesize speech with reasonable quality and has very high robustness against fundamental frequencies. Future work includes to investigate the optimal network architecture of the prenets and to introduce GAN training to boost the performance of the proposed system.