Unified Source-Filter GAN: Unified Source-filter Network Based On Factorization of Quasi-Periodic Parallel WaveGAN

PWG is a GAN-based method for generating raw waveforms. It is a compact model without an autoregressive structure or a causal mechanism, and can achieve fast speech generation with high fidelity. The model consists of two networks, generator (G) and discriminator (D). The WaveNet-based generator, which is conditioned by auxiliary features, learns to make the discriminator recognize the generated sample as $real$. This process can be written as follows:

$$\mathcal{L}_{adv}(G,D)=\mathbb{E}_{\bm{z}\sim\mathcal{N}(0,I)}\left[(1-D(G(\bm{z})))^{2}\right],$$

(1)

where $\bm{z}$ is random noise distributed from Gaussian distribution. Note that all auxiliary features of the generator are omitted in this paper for simplicity.

The discriminator learns to identify the generated sample as $fake$ and the natural sample as $real$. This process can be written as follows:

$$\begin{split}\mathcal{L}_{D}(G,D)=\mathbb{E}_{\bm{x}\sim p_{data}}\left[(1-D(\bm{x}))^{2}\right]&\\ +\mathbb{E}_{\bm{z}\sim\mathcal{N}(0,I)}\left[D(G(\bm{z}))^{2}\right]&,\end{split}$$

(2)

where $\bm{x}$ denotes the natural samples and $p_{data}$ denotes the data distribution of the natural samples. PWG also adopts multi-STFT loss [3] as an auxiliary loss $\mathcal{L}_{aux}(G)$ to improve the training stability. In conclusion, the final loss function of the generator can be written as a weighted sum of $\mathcal{L}_{aux}$ and $\mathcal{L}_{adv}$ as follows:

$$\mathcal{L}_{G}(G,D)=\mathcal{L}_{aux}(G)+\lambda_{adv}\mathcal{L}_{adv}(G,D)$$

(3)

where $\lambda_{adv}$ is hyperparameter for weight and empirically set to 4.0 in this paper.

Although PWG achieves high-fidelity speech generation, the fully data-driven nature makes PWG lack the explicit controllability of each speech component especially when unseen auxiliary features are given such as $F_{0}$ outside the $F_{0}$ range of training data. To alleviate this issue, in QPPWG pitch-dependent dilated convolution neural networks (PDCNNs), which dynamically changes dilation sizes adapting to pitch, are introduced to PWG. PDCNN facilitates QPPWG to capture the very long-term dependencies of periodic components and makes the pitch of the QPPWG-generated speech more consistent with the auxiliary $F_{0}$.

For the ordinary dilated convolution neural networks, the dilation size $d$, is predefined and time-invariant. The dilation sizes $d_{t}$ of PDCNNs are dynamically defined at each time step $t$ as follows:

$$d_{t}=d\times f_{s}/(f_{0,t}\times a)$$

(4)

where $f_{s}$ is the sampling rate, $f_{0,t}$ is an $F_{0}$ value at time $t$, and $a$ is a hyperparameter called dense factor, which determines the sparsity of the PDCNNs, and empirically set to 4.0 in this paper.

NSF is a neural vocoder based on source-filter model, and divided into three modules: a condition module, a source module, and a filter module. An $F_{0}$ sequence $f_{0,1},\cdots,f_{0,T}$ and a spectral feature sequence (e.g. mel-spectrogram) are used as an input of NSF. The condition module upsamples the input features and extracts feature embeddings for the resonance filtering. In the source module, by treating $f_{0,t}$ as the instantaneous frequency, the fixed number of sinusoidal wave basis signals is generated, where the $h$-th basis signal $e_{t}^{(h)}$ is given by

$$e_{t}^{(h)}=\begin{cases}\sin\left(\displaystyle{\sum_{k=1}^{t}}2\pi\frac{hf_{0,k}}{f_{s}}+\phi\right)+n_{t}&\mbox{if}~{}f_{0,t}>0\\ \displaystyle{\frac{1}{3\sigma}n_{t}}&\mbox{if}~{}f_{0,t}=0,\end{cases}$$

(5)

where $f_{s}$ is sampling frequency, $\phi\in[-\pi,\pi]$ is a random initial phase, $n_{t}\sim\mathcal{N}(0,\sigma^{2})$ is a Gaussian noise, and $h$ is the scale factor of the fundamental frequency. These signals are merged using a feed-forward network to output the source excitations. The filter module modulates the source signal using multiple stages of dilated convolution and affine transformations similar to those in ClariNet [20]. NSF adopts multi-resolution STFT loss to learn the difference between the output and the target waveform in the spectral domain. Unlike that of PWG, it calculates the mean square error (MSE) for the log power spectrum.

As the proposed architecture shown in Fig.1, the generator of uSFGAN receives random noise $\bm{z}$ sampled from Gaussian distribution, an $F_{0}$ sequence $\bm{f}$, and an auxiliary feature sequence $\bm{c}$ as the input, where $\bm{f}$ and $\bm{c}$ are assumed to be extracted per frame. A sinusoidal signal $\bm{v}=v_{1},\cdots,v_{T}$ is first generated on the basis of upsampled $\bm{f}$ as follows:

$$v_{t}=\begin{cases}\sin\left(\displaystyle{\sum_{k=1}^{t}}2\pi\frac{f_{0,k}}{f_{s}}\right)&if~{}f_{0,t}>0\\ 0&if~{}f_{0,t}=0,\end{cases}$$

(6)

where $f_{0,t}$ is the instantaneous frequency at time $t$, and $f_{s}$ is the sampling frequency. Unlike QPPWG adopting only noise inputs, the sinusoidal signal input is used to make the estimation of the harmonic components easier and improve the learning efficiency of the proposed source-network. Then, $\bm{v}$ is combined with $\bm{z}$ as a two-channel input of the source-network. The source-network performs the pitch-dependent dilated convolution conditioned on the upsampled auxiliary features $\bm{c}$ to output the source excitation signal $\hat{\bm{e}}$. The generated source excitation signal is used as the input to the filter-network and is also used to calculate the spectral envelope regularization loss, which is used for the auxiliary loss, as mentioned in Section 3.2. In the filter-network, non-causal dilated convolution with fixed dilation sizes is performed. The output waveform is input to the discriminator and also used to calculate the multi-resolution STFT auxiliary loss.

To encourage the source-network to output a reasonable source excitation signal, one constraint is imposed on the output of the source-network. As in the traditional source-filter vocoders, such as STRAIGHT [29] and WORLD [30], we assume that the spectral structure of the source excitation signal consists of harmonic components and stochastic components, and its spectral envelope is flat, i.e., the power of spectral envelope is constant over all frequency. We adopt regularization to satisfy this assumption on the spectral envelope of the source excitation signal in the output of the source-network.

We use a simplified algorithm of cheaptrick [33] to extract the spectral envelope from the output signal of source-network. The original algorithm of cheaptrick is composed of three steps: (1) $F_{0}$ adaptive windowing and calculation of log power spectrum, (2) $F_{0}$ adaptive smoothing in the spectral domain, and (3) $F_{0}$ adaptive liftering in the cepstrum domain. In order to speed up the spectral envelope estimation process, we apply several modifications to the cheaptrick algorithm. First, we directly use the $F_{0}$ values given as the auxiliary feature $\bm{f}$ rather than extracting $F_{0}$ values from the output signal. Those $F_{0}$ values are further rounded to integers, and then, the corresponding windows and liftering functions are obtained in advance. Moreover, the step (2) is omitted because this process requires a relatively large processing time and the $F_{0}$ adaptive spectral envelope extraction is still performed by the $F_{0}$ adaptive liftering in the step (3). Although these modification causes slight degradation of the spectral envelope estimation accuracy, it doesn’t cause any significant issues as the precise spectral envelope estimation is not necessary for the regularization.

The spectral envelope regularization loss is given by

$$\mathcal{L}_{reg}(G)=\frac{1}{2}\sum_{n=1}^{N}\sum_{k=1}^{K}\hat{E}_{k}^{(n)2},$$

(7)

where $\hat{E}_{k}^{(n)}$ is the $k$-th frequency component of log power spectral envelope extracted from $\bm{\hat{e}}$ at $n$-th time frame by the simplified cheaptrick algorithm. Note that when this loss reaches to 0, linear power values of the spectral envelope are 1 over all frequency and time frames.

The same adversarial losses as QPPWG but different auxiliary losses are adopted in uSFGAN training. Our method uses two types of auxiliary losses: multi-resolution STFT loss and the spectral envelope regularization loss. The STFT loss is defined as follows:

$$\mathcal{L}_{s}(G)=\frac{1}{2}\sum_{n=1}^{N}\sum_{k=1}^{K}\left[\log\frac{\mbox{Re}(Y_{k}^{(n)})^{2}+\mbox{Im}(Y_{k}^{(n)})^{2}}{\mbox{Re}(\hat{Y}_{k}^{(n)})^{2}+\mbox{Im}(\hat{Y}_{k}^{(n)})^{2}}\right]^{2},$$

(8)

where $Y_{k}^{n}$ and $\hat{Y}_{k}^{n}$ are the $k$-th STFT component at the $n$-th time frame of a natural waveform and the output waveform. Re and Im denote real part and imaginary part, respectively. This STFT loss is different from that of PWG [3], but the same as that of NSF [21, 22]. Finally, our auxiliary loss is represented as follows:

$$\mathcal{L}_{aux}(G)=\frac{1}{M}\sum_{m=1}^{M}\mathcal{L}_{s}^{(m)}(G)+\lambda_{reg}\mathcal{L}_{reg}(G),$$

(9)

where $M$ is the number of STFT losses using various STFT parameters, and $\lambda_{reg}$ is a hyperparameter balancing the two auxiliary losses and empirically set to 1.0 in this paper.

To investigate the effectiveness of our proposed method, we compared four different models: publicly available pretrained NSF (hn-sinc-nsf-9 [34]) model referred to as PT-NSF, NSF with WORLD features referred to as WORLD-NSF, QPPWG, uSFGAN, and uSFGAN without the spectral envelope regularization loss. We adopted $F_{0}$ conversion to evaluate the controllability.

For the training data, we used 4000 utterances from CMU-ARCTIC database [35] consisting of more than 1000 utterances each from four speakers: slt, bdl, clb, and rms. We used a set of 264 utterances consisting of 66 utterances from each speaker as validation data and another set of 264 utterances as test data. The sampling frequency was set to 16000 Hz by down-sampling.

WORLD-NSF used the same architecture as PT-NSF. QPPWG used 10 adaptive blocks and 10 fixed blocks as the optimized setting. uSFGAN used 30 adaptive blocks for the source-network, and 30 fixed blocks for the filter-network. uSFGAN was trained with the RAdam optimizer [36] ($\epsilon=10^{-6}$) with 400 k iterations as in QPPWG. The generator of uSFGAN is trained with only auxiliary loss in the first 100 k iterations, and then trained with the adversarial loss as well as the auxiliary loss in the remaining 300 k iterations. The parameter settings of the multi-resolution STFT loss are shown in Table 1.

In PT-NSF, $F_{0}$ extracted by YAAPT [37] and mel-spectrum extracted by STFT-based method were used for the input. In the other models, $F_{0}$, spectral envelope, and aperiodicity extracted by WORLD [30] were used. The window length was set to 64 ms and the shift length was set to 5 ms in all models. The spectral envelope was parameterized into 25-dimensional mel-cepstral coefficients, and aperiodicity was coded into 1-dimension. The unvoiced/voiced intervals were represented as a binary feature. For PT-NSF, the target speech used for training was normalized after feature extraction. On the other hand, for the other models, no normalization process was applied to the target speech.

As objective evaluation indexes, we used root mean square error of log $F_{0}$: RMSE [Hz], unvoiced/voiced decision error: $U/V$ [%], mel-cepstral distortion: MCD [dB], and log spectral distortion: LSD [dB]. We conducted all calculations after normalizing the power. For the calculation of RMSE when $F_{0}$ was transformed, we considered $F_{0}$ values extracted from natural speech multiplied by the scale factor as reference $F_{0}$ values.

The objective evaluation results are shown in Table 2. It shows that uSFGAN tends to generate speech well conveying the information of the given auxiliary features with higher accuracy than PT-NSF, WORLD-NSF, and QPPWG. In addition, the spectral envelope regularization loss significantly improves the $F_{0}$ transformation accuracy in uSFGAN. An example of spectrograms and waveforms of the output source excitation signals from uSFGAN with and without the spectral envelope regularization loss are shown in Figs. 3 and 3. We can find that the use of the spectral envelope regularization loss is effective for generating a reasonable source excitation signal of which spectral envelopes are flat and both harmonic components and periodic waveform shape well correspond to $F_{0}$ values.

We conducted an opinion test on speech quality. Natural speech and synthetic speech from WORLD and three models: PT-NSF, QPPWG, and uSFGAN were evaluated by 10 subjects. We evaluated 160 utterances per each method per $F_{0}$ scaling factor. The synthetic speech was generated by scaling $F_{0}$ values by 1.0, 2.0 and 0.5 times.

As the experimental result, mean opinion scores (MOS) are shown in Table 3. It shows that uSFGAN significantly outperforms WORLD, PT-NSF, and QPPWG in both $1.0\times F_{0}$ and $0.5\times F_{0}$. In $2.0\times F_{0}$, although WORLD still achieves the best speech quality, uSFGAN achieves comparable speech quality to PT-NSF and significantly better speech quality than QPPWG.

To investigate the controllability, we conducted an ABX test to evaluate the perceptual accuracy of $F_{0}$ conversion by 10 subjects. We evaluated 100 utterances per each method per $F_{0}$ scaling factor. The synthetic speech by WORLD was used as a reference speech for three models: PT-NSF, QPPWG, and uSFGAN. The subjects selected a speech sample of which pitch was closer to that of the reference speech.

The experimental results are shown in Table 4. Each entry in the table shows the number of times selected as closer to the reference pitch in each pair comparison from the three models. It shows that uSFGAN significantly outperforms the other models in both $2.0\times F_{0}$ and $0.5\times F_{0}$. Our samples can be found on our demo page [38].