DeepA: a deep neural analyzer for speech and singing vocoding

The early vocoders did not achieve good quality of speech reconstruction, however that changed with the development of STRAIGHT [7] and then WORLD [8]. They achieve this by decomposing speech into three temporal sequences of parameters – the fundamental frequency (F0), smoothed spectrogram (roughly corresponds to the voice timbre) and aperiodicity (correlates with “voice quality” attribute [15]). This strategy allowed for high-quality speech reconstruction and (within a certain range) manipulation.

Conventional vocoders are still used today, mainly either as voice manipulation tools (for example, for changing F0 in voice conversion tasks) or as the provider of interpretable features for training various neural frameworks [16, 17, 18]. However, being speech processing tools built upon certain assumptions about the analyzed signal, conventional vocoders are not flexible enough to be used for high-quality singing voice synthesis. Even though both WORLD and STRAIGHT were either extended to be able to solve the task [15] or applied directly (with some additional tools) [19], both approaches still use a lot of assumptions and heuristics. It should be mentioned that parameter estimation from data via numerical optimization is often employed by those systems, but the number of these parameters is usually quite small, which makes the whole system to rely on the highly constrained model. While a well-engineered model in tandem with a relatively restricted numerical parameter estimation can outperform the more general data-driven neural frameworks in a task the system was designed for, the latter system is more flexible in terms of application and is more adaptable to new data.

In this paper we focus on utilizing these properties of neural frameworks in combination with structural constraints, inspired by conventional vocoders, to learn a compact singing voice representation, which allows for both the accurate speech reconstruction and manipulation.

Recent development in the field of neural networks showed that those systems are capable of high-quality speech synthesis, WaveNet [9], WaveGlow [20], WaveRNN [10], GAN-based models [21, 22], LPCNet [23], neural homomorphic vocoder [24] and NSF models [18] being the few examples. Since they reconstruct speech using F0 and/or other acoustic features (e.g. spectrogram, phoneme encoding etc.) a term “neural vocoder” was coined to describe such systems. The architecture and idea behind each neural vocoder may differ (for example, WaveNet generates raw audio, sample by sample, in autoregressive manner, while hn-NSF essentially implements a harmonic-plus-noise model [25] within a neural framework), but all of those systems are flexible in terms of input features.

However, they are not “vocoders” in full sense of the word as they do not perform speech analysis, but rather rely on another module or algorithm, such as STRAIGHT, WORLD, and short-time Fourier transform, for feature extraction. In that regard such frameworks are essentially neural synthesizers, which rely on the assumption that extracted features are adequate and are free from errors. Even though some of the frameworks attempt to mitigate errors in the input features, speech manipulation using such systems remains a challenge. For example, hn-NSF model is able to deal with F0 errors to some extent, but when combining the modified F0 and the original spectrogram as an input, the resulting speech sample has a pitch that is different from what can be inferred from either of the input parameters.

Given this issue and the fact that there appears to be no study dedicated to the development of neural vocoder in full sense, i.e., a complete neural analyzer and synthesizer, the focus of this paper is the development of such a neural vocoder system. In particular, we employ hn-NSF model as a neural synthesizer to be trained jointly with a neural analyzer, architecture of which was inspired by both conventional vocoders and the hn-NSF model itself.

Auto-encoder [26] is a system that aims to learn the latent, usually – more compact and/or meaningful, representation of the input. Auto-encoders may be viewed as systems that perform dimensionality reduction, but in contrast to principal component analysis the components of the encoding can be constrained or “conditioned” during the training, making the network to converge to the representation with desired properties. For example, the encoding can be conditioned to be “smooth” in a way that makes sense in terms of the task at hand, which may be used for data denoising [27].

Variational auto-encoder (VAE) [28] extends the basic architecture with an additional constraint – the obtained latent representation should resemble a sample from a certain well-known distribution. Gaussian distribution is a common choice for the distribution in question, since an expression for Kullback–Leibler divergence (which is used as part of the loss) between two Gaussian distributions is quite simple and can be derived analytically. The main idea of VAE is to implicitly estimate the distribution of the input data and learn the effective encoding based on that information.

VAE-based models are widely used for learning disentangled representation of some parameters, that may be used for reconstructing the input. For example, it was used for learning the disentangled representation of pitch and timbre of musical instruments [12] and for learning the latent representation of speaking style in a TTS framework [29]. The framework proposed in this paper is built as a variational auto-encoder in order to learn independent representations of harmonic and noise parts of the input sample, which allows one to use the system for speech decomposition, manipulation and reconstruction, similar to the conventional vocoders.

Vocoder parameters are designed to be independent, but they are expected to be consistent with each other. For example, both F0 and aperiodicity generated by the WORLD vocoder have clearly defined voiced and unvoiced segments and ideally those segments should overlap each other exactly. However, sometimes that is not the case. As shown in Fig. 1, F0 and aperiodicity, calculated by WORLD [8], may predict different “voicedness” (V/UV flag) of the same frame range (highlighted by a grey rectangle in the picture). The sample, reconstructed using these parameters, exhibits distortion in the interval of inconsistency.

Even when F0 is consistent with aperiodicity, the former may still contain noticeable errors. As shown in Fig. 2, the first group of highlighted frames (at the very beginning of the sample) has no periodic structure according to the mel-spectrogram. However, the vocoder estimated F0 of that range as close to the mean pitch of the sample. The second highlighted group exhibits the opposite problem – according to F0 the frames are unvoiced even though the spectrogram clearly shows the continuous periodic structure in that area.

A vocoder seeks to decompose speech or singing vocals into interpretable vocoder parameters. While text-to-speech systems may allow rare mild distortions, the high dynamic range of the singing voice makes those distortions more prominent, which may significantly degrade the perceived singing quality. Such distortions in conventional vocoders usually arise from hard-coded human assumptions in the system. We believe that a neural analyzer with data-driven modeling training could be a solution, that we will discuss next.

The encoder consists of two parallel C-blocks (see Fig. 4), which map the input mel-spectrogram to the latent representations of the noise (matrix $a$ in a diagram) and harmonic (matrix $s$ in a diagram) parts of the input. Both matrices are $L$ rows by $T$ columns, where $L$ is the latent dimension and $T$ is the number of temporal frames in the input. The full latent representation $z\in\mathbb{R}^{2L\times T}$ is defined as a vertical stack of $a$ and $s$. It should also be mentioned that the last row of the matrix $s$ (which conveniently corresponds to the last row of the latent representation itself) is conditioned to resemble the normalized ground truth F0. The encoder architecture is inspired by a hn-NSF [18] neural synthesizer, which uses two separate blocks for generating harmonic and noise parts of the speech signal independently and then adds them together to produce the final output. The same idea is used for the decoder architecture as well, which is described in the next subsection.

As can be seen from the diagram (Fig. 3), the decoder is structurally similar to encoder, using TC-blocks (see Fig. 4) instead of C-blocks. The latent speech representation $z$, which is passed to the decoder as an input, is decomposed into matrices $a$ and $s$, which are then processed by the decoder separately. This structural constraint is proven to be essential for learning disentangled representations of harmonic and noise parts of the spectrogram. The decoder transforms those representations into masks, which are then multiplied by the generated white noise mel-spectrogram (NS block in Fig. 3) or a mel-spectrogram of a harmonic signal with ground truth F0 as a fundamental (HS block in Fig.3).

We are particularly interested in the performance of F0 estimation and rendering. Therefore, we conduct experiments on singing voice samples from the NUS-HLT Speak-Sing (NHSS) database [32], which constitutes approximately 7 hours of audio. All samples were downsampled to 16 kHz, mel-spectrograms of order 80 were used as input features, ground truth F0 was extracted using the WORLD [8] vocoder and the latent dimension $L$ was experimentally chosen to be 16. The framework also uses the modified VAE loss in order to improve the system stability. The full loss of the system is given by the following expression:

where $Loss_{KL}$ is the averaged over time Kullback-Leibler divergence between $\mathcal{N}(\mu,e^{\nu})$ and $\mathcal{N}(0,I)$, which is essentially the distance between two distributions (see [28] for derivation) and $Loss_{NSF}$ is a hn-NSF synthesizer loss defined in [18]. Basically, it is a sum of three STFT losses between the ground truth waveform and the generated one, which represents the speech reconstruction loss in frequency domain. The first quadratic term of $Loss$ represents F0 reconstruction loss, and the third quadratic term of $Loss$ represents the VAE reconstruction loss (mean squared error between the input mel-spectrogram and the reconstructed one). Here $T$ is the number of temporal frames in the input, $\mu$ and $\nu$ denote the predicted mean and log-variance of the latent representation $z$; $F0,x$ and $y$ denote ground truth F0, mel-spectrogram and singing waveform respectively, while the corresponding circumflexed variables ($\widehat{F0},\hat{x}$ and $\hat{y}$) denote the generated ones. $\alpha_{1}$ and $\alpha_{2}$ are positive weights determined experimentally. The discovered working configuration is $(\alpha_{1}=10^{2},\alpha_{2}=10^{-2})$.

In this paper three systems are compared: WORLD vocoder, hn-NSF synthesizer using WORLD vocoder parameters, and the proposed neural analyzer followed by hn-NSF synthesizer. WORLD vocoder parameters were extracted using 5 ms frame period and 1024 frequency bins (for spectrogram and aperiodicity), a baseline hn-NSF synthesizer used F0 extracted by WORLD and mel-spectrogram of order 80 and DeepA used just a mel-spectrogram of order 80 with the hn-NSF synthesizer within the proposed framework using latent spectrogram representation $z$ of order 32 as its input.

MCD:

As can be seen from F0 deviation values, the RMSE value appears to be quite large. This is due to the fact that RMSE is sensitive to outliers. When several samples with the highest RMSE values were investigated, it turned out that the large error is caused by those frames, for which the WORLD vocoder and DeepA predict the opposite V/UV flag. This made RMSE a very useful indicator to highlight the discrepancy between the two systems. By inspecting 20 worst samples in terms of RMSE, it turned out that in almost all the cases the proposed model predicted the correct V/UV flag (two examples of the F0 contour comparison are shown in Fig. 5).

To accurately represent the overall performance of the proposed framework, the median of absolute difference (MD) was employed. MD is insensitive to outliers, thus giving the robust approximation of the central tendency. The MD value for the DeepA is 4 Hz, which, in conjunction with the observations mentioned in the paragraph above, shows that the system is both in good agreement with WORLD vocoder and robust, predicting more accurate results when WORLD fails.

We report the mean-opinion score (MOS) in Table 2. This test seeks to determine the overall quality of speech generated by WORLD, hn-NSF (which uses WORLD F0 and mel-spectrogram as input) and the proposed framework (DeepA analyzer + hn-NSF synthesizer) with respect to the ground truth. 10 subjects participated in the test and each listened to 96 utterances in total. From Table 2 it can be seen, that the proposed framework achieved the quality comparable to the other systems. These results together with the objective evaluation in the previous subsection suggest that the proposed framework is capable of both analyzing and reconstructing singing voice with appropriate quality. The advantage of the neural analyzer-synthesizer pipeline lies in the fact that it employs interpretable latent representation that allows direct F0 manipulation in the same way as conventional vocoders.