Two-stage training method for Japanese electrolaryngeal speech enhancement based on sequence-to-sequence voice conversion

Seq2seq VC model can encode an input source speech $\textbf{x}_{1:n}=\{\textbf{x}_{1},\textbf{x}_{2},...,\textbf{x}_{n}\}$ into a sequence of hidden representations $\textbf{h}_{1:n}=\{\textbf{h}_{1},\textbf{h}_{2},...,\textbf{h}_{n}\}$. During the autoregressive decoding process at each time step, the attention mechanism considers the hidden representations of the encoder outputs at all previous time steps and then generates the context vector. After that, the decoder predicts the output acoustic features of the context vector and finally generates the converted speech $\textbf{y}_{1:m}=\{\textbf{y}_{1},\textbf{y}_{2},...,\textbf{y}_{m}\}$ by employing post-network optimization [13]. Based on the above mechanism, the seq2seq VC model can relax the alignment restriction to realize the mapping between different lengths, i.e., $n$$\neq$$m$.

A plethora of works on constructing seq2seq VC are comprised of either recurrent neural networks (RNNs) [14] or convolutional neural networks (CNNs) [15]. In general, RNNs have difficulties in discovering effective long-term dependencies due to the properties of recursive layers, while CNNs rely on the number of the different kernels in convolutional layers, particularly in handling long sequences. Conversely, having great potential in voice processing, the Transformer [16], possesses multi-head self-attention mechanism and positional embedding, making it capable of perceiving global inputs efficiently. In our work, we find out that EL2SP can inherit the hidden representations of normal-speaker VC. Inspired by [8], the system we built is able to share the hidden representations from Transformer-based TTS model to the VC model through pretraining.

Data augmentation is an efficient approach to increase the variety and quantity of the dataset by generating synthetic training data, thus resolving the issues of non-parallel, semi-parallel, or limited data in VC [11]. [17] presented a data augmentation method named ParaGen for frame-to-frame non-parallel VC, but required a sophisticated architecture with accurate hyperparameters and costly training for speaker disentangling to produce the augmented speech. And [10] proposed a VC system named Parratron to normalize arbitrary speaker’s speech into a single canonical target speech by leveraging synthetic data. However, building the TTS model relies on huge amount of data. By and large, most studies focus on generating high-quality augmented data. In our study, large amount of original dataset is assumed unavailable. The trained TTS models are directly implemented to produce relatively low-quality SPD for VC.

We adopt the transformer-based TTS pretraining for building a one-to-one VTN. The TTS model is similar to the VC model in decoding hidden representations into speech, while only a single speaker’s dataset and corresponding transcriptions are needed. This naturally motivates us to transfer the ability from the TTS to the VC during pretraining stages. Fig. 1 (a) depicts the specific architecture containing two pretraining stages.

Given a large normal TTS corpus, $D_{TTS}$={$T_{TTS}$, $S_{TTS}$}, where $T_{TTS}$ and $S_{TTS}$ denote the text and speech dataset respectively. During the decoder pretraining stage, we use $D_{TTS}$ for typical TTS training. Encoder can effectively produce the hidden representations of pure linguistic information of text, thanks to which the decoder has a robust capability of capturing the relation between various speech features and linguistic information. In the encoder pretraining stage, an autoencoder is involved in training with a reconstruction loss instead of the original TTS encoder part, while freezing the pretrained decoder. Here, the $S_{TTS}$ is used as both inputs and target dataset. In this manner, the encoder can be forced to accurately learn to extract analogous linguistic hidden representations from the speech instead of the text, provided that the decoder maintains the ability to recognize linguistic information from the encoded text.

The above is the complete process of VTN pretraining. Note that we choose normal human corpus for pretraining because it can be easily obtained, and the goal for both EL2SP and normal VC is to convert the hidden linguistic representations to normal speech, thus enabling the EL2SP to inherit the useful capability. After that, fine-tuning stage can be implemented using a EL2SP corpus different from the $D_{TTS}$.

As illustrated in Fig. 1 (b), this process aims to produce EL and normal SPD from the TTS systems. Therefore, we train not only TTS for normal speech generation but also for EL speech generation. However, the initial source EL corpus, $D_{EL}$={$T_{EL}$, $S_{EL}$}, and the target normal corpus, $D_{N}$={$T_{N}$, $S_{N}$}, are too small to be trained from scratch. We perform a fine-tuning with the TTS model obtained by the decoder pretraining stage based on the two corpus, respectively, obtaining the corresponding models, $TTS_{EL}$ and $TTS_{N}$. Note that the performances of $TTS_{EL}$ and $TTS_{N}$ are still poor due to the original dataset being too small. Finally, we input an external set of text to the two systems simultaneously to generate low-quality EL and normal SPD, denoted by $S_{EL}^{SPD}$ and $S_{N}^{SPD}$, respectively.

As presented in Fig. 1 (c), the parameters of $VTN_{pt}$ from Section 3.1, can be used as valid a priori information to initialize the VTN-based EL2SP model, thus achieving better performance and extremely efficient convergence compared to training from scratch. Hereafter, SPD and original natural speech pairs are concurrently pooled together as the training set to obtain the EL2SP model, $VTN_{1st}$, which is denoted as Stage I. Although SPD can model the intonation and speaker identity of normal speech and EL speech, it contains some misinformation due to the limited performance of the TTS models, which may inhibit EL2SP from building perfect hidden representations. Furthermore, we hypothesize that the limited natural dataset can provide a correction for training. So, stage II is set up to obtain the final EL2SP model, $VTN_{2nd}$, during which we initialize the parameters of $VTN_{1st}$ and re-input the original natural data pairs for training.

We built two original small-scale Japanese datasets to evaluate our proposed method. In one, a healthy male speaker recorded parallel utterances using an electrolarynx and his normal voice, denoted by SimuEL and NormSP, respectively. Each pair contained 413 sentences, totaling to 20 minutes. The other EL set came from a real male laryngectomee, denoted as RealEL. Since we could not record the normal corpus from this laryngectomee, we shared NormSP as the reference target speech. Note that RealEL contains only 200 sentences within 10 minutes, whose utterance contents are only part of NormSP, so this dataset belongs to a semi-parallel corpus.

Two experiments were designed based on this two datasets. For case 1, our goal was to convert SimuEL to NormSP (SimuEL-NormSP), which can simulate the normal corpus of patient was available. For case 2, we contrived to convert RealEL to NormSP (RealEL-NormSP), which was a more severe but practical case. Here, we used 20 utterances from the respective dataset as a development set, another 40 as an evaluation set, and the rest were used as a training set. To address the non-parallel part in RealEL-NormSP, we first supplemented the corresponding synthetic EL speech to utilize all feasible data for our method. Note that due to the datasets provided by two cases were different, for the sake of simplicity, we built different evaluation sets.

We used ESPnet [18] to implement the proposed system, where all the speech signals were re-sampled at 24khz, and the 80-dimensional mel filterbanks with 2048 FFT points and a 300-point frame-shift were used to extract the acoustic features. We followed Section 3.1 to build the pretrained VTN using the Japanese corpus named JSUT database [19], which contains 7696 utterances recorded by a female speaker, roughly 10 hours long. In addition, text data of JSUT database was also chosen for generating external SPD. VTN for constructing EL2SP and the TTS models for generating SPD have similar base configurations, using six-layer blocks with four attention heads in the encoder and decoder. The input feature sequence of the TTS model contains phoneme and pause information. For the waveform synthesis module, we used a Parallel WaveGAN (PWG) neural vocoder [20] to reconstruct SPD and the output of EL2SP. The corresponding speaker-dependent PWG vocoders were trained separately using SimuEL, NormSP, and RealEL. Note that the PWG vocoder for NormSP was trained from scratch, while for SimuEL and RealEL were pretrained from an EL dataset with 1000 utterances. Moreover, we set up typical Transformer-based baseline models for case 1 and case 2. Their framework and pretraining process are identical to the proposed system but use only original parallel pairs without SPD during EL2SP training, i.e., 353 for case 1 and 140 for case 2.

We performed objective evaluations using two several metrics, 1) mel cepstrum distortion (MCD), which was calculated between the ground truth samples and the generated samples to measure the spectral envelope distortion, and 2) the character error rate (CER) to evaluate the intelligibility. Meanwhile, as the quality of SPD needed to be examined in this study, we used CER as a rating criterion for SPD. Here, the ASR engine used to calculate CER was conformer-based architecture [21] pretrained from Japanese laboroTV database [22] and fine-tuned as in [23]. It could recognize all original EL and normal speech with the best CER (in %) of 14.7 and 6.9, respectively, which can be regarded as the upper bounds of ASR results to reflect the quality of SPD.

We also performed subjective tests to evaluate the perceptual performance of VC on the generated speech in terms of naturalness. An opinion test to assess naturalness was conducted in terms of mean opinion score (MOS). Listeners were asked to rate the naturalness of each given speech (in the scale of one to five). For case 1 and case 2, twenty random utterances were chosen for the naturalness tests, respectively. More than thirty listeners who can speak Japanese were recruited. Audio samples are available online¹¹1https://forpapersub.github.io/For_SLT_2022/.