Syntactic Representation Learning for Neural Network based TTS with syntactic parse Tree Traversal

To provide high-level syntactic representations with rich syntactic information to neural network based TTS system, we propose a syntactic representation learning network based on syntactic parse tree traversal. Constituent parse trees are extracted including labels and tree structure of constituents.

To represent the tree structure for neural network based TTS, depth-first traversals are of possible to use to linearize the syntactic parse tree to constituent sequences. Since any single tree traversal algorithm will map multiple syntactic parse trees to a same sequence, both left-first and right-first are proposed to use to alleviate the ambiguity. The sequence of constituent labels generated by the two traversals can be formulated as the following equations:

$$\begin{split}\small C_{l}&=[c_{l}^{1},...,c_{l}^{m}]\\ C_{r}&=[c_{r}^{1},...,c_{r}^{m}]\end{split}$$

(1)

where $C_{l}$ and $C_{r}$ are the constituent label sequences generated from the left-first and right-first traversals respectively, $c_{l}^{i}$ and $c_{r}^{i}$ are constituent labels, $m$ is the length of sequences. The constituent labels are then embedded by a shared embedding layer and modeled by two different uni-directional GRU networks, one GRU network for each sequence. The process can be represented as:

$$\begin{split}\small&\hat{C}_{l},\hat{C}_{r}=Embedding(C_{l},C_{r})\\ &O_{l}=GRU_{l}(\hat{C}_{l})\\ &O_{r}=GRU_{r}(\hat{C}_{r})\end{split}$$

(2)

where $\hat{C_{l}}$ and $\hat{C_{r}}$ are embedding sequences of constituent labels, $GRU_{l}(\cdot)$ and $GRU_{r}(\cdot)$ are two different uni-directional GRUs, $O_{l}$ and $O_{r}$ are the outputs of $GRU_{l}(\cdot)$ and $GRU_{r}(\cdot)$ respectively.

Syntactic features are the concatenations of the outputs of GRUs for each word, which can be formulated as:

$$\small f_{i}=[O_{l}^{p_{l}^{i}},O_{r}^{p_{r}^{i}}],i\in\{1,2,...,w\}$$

(3)

where $p_{l}^{i}$ and $p_{r}^{i}$ are the positions of the $i$-th word in $C_{l}$ and $C_{r}$, $w$ is the number of words of the input text, $f_{i}$ is the learnt syntactic representation.

To improve the discriminability and diversity of the embeddings of the syntactic labels, global nuclear-norm maximization loss (NML) [15] is proposed to increase the rank of the embeddings of all possible constituent labels. The NML is defined as:

$$\begin{split}\small&\hat{C}=Embedding(C)\\ &\mathcal{L}_{NML}=-\frac{1}{N}{\|\hat{C}\|}_{*}.\end{split}$$

(4)

where $C$ is the set of all possible constituent labels, $\hat{C}$ and $N$ are the embedding and length of $C$ respectively. ${\|\hat{C}\|}_{*}$ is computed as:

$$\small{\|\hat{C}\|}_{*}=\sum_{i}\sigma_{i}$$

(5)

where $\sigma_{i}$ is the $i$-th the singular value of $\hat{C}$.

The learnt syntactic representations are word related, which are upsampled to phoneme level and concatenated with phoneme embeddings. Syntactic representation is copied to match the phoneme sequence length of current word. Tacotron 2 [4] is employed to generate spectrogram from the concatenated syntactic representations and phoneme embeddings, and Griffin-Lim [14] is further utilized to reconstruct the waveform. The whole model is trained with a loss function which can be formulated as:

$$\small\mathcal{L}=\mathcal{L}_{TTS}+\lambda\mathcal{L}_{NML}$$

(6)

where $\mathcal{L}_{TTS}$ is the loss function defined in Tacotron 2 and $\lambda$ is the loss weight for NML.

We train models on public Chinese female corpus [16], which includes 10-hour professional speech and 10000 sentences. 500 sentences are used for validation and other sentences are used for training. We down-sample the speech to 16k Hz sampling frequency, The tacotron 2 part in our model is trained with vanilla setups except setting frequency to match our speech. The learning rate is fixed to $10^{-3}$ and the loss weight of NML is 0.05.

We train the WRF based TTS [12] as baseline approach. The parser used in WRF [17] is replaced by state-of-the-art syntactic parsing model Benepar [18].

We program all the models based on an open sourced Tensorflow implemention of Tacotron 2 [19]. We train all the models for 50k iterations with a batch size of 16 on a NVIDIA 2080 Ti GPU.

We randomly select 30 sentences from Internet as test set, 5 of which are sentences with multiple different syntactic parse trees. Synthesized speeches are shifted in random order and rated by 20 native speakers on a scale from 1 to 5, from which a subjective mean opinion score (MOS) is calculated.

As show in Table.1, the proposed system receives a MOS of $3.82$ while the baseline approach receives a MOS of $3.70$, with a comparable variance. T-test reveals that our proposed approach significantly outperforms the baseline with a $p$ of $0.0079$.

We also conduct a ABX preference test between pairs of systems on the synthesized speech. The listeners are presented with the speeches synthesized by the baseline and proposed approaches in random order, and decide which one has the better prosody and naturalness. As show in Table.2, the proposed approach receives $45.8\%$ preference rate exceeding the baseline approach by $17\%$.

To visualize the contribution of NML, we train our model without NML with the same settings. Another ABX preference test is conducted on same test set and listeners. The listeners are presented with the speeches synthesized by our models with and without NML, and decide which one has the better prosody and naturalness. As show in Table.3, the approach with NML receives $58.3\%$ preference rate exceeding the approach without NML by $27\%$.

We visualize the learnt embeddings of constituent labels with and without NML by principal components analysis (PCA). As show in Fig.2, embeddings with NML are more scattered than the embeddings without NML, which demostrates the effectiveness of NML in improving discriminability and diversity.

We further conduct case studies by comparing the spectrogram and pitch contours of the speeches generated from different methods, as shown in Fig.3. The speech from baseline approach has an unexpected long pause between the 5-th character and 6-th character in such long sentence.

Besides, the word from the 16-th to 17-th character, “ji4 yi4” that are both of fourth tone in Chinese, is synthesized with an unnatural up-and-down tone in WRF. Instead, the spectrogram and pitch contour of the speech synthesized by our proposed method are more stable, indicating the stability of the proposed syntactic representations from bidirectional traversals. For the last three characters, there is an unexpected stress (high pitch value) on 23-th character “xia4” in WRF result, while proposed method shows a gradually decreasing pitch contour at the end of the sentence leading to higher naturalness. This is cause by the WRF features in baseline approach which consider a uni diridectional information in the syntactic parse tree.

With the proposed syntactic representation learning method, it is possible that a sentence with the same text but different syntactic parse trees might lead to synthesized speeches with different prosody expressions, which provides a possibility for prosody control of speech synthesis. To validate this, we conduct further experiments by inputting sentences with multiple different syntactic parse trees to the proposed model. One example is shown in Fig.4, from which the prosodic differences of the synthesized speeches can be clearly observed. Upper structure regards the first three characters as a word and the sentence means “White swan swims on the lake”, while below treats these characters as two words and the sentence means “During the day, goose swims on the lake”. Although the graphemes and phonemes are same, the meanings of the sentences are different with each tree. And the prosodic differences match the meanings respectively. Also the prosody of last five characters in either synthesized speech are similar since their corresponding syntactic structure information is similar.