A Deep-Bayesian Framework for Adaptive Speech Duration Modification

Let $X\in\mathbb{R}^{D\times T_{s}}$ denote the input speech. In this work, $X$ corresponds to the filter-bank energies, where $D$ is the number of filter-banks, and $T_{s}$ is the number of temporal frames in the utterance. Similarly, we denote target speech as $Y\in\mathbb{R}^{D\times T}$. Notice that the target utterance length $T$ may differ from $T_{s}$.

Our generative process for the target speech is as follows:

where $T$ is the length of the target utterance, and $Y_{t}$ is the target features at time $t$. The parameters $\{T^{0},b_{T},Y_{t}^{0},b_{y}\}$ of the distributions are unknown and we implicitly estimate them via a deep neural network parameterized by $\gamma$ and $\theta$.

By treating the unknown parameters as functions of the input $X$, we obtain the following estimating equations for the target sequence length and frame-wise filter-bank energies:

The functions $f_{\gamma}(\cdot)$ and $f_{\theta}(\cdot,\cdot)$ correspond to deep networks. The variable $A_{t}\in\mathbb{R}^{T_{s}}$ is an attention vector that combines frame-wise features of the source utterance $X$ to generate the target frame $\hat{Y}_{t}$. Notice that the residual, which cannot be explained by the input utterance, depends on the predictions $\hat{Y}_{0:t-1}$ at previous time steps. This autoregressive property allows the neural network to learn a time-varying component that can differentiate between the speakers or emotions.

During training, we use paired data $(X,Y)$ to maximize the likelihood of the target speech signal with respect to the neural network weights $\{\theta,\gamma\}$. This likelihood can be written as:

where, the second term of Eq. (7) can be expanded as follows:

The variable $M$ in Eq. (8) denotes the attention mask and is introduced for convenience; it is a deterministic function of the source speech length $T_{s}$ and the estimated target length $\hat{T}$.

We use a variational free energy formulation [23] to derive an upper bound to our data log-likelihood (see supplemental materials for complete derivation). This bound can be translated into the following neural network loss function:

Here, $\lambda_{1}$ and $\lambda_{2}$ are model hyperparameters and implicitly contain the variances of Laplace distributions in Eq. (1). The distribution $q_{\theta}$ is a variational distribution which is approximated by the fully convolutional neural network in Fig. 3.

The mask $M$ is used to constrain the scope of the attention mechanism to be similar in time-scale to the input. This procedure is important for two reasons. From a speech quality perspective, large swings in speaking rate may generate unintelligible speech. From an estimation perspective, the utterances contains hundreds (sometimes thousands) of frames. It is difficult to robustly train a deep network to generate such long attention vectors using smaller datasets.

We use the masks derived from Itakura parallelogram [24], as illustrated in Fig. 2. The Itakura parallelogram is used to speedup DTW algorithms when the speaking rates in the source and target utterances are expected to be similar [24]. The slope of the Itakura parallelogram specifies the minimum and maximum speaking rates that the reconstructed utterances are allowed to possess in comparison to the input speech.

We adapt the neural network architecture from [25] by adding skip connections to the last layer and changing the configuration of the attention module. Fig. 3 shows the encoder, decoder and the new attention module of the convolutional neural network. The encoder is responsible for generating feature embeddings for the decoder and for predicting the relative length of target speech. The sample operation in Fig. 3 is responsible for generating a sample from the attention distribution required for reconstruction and backpropagation.

We train our model using the Adam optimizer [26] with a fixed learning rate of $10^{-4}$. The input $X$ is an 80-dimensional vector of Mel-filterbank energies. The projection layer expands this input to $256$ dimensions. Both the encoder and decoder consist of $10$ convolutional layers, each followed by gated linear unit. We use data augmentation to stabilize the network. Specifically, we reverse the input-output sequences and randomly extract intervals (with probability $0.5$) from the full utterance. Our full model training procedure is described in the supplementary materials. The source code can be download from: https://engineering.jhu.edu/nsa/links/.

Our final step is to use the attention map produced by the decoder as a proxy for the DTW similarity matrix between the source and target speech frames. Effectively, we use the robust dynamic programming operation to get a path of alignment within the mask boundary, rather than rely on the noisy spectral reconstruction (see Algorithm 1). To avoid skipping phonemes, the path is constrained to take at most one horizontal or vertical step consecutively while backtracking. We finally use this alignment as a lookup table to synthesize the target speech from the input via the WORLD vocoder [27].

CMU-ARCTIC has 4 American English speakers (two male, two female), who are paired according to gender for voice conversion. We train our duration modification framework using $2164$ utterances from the database and use the remaining $100$ utterances for testing the open-loop modification properties.

VESUS is an emotional speech corpus containing $250$ phrases read by $10$ speakers in $4$ emotion classes: neutral, angry, happy, and sad. VESUS also contains crowd-sourced emotional annotations. Here, we primarily use those utterances that are correctly annotated by at least half of the listeners.

We train three duration models corresponding to the three neutral-emotional pairs. This results in the following split:

• •

  Neutral to Angry Conversion: 2385 utterances for training, 72 for validation and, 61 for testing.

• •

  Neutral to Happy Conversion: 2431 utterances for training, 43 for validation and, 43 for testing.

• •

  Neutral to Sad Conversion: 2371 utterances for training, 75 for validation and, 63 for testing.

Given the small sample size due to shorter sequences, we fine-tune the model trained on CMU-ARCTIC for each emotion conversion task in lieu of training the networks from scratch.

As seen in Fig. 3, we use the encoder embeddings to predict the length of the target utterance as a ratio of the source utterance length. Fig. 4 shows the error in predicting this ratio in a ms/sec. Notice that our framework mispredicts the utterance lengths by only $40$ms/sec and $65$ms/sec on CMU-ARCTIC and VESUS, respectively. Duration prediction is particularly challenging on VESUS due to marked differences between neutral and emotional utterances. However, our framework performs well even in this challenging scenario, likely due to our fusion of deep representation and Bayesian regularization.

Next, we compare the alignment between source and target speech frames using our method with the original DTW algorithm. Recall that DTW requires access to the target speech utterance, whereas our approach does not. To compare the warping paths, we code the horizontal, diagonal, and vertical moves of the backtracking procedure into three classes. We then compute the edit distance between the DTW alignment and the attention map based alignment. Fig. 5 illustrates the match ratio normalized by the average length. As seen, the match ratio varies between $0.70$ and $0.85$, which suggests that our approach captures the general characteristics of an unseen target utterance. To our knowledge, this is the first demonstration of an adaptive duration modification framework.

Fig. 7 shows the effect of modifying the slope of the Itakura parallelogram and the horizontal/vertical movement constraint during DTW. As expected, relaxing the slope constraint and increasing the number of horizontal/vertical moves provide more flexibility in adjusting the speaking rate of generated speech. However, this flexibility can lead to missing or distorted phonemes, suggesting a trade-off between changing the speaking rhythm and preserving naturalness. Our framework allows the user to tune these knobs for their own application.

Finally, we crowd source the mean opinion score (MOS) for the re-synthesized speech in the test set using Amazon mechanical turk (AMT). As seen in Fig. 6, our framework achieves an average MOS between $3.7-4.0$ across the four tasks. This performance is at par with state-of-the-art neural vocoders trained on hundreds of hours of speech. We note that CMU-ARCTIC task has the lowest MOS, perhaps due to the longer and more complex utterances. Interestingly, the MOS is unaffected by errors in length prediction, as evidenced by the VESUS neutral-angry emotion conversion task. This suggests that our approach of combining the neural network attention weights with a structured DTW algorithm provides robustness to both the speech characteristics and estimation errors.

We use a convolutional neural network to predict the length and target speech frame using the following expression:

We maximize the log likelihood of the observed data $(X,Y)$ to estimate the weights of the neural network denoted by $\{\theta,\gamma\}$. This data likelihood can be written as:

By expanding the second term of Eq. (7) and using the conditional independece from the graphical model, we have:

The attention mask $M$ is a deterministically constructed from the source speech length $T_{s}$ and the estimated target length $\hat{T}$.

In this work, we encode the attention $A_{t}$ as a one-hot vector across the $T_{s}$ input frames of the source speech. Therefore, it follows a multinomial distribution. For simplicity, we model $A_{t}$ as conditionally independent of the utterance length $T$ given the mask $M$ and the input $X$. Specifically, taking the $\log(\cdot)$ of Eq. (7) and combining with Eq. (8) yields:

The distribution $q_{\theta}(\cdot)$ above is an approximating distribution for the attention vectors implemented by a convolutional network. The first inequality uses the convexity of the $-\log$ function, and the second inequality comes from the fact that entropy $H(q_{\theta})\geq 0$. Notice that we have implicitly assumed $P(A_{t}|X,\hat{Y}_{0:t-1},M)$ has a uniform distribution over the masked region. This is a reasonable assumption given that the masking process reduces the attention domain to a small region. However, the form of $q_{\theta}$ is not penalized for deviating from this uniform distribution during training. This flexibility allows the network to learn realistic attention vectors during autoregressive decoding. Eq. (9) can be easily translated into a neural network loss function which we minimize for $\{\theta,\gamma\}$:

$\lambda_{1}$ and $\lambda_{2}$ are the model hyperparameters that adjusts the trade-off between the two objectives and implicitly contain the variances of the Laplace distributions introduced in the main text. Notice that the loss in Eq. (10) computes an expectation over the attention maps. We use the Monte-Carlo estimate by sampling from the attention map at each time-step. The training procedure is therefore stochastic in nature due to this random sampling. We mix this stochastic version with the maximum aposteriori estimate (MAP) of the attention vector with probability of 0.1 in the beginning of training procedure.

We start with a small threshold in line 8 (i.e., low contribution of the stochastic loss) to prevent the model from diverging in sub-optimal directions. The MAP estimate helps in this regard. Once, the number of training epochs exceed a fix value, we increase threshold to place more emphasis on the stochastic loss. Empirically, we found this to be extremely helpful in generating monotonic attention. We fixed the slope of attention mask in line 5 to $1.25$ based on the relative difference in length observed in the training datasets.