Global Rhythm Style Transfer Without Text Transcriptions

Figure 1 shows two utterances of the same word “Please call Stella”, with the phone segments marked on the $x$-axis. As shown, although the content is almost the same across the two utterances, each phone in the second utterance is repeated for more frames than in the first utterance. In this paper, we will measure rhythm as the number of frame-aligned repetitions of each phone symbol.

Formally, denote $\bm{X}(t)$ as the spectrogram of a speech utterance, where $t$ represents the frame index. Denote $\bm{S}$ as the vector of phone symbols contained in an utterance, which we also refer to as the content information. Denote $\bm{R}$ as a vector of the number of repetitions of each phone symbol in $\bm{S}$, which we also refer to as the rhythm information. Then $\bm{S}$ and $\bm{R}$ represent two information components in $\bm{X}(t)$. Therefore, the task of disentangling rhythm involves removing phone repetitions while retaining phone identities. Formally, we would like to derive a hidden representation $\tilde{\bm{Z}}$ from the speech $\bm{X}$, i.e. $\tilde{\bm{Z}}=f(\bm{X})$, according to the following objective

where $I(\cdot;\cdot)$ denotes mutual information. If the text transcriptions were available, this objective could be achieved by training an asynchronous sequence-to-sequence model, such as the Transformer (Vaswani et al., 2017), to predict the phone sequence from input speech. However, without the phonetic labels, it remains an unresolved challenge to uncover the phonetic units in an unsupervised manner.

Existing unsupervised rhythm disentanglement techniques seek to obscure the rhythm information by temporally resampling the speech sequence or its hidden representation sequence. One common resampling strategy, as proposed by Polyak & Wolf (2019), involves three steps. First, the input sequence is separated into segments of random lengths. Second, for each segment, a sampling rate is randomly drawn from some distribution. Finally, the corresponding segment is resampled at this sampling rate.

To gauge how much rhythm information the random resampling can remove, consider two hypothetical speech sequences, $\bm{X}(t)$ and $\bm{X}^{\prime}(t)$, with the same content information $\bm{S}$, but with different rhythm information $\bm{R}$ and $\bm{R}^{\prime}$ respectively. If random resampling is to reduce the rhythm information in each utterance, there must be a non-zero probability that the random resampling temporally aligns them (hence making them identical because their only difference is rhythm). Thus, the original rhythm information distinguishing the two utterances is removed. Formally, denote $\tilde{\bm{Z}}(t)$ and $\tilde{\bm{Z}}^{\prime}(t)$ as the resampled outputs of the two unaligned but otherwise identical sequences. Then a necessary condition for reduction in rhythm information is that

Higher probability removes more rhythm information. If the probability is zero, then $I(\bm{R};\tilde{\bm{Z}})$ will reach its upper bound, which is $I(\bm{R};\bm{X})$. If the probability is one, then $I(\bm{R};\tilde{\bm{Z}})$ would reach its lower bound, which is $I(\bm{R};\bm{S})$. Please refer to Appendix A for a more formal analysis.

Therefore, we argue that the above random resampling algorithm does not remove much rhythm information, because it has a low probability of aligning any utterance pairs with the same content but different rhythm patterns. Figure 1 is a counterexample, where the duration of each phone changes disproportionally. The vowel “ea” gets more stretched than the consonants. Consider the case where the entire sequence falls into one random segment in the first step of the resampling algorithm. Then, due to the disproportional stretch among the phones, it is impossible to simultaneously align all the phones by uniformly stretching or squeezing the two utterances. If the utterances are broken into multiple random segments, it is possible to achieve the alignment, but this requires the number of random segments is greater than or equal to the number of phones, with at least one segment boundary between each pair of phone boundaries, whose probability of occurring is extremely low. In short, to overcome the limitation of the existing resampling scheme, we need better ways to account for the disproportional variations in duration across different phones.

As shown in Figure 2, AutoPST adopts a similar autoencoder structure as AutoVC (Qian et al., 2019). The decoder aims to reconstruct speech based on the output of the random resampling module and the domain identifiers, e.g., the speaker identity. AutoVC has been shown effective in disentangling the speaker’s voice via its information bottleneck, but it does not disentangle pitch and rhythm.

Therefore, AutoPST introduces three changes. First, instead of spectrogram, AutoPST takes the 13-dimensional MFCC having little pitch information. Second, AutoPST introduces a novel resampling module guided by self-expressive representation learning (Bhati et al., 2020), which can overcome the challenges mentioned in the previous section. Finally, AutoPST adopts a two-stage training scheme to prevent leaking rhythm information.

Formally, denote $\bm{X}(t)$ as the speech spectrogram, $\bm{C}(t)$ as the input MFCC feature, and $D$ as the domain identifier. The reconstruction process is described as follows

where $\mathrm{Enc}$, $\mathrm{Res}$, $\mathrm{Dec}$ stand for the encoder, the resampling module and the decoder, respectively. Sections 4.2 to 4.4 will introduce the random resampling module, and Section 4.6 will discuss the training scheme.

Our resampling scheme capitalizes on the observation that the relatively steady segments in speech tend to have more flexible durations, such as the “ea” segment in Figure 1. We thus modify the self-expressive autoencoder (SEA) algorithm proposed by Bhati et al. (2020) into a similarity-based downsampling scheme. SEA derives a frame-level speech representation, which we denote as $\bm{A}(t)$, that contrastively promotes a high cosine similarity between frames that are similar, and a low cosine similarity between dissimilar frames. We then create a Gram matrix, $\mathsf{G}$ to record the cosine similarities between any frame pairs:

More details of the SEA algorithm can be found in Appendix B.1 and the original paper (Bhati et al., 2020).

As shown in the left panel of Figure 3(3(a)), our downsampling scheme for $\bm{Z}(t)$ involves two steps. First, we break $\bm{Z}(t)$ into consecutive segments, such that the cosine similarity of $\bm{A}(t)$ are high within each segment, and that the cosine similarity drop across the segment boundaries. Second, each segment is merged into one code by mean-pooling.

Formally, denote the $t_{m}$ as the left boundary for the $m$-th segment. Boundaries are sequentially determined. When all the boundaries up to $t_{m}$ are determined, the next boundary $t_{m+1}$ is set to $t$ if $t$ is the smallest time in $(t_{m},\infty)$ where the cosine similarity between $t$ and $t_{m}$ drops below a threshold:

$\tau(t)$ is a pre-defined threshold that can vary across $t$. Section 4.3 will discuss how to set the threshold. After all the segments are determined, each segment is reduced to one code by mean pooling, i.e.

Figure 3(3(a)) shows a toy example, where the input sequence of length four. The second and the third codes are very similar. Then with a proper choice of $\tau(t)$, the downsampling would divide the input sequence into three segments, and collapse each segment into one code by mean-pooling. Note that the threshold $\tau(t)$ governs how tolerant the algorithm is to dissimilarities. If $\tau(t)=1$, each code will be assigned to an individual segment, leading to no length reduction.

Figure 4 shows the segmentation result of the two utterances shown in Figure 1, where the vertical dashed lines denote the segment boundaries. We can see that despite their significant difference in length, the two utterances are broken into approximately equal number of segments and the segments have a high correspondence in terms of content. Since the downsampled output is obtained by mean-pooling each segment, we can expect that their downsampled output would be very similar and temporally-aligned, which implies that the necessary condition for rhythm information loss (Equation (3)) is approximately satisfied.

For any fixed threshold $\tau$ in Equation (6), there is a trade-off between rhythm disentanglement and content loss. The lower the $\tau$, the more rhythm information is removed, but the more content is lost as well. Ideally, during testing, we would like to set the threshold to $1$ to pass full content information to $\tilde{\bm{Z}}(t)$, and make the decoder ignore all the rhythm information in $\tilde{\bm{Z}}(t)$. This can be achieved with a randomized thresholding rule.

To see why, notice that if the decoder were to use the rhythm information in $\tilde{\bm{Z}}(t)$, it must know the value of $\tau$, because how the decoder (partially) recovers the rhythm information depends on how the rhythm information is collapsed, which is governed by $\tau$. However, the large variations in speech rate, utterance length and rhythm patterns in the training speech would overshadow the variations in $\tau$, making it extremely hard to estimate the value of $\tau$. Thus, the decoder will ignore whatever rhythm information remains in $\tilde{\bm{Z}}(t)$.

We adopt a double-randomized thresholding scheme. We first randomly draw a global variable $G\sim\mathcal{U}[u_{l},u_{r}]$ that is shared across the entire utterance, where $\mathcal{U}[u_{l},u_{r}]$ denotes the uniform distribution within the interval $[u_{l},u_{r}]$. Then to determine if time $t$ should be the next segment boundary (i.e., $t_{m+1}$ in Equation (6)), we draw a local variable $L(t)\sim\mathcal{U}[G-0.05,G+0.05]$. Then

$q\mathrm{-quantile}[\cdot]$ denotes taking the $q$-quantile, and b denotes the length of the sliding window within which the threshold is computed, which is set to 20 in our implementation.

The motivation for setting the two levels of randomization is that $G$ can obscure the global speech rate information and $L(t)$ can obscure the local fine-grained rhythm patterns.

To further obscure the rhythm information, we generalize our resampling module to accommodate upsampling. Just as downsampling aims to mostly shorten segments with higher similarity (hence decreasing the disproportionality), upsampling aims to mostly lengthen segments with higher similarity (hence increasing the disproportionality).

In the downsampling case, $\tau=1$ implies no length reduction at all. We thus seek to extrapolate the case to $\tau>1$, where the higher the $\tau$, the more the sequence gets lengthened. Our upsampling algorithm achieves this by inserting new codes in between adjacent codes. Specifically, suppose all the boundaries up to $t_{m}$ are determined. When $\tau(t)>1$, according to Equation (6), $t_{m+1}$ will definitely be set to $t$. In addition to this, we will add yet another sentence boundary to $t$, i.e. $t_{m+2}=t$, if

In other words, we are inserting an empty segment for the $(m+1)$-th segment (because $t_{m+1}=t_{m+2}$). During the mean-pooling stage, this empty segment will be mapped to the code at its left boundary, i.e.,

The non-empty segments will still be mean-pooled the same way as in Equation (7).

The left panel of Figure 3(3(b)) illustrates the upsampling process with the length-four toy example. Similar to the case of $\tau=1$, all the codes are individually segmented. The difference is that a new empty segment is inserted after the third code, which is where the cosine similarity is very high. At the mean-pooling stage, this empty segment turns into an additional code that copies the previous code.

To sum up, our resampling algorithm goes as follows.

$\bullet$ For each frame, a random threshold $\tau(t)$ is drawn from the distribution specified in Equation (8).

$\bullet$ If $\tau(t)<1$, the current frame would be either merged into the previous segment, or start a new segment, depending on whether its similarity with the previous segment exceeds the $\tau$, hence achieving downsampling (as elaborated in Section 4.2).

$\bullet$ If $\tau(t)\geq 1$, the current frame would form either one new segment or two new segments (by duplicating the current frame), depending on whether its similarity with the previous segment exceeds $1-\tau(t)$, hence achieving upsampling (as elaborated in Section 4.4).

$\bullet$ Move onto the next frame and repeat the previous steps.

Because the threshold for each frame is random, an utterance could be downsampled at some parts, while upsampled at others. This would ensure the rhythm information is sufficiently scrambled. As a final remark, the random threshold distribution (Equation (8)) is governed by the percentile of the similarity, because the percentile has a direct correspondence to the length of the utterance after resampling. Appendix D.1 provides a visualization of how different thresholds affect the length of the utterance after resampling.

Despite the resampling module, it is still possible for the encoder and decoder to find alternative ways to communicate the rhythm information that is robust against temporal resampling. Thus we introduce a two-stage training scheme to prevent any possible collusion.

The first stage of training, called the synchronous training, realigns $\tilde{\bm{Z}}(m)$ with $\bm{Z}(m)$, as shown in the right panels of Figure 3. Specifically, for the downsampling case, we copy each $\tilde{\bm{Z}}(m)$ to match the length of the original segment from which the code is mean-pooled; for the upsampling case, we delete the newly inserted $\tilde{\bm{Z}}(m)$. The network is then trained end-to-end to reconstruct the input with the realignment module, as shown in Figure 5(a). Since the decoder has full access to the rhythm information, the encoder will be trained to pass the content information and not the rhythm information. The second stage, called asynchronous training, removes the realignment module, freezes the encoder, and only updates the decoder, as shown in Figure 5(b).

Architecture The encoder consists of eight $1\times 5$ of convolution layers with group normalization (Wu & He, 2018). The encoder output dimension is set to four. The decoder is a Transformer with four encoder layers and four decoder layers. The spectrogram is converted back to waveform using a WaveNet vocoder (Oord et al., 2016). More hyper-parameters setting details can be found in Appendix C.

Dataset Our dataset is VCTK (Veaux et al., 2016), which consists of 44 hours of speech from 109 speakers. We use this dataset to perform the voice style transfer task, so the domain ID is the speaker ID. We use 24 speakers for training and follow the same train/test partition as in (Qian et al., 2020b). We select the two fastest speakers and two slowest speakers from the seen speakers for evaluating rhythm style transfer. For further evaluation, we select two other speaker pairs with smaller rhythm differences. The first pair consists of speakers whose speech rates are at 25% and 75% percentiles among all the test speakers; the second pair at 40% and 60%, respectively. More details can be found in Appendix C.

Baselines We introduce two baselines. The first is the F0-assisted AutoVC (Qian et al., 2020a), an autoencoder-based voice style transfer algorithm. For the second baseline, we replace the AutoPST’s random resampling module with that of the SpeechSplit (as introduced in Section 3.2). We refer to this baseline as RR (random resample).

For a fast-slow speaker pair and one of their parallel sentences, “One showing mainly red and yellow”, Figure 6 shows the spectrograms of conversions from the slow speaker to the fast (left panel), and from the fast speaker to the slow (right panel). The word alignment is marked on the $x$-axis. As shown, all the algorithms can change the voice to the target speaker, as indicated by the average F0 and formant frequencies. However, only AutoPST  significantly changes the rhythm towards the desired direction. AutoVC and RR barely change the rhythm. It is also worth noting that AutoPST indeed learns the disproportionality in duration changes across phones, most duration changes occur in the steady vowel segments, e.g., “ow” in “yellow”. This verifies that our similarity-based resampling scheme can effectively obscure the relative length of each phone.

To objectively measure the extent to which each algorithm modifies rhythm to match the style of the target speaker, for each test sentence and each fast-slow speaker pair (the test set is a parallel dataset), we generate a fast-to-slow and a slow-to-fast conversion. If an algorithm does not change the rhythm at all, then the fast-to-slow version should have a shorter duration than its slow-to-fast counterpart; in contrast, if an algorithm can sufficiently obscure and ignore the rhythm information in the source speech, then it can flip the ordering. We compute relative duration difference as

where $L_{\textrm{F2S}}$ and $L_{\textrm{S2F}}$ denote the lengths of fast-to-slow and slow-to-fast conversions, respectively. If the rhythm disentanglement is sufficient, this difference should be positive.

Figure 7(7(a)) shows the box plot of the relative duration differences across all test sentences and all the top four fastest-slowest speaker pairs. Figure 7(7(b)) and Figure 7(7(c)) show the results on speaker pairs with smaller rhythm differences (75% vs 25% and 60% vs 40% respectively). As shown, only AutoPST can achieve a positive average relative duration difference, which verifies its ability to disentangle rhythm. AutoVC gets the most negative relative duration differences, which is expected because it does not modify duration at all. RR achieves almost the same negative duration differences, which verifies that its rhythm disentanglement is insufficient.

To better evaluate the overall quality of prosody style transfer, and whether prosody style transfer improves the perceptual similarity to the target speaker, we performed a subjective evaluation on Amazon Mechanical Turk. Specifically, in each test unit, the subject first listens to two randomly ordered reference utterances from the source and target speaker respectively. Then the subject listens to a converted utterance from the source to the target speaker by one of the algorithms. Note that the content of the converted utterance is different from that in the reference utterances. Finally, the subject is asked to assign a score of 1-5 to describe the similarity to the target speaker in one of the three aspects: prosody similarity, timbre similarity, and overall similarity. A score of 5 means entirely like the target speaker; 1 means completely like the source speaker; 3 means somewhat between the two speakers. Each algorithm has 79 utterances, where each utterance is assigned to 5 subjects.

Table 1 shows the subjective similarity scores. As shown, AutoPST  has a significant advantage in terms of prosody similarity over the baselines, which further verifies that AutoPST can generate a prosody style that is perceived as similar to the target speaker. In terms of timbre similarity, AutoPST  performs on-par with AutoVC, and the gaps among the three algorithms are small because all three algorithms apply the same mechanism to disentangle timbre.

For the overall similarity, it is interesting to see how the subjects weigh the different aspects in their decisions. Specifically, although AutoVC can satisfactorily change the timbre, it is still perceived as only very slightly similar to the target speaker, because the prosody is not converted at all. In contrast, the AutoPST  results, with all aspects transferred, are perceptually much more similar to the target speaker. This result shows that prosody indeed plays an important role in characterizing a speaker’s style and should be adequately accounted for in speech style transfer.

So far, our experiments have mostly focused on the overall speech rate. One question we are interested in is whether AutoPST can recognize fine-grained rhythm patterns, or can only adjust speech rate globally. To study this question, we modify the utterance “One showing mainly red and yellow” in Figure 6(a) by stretching “yellow” by two times, creating an abnormal rhythm pattern. We then let RR and AutoPST  reconstruct the utterance from this abnormal input. If these algorithms can recognize fine-grained rhythm patterns, they should be able to restore the abnormality.

Figure 8 shows the reconstructed spectrogram from the abnormal input. As shown, RR attempts to reduce the overall duration, but it seems unable to reduce the abnormally long word “yellow” more than the other words. In contrast, AutoPST not only restores the overall length, but also largely restores the word “yellow” to its normal length. This shows that AutoPST can indeed capture the fine-grained rhythm patterns instead of blindly adjusting the speech rate.

Although AutoPST is designed for voice style transfer, we nevertheless also test AutoPST on the much harder non-parallel emotion style transfer to investigate the generalizability of our proposed framework. We use the EmoV-DB dataset (Adigwe et al., 2018), which contains acted expressive speech of five emotion categories (amused, sad, neutral, angry, and sleepy) from four speakers. During training, two emotion categories are randomly chosen for each speaker and held out, for the purpose of evaluating generalization to unseen emotion categories for each speaker.

Among the five emotions, neutral has the fastest speech rate and sleepy has the slowest. We thus follow the same step in Section 5.3 to compute the relative duration difference between fast-to-slow and slow-to-fast emotion conversions for each speaker. Figure 7(7(d)) shows the box plot. Consistent with the observations in Section 5.3, AutoPST can bring most of the relative duration differences to positive numbers, whereas the baselines cannot. This result shows that AutoPSTcan generalize to other domains. Additional results can be found in Appendix D.