Naturalization of Text by the Insertion of Pauses and Filler Words

Electronic systems that interact with humans are making their way into everyday life. From robots in the house to personal assistants on our phones, these systems aid us in completing both simple and complex tasks. Interfacing with these electronic systems has also evolved. The focus has shifted to voice-based interactions over the more traditional text-based interactions seen in legacy systemsvoice . Voice commands and feedback provide a more natural way for us to interface with these systems. However, even with the rapid development of the state of the art systems, we still largely interact with computerized voices. With this increase in voice-based interaction, we propose a set of methods to naturalize them. Naturalization of speech can be achieved in multiple ways. In this article, we transform text by adding pauses and filler words such as ’uh’ and ’um’ to an input sentence. Google’s Duplexduplex , can make calls on your behalf to book appointments or reserve tables and was unveiled in 2018. Despite being a computerized voice, pauses and filler words make the conversation more natural. Motivated by this observation, we propose the following methods. Moreover, these methods can be integrated as a module in a pipeline that takes a voice query as input and outputs a voice response. This module would take the response text from an information retrieval system as input and output naturalized text which can be input for a text-to-speech (TTS) module.

Previous cognitive studies of natural human speech indicate that the addition, deletion, modification of words, sentences of standard speech are observed in conversationscognition . These can be termed as disfluencies in speech. To achieve naturalization of speech, we resort to the addition of pauses and filler words such as ’uh’ and ’um’. These pauses and filler words are an important part of natural conversations and unconsciously end up as a part of them. In natural conversations, pauses and filler words are often used when a conversationalist is either emphasizing on a part of the conversation or gathering their thoughts. By inserting them in standard text, the text is transformed into a more natural text. An example

As observed, in a conversational context, the transformed text sounds more natural than the standard text. After studying the structure of conversations, we observe the frequency of these filler words in conversationsfreq and the roles they playdisc . Previously, in stolcke1 a language model is developed that predicts disfluencies probabilistically and uses an edited, fluent context to predict the following words. In stolcke2 , a combination of prosodic cues modeled by decision trees and word-based event N-gram language models automatically detects disfluencies. These models detect disfluencies such as pauses and filler words in speech. In local , an ”Editing Term” is inserted from the speech synthesis inventory and local prosodic modifications are applied at units adjacent to it. This suggests synthetic disfluent speech can be generated by making local changes. In sundaram , an empirical text transformation method is proposed for spontaneous speech synthesizers which are trained on transcribed lecture monologue. The novelty of this article includes (a) using transcribed movie dialogues as training data, (b) a hybrid approach in inserting pauses and filler words to increase confidence, (c) tunable degree of naturalization, (d) blind survey to assess the quality of the transformed text.

First, we discuss data preprocessing followed by the two proposed methods. The bigram method involves the storing of bigrams with a pause or a filler word and its corresponding frequency in the training data. During transformation, a probabilistic approach is adopted to insert an appropriate pause or filler word. The hybrid approach first transforms text using the bigram method. A sequence to sequence model tuned to the context is then employed to validate these insertions.

The closest imitation of natural human speech which is well recorded and structured is in the form of dialogues from movie scripts. We use the Movie Dialogue Corpus from Kagglecornell as well as the Baskin Engineering, Film Corpus 2.0ucsc . Combined, this forms a corpus of over $2.2\times 10^{6}$ lines of text. Data preprocessing (I) outputs the training data and the steps involved are

1. 1.

   Extracting the lines containing disfluencies such as pauses and ”uh”, ”um”

2. 2.

   Care is taken to identify and extract multiple variations of the above disfluencies such as ”uhh” and ”uhm”

3. 3.

   Two or more periods or dashes are also considered as pauses and are extracted forming an intermediate corpus

4. 4.

   Identified pauses are replaced by ”(pause)” and ”uh”, ”um” are replaced by ”(uh)”, ”(um)” to maintain uniformity in the final corpus

This forms a corpus of over $2\times 10^{5}$ lines and serves as the training data.

As there are multiple ways of achieving natural speech, we aim to transform the text into the most natural-sounding version. To illustrate this, following is an example

Therefore, we use a medium-sized pool of 275 odd individuals to evaluate the outputs and determine if they sound natural or not. We refer to them as respondents hereafter. The outputs are generated by both, the bigram and the hybrid method. Degree of naturalization is 9.5% for both methods. While answering the question — ’Which of the following sounds more natural?’, judging between two text excerpts is more difficult than judging between two voice clips of the same text excerpts. Hence, the respondents were presented with two voice clips. One was a performance of an original interview’s transcript and the other was a performance of the transformed text of the same interview’s transcript. Both were performed by the same voice actor. Here is an example of the output text from the bigram method followed by an example from the hybrid method

A ”Cannot Determine” choice was also included as this would indicate near-natural transformation by our methods. This was repeated twice per method with different interviews to test it’s robustness when the context of the input text was varied. Table 3 and 4 list results of each sample, each method. Figures 3 and 4 illustrate only the summary for each method.

If ”Generated” and ”Cannot Determine” together form the majority of responses ($\geq 50\%$), our methods can be said to effectively naturalize text. The ”Cannot Determine” choice indicates that the transformation was indiscernible from the original text. Counter-intuitively, the hybrid method performs not as well as the bigram method. However, this could be a result of the bias concerning the context of the sample interviews. The methods may perform poorly in certain contexts. We reiterate that the methods are highly sensitive to the context of the training data. As the size of the training data and the breadth of the context increase, the methods get more robust and consequently, the transformations more natural.

As stated previously, our methods of text transformation are fast. This is to ensure that this text transformation module can be integrated before a TTS module. Table 5 details the timing data in seconds for both methods and their averages. All runs are performed with equal computing power to maintain uniformity. All samples are long sentences and the degree of naturalization is 9.5% for both methods. Sentence length and degree of naturalization are directly proportional to the time taken to generate an output sentence. We observe that the bigram method is multiple factors faster than the hybrid method.

In this article, we propose a novel, fast transformation method that has tunable naturalization. It also uses a hybrid approach to confirm the transformations. We evaluate our methods using a medium-sized pool of individuals to evaluate the outputs and determine if they are natural. We conclude that our methods yield near-natural transformations of text and can be integrated as a module into a voice query to voice response pipeline.