LaughNet: synthesizing laughter utterances from waveform silhouettes and a single laughter example

Figure 1 illustrates the waveform silhouette used by LaughNet. Waveform silhouettes are two-dimensional frame-based features extracted from a raw waveform using a series of overlapping windows. This is similar to how a spectrogram is extracted but instead of short-time Fourier transform (STFT), simple max and min pooling operations are used to obtain the ‘shape’ of the waveform as we want an abstract representation. The continuous min and max values are quantized into several bins to further abstract the representation. By changing the configurations of the pooling window and quantization, we can adjust the level of abstraction of the waveform silhouettes. A high-level abstract interface enables more intuitive control for humans but is less detail oriented, and vice versa for a low-level abstract interface.

Investigating different levels of abstraction is the main objective of this research, but as a preliminary study, two levels of quantization, as shown in Figure 1, are tested as the input features for laughter synthesis. Our study shares a similar motivation with that of Greshler et al. [11] but instead of changing a low sampling rate waveform to change the generated content, we use an engineered waveform silhouettes as the control interface.

To test our hypothesis, we used HiFi-GAN [12], a generative adversarial network (GAN)-based [13] vocoder [14] that transforms a mel-spectrogram into a raw waveform, as the basis of our laughter synthesis system. As HiFi-GAN is not autoregressive, it can generate waveforms efficiently and with high-fidelity, which enables us to quickly test our ideas. The main difference is that we use waveform silhouettes as the input of the generator instead of mel-spectrograms. Readers can refer to the original paper [12] for details. We only summarize the main components in this paper.

The general structure of LaughNet is illustrated in Figure 2. The generator (G) takes a frame-based silhouette and up-samples it to the designated sampling rate. G consists of many up-sampling layers and several multi-receptive field fusion modules that add features from residual blocks of different kernel sizes and dilation rates. The amount of parameters can be adjusted, but for LaughNet model, we simply use the $V1$ configuration described in the original study [12]. In a typical GAN-based fashion, G was trained alternatively with one or several discriminators in a minimax game. The same as HiFi-GAN, our model includes two types of discriminators: the multi-period discriminator which learns to discriminate real and fake segments by processing samples at a particular period; and the multiscale discriminator [15, 14] which learns to discriminate by processing samples at a down-sampling scale. To train G, both feature-matching loss [14] and mel-spectrogram loss [12] are used.

As high-quality conversational laughter data [16] is spontaneous and difficult to obtain, we decided to used acted laughter, which have clear emotion intents and more isolated, for our experiments. Specifically, we used video-game-development assets purchased from Unity Asset Store¹¹1https://assetstore.unity.com/. It is a common practice for small and independent game developers to use these assets in their games. Video game development is also a practical application scenario of the proposed system. The Laugh SFX package we purchased has laughter of several male and female actors. For training, eight utterances, four from male and four from female, were used as the targets. As these sound assets are not designed for research, information about speakers were not provided, we simply selected utterances with perceivable differences in voice characteristics as different targets. Another set of eight laughter utterances were selected as template, or ‘source’ utterances, the silhouettes of which were used as the inputs for synthesizing laughter utterances. Our laughter synthesis system, in a way, can be regarded as a very special voice conversion (VC) system [17], hence the use of similar terminologies. The experiment and the evaluation were also structured similar to that of VC.

As we need to train a laughter synthesis system with a single example, to achieve a stable and decent performance, we first train the model with a multi-speaker speech corpus. Specifically, we used 24.4 hours of speech from 100 speakers of the VCTK corpus [18]. Prior studies showed that transfer learning from speech can help improve the performance of a laughter-synthesis model [7].

We used 24-kHz waveforms for the experiments. A sliding window with 1024 samples in length and moving 256 points each step was used to extract silhouettes. Each frame is a two-dimensional feature vector consists of the min and max value within the window. The continuous values were then quantized into a set of values. We split them into equally distributed bins (linear) or using the $\mu$-law algorithm. Three versions of LaughNet were evaluated: linear 256-bin LN256, 8-bit $\mu$-law MU256, and 4-bit $\mu$-law MU016. The hyperparameters used in training were the same as those used for HiFi-GAN [12] which includes both a mel-spectrogram and a feature-matching losses in the generator optimization loss. Mel-spectrograms were extracted using the same sliding-window configuration as the waveform silhouette.

We first initialized these using VCTK speech data for 150000 steps with each training batch consisting of 16 6-second segments randomly extracted from training utterances. Waveforms were also randomly scaled by a factor $\lambda\in[0.3,1.0]$. This helps increase the variety of waveform silhouettes. We then fine-tuned the models to the target laughter for another 50000 steps. For evaluation, we extracted two 6-second segments from each source utterance at a random position to create a test set of 16 silhouettes. Even though the end goal is creating a system in which users can control the synthetic laughter by directly changing the input, we used silhouettes extracted from source laughter utterances for testing, as this was a preliminary study. Given this arrangement, our experiment was quite similar to voice conversion, as we essentially wanted to change the characteristics of the laughter while retaining the ‘shape’ of the source waveform. In summary, each version of LaughNet created 8 models by fine-tuning to the 8 target laughter utterances, which were used to generate new laughter utterances from 16 silhouettes extracted from source laughter utterances.

We evaluated three versions of LaughNet by conducting listening tests in which participants were asked to judge the quality and speaker similarity of the laughter samples. We gathered 16 participants, each did 8 sessions, in which 16 quality and 16 similarity questions were included. For the quality evaluation, listeners were presented with a natural laughter sample (NAT) of a target or a sample from one of the synthesis models and asked to judge quality. Figure 3a shows the results in which all of the models received low score with MU016 scoring slightly better than the other two. For the similarity evaluation, listeners were presented with two samples, one from a target and the other from one of the evaluated models. These included natural samples of the same target (NAT) and the natural laughter sample of a random speaker with a different sex than the target (CROSS). Figure 3b shows the results of the similarity evaluation, in which all three LaughNet models had lower scores than NAT but higher than CROSS. Out of all three LaughNet models, MU256 scored slightly better than the other two.

In summary, the synthesized laughter samples had low quality, but express a certain level of individuality as indicated from the similarity-evaluation results.

Since the purpose of this research was not simply synthesizing new laughter utterances but developing a novel control interface for synthesis systems, we need a method to evaluate the performance of waveform silhouettes as means of control. Table 1 shows the mean squared error (MSE) between the input silhouettes and silhouettes of the generated waveforms. More specifically, the error was calculated using the silhouette before quantization. Simply speaking, if we had an accuracy interface, then the error between the input and output silhouette would be minimal. Interestingly, the two models used $\mu$-law for quantization (MU256 and MU016) had the same error, which is lower than the model using the linear function for quantization (LN256). We can say that MU256 and MU016 have a more precise control interface than LN256.

While MSE provide some objective evaluations over the control accuracy, it is difficult for humans to understand. Figure 4 presents examples of input silhouettes and silhouettes of the synthesized laughter. Subjectively speaking, we can see that the input silhouette successfully guided the shape of the generated waveform. Furthermore, we performed simple manipulation operations (stretch the silhouettes vertical and horizontally) to test the feasibility to use waveform silhouette as a mean of control. These samples were not evaluated subjectively, but readers can listen to them in the associated web page²²2Laughter samples are available at https://nii-yamagishilab.github.io/sample-laughnet-waveform-silhouette/. It is important to note that the accuracy of the control needs to be put in the context of a synthesis system, meaning that while we want an accuracy control, we also want LaughNet to synthesize laughter with the characteristics of the training example. Balancing between the two objectives is a research topic that we want to explore more in the future. One direction to solve this problem is increasing the level of abstraction of the waveform silhouette and allow the synthesis model to automatically determine the details of the waveform