Generating Diverse Vocal Bursts with StyleGAN2 and MEL-Spectrograms

The Emotion Generation Track (ExVo Generate) of the ICML Expressive Vocalizations Competition presents a unique opportunity to examine generative models and optimize their ability to create audio samples that resemble a novel dataset of non-verbal vocal bursts from (Cowen et al., 2022). The hope is that the resulting models from this endeavor can provide exciting insights into the interplay between emotion and audio.

We train a conditional variant of StyleGAN2 (Karras et al., 2020b) on various processed mel-spectrograms generated from the data and find substantial improvements in generated sample quality over the baseline described in (Baird et al., 2022).

While research on mel-spectrograms as a means to go between the audio and the visual domain is extensive, most work focuses on mapping mel-spectrograms to audio (using GANs or other techniques) and not the unconditional generation of the mel-spectrograms themselves (Kumar et al., 2019; Shen et al., 2018; Prenger et al., 2019; Wang et al., 2018). By combining the methods above with an additional network that maps text sequences to mel-spectrograms, state-of-the-art voice synthesis is possible (Tan et al., 2021).

However, as the first part of this process often depends on being provided a text sequence as input: unconditional generation of mel-spectrograms is as of yet still poorly understood. This area is crucial for the non-verbal domain, where we cannot provide complete text input to the model. Instead, the model must perform either completely unconditional or conditional generation with very little information (e.g., just being provided a specific emotion to generate).

The closest work examining the unconditional generation of non-verbal audio samples is Afsar et al. (2021), that uses an MSGGAN to generate realistic laughter from a relatively small dataset of laughter samples. Conditional generation of laughter using additional information such as gender and age is also briefly explored, and interpolations between genders demonstrate the possibility of learning a meaningful latent space.

At a high level, our model’s audio generation process consists of 4 steps:

1. 1.

   Preprocess the audio to improve sample quality (for all emotions).

2. 2.

   Convert the processed audio to square mel-spectrograms of dimension $128\times 128$ or $256\times 256$.

3. 3.

   Train StyleGAN2 on the mel-spectrograms and use it to generate new ones.

4. 4.

   Convert back the generated mel-spectrograms to audio.

The StyleGAN2 architecture learns to produce realistic mel-spectrograms in both the unconditional and conditional cases with a relatively little tuning. Audio samples from the best conditional model checkpoint are made available¹¹1https://bit.ly/39Ea0gk. The FAD (Fréchet Audio Distance), HEEP (Human-Evaluated Expression Precision) and $S_{GEN}$ values of the best model can be found in Table 1. Overall, we find that our StyleGAN2 based method significantly improves the generation of each emotion in terms of FAD and $S_{GEN}$ scores while HEEP is significantly improved for each emotion except Awe/Horror which are marginally worse.

Before continuing, we distinguish between visual FID (Heusel et al., 2017) and the evaluation metric mentioned in (Baird et al., 2022) which we designate as FAD (Fréchet Audio Distance), as it still examines the Fréchet distance of Gaussians but relies on the features from the last layer of a model trained on audio data instead of those from Inception Net (Szegedy et al., 2016).

HEEP on the other hand uses human evaluations of the submitted samples where each sample is assigned a rating for each emotion based on how intensely the sample represents that emotion. The resulting matrix $H$ is then compared to the matrix $T$ of one-hot encoded samples (i.e. 1 for the emotion the sample is supposed to represent and 0 for the others), using:

Finally, $S_{GEN}$ is an aggregate measure that combines FAD and HEEP:

Figure 3 compares FAD and FID as training progresses. Even after only a few epochs, the generator learns to produce mel-spectrograms that, to the untrained eye, resemble those obtained from the real samples. Then, for most configurations we experimented with, the lowest visual FID is achieved roughly halfway through training. Nonetheless, qualitatively, sample quality keeps increasing before eventually training breaks as we observe mode collapse for all models (similarly to the collapse observed in (Brock et al., 2019)). Visually, this progression can be observed in Figure 2.

To generate the samples provided for the competition, we ultimately select the conditional model at 128x128, sampled at a checkpoint slightly before mode collapse. We find, from manual inspection combined with FAD that it yields the highest quality samples. No truncation is used to improve the precision of generated samples.

While we do not report results for the models trained on 256x256 mel-spectrograms we find that even though they achieve decent results, the 128x128 equivalent was almost always better in terms of sample quality. Nonetheless, the higher resolution models also suffered from the same issue of mode collapse.

Generally, we find that StyleGAN2 is a suitable architecture for generating realistic mel-spectrograms as long as early-stopping or some manual equivalent is used with respect of FAD due to the presence of mode collapse.

While StyleGAN2 has generally been more resistant to mode collapse than other architectures the issue in this case is perhaps exacerbated by a relatively high resolution of images combined with a small dataset. Further work is needed to better understand the cause and methods to avoid/delay it.

As shown in Figure 3, even after FID starts increasing, FAD keeps improving (as well as our qualitative evaluation of the generated audio samples). This discrepancy seems to indicate two things. FAD is a better metric for evaluating the generator than FID and StyleGAN2 is able to learn domain-specific features that are ordinarily not required for visual learning tasks (or at least these features are not learned by InceptionNet). The difference could also potentially explain why it is hard to visually observe the quality of the mel-spectrograms increasing (after the initial bit of training) even though the quality of the generated audio is still improving.

The experiments with 256x256 mel-spectrograms illustrate a tradeoff that needs to be made between improving the inversion quality (a higher resolution mel-spectrogram will be inverted to higher quality audio) and the difficulty of the image-generation task (it is harder to learn to generate higher resolution images). While 128x128 seemed to offer a good middle ground (a not too significant loss in inversion quality with a resolution that StyleGAN2 has generally performed well on), it is possible that a lower or higher resolution would better optimize that tradeoff.

While this initial foray into unconditional, non-verbal audio generation is promising, multiple parts of this audio generation process can be further optimized.

Due to time constraints, little work was done in regards to hyperparameter exploration (amount of preprocessing, mel-spectrogram generation parameters, StyleGAN2 training parameters, etc.). A more principled search of these hyperparameters (especially the resolution of the images) as well as the addition of some form of extra regularization could significantly improve performance. For example (Tseng et al., 2021) finds that a regularization term that penalizes the discriminator for having different outputs for real/fake images helps postpone mode collapse and stabilize training. Otherwise, recent work has been done on optimizer related techniques to stabilize training of GANs. Exploring other optimizers than Adam such as ExtraAdam (Gidel et al., 2019) could also address mode collapse.

Additionally, instead of using the Griffin-Lim algorithm for mel-spectrogram inversion, taking advantage of a model such as MelGAN (Kumar et al., 2019) could improve inversion quality and speed. It would be particularly interesting to see if the two sets of networks could perhaps be trained jointly or if it’d be better to train them separately (we could also potentially get better inversion from low resolution mel-spectrograms using MelGAN).

While StyleGAN2 works well as a general purpose image generation framework, many of the optimizations it introduced are not designed with the domain of mel-spectrogram images in mind. Further domain-specific techniques (for example adapting the architecture to be able to generate rectangular images which are more suitable for audio) could bring about significant improvements in generation capabilities of mel-spectrograms.

Other than just improving generation capabilities, a principled investigation of the latent space of the trained GANs could yield interesting insights. It would be particularly relevant to attempt interpolation along different emotions. We could then perhaps answer questions such as: what sound is halfway between horror and awe?