TräumerAI: Dreaming Music with StyleGAN

We utilized a music auto-tagging model as a fixed music encoder, which is a short-chunk CNN with residual connection based on won2020eval (8), and trained the model with MagnaTagATune dataset law2009evaluation (9) and its top 50 tags. We used the output of the last CNN layer as an embedding of the audio.

For image generation, we used StyleGAN karras2019style (10) due to its capacity of generating high-resolution images of quality. Our system was made from a public PyTorch implementation²²2https://github.com/rosinality/stylegan2-pytorch of StyleGAN2 karras2020analyzing (11) and a pre-trained model that was trained with WikiArt Dataset³³3https://github.com/pbaylies/stylegan2.

Rather than establishing an objective metric between musical and visual semantics, we manually labeled the pairs in a subjective manner. One of the authors listened to 100 music clips of 10 seconds long and selected an image that suits the music among the 200 StyleGAN-generated examples. If the annotator could not find an appropriate image, he generated another 200 images in random or based on a single image from the previous step. The data covers various genres including classical, jazz, pop, ballad, R&B, new age, K-pop, J-pop, rock, electronic, hip hop, and trot.

During the process, the annotator considered his emotional impressions such as arousal and valence, genre and era, and timbral characteristic such as instrumentation. We intentionally avoided mapping vocal sounds to a portrait unless the vocal is strongly dominant or any other instruments does not appear, because portraits have relatively low variance on visual style or perceptive emotion.

Based on the collected data, we trained a simple transfer function that converts an audio embedding $z_{mu}$ to a style embedding $w_{st}$. This transfer function is similar to the one used for zero-shot learning between words and images in socher2013zero (12), with mean and deviation $\mu_{st}$, $\sigma_{st}$ of $w_{st}$ from random sampled $z$ of StyleGAN.

$$\mathcal{L}(w_{st},z_{mu})=\sum{\lvert w_{st}-(2\sigma_{st}\tanh(Wz_{mu}+b)+\mu_{st})\rvert}$$

(1)

The system takes audio input and extracts a sequence of audio embedding. The sequence is converted to a sequence of styles, which is then generated as a sequence of images by StyleGAN. During the experiment, linear interpolation between coarsely sampled sequence( 3 sec) results in abrupt acceleration. Therefore, we sampled the 30 audio embeddings per second so that each frame of video is generated from the corresponding audio embedding. Since our mapping conserves the temporal progress of embeddings, the progress of the video followed structural change of music. Also, smoothing with an averaging window is applied to the style sequence to prevent the generated images from changing too rapidly. The window size differs by style hierarchy, so that coarse, middle, and fine styles are smoothed with a window of 3 sec, 2 sec, and 0.3 sec, respectively.

The selected music and corresponding images during the annotation process are presented in Figure 2. The images were selected from random generation of StyleGAN without truncation to attempt to fully exploit its diversity in expression. Some images like No. 28 and No. 67 are extremely distant from other images in terms of style latent vector, as we did not take into account how extreme the image is in the style latent space. The annotation process took about 10 hours, and we did not modified any annotation after or during the process. To compensate these outliers, we used L1 loss and tanh non-linearity. To be clear, the 100 labeled music clips do not include any music of the artists who are selected for video demonstration.

The annotation process became time consuming when to cover extremely different genre like Korean trot. If an annotator limits the genre of music and style of image in certain level annotation process can become much faster. Therefore, We expect that other users can make their own version of audio-visual mapping by their preference in shorter time.

Figure 3 shows examples of generated images. Again, we encourage the readers to watch videos on https://jdasam.github.io/traeumerAI_demo/, since how the image is changed throughout the music is the main part of our contribution.

Figure 4 demonstrates how audio embeddings and its corresponding mapped style embeddings changes as music progresses, represented in 2D PCA. The presented result shows that our audio encoder extracts different audio embedding for each sub-genre of the music, and also that the mapped style vectors are conserving the tendency in certain level.