Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment

We use ResNet-50 (He et al, 2016) for the image encoder. To effectively handle a wide range of visual contents, we train the image encoder in a self-supervised manner (Caron et al, 2020) using ImageNet (Deng et al, 2009) without relying on labels.

We utilize BigGAN (Brock et al, 2018) architecture to generate images with diverse visual scene contents. We adapt the input structure based on modifications from ICGAN (Casanova et al, 2021) to make BigGAN as a conditional generator. This setup allows us to train the generator to produce photo-realistic images at a resolution of $128\times 128$ using the conditional visual features $\mathbf{z^{V}}$ obtained from the image encoder. The generator is trained on ImageNet in a self-supervised manner without labels, while the image encoder is pre-trained and remains fixed.

We use ResNet-18, which takes the audio spectrogram as input. Following the final convolutional layer, adaptive average pooling aggregates the temporal-frequency information into a single vector. This pooled feature is then fed into a linear layer to produce the audio feature $\mathbf{z^{A}}$. The audio encoder is trained using either the VGGSound (Chen et al, 2020a) or VEGAS (Zhou et al, 2018) datasets, applying the loss defined in Eq. (2), according to each target benchmark.

We utilize the VGGSound (Chen et al, 2020a) and VEGAS (Zhou et al, 2018) datasets for training and testing our method. VGGSound contains approximately 200K videos, from which we chose 50 classes and follow the provided training and testing splits. VEGAS, on the other hand, includes about 2.8K videos with 10 classes. To maintain data balance, we use 800 videos for training and 50 videos for testing from each class. We use the test splits from both datasets for subsequent qualitative and quantitative evaluations.

We use both objective and subjective metrics to evaluate our method.

• •

  CLIP (Radford et al, 2021) retrieval : Inspired by the CLIP R-Precision metric (Park et al, 2021), we evaluate the generated images by conducting an image-to-text retrieval test, measuring recall at $K$ ($R@K$). We input the generated images along with audio category names (text) into CLIP, then measure the similarity between the image and text features to rank the text descriptions for each query image.

• •

  Fréchet Inception Distance (FID) (Heusel et al, 2017) and Inception Score (IS) (Salimans et al, 2016) : We use FID to measure the Fréchet distance between features obtained from real and synthesized images using a pre-trained Inception-V3 model (Szegedy et al, 2016). This same model is also used IS, computing the KL-divergence between the conditional and marginal class distributions.

• •

  Human evaluations : We recruit 70 participants to analyze the performance of our method from a human perception perspective. Participants are asked to compare our model with an image-conditioned generation model (Casanova et al, 2021) and to assess whether the images generated by our model accurately correspond to the input sounds. Further details are available in Sec. 4.3.

The audio encoder takes a $1004\times 257$-dimensional log-spectrogram, which is converted from 10 seconds of audio, to extract audio features. Simultaneously, the video frame is resized to $224\times 224$ and fed into the image encoder to extract visual features. We train our model on a single GeForce RTX 3090 for 50 epochs with early stopping. The Adam optimizer is used, with a batch size set at 64, a learning rate of $10^{-3}$, and a weight decay of $10^{-5}$.

Sound2Vision generates visually plausible images from a single input waveform, as shown in Fig.1 and 4. Unlike previous work (Fanzeres and Nadeu, 2021; Wan et al, 2019), it is not limited to a small number of categories but instead handles a diverse range, including animals, vehicles, sceneries, etc. We highlight that our model can even distinguish subtle differences in similar sound categories, such as “Waterfall Burbling” and “Sea Waves” sounds, and produces accurate and distinct images.

After presenting the results of Sound2Vision in generating a wide variety of images, we analyze which parts of the audio the model uses to generate images and whether it is attending to the true “semantic” context in the audio. With the trained Sound2Vision, we visualize a coarse localization map on the spectrogram, indicating areas the model highlights for image generation. Using the same technique of Gradient-weighted Class Activation Mapping (Grad-CAM) (Selvaraju et al, 2017), we compute the gradient of the generated image with respect to the feature map activation of the audio encoder’s last convolutional layer to produce the coarse localization map (Fig. 5). As shown, each highlighted moment is visualized in a heatmap, with the most emphasized regions transitioning from blue to red. For example, (a) shows that the model focuses on a certain region of the sound that is dominant throughout the duration, e.g., elk bulging, for generating an image. On the other hand, (b) demonstrates that the model focuses on both regions of skiing sound (wind blowing and footsteps on snow) and human sound, to produce a composite image.

We conduct a series of experiments to validate our proposed method on VGGSound and VEGAS, compared with several closely related baselines, as detailed in  Table 1 (a). First, our model (B) is compared with an image-to-image generation model identical to ICGAN(Casanova et al, 2021) (A). Although (B) shares the same image generator as (A), it utilizes a different encoder type and input modality. Results show that (B) performs favorably against (A) across all metrics. We attribute this to the noisy nature of video datasets, which may disturb (A) from extracting informative visual features for image generation, whereas audio input proves to be more robust to these disturbances, thus resulting in more plausible images. Additionally, we evaluate our model (B) against a retrieval system (C) that serves as a strong baseline. The retrieval system uses the same audio encoder as (B), while the image generator $G$ is replaced with the same memory-sized database of images from the training data. This system finds the closest image from the database, given the input audio feature. Compared to (D)—an upper bound where video frames are directly used— (C) shows a significantly smaller performance gap with (D) compared to the gap between (B) and (D), validating the effectiveness of our audio encoder in mapping audio to the shared embedding space. While (C) surpasses (B) in $R@1$ for both datasets, (B) performs comparably with (C) in $R@5$, demonstrating that our method approaches the performance of this strong baseline.

The user study results are summarized in Table 1 (b) across two experiments: (i) a comparison with ICGAN, and (ii) an evaluation of the plausibility of image generation from given audio. Each experiment consists of 20 questions where participants are presented with audio and multiple images. In (i), participants select images they believe best represent the audio, with two images generated by both our model and ICGAN, and the remaining images randomly chosen from either method. We measure preference by comparing the recall probability of ICGAN with our model. In (ii), all four provided images are generated by our model, but only one corresponds to the given audio. Participants then select the image they find most related to input audio. In (i), our model is preferred. Additionally, (ii) demonstrates that our method achieves a precision rate of 83.8%, supporting that our model generates images highly correlated with the provided audio.

The ablation studies conducted to assess various design choices are summarized in Table 2. We evaluate the performance of using different distillation losses: a straightforward $L_{2}$ loss between the visual and audio features, and an InfoNCE loss (Oord et al, 2018) with cosine similarity distance measurement, $L_{nce}$, rather than using $L_{2}$ distance as in  Eq. (2). The results of experiments (A), (B), and (F) indicate that our chosen loss (F) leads the model to produce more diverse and improved image quality. Furthermore, the comparison between (C) and (F) demonstrates that our audio-visual pair selection method, as discussed in Sec. 3.3, contributes to performance improvements. Lastly, we examine the effect of audio duration by training models with 1, 5, and 10 seconds of audio while keeping other experimental settings the same. The results from (D), (E), and (F) show that longer audio durations consistently enhance performance, likely because longer ones may capture more comprehensive semantics, whereas shorter durations may miss critical details.

As discussed in (Zhang et al, 2023), the cross-modal transferability is a phenomenon that allows the learned shared embedding space to make different modalities interchangeable for cross-modal tasks. According to Zhang et al., this intriguing phenomenon is enabled by the unique geometry of the modality gap:

• •

  The modality gap approximates a constant vector. This is verified by computing distributions over $\lVert\boldsymbol{g}\rVert$ (magnitude), where $\boldsymbol{g}=\mathbf{z^{V}}-\mathbf{z^{A}}$ is the modality gap between the paired visual and audio features.

• •

  The modality gap is orthogonal to the span of the features, and features have zero mean in the subspace orthogonal to the modality gap. We verify this by computing distributions over $\text{cos}(\mathbf{z^{V}}-\mathbb{E}_{\mathbf{z^{V}}}[\mathbf{z^{V}}],\mathbb{E}_{\boldsymbol{g}}[\boldsymbol{g}])$ (orthogonality) and $\mathbb{E}_{\mathbf{z^{V}}}[\mathbf{z^{V}}-(\mathbf{z^{V}})^{T}\boldsymbol{g^{\prime}}\boldsymbol{g^{\prime}}]_{i}$ (center), where $\boldsymbol{g^{\prime}}=\mathbb{E}_{\boldsymbol{g}}[\boldsymbol{g}]/\lVert\mathbb{E}_{\boldsymbol{g}}[\boldsymbol{g}]\rVert$ and $i$ denoting $i$-th dimension of each feature.

As demonstrated in Fig. 6 (a), we find that optimizing InfoNCE loss (Oord et al, 2018) with cosine similarity distance measurement, denoted as $L_{nce}$, results in a constant magnitude of the modality gap, while the orthogonality and centering values are near zero. This result supports the idea that the objective of our model preserves geometric properties that facilitate cross-modal transferability.

Although $L_{nce}$ objective has proven to be effective in learning aligned audio-visual features, we further explore how closing the multimodal gap affects sound-to-image generation performance. We compare different configurations of the loss function, from $L_{nce}$ to our final objective, $L_{total}$. Figure 6 (b) illustrates the geometry of the modality gap learned from $L_{total}$. The magnitude of the multimodal gap is significantly reduced compared to that using $L_{nce}$, while other properties, such as orthogonality and centering, remain close to zero.

Building on this, we analyze the relationship between the modality gap and sound-to-image generation performance. Figure 7 shows the t-SNE (Van der Maaten and Hinton, 2008) visualization of audio and visual features learned by different loss functions. While the features in Figure 7 (a), learned with $L_{nce}$, exhibit a noticeable gap in the latent space, the features in Figure 7 (b), learned with $L_{total}$, show less of a gap, with the two modalities overlapping. We observe significant improvements in overall metrics as we reduce the modality gap between audio-visual features, including quantitative metrics and the multimodal alignment measurement introduced in  (Goel et al, 2022). This analysis suggests that learning to closely align audio features with visual features, particularly by reducing the gap between them, is essential for cross-modal generative tasks to produce diverse and visually plausible images.

Humans are capable of estimating the rough distance or size of an object based on the volume of its associated sound. To verify whether our model also has a similar ability to understand volume differences, we experiment by both reducing and increasing the volume of the reference audio. Each modified audio waveform input is fed into our model with the same noise vector. As shown in Fig. 8, the objects in the generated images appear larger as the volume increases. Notably, in the “Water Flowing” example, volume adjustments result in images that depict varying strengths of water flow, while in the “Rail Transport” example, they illustrate a train appearing progressively closer in the scene. These observations indicate our model’s ability to not only recognize class-specific features but also understand the relationship between audio volume and visual changes. This suggests that visual supervision, rather than class labels, enables the model to capture such dynamic and meaningful audio-visual relationships.

We explore whether our model can reflect the presence of multiple sounds in a single generated image. To test this, we combine two distinct waveforms into one and input the mixed waveform into our model. As illustrated in Fig. 9, our model successfully generates images that capture the composite audio semantics. For example, mixing the “Skiing” sound with others results in images where elements such as a railroad or a bird emerge within a snowy scene. Similarly, when the “Hail” sound is mixed, the generated images show both a train and a bird appearing in a misty scene. The task of detecting multiple distinct sounds from a single combined audio input, known as audio source separation (Gao and Grauman, 2019), and visually representing them accurately within a single context is not trivial. Nevertheless, our findings indicate that our model is capable of achieving this to a certain extent.

In this experiment, we manipulate the input waveform by combining multiple waveforms and simultaneously adjusting the volume of each waveform. In Fig. 10, we combine the “Wind” sound with either the “Bird” or “Dog” sounds, adjusting their volumes. As the volume of the “wind” sound increases and the “bird” sound decreases, the bird appears smaller in the image, eventually getting covered by bushes. Similarly, as the “dog” sound grows louder, the generated image transitions to a close-up of a dog indoors as the “wind” sound fades. However, when the “wind” sound becomes dominant again, the scene shifts to a wider shot of the dog outdoors. These results demonstrate that our model can detect subtle changes in the audio and accurately reflect these variations in the generated images.

Given an image and a semantic condition from audio, our model can generate target images in a compositional manner. This requires understanding the semantic coupling between the given image content and the audio condition. This task is similar to the recent trend of compositional image retrieval. However, here, we aim to demonstrate the compositional image generation ability of our model using audio conditions. We use simple multimodal embedding space arithmetic for image and audio-conditioned image generation. Given an image and audio, we extract visual features $\mathbf{z^{V}}$ and audio features $\mathbf{z^{A}}$. We then interpolate these features in the latent space to obtain a new feature: $\mathbf{z^{new}}=\lambda\mathbf{z^{V}}+(1-\lambda)\mathbf{z^{A}}$, where $\lambda$ varies across examples. This new feature is fed into the image generator to synthesize an image. As shown in Fig. 11, this simple approach effectively incorporates the sound context into the given scene, such as adding parade-looking people with marching sounds, stylizing the given building image with hail or skiing sound, and adding various vehicles on the road with respective sounds.

We further use this approach to compare our method with specifically designed the sound-guided image manipulation approach (Lee et al, 2022) in Fig. 12. Although this task is not explicitly targeted by our model, it emerges as a natural outcome of our design. While the results from Lee et al. (Lee et al, 2022) maintain the overall content of the original image, they fail to insert objects that correspond to the sound. In contrast, our method successfully generates an image (nearly similar to the given one) by conditioning on both modalities, for instance, it adds an explosion and a tractor to the scene or makes the ocean view appear cloudy in response to hail sounds.

Here, we explore sound-guided image editing from a different perspective by manipulating inputs in the latent space. Utilizing GAN inversion techniques (Abdal et al, 2019; Richardson et al, 2021), we extract a visual feature $\mathbf{z^{V}_{inv}}$ and a corresponding noise vector $\mathbf{z^{N}_{inv}}$ for the reference image. We also change the volume of the associated audio to extract two distinct audio features in the embedding space, $\mathbf{z^{A}_{1}}$ and $\mathbf{z^{A}_{2}}$. We then adjust the visual feature by moving it in the direction of the difference between these two audio features, resulting in a new feature: $\mathbf{z^{new}}=\mathbf{z^{V}_{inv}}+\lambda(\mathbf{z^{A}_{1}}-\mathbf{z^{A}_{2}})$, thereby guiding the visual manipulation with audio cues. This new feature is fed into the image generator $G(\mathbf{z^{N}_{inv}},\mathbf{z^{new}})$, enabling us to edit the original image based on corresponding sounds. As shown in Fig. 13, simply by adjusting the volume and navigating through the latent space, we can modify visual elements such as the size of an explosion, the flow of a waterfall, or the intensity of ocean waves.

The family of diffusion models (Ho et al, 2020; Dhariwal and Nichol, 2021; Nichol and Dhariwal, 2021) aims to learn the underlying probabilistic model of the image data distribution $p(\mathbf{x})$. Building on these successes, LDM (Rombach et al, 2022) learns such a distribution within the latent space of the variational autoencoder, $p(\mathbf{z})$. This involves learning the reverse Markov process over a sequence of $T$ timesteps in the latent space. For each timestep $t=0,\dotsc,T$, the denoising function $\epsilon_{\theta}:R^{d}\rightarrow R^{d}$, where $d$ is the dimension of the latent, is trained to predict a denoised version of the perturbed $\mathbf{z_{t}}$ at timestep $t$, as $\epsilon_{\theta}(\mathbf{z_{t}},t)$. The corresponding objective can be simplify written as follows:

The variant of LDM that accepts conditional inputs formulates the conditional distribution as $p(\mathbf{z}|\mathbf{c})$, where $\mathbf{c}$ serves as the conditioning input for generation. This model integrates a cross-attention mechanism, allowing it to be conditioned with different modalities. This is done by modifying the objective function to integrate this conditioning as:

In our case, the LDM is initially trained to use the visual features extracted by the Vision Transformer (ViT) (Dosovitskiy et al, 2020) as the conditioning vector $\mathbf{c}$. Subsequently, we train the audio encoder to learn to align with these visual features, which are then used as the conditioning for sound-to-image generation.

Figure 14 shows the qualitative comparison between images generated by GAN image generator (denoted as GAN) and those produced using the Latent Diffusion Model image generator (denoted as LDM). With the same input audio from various categories, the results from LDM clearly demonstrate improved image quality with more fine-grained details. This enhancement is further supported by the quantitative evaluations summarized in Table 3. Using the LDM as an image generator shows enhanced performance across all datasets and evaluation metrics. Furthermore, compared to AudioToken (Yariv et al, 2023), which are also based on LDM and trained with VGGSound, our model shows favorable results in overall metrics.

We also compare both of our models with other prior arts, S2I (Fanzeres and Nadeu, 2021)¹¹1https://github.com/leofanzeres/s2i and Pedersoli et al. (Pedersoli et al, 2022)²²2https://github.com/ubc-vision/audio_manifold. Note that Pedersoli et al., although not targeted for sound-to-image generation task, utilizes a VQVAE-based model (Van Den Oord et al, 2017) for generating sound-to-depth or segmentation images. Despite our models’ ability to handle a broader range of in-the-wild audio, we adopt the same training setup as S2I by training our models and Pedersoli et al. with five categories from the VEGAS dataset for a fair comparison. As shown in Fig. 15, both of our models surpass all other methods. Additionally, our proposed models generate visually plausible images, whereas previous methods often fail to produce recognizable results. These outcomes highlights the effectiveness of our training approach, where learning visually enriched audio features combined with a robust image generator leads to superior performance.

The model with the LDM image generator can generate images from a broader range of sound categories that are unattainable with the GAN-based model. As described in (Sung-Bin et al, 2023), while VGGSound contains 310 categories, 50 categories among these are selected for training the model. The choice to use a subset of categories is because of the limitations of the GAN-based model’s generative power, which struggles to produce plausible images across all categories. Training on all categories results in degraded image quality and lower CLIP retrieval performance. However, with the LDM model, we demonstrate that the model can handle more diverse sound categories while maintaining the image quality and reasonable range of CLIP retrieval scores.

Figure 16 (a) shows the CLIP retrieval R@5 results for both the GAN and LDM models, demonstrating how performance varies as the number of training sound classes increases from 50 to 200. As shown, GAN shows a drastic performance drop in R@5, whereas the degradation in LDM follows a more gentle slope. Interestingly, the performance of training 100 classes in GAN is similar to that of training 150-200 classes in LDM. One of the reasons for this degradation in GAN is due to its weaker generative power, especially in human-related classes, a limitation also discussed in (Sung-Bin et al, 2023). As the number of sound classes increases, the likelihood of including audio-visual pairs related to human actions also increases, resulting in poorer CLIP R@5 on GAN.

In Fig. 16 (b), we also provide a qualitative comparison between GAN and LDM along with CLIP R@5 results for each category, where both models are trained with 200 classes. In classes related to animals, both GAN and LDM produce plausible images, resulting in similar R@5 scores. For instance, the “Dog Baying” and “Donkey Braying” classes show approximately a 16% degradation in CLIP R@5 from LDM to GAN, while the “Cuckoo Bird Calling” class shows comparable results. However, in human action-related classes, GAN generates unrecognizable images while LDM consistently produces high-quality images with identifiable human actions. Moreover, LDM outperforms on CLIP R@5, showing approximately a 50% drop from LDM to GAN.

Figure 17 presents additional results across new categories that the GAN-based model could not handle. LDM can generate images from various transportations and environmental sounds, as well as human-related sounds, accurately reflecting the gender or age range.

Similar to Sec. 5, the LDM version of our model also supports the capability of generating controllable outputs by manipulating inputs in both waveform and latent spaces to some extent. As demonstrated in Fig. 18, the model can respond to variations in the volume of the same audio, i.e., manipulation in the waveform space. For instance, increasing the volume results in a more forceful waterfall and a larger depiction of thunder. Figure 19 illustrates the results of image and audio conditioned image generation, which involves manipulation in the latent space. We specifically refer to (Liu et al, 2022) and employ compositional generation techniques to incorporate multiple concepts from both image and audio. As shown, the model successfully manipulates the given car image with several environmental sounds, such as a burbling stream and skiing sound. In other examples, the model successfully inserts sound sources into the given images. One clear observation, as also presented above, is that the quality of the generated images is higher than that of the GAN version (see Fig. 11). Additionally, it maintains the integrity of the given image better than the GAN version while manipulating it with the sound. However, this is primarily due to the superior image generation ability of the LDM. As all the examples and results in this section show, regardless of the image generator used, the alignment of the cross-modal signal is the key factor in achieving these capabilities, and our learning objective facilitates this.

Figure 20 shows faces generated from diverse types of input speech. Interestingly, without using explicit complex modeling to learn the relationship between speech and facial attributes, the model effectively distills rich visual information into audio features. It successfully extracts facial attributes from the input speech and accurately reflects them in the generated images, including ethnicity (first row), gender (second row), and age range (third row). Moreover, the model captures subtle cues from the speech, such as a vintage audio effect from old black-and-white films, and incorporates this cue by generating faces with grayish colors, mimicking the style of old black-and-white movies, as shown in the last row.

Since the original and generated faces have diverse head poses and are captured from different camera viewpoints, we facilitate comparison by reconstructing these into canonical 3D meshes, displayed in the third and fourth columns. Specifically, we feed both the original and generated images into MICA (Zielonka et al, 2022) to reconstruct 3D faces and use FFHQ-UV (Bai et al, 2023) to apply textures to the reconstructed faces. This visualization method allows us to clearly observe that the model can effectively generate faces from speech, resembling the appearances similar to the original faces.

Having trained one audio encoder on environmental in-the-wild sounds as described in Sec. 6.1 and another on a speech dataset as in Sec. 6.2, we are able to generate images that reflect both environmental and speech sounds in a single image. Given an audio feature from environmental sounds, $\mathbf{z^{A}_{E}}$, and another from speech, $\mathbf{z^{A}_{S}}$, we can interpolate between these features to define a composite audio feature, $\mathbf{z^{new}}=\lambda\mathbf{z^{A}_{E}}+(1-\lambda)\mathbf{z^{A}_{S}}$, where $\lambda$ varies across examples. This new audio feature is then fed into the LDM to generate images that incorporate both types of content. Figure 21 shows the images that integrate both signals into one image. The leftmost image is generated solely from environmental sound, the rightmost image from speech, and the center image reflects both interpolated signals. Interestingly, this simple interpolation in the latent space enables the placement of humans in diverse environments, such as snowy or cloudy scenes, while preserving their identity.