What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models

Much of the work on reasoning in vision MLLMs builds on the Visual Question Answering (VQA) (Antol et al., 2015), a captioning task that arguably requires a model to demonstrate complex reasoning capabilities. Modifications to the original VQA approach include using different metrics that better reflect real-world visual concepts (Kervadec et al., 2021) and visualization approaches that allow for finer-grained investigation of vision MLLMs’ reasoning capabilities (Jaunet et al., 2021). Recent evaluations on visual reasoning have shown that the representation of visual information in MLLMs is a bag of words, rather than an ordered representation, as evidenced by the inability of the models to answer any questions related to the order of the objects in the images (Yuksekgonul et al., 2022). Vision MLLMs also lack spatial reasoning capabilities when queried about the left-right location of objects in an image (Kamath et al., 2023). Another problem with vision MLLMs is their lack of understanding of relationships between objects, such as when they incorrectly assign human actions to animals, and vice-versa (Thrush et al., 2022). Research into visual reasoning capabilities in vision MLLMs is facilitated by the development of benchmarks, which have demonstrated some of their ongoing reasoning and hallucination issues (Fu et al., 2023; Fan et al., 2024). These include vision MLLMs performing worse than LLMs on instruction following (Zeng et al., 2023) and vision MLLMs exhibiting a lack of semantic grounding, which limits their ability to leverage textual relationships present in the LLM in the visual modality (Lu et al., 2024).

(Kervadec et al., 2019; Song et al., 2023) to facilitate reasoning on more fine-grained aspects of the data. While generating more samples is a sufficient solution for specific abilities, it requires having matching data samples from each modality. As a result, it becomes even harder to come up with sufficient training data for data-hungry deep-learning models. A well-aligned model would be able to use abilities gained in one modality in another without requiring any extra data. For example, if a model can converse about modes of transportation, it should be able to show similar reasoning capabilities given an image of a car.

In-context learning through prompts is used to facilitate and analyze reasoning in MLLMs (Zhao et al., 2023). Reasoning abilities typically improve when a model is forced to take specific steps through its prompt (Kojima et al., 2022; Yao et al., 2022) and prompt probing (including prompts related to visual, textual, and outside information) has been key in the understanding of MLLMs visual reasoning limitations (Qi et al., 2023). Such probing has shown that in MLLMs, the use of non-linguistic prompts can increase the risk of catastrophic forgetting (Wang et al., 2023), reducing any reasoning capabilities that they do exhibit. It also underlines the importance of incorporating outside knowledge in the testing paradigm when assessing reasoning (Marino et al., 2019) as a way to assess the contribution of how much text-based reasoning concepts in the language model are being leveraged in the MLLM. Typically multi-step training is employed to mitigate catastrophic forgetting, the second modality is aligned to the frozen LLM and then fine-tuned with LLM low-rank adaption (LoRA) (Alayrac et al., 2022; Ye et al., 2023).

Audio MLLMs are a more recent development than vision MLLMs (Deshmukh et al., 2023; Silva et al., 2023; Gong et al., 2023b; Tang et al., 2023, 2024, e.g.,). Arguably, the first audio MLLM was Pengi (Deshmukh et al., 2023), which frames all audio tasks as a text generation task in order to leverage the causal capabilities of the integrated LLM on the input audio prompt. It uses the hierarchical audio transformer HTS-AT (Chen et al., 2022) for its audio encoder, CLIP (Radford et al., 2021) for its text encoder, and applies contrastive pre-training with CLAP (Elizalde et al., 2023). While Deshmukh et al. (2023) evaluate Pengi on a range of open-ended tasks, such as audio captioning and audio question/answering, and closed tasks related to classification and retrieval, they do not evaluate Pengi’s reasoning capabilities. Some slightly more recent audio MLLMs that explore the concept of reasoning are Listen, Think, Understand (LTU) (Gong et al., 2023b) and SALMONN (Tang et al., 2023). Figure 1 provides a generalized overview of the architecture of these models.

LTU uses Audio Spectrogram Transformer (AST) (Gong et al., 2021) for its audio encoder and the LLaMA text encoder (Touvron et al., 2023) along with CAV-MAE (Gong et al., 2022) for contrastive pre-training. Like Pengi, LTU treats all audio as input for automatic audio captioning (AAC), from which it leverages the reasoning abilities of LLaMA to reason about the audio captions. They freeze AST and use Low-rank Adaptation (LoRA) (Hu et al., 2021) to force the model to condition on the audio captions and not rely simply on the language model, which helps to minimize hallucinations. In evaluating LTU’s reasoning capabilities, Gong et al. (2023b) are concerned with the model’s ability to “think”, which the authors argue is demonstrated by tasks where the model explains an audio caption, and “understand”, which they further argue is demonstrated by tasks where the model has to infer further action. Gong et al. (2023a) also released LTU-AS, which integrates a speech model.

SALMONN uses two audio encoders one from a speech model and one from a generic audio model, which gives it automatic speech recognition (ASR) capabilities, like LTU-AS. SALMONN feeds the output of Whisper’s speech encoder (Radford et al., 2023) and BEAT’s audio encoder (Chen et al., 2023a) for generic audio sound into Q-Former query transformer (Li et al., 2023) to generate audio tokens for input into Vicuna (Chiang et al., 2023). Tang et al. (2023) specifically evaluate the effect of fine-tuning on the reasoning tasks. Since most of their data consists of ASR and AAC instructions, the model tends to ignore the prompt and respond with one of these answers. To address this, activation tuning is applied, which is lowering the scaling factor of the LoRA method. As with Pengi and LTU, the reasoning tasks in SALMONN are performed on the text generated from the audio samples, either captions or transcribed speech. Evaluation of SALMONN revealed a similar phenomenon to those observed in visual MLLMs, where the model forgets some of the text-based commonsense knowledge available in the LLM.

Evaluations of reasoning in audio MLLMs, primarily with LTU and SALMONN, have examined and proposed ways to address the observation made in visual MLLM literature that MLLMs exhibit issues with instruction following compared to LLMs. The lack of semantic grounding, however, has not been evaluated and is the focus of this paper.

We used the pre-trained LTU model as our base model in these experiments. The LTU model was trained using existing audio datasets, their labels, descriptions, transcripts, and metadata as available. As described in Section 2.2, LTU generates text captions for an input sound. These captions can be utilized for classification by calculating the similarity between them and the expected labels. This similarity is then interpreted as a confidence score for each label. We adopted the methodology outlined in the LTU paper for this task, employing OpenAI’s text-embedding-ada-002 model to obtain text embeddings for both the labels and model outputs. For each sample, we computed the cosine similarity between the label embeddings and the model’s output or caption. These similarity scores were then used as confidence indicators for the corresponding label of the given data point.

The advantage of relying on the similarity between text embeddings is that it enables the matching of related captions with the correct label, even if the given label does not appear in the caption. However, a potential disadvantage is the loss of semantic information, which could lead to incorrect label matching due to the diverse information contained within the caption. In order to quantify this, we conducted the same experiments repeatedly, with the variance being 0.7%, a figure negligible in comparison to the changes in results—up to 20%—observed between different experiments, as demonstrated in Tables 2 and 1.

We ran this experiment on EDANSA (Çoban et al., 2022), a bioacoustics audio dataset collected in Alaska, which comprises 10,782 10-second samples, totaling 27 hours of audio. This multi-label dataset contains 28 distinct labels, organized in a hierarchical structure. We selected 12 labels that represent main events and have more than 400 samples in total, namely: Anthrophony, Aircraft, Biophony, Songbird, Bird, Grouse, Insect, Waterfowl, Geophony, Rainfall, Wind, and Silence. Since the LTU model does not use EDANSA, nor any other ecological soundscapes, in its training set, it is an ideal choice for testing LTU’s out-of-distribution performance. Furthermore, labels such as Geophony and Biophony are not common in LTU’s training set, allowing for better evaluation of the effect of fine-tuning.

While LTU has been shown to underperform compared to supervised approaches for classification tasks (Gong et al., 2023b), we are interested in its in-context classification capabilities using knowledge from the text domain. Therefore, we tested the original LTU model on EDANSA labels to establish a baseline performance. In this experiment, we employed the most effective prompt from the original LTU classification experiments: ‘write an audio caption describing the sound.’ We noted that the model often failed to output expected labels such as biophony due to a lack of examples in the training. Consequently, we devised alternative labels (Appendix B) that are synonymous and incorporated these into our experiments.

However, even if using similarity measures aids in classification, LTU’s open-ended captions may not always match expected labels, even after fine-tuning. This is a limitation compared to supervised models with fixed output labels, as the model might not use exact labels or might miss silent or secondary sound events. To address this, we included expected labels in the prompts, such as “Write an audio caption describing the sound. Could the sound be {list of comma separated labels}?".

Subsequently, we fine-tuned the LTU model on the EDANSA dataset using the suggested train, test, and validation split from the original paper. We employed LoRA to fine-tune the language model, and for the audio encoder, we experimented with full fine-tuning and only training the projection layer. With LoRA, we used a learning rate of 16, as suggested by the original LTU paper (Gong et al., 2023b), and also experimented with values of 2, 4, and 8. We opted for lower values based on research indicating that a lower learning rate enhances performance by preventing catastrophic forgetting (Tang et al., 2023). We used a batch size of 256, a LoRA alpha of 1, and a LoRA dropout of 0.05. We applied early stopping if validation did not improve after 10 epochs and used the checkpoint with the lowest validation score.

For these experiments, we primarily used the 13B version of the LTU model. The in-context learning experiments in this Section 3 were run with both 7B and 13B models, but we found the performance of the 13B model to be superior, and thus its performance is reported for all experiments.

To conduct these experiments, we utilized two NVIDIA A40 GPUs, each with 48GB of memory. Each fine-tuning run took approximately 3 hours, and we conducted a total of 46 runs. This amounted to 11.5 days of GPU time in total.

The results of our audio classification experiments are encapsulated in Table 1. As anticipated, the LTU model’s performance in classifying samples from the EDANSA dataset falls short when compared to the supervised model. The supervised approach achieves a mean AUC score across labels of 0.97, whereas the LTU model lags considerably behind with a mean AUC of 0.69. Upon training the audio projection layer (“Partially Fine-tuned"), which serves as the intermediary between the LLM and the audio encoder, the performance experiences a marginal increase of 0.03 points. This layer is instrumental in aligning audio embeddings with input tokens, and the modest improvement suggests that the LLM may necessitate additional training to effectively recognize out-of-distribution sounds. When we proceed to fully fine-tune both the audio encoder and the LLM (“Fully Fine-tuned"), the performance is enhanced by 0.09. This improvement suggests that the LLM is acquiring new audio concepts and refining the mapping from audio to text for novel classes.

Table 2 compares the performance of different prompt strategies with the fully fine-tuned LTU model. Interestingly, supplying potential labels in the prompt to steer the model towards outputting labels of interest does not enhance the average performance. While some labels exhibit a slight improvement, others deteriorate, indicating that the model does not accord substantial attention to the information presented in the prompt.

Our final experiment with prompt modifications, which involved incorporating descriptions of Grouse calls into the prompt, also failed to meaningfully change the results. Although the AUC score for Grouse classification decreased by 0.06, the F-1 score rose by 0.05 to 0.46. Note, however, this score is still significantly lower than the performance achieved by the supervised model. The F-1 score was selected by calculating the F-1 for all threshold values with 0.001 increments on the validation set and choosing the threshold that yielded the optimal F-1 score for the reported test set results.

We might expect that given the extensive training of the audio encoder, it would summarize the general properties of any audio input first as embeddings and then audio tokens. However, our results reveal a more complex interaction. The performance improvement observed when transitioning from partial to full fine-tuning suggests that the LLM is not merely processing the general audio properties encapsulated in the embeddings. Instead, it appears to focus on specific audio tokens that correspond to the related audio class, indicating that the LLM is mapping specific sound events to their associated words. This exposes a limitation. Ideally, the LLM should leverage the general properties in the embeddings to be able to reason with audio. However, its focus on specific audio tokens suggests it is not fully utilizing the information in the embeddings, potentially limiting its generalization capabilities. These findings underscore the challenges inherent in leveraging LLMs for in-context audio classification tasks, particularly when dealing with out-of-distribution sounds. They also highlight the potential benefits and limitations of various fine-tuning strategies and prompt modifications. While the LLM’s ability to map specific sound events to associated words is beneficial, it is also a limitation in generalization to a wider range of audio inputs. The minimal impact of text prompts further suggests a lack of direct linkage between audio and text representations within the LLM.

LLMs model semantic relationships and can answer questions requiring reasoning, such as hypernyms and synonyms. Hypernyms represent a type of semantic relationship where one term serves as a broader category encompassing a set of other terms. For instance, ‘fruit’ is a hypernym for ‘apple’ and ‘orange’. Synonyms represent another type of semantic relationship where two different terms share a similar meaning. We use this property of the hypernym and synonym relationships to construct the text prompt P3 from Table 3. For synonyms, we pose the question “Is concept similar to synonym?" which maps to the text prompt P1 from Table 3. We expect LLMs to be able to answer these questions correctly if they understand the relationship between these concepts. This is because the ability to correctly identify and use hypernym and synonym relationships is a key aspect of understanding semantic relationships in language, and is a strong indicator of a system’s ability to reason and generate meaningful responses in natural language processing tasks. The hypernym and synonym approaches can also be applied to the audio modality. If the model has learned the semantic relationship between the concepts from textual data, it should be able to transfer this understanding to audio data. Given the sound of a concept, the model should be able to identify it as a type of hypernym or as similar to a synonym, demonstrating its ability to reason. To test this, we pose a question using an audio file that represents a concept. The question is framed as “Is the sound of the object in this audio signal a type of hypernym?" for hypernyms, and “Is the sound of the object in this audio signal similar to synonym?" for synonyms. These question formats correspond to the audio prompts P4 and P2 from Table 3, respectively. For example, if we have an audio file of a songbird’s chirping, the question would be “Is the sound of the object in this audio signal a type of bird?", to which the answer should be yes. We also test a condition where a silent audio file is provided with the text prompt to assess whether the mere presence of audio changes the MLLM’s reasoning processes.

We have constructed a concise benchmark that comprises 12 concept words. Each word is associated with up to 4 hypernyms, 4 synonyms, and 4 unrelated terms, culminating in a total of 111 word relationships (see Appendix C for a full list). We expect the model to respond favorably to all prompts involving concept hypernyms and synonyms, while showing a negative response to pairs of concepts and unrelated terms. For each word, we employ 4 audio files, each repeated 4 times, resulting in 16 queries. Consequently, we have 16 outputs per word pairs.To interpret these outputs, we employ regular expressions to discern whether the model’s response aligns with the question, i.e., whether it is responding positively or negatively. A positive response, or an affirmation, is classified as a ’yes’ response. We then compute the ’yes’ rate, which is defined as the percentage of ’yes’ responses out of the total 16 responses for each word pair. Notably, a ’no’ response to an unrelated term is correct in our context and is thus flipped to ’yes’ for consistency in the ’yes’ rate, ensuring it accurately represents the model’s correct identifications of both related and unrelated terms. Our samples are carefully selected from the evaluation set of AudioSet. We specifically opt for samples that contain only the target label, deliberately excluding any other sound events. A full list of the audio files we used is available in Appendix D. We use the same compute resources described for experiment 1, however we only run models in inference mode, adding up to less than 30 hours of GPU time.

Figure 2 presents the results of our experiments on concept representations in an audio MLLM. In the similarity category, the text-only approach and the text with silent audio approach both showed perfect performance. This indicates that the model was able to perfectly identify synonyms in a text-only context and when paired with silent audio. However, when actual audio data was introduced, there was a slight decrease in performance. The AudioSet condition had a median yes rate of 1.0 but with a slightly wider IQR of 0.031, indicating a small amount of variability in the responses. The EDANSA condition showed a further decrease in performance, with a median yes rate of 1.0 but an even wider IQR of 0.093, suggesting greater variability in the model’s responses.

In the hierarchy category, the text-only approach and the text with silent audio approach again showed perfect performance. This demonstrates the model’s proficiency in identifying hypernyms in a text-only context and when paired with silent audio. However, the introduction of actual audio data resulted in a significant decrease in performance. The AudioSet condition had a median yes rate of 28.1% and an IQR of 0.234, indicating a substantial amount of variability in the responses. The performance further deteriorated with the EDANSA condition, which had a median yes rate of 0.0 and an IQR of 0.265625.

Our empirical findings show a distinct performance disparity in the Large Language Model (LLM) when tasked with answering questions pertaining to text-to-text versus audio-to-text relationships. In the similarity task, the model adeptly reasons about tokens, irrespective of their origin from audio captions or text, demonstrating its proficiency in identifying related words. However, a slight performance edge is observed in text-based tasks, indicating the model’s stronger affinity for its native modality. In contrast, the hierarchy task presents a more complex challenge. While the LLM effectively leverages its reasoning capabilities with text, its performance wanes when presented with sound, suggesting a lack of connections between audio and textual concepts. This limitation is echoed in the findings from Experiment 1, where the model excelled in tasks involving one-to-one mapping, akin to the similarity task, but struggled with tasks requiring a broader understanding of relationships between concepts, as in the hierarchy task. We also observed that the model’s performance is worse with EDANSA samples compared to AudioSet samples, indicating potential difficulties in processing and understanding out-of-distribution sound data.