PAM: Prompting Audio-Language Models for Audio Quality Assessment

Audio quality. The term implies a variety of properties in various contexts. For this work, we consider audio to be high quality when the presence of artifacts and noise is imperceptible. For example, white noise, clipping, and other distortions. We did not consider non-speech audio as noise, such as sound events, music, reverb, echo, and in general all naturally occurring sounds.

The user can provide an audio file and text class of “the sound is clear and clean” and determine the audio-text similarly using the model. The similarity can be squashed between 0 and 1 and used as a score. Though this method is valid and used for multiple tasks (Liu et al., 2023a; Kreuk et al., 2022; Ghosh et al., 2023), we see prompting with just one class of “the sound is clear and clean” leads to a poor correlation with human perception and distortions across various tasks and distributions. One of the reasons this strategy does not work is due to linguistic ambiguity. Particularly, if the prompt is “the sound is clear and clean”, then depending on the context, the model can infer: (1) The sound is easy to understand, see, or hear, without any distortion, noise, or interference. (2) The sound is pure, crisp, and pleasant, without any harshness, dullness, or muddiness. This meaning is based on the definition of ‘clean’ as ‘having a pure, fresh, or smooth quality.’, or (3) The sound is honest, accurate, and truthful, without any deception, manipulation, or bias. This meaning is based on the definition of ‘clean’ as ‘free from dishonesty or corruption’.

To address this problem, we prompting a strategy that will minimize ambiguity in the latent space. This is achieved by using multiple prompts that force the model to make similarity calculations along the latent subspace of audio quality (e.g., “the sound is clear and clean” and “the sound is noisy and with artifacts”). This not only reduces the ambiguity but allows us to audio quality measurement as a binary classification problem, where the final score is between 0 to 1 and regarded as a relative similarity. The PAM computation is explained in Section 2.4

The choice of text prompts has an impact on the similarity and if not optimal leads to spurious correlates being measured rather than audio quality. The PAM score uses ‘opposite” text prompts of “the sound is clear and clean” and “the sound is noisy and with artifacts”. These prompts are chosen based on analysis of CLAP’s (Elizalde et al., 2023b) training data and tested across various setups, tasks, and distributions. Specifically, the text descriptions in training data are combined and a frequency count per word is obtained (unigram) after removing stop-words. Then, we computed BERT embeddings for each word and took cosine similarity between filtered words and the word “noise”. We use the top 10 words from the list to form the two text prompts. In the rest of the paper, PAM implies the usage of the above prompts.

However, to get more insight into the type of artifacts and noise, the prompts can be changed. That is the prompts can be designed for specific tasks and setups in mind. For example, in the definition of audio quality 2.1, reverb and echo are not considered as noise and PAM score does not degrade with Reverb addition 3. Therefore, our general PAM score cannot be used as a metric for the specific task of Acoustic Echo Cancellation (AEC) to measure echo suppression. Therefore, we can design attribute-specific prompts for audio quality outside of our definition.

The PAM computation is shown in Fig 2, right section. The user provides an audio file which is converted into a mel-spectrogram ($x\in\mathbb{R}^{T\times F}$) and passed to CLAP’s audio encoder to produce an audio embedding $v\in\mathbb{R}^{1\times d}$. In parallel, the two “opposing” prompts about quality (“the sound is clear and clean” and “the sound is noisy and with artifacts”) are tokenized and embedded using the text encoder to produce text embeddings $u\in\mathbb{R}^{d\times N}$. After projection into the multimodal space, the dot product is computed between the two embeddings, followed by softmax: $p_{\mathrm{h}}=\frac{e^{z_{\mathrm{h}}}}{\sum_{j=1}^{2}e^{z_{j}}}$, where $\mathrm{h}$ is the index of the prompt related to high quality, $z_{j}=u_{j}\cdot v$, $(\cdot)$ denotes the dot product, and $p\in\mathbb{R}^{1\times 2}$. The value of $p_{\mathrm{h}}\in[0,1]$ is the PAM score and informs about the quality of the audio.

An audio quality metric should degrade with the presence of distortions and artifacts in the audio. To verify this, we add common simulated distortions to audio sourced from a professionally recorded sound effect pack. The four types of distortions used are (1) Gaussian Noise with increasing standard deviation (2) Gaussian Noise addition with particular SNR (3) Tanh distortion (4) Mu Law compression. Lastly, we add Reverb, which by the definition in Section 2.1 is not considered an artifact or distortion. Figure 3 shows the effect of distortions on PAM score when tested on a professionally recorded sound effect pack. The PAM score degrades as the noise is added except for Reverb. For Reverb, the PAM score is fairly constant, i.e., changes from 0.76 to 0.81. While for others we see considerable degradation in PAM score. To check robustness across source distribution, we change the dataset from professionally recorded to AudioCaps (audio from YouTube videos containing sound events), MusiCaps (music tracks from YouTube), and LibriTTS (speech, audioboks). We see similar trends of PAM score degrading with the addition of distortions and consistent scores across Reverb. The details can be found in Appendix B.

Figure 2 shows two different prompting strategies that can be used to get a quality-related score. The figure on the left shows naive prompting and the figure on the right shows the opposite prompting strategy of PAM. The advantages of the opposite prompting setup and the limitations of the naive prompt are explained in Section 2.2. In this section, we perform experiments to compare two setups with human listening scores.

We use the NISQA (Non-Intrusive Speech Quality and TTS Naturalness Assessment) (Mittag et al., 2021) dataset to check the correlation between PAM, the single prompt strategy, and human perceptual evaluation. The NISQA Corpus includes more than 14,000 speech samples with simulated (e.g. codecs, packet-loss, background noise) and live (e.g. mobile phone, Zoom, Skype, WhatsApp) conditions. Each file is labeled with subjective ratings of the overall quality. We use simulated and live talk corpus from NISQA. The simulated corpus contains simulated distortions with speech samples from four different datasets and the live talk corpus contains recordings of real phone and VoIP calls. Unlike PAM, NISQA considers sounds events as noise, so human raters labelled the recordings as low quality. Therefore, we created a filtered NISQA set and applied four distortions: (1) white noise addition with a particular SNR (2) live talk on a laptop or smartphone (3) low bandpass filter (4) high bandpass filter. We check the correlation of the single prompt strategy and the opposite prompt strategy against the Mean Opinion Score (MOS) from human listeners. MOS is a numerical measure of the human-judged overall quality and it is the arithmetic mean of the ratings given by subjects on a predefined scale. We used the existing MOS numbers from NISQA. The Pearson Correlation Coefficient (PCC) measures linear correlation between two sets of data (Pearson, 1920) and it is shown in Figure 4. PCC ranges from -1 to 1, where -1 indicates a perfect negative correlation, 0 indicates no correlation, and 1 indicates a perfect positive correlation. We see that the single-prompt strategy does not correlate with MOS, the human perceptual evaluation. While PAM not only correlates, but achieves a PCC greater than 0.7 on (1) white noise distortion and (2) real-world talk recorded from laptops and smartphones.

An audio quality metric should give high scores to audio that is free from distortions. For example, professionally recorded and edited audio should achieve a higher PAM score compared to audio sourced from YouTube, which is generally recorded with handheld devices and may contain noise or distortions. To confirm this hypothesis we carried on the following setup. We compare PAM among three sets in Table 2. (1) AudioCaps dataset sourced from YouTube containing sound events (clapping, alarms, dog barking, etc). (2) MusicCaps data sourced from YouTube with additional filtering to retain high-quality and remove low-quality music recordings. (3) Professionally recorded audio containing sound events.

TTA generation models synthesize non-speech non-music audio (sounds) from text descriptions. Although there are established metrics available, evaluating the generation quality of these models is still an open research question.

Table 1 shows the evaluation of TTA with objective metrics from in literature (Liu et al., 2023a; Kreuk et al., 2022). These metrics do not consider any type of perceptual aspect and consist of a distance between the generated audio and a distribution from a reference set. The objective metrics for all the systems are in Appendix C. We use publicly available variants of AudioLDM (Liu et al., 2023a), AudioLDM2 (Liu et al., 2023b), AudioGen (Kreuk et al., 2022) and MelDiffusion (See Appendix for details C). We choose the variant of the model corresponding to the largest parameter count, because it usually correlates better with higher performance. The captions from the AudioCaps test set (747 captions) are used to generate audio from the above 4 models and their variants. Captions are textual descriptions of the sounds, i.e. “A drone is whirring followed by a crashing sound”.

We carry out a human listening experiment to compute the correlation between metrics and human perception. We randomly picked 100 captions and their corresponding generated audio from the test set. During the experiment, each participant was asked to rate each audio in terms of Overall Quality - OVL and Relation of audio to text caption - REL on a five-point Likert scale. The order of audios was fully randomized and each audio was rated by 10 participants. Raters were recruited using the Amazon Mechanical Turk platform. To ensure quality in the annotations, participants who consistently provided identical scores in every HIIT (e.g., all 1s) or who completed the task in less than 10 seconds were excluded.

Figure 5 summarizes the PCC between per-model metrics and OVL and REL respectively. PAM correlates correlates significantly better with human perception of quality (OVL and REL) than the task-specific metrics of KL softmax and KL sigmoid. The KL metric uses the CNN14 (Kong et al., 2020b) model to extract audio embeddings for the generated and reference set. The CNN14 model is trained to classify audio into different sound events and hence does well at recognizing the presence of sound events rather than overall quality. Also, a recent work (Liu et al., 2023b) observed that reference-free metrics like KL provide high scores when the generation model is trained on the same distribution data as the KL reference set. PAM is a no-reference metric so it does not have these drawbacks.

TTM generation models synthesize music based on text descriptions. Although objective performance metrics exist, evaluating the subjective quality of these models remains an open research question.

Subjective performance can be described in terms of Acoustic Quality (AQ), which measures whether the generated sound is free of noise and artifacts, and Musical Quality (MQ), which measures the quality of the musical composition and performance.

A commonly used reference set for evaluating TTM models is MusicCaps (Agostinelli et al., 2023), a music subset of AudioSet (Gemmeke et al., 2017) that contains rich text captions provided by musical experts. A recent work (Gui et al., 2023) used GPT-4 to derive AQ and MQ ratings for MusicCaps audio samples via text-analysis of the corresponding captions. Each MusicCaps song was assigned one AQ and one MQ label ”high”, ”medium”, or ”low”. If AQ or MQ could not be inferred from the caption text, the label ”not mentioned” was assigned. These text-derived labels were shown to correlate reasonably well with human perception (Gui et al., 2023). We compare these text-based AQ and MQ labels with an audio-only analysis via PAM. The results shown in Table 4 indicate similar trends of audio-only PAM and text-based analysis via GPT-4.

For a direct comparison with human perception, we calculate PAM on a set of real and generated music recordings. Subjective AQ and MQ labels were collected by authors in (Gui et al., 2023) as MOS scores from several human judges. The real samples were taken from the Free Music Archive (FMA) and MusicCaps. For TTM generation, publicly available variants of MusicLM (Agostinelli et al., 2023) and MusicGen (Copet et al., 2023), as well as Mubert (Mubert-Inc, 2023) were used.

Figure 6 shows PCC between the MOS ratings and PAM. For comparison, the (absolute) PCC of the Fréchet Audio Distance (FAD) is shown for two pretrained models. Recall that FAD requires a pretrained model to compute audio embeddings on a reference dataset. Here, we used MusCC, a set of studio quality music (Gui et al., 2023), as a reference for FAD. PAM performs competitively, outperforming the commonly used FAD-VGGish metric in all comparisons.

Table 3 shows the evaluation of TTM with objective metrics and the proposed PAM. The samples are generated using MusicCaps captions as prompts. The row in grey shows results for the original MusicCaps audio and constitutes an upper performance bound for objective metrics that use MusicCaps audio as a reference, including FD, FAD, and KL. However, as the quality of MusicCaps samples varies significantly (cf. Table 4) TTM models may outperform MusicCaps in perceptual quality, as PAM indicates for MusicGen-l. We observe similar trends for PCC results (Table 5) corresponding to Table 3.

Speech Synthesis involves creating artificial speech, either by converting text to speech (TTS) or altering existing speech to sound like a different speaker or style, known as voice conversion. In our study, we examine the effectiveness of PAM in the above two tasks. For TTS, A recent work (Alharthi et al., 2023) conducted human evaluation studies for different TTS systems. The study used StyleTTS (Li et al., 2022), MQTTS (Chen et al., 2023), and YourTTS (Casanova et al., 2022) to generate speech for 100 sentences from the LibriTTS dataset (Zen et al., 2019). Each generated sample was rated by 10 raters. We use this dataset and compare PAM with existing metrics. The absolute results are shown in Table 6 and the PCC correlation with human evaluation in Figure 7. On average, PAM correlates better with human perception of speech quality than existing metrics.

The second speech synthesis task we consider is Voice Conversion (VC), where the aim is to convert audio containing the original speech to audio containing the target speaker’s voice. For this, we use the VoiceMOS 2022 challenge dataset (Huang et al., 2022b), specifically the VCC subset. The VCC subset includes 3,002 utterances from 79 systems. We test PAM on this dataset and compare it with existing metrics of MOSNet (Lo et al., 2019), MOS-SSL (Cooper et al., 2022), and SpeechLMScore (Maiti et al., 2023). PAM performs worse than other speech-based finetuned metrics.

On both the setup of TTS and Voice Conversion, the literature metrics like MOSNet, and MOS-SSL were trained on the train split of data. Therefore, all the evaluation setup is in-distribution for the existing metrics. To check out-of-distribution performance, we consider an out-of-domain (OOD) subset of the VoiceMOS challenge. the OOD subset is sourced from the 2019 Blizzard Challenge (Wu et al., 2019), and contains 136 Chinese TTS samples. The PCC results of metrics are shown in Table 7. In the OOD setup, PAM correlates better than existing metrics that are not trained on the OOD data. This showcases the ability of PAM to be a zero-shot audio quality metric.

Overall, PAM can detect audio quality and distortions in generated speech. For speech tasks, it falls short of task-specific metric, where the generated speech is rated based on intelligibility or speaker characteristics. This is explained in the Section 6.

Noise and artifacts negatively impact perceived speech quality, e.g., in voice communication systems (Reddy et al., 2021b). Deep Noise Suppression (DNS) aims at enhancing speech quality by suppressing noise. MOS derived from listeners judging the output of a DNS model provides a subjective performance metric to develop or tune the model. Machine-Learning based blind MOS estimators such as DNS-MOS have shown to outperform existing objective metrics for estimating the speech quality of DNS models (Avila et al., 2019; Reddy et al., 2021a). We compute PAM on the output of models participating in the ICASSP 2021 DNS challenge (Reddy et al., 2021b) and compare it against a state-of-the-art DNS-MOS estimator (Reddy et al., 2021a). Unlike generative models, DNS involves removing unwanted signals, hence its perceptual quality is impacted both by the quality of the (desired) speech as well as the quality and suppression of noise. We hypothesise that estimating such multifaceted quality may benefit from more comprehensive prompts. As a proof of concept, we calculate PAM with two prompt averaging strategies (see appendix for details). Table 8 summarizes the results in terms of Spearman’s Rank Correlation Coefficient (SRCC) between the average human-labeled MOS of each tested DNS model and the average DNS-MOS or PAM. The SRCC indicates how well the ranking of the tested DNS models in terms of their subjective quality is preserved (Reddy et al., 2021a). PAM performs competitively compared to the state-of-the-art MOS estimator trained specifically for this task.