The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction

This year’s challenge tracks included MOS prediction for “zoomed-in” systems, MOS prediction for singing voices, and semi-supervised MOS prediction for noisy, clean, and enhanced speech. Table 1 summarizes the datasets used in each track. The details are described next.

This year’s challenge was held on CodaBench¹¹1https://www.codabench.org/competitions/2650/, an advanced version of the previous CodaLab platform. It is an open-source web-based platform for machine learning competitions and reproducible research. The participants could choose to participate in any track as long as they submitted their answers in the evaluation phase.

To promote research reproducibility, we prohibited the use of in-house datasets unless the participants were willing to (1) publicize the datasets after the challenge, or (2) open-source their system, along with any ready-to-use model checkpoints. In addition, to enforce a complete semi-supervised setting for track 3, we further limited participants to use only the labeled data ($\langle$speech, subjective rating$\rangle$ pairs) provided by the organizers. This implies that other off-the-shelf subjective speech quality estimators (including but not limited to PESQ, STOI, and any other open-sourced data-driven quality estimators, such as SSL-MOS [22]) could not be used for system development and training.

In the training phase, we directly provided links for downloading the training and validation sets, including the speech samples and ratings. Track 1 was an exception, as we only provided instructions for obtaining the BVCC dataset. We also provided the set of sample IDs for the validation set. Participants were about to upload their validation results to the system and submit their scores to the leaderboard, such that they could get to know how their systems performed and see the results of other teams’ submissions.

For the evaluation phase, as all track 1 and 2 samples were accessible in the training phase, we released only the sample IDs in the evaluation set. For track 3, we provided both the samples and the IDs. Participants were allowed to make up to ten submissions, and the leaderboard was only used for sanity checking on whether the sample IDs and samples matched. After completing the analysis of the submissions from participants, we returned to the participants (1) the results, (2) their unique team ID, and (3) the ground-truth labels to the evaluation set of each track.

For tracks 1 and 2, following previous VMCs, we computed both utterance-level and system-level mean squared error (MSE), linear correlation coefficient (LCC), Spearman rank correlation coefficient (SRCC), and Kendall’s Tau rank correlation coefficient (KTAU). Our primary metric was system-level SRCC with KTAU as a secondary metric because, in a real-life MOS prediction task, we mainly want to know the rankings of the systems under consideration. For track 3, the utterance-level correlation is selected as the evaluation metric, following prior works on non-intrusive speech quality prediction models in denoising tasks [25, 26]. We calculated the utterance-level LCC of SIG, BAK, and OVRL. We considered a “system” as a unique combination of speech enhancement method, SNR, and noise type.

Table 3 shows the results from the baseline and all participating teams, and the bar plot of the main metric, system-level SRCC, is shown in Figure 1. First, we found that given only the original BVCC labels, most systems (including the baseline) struggled to rank the systems in the 12% zoom-in rate, except for T03 and T06. This result shows that the 12% zoom-in rate labels are indeed harder to predict compared to 25%. The baseline ranked 4th and 6th w.r.t. the 25% and 12% zoom-in rates, showing that participants have indeed advanced the prediction of zoomed-in systems. Remarkably, T05 and T06 were the top two systems in all eight metrics w.r.t. both 25% and 12% zoom-in rates.

For track 2, Table 5 shows the results from the baseline and all participating teams, and the bar plot of the main metric, system-level SRCC, is shown in Figure 2. If we only look at the system-level SRCC, unfortunately, no participant outperformed the baseline. Nonetheless, the baseline ranked first in only the system-level SRCC and system-level KTAU, and the T06 system ranked first in all four utterance-level metrics. Moreover, we observed that in Table 5, the difference between the worst system and the top system in each metric was in fact small. This indicates that the participants could not identify an effective method for this track.

The overall evaluation results of track 3 are presented in Table 6. Among the three metrics, SIG is the most difficult to predict, as indicated by lower correlation scores compared to BAK and OVRL. Based on the submitted systems, we noticed that all systems incorporated large pretrained models, either SSL models (T03, T04, T06) or Whisper (T01). Moreover, predicting SIG, BAK, and OVRL with very limited labeled training utterances is quite challenging, especially when dealing with unseen scenarios. In this context, directly fine-tuning SSL models with corresponding speech assessment labels using the regression loss between the estimated scores and the corresponding labels (T04, T06) showed better effectiveness than performing additional fine-tuning of SSL models with extra training data using the corresponding SSL loss (T03). We also observed that fusion estimation (combining estimated scores from two different modules) can provide better performance with limited training data, as indicated by the good performance of T06, where the predicted MOS is combined with the MOS retrieved from the datastore using k-nearest neighbors (kNN) method. Additionally, we noted that no single system excels across all metrics. Specifically, T06 performs best in predicting SIG and OVRL, while T04 performs best in predicting BAK. Interestingly, for BAK estimation, first fine-tuning the SSL model for SNR estimation and then fine-tuning it for BAK score estimation can significantly improve prediction performance.

Comparing VQScore (B01) and the submitted systems to DNSMOS P.835, we observed that large-scale labeled training data may not be necessary to train a speech quality predictor. We also observed that the average standard deviation (std) among labelers in the evaluation set is 0.686, 0.571, and 0.559 for SIG, BAK, and OVRL, respectively. SIG has the highest std, implying it is difficult for labelers to obtain aligned ratings. However, although it was also found in [15] that SIG is the most difficult metric to predict, the underlying reason remains unclear.

For tracks 1 and 2 where we did not pose any constraint on the data usage, to our surprise, many teams did not use additional labeled datasets⁴⁴4Here, by saying “additional labeled dataset”, we mean datasets with subjective ratings as labels. We do not take into account, for instance, data used for self-supervised pre-training.. For track 1, four out of the six⁵⁵5T07 was not counted. teams did not use additional labeled data, and for track 2, three out of six teams did not use additional labeled data. Popular additional dataset choices included SOMOS [27] and past Blizzard Challenges. T03 even tried to use the ConferencingSpeech 2022 training sets [28], which mainly consist of noisy speech samples and their ratings.

For track 3 where we did not allow using additional datasets with subjective labels, two out of the four participants did not use additional noise or speech data. Other participants used datasets like the AudioSet [29], DNS [30], ESC-50 [31], MS-SNSD [32], and ASVSpoof 2019 [33] to either pre-train the model or perform data augmentation.

As in previous challenges, using self-supervised learning (SSL) representations still remained a popular approach, as almost all systems (including the baseline) used features from models including wav2vec 2.0 [34], XLSR [35], WavLM [36] and BYOL-S [37]. While previous challenge participants only used one or more SSL features, some participants this year attempted to use of other features. For instance, T01 used magnitude spectrograms, mel spectrograms and Whisper features [38], T04 used features from HiFi-GAN discriminators [39], T05 used mel spectrograms, and T08 used pitch histograms. On the other hand, listener modeling [40], which was a widely used technique in VMC 2022, was rarely used this year. This is possibly due to the fact that the datasets of tracks 2 and 3 did not come with listener-wise labels.

As for the model architecture, most teams follow the design of SSL-MOS, i.e., attaching a simple linear or recurrent neural network layer to predict the final score on top of the feature extractor (in most cases SSL model). Some teams tried to use more complicated models, such as EfficientNetV2 [41]. Finally, model ensembling was an effective approach, as they were used by top teams including T04 and T05.