THE T05 SYSTEM FOR THE VOICEMOS CHALLENGE 2024:
Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech

The field of computer vision using deep learning has significantly advanced in recent years, and applying DNN-based models (i.e., deep image classifiers) to spectrograms has demonstrated promising results in some audio/speech processing tasks  [13, 14, 15]. Our system thus leverages the features extracted from spectrograms using a convolutional neural network (CNN) pretrained on a large image dataset. The architecture of our spectrogram feature extractor is shown in Figure 1.

In spectrogram feature extraction, the input speech waveform is first transformed into multiple mel-spectrograms. Each mel-spectrogram is extracted with different short-term Fourier transform (STFT) settings, which aims to mitigate the problem of the trade-off between frequency resolution and time resolution determined by the window size [13]. Let $\textrm{\boldmath$x$}=[\textrm{\boldmath$x$}_{1}^{\top},\ldots,\textrm{\boldmath$x$}_{K}^{\top}]^{\top}$ be $K$ speech frames, where $\textrm{\boldmath$x$}_{k}$ denotes the $k$th frame consisting of $L$ samples. These frames are randomly extracted from the input speech waveform. Multiple mel-spectrogram transformations with $N$ different window sizes $(w^{(1)},\cdots,w^{(N)}),w^{(n)}\in\mathbb{N}$, $\mathrm{MelSpec}_{w^{(n)}}(\cdot)$, are applied to each extracted audio frame:

where $\textrm{\boldmath$y$}_{k}^{(n)}$ denotes the $n$th mel-spectrogram extracted from the $k$th speech frame using the window size $w^{(n)}$.

These mel-spectrograms are then regarded as images rather than speech parameter sequences and fed into CNNs pretrained on ImageNet [16], following previous work [15]. The shape of mel-spectrogram image is fixed as $(F,F)$, where $F$ represents the number of mel-bands, regardless of the window size setting. Multiple CNNs are prepared, where each network receives a spectrogram with a different window size $w^{(n)}$ as input, extracting an image feature as follows:

The features obtained from $\textrm{\boldmath$y$}_{k}^{(n)}$ through the CNN for each window width setting, i.e., $(\textrm{\boldmath$h$}_{k}^{(1)},\ldots,\textrm{\boldmath$h$}_{k}^{(N)})$, are aggregated using a weighted sum $\tilde{\textrm{\boldmath$h$}}_{k}=\sum_{n=1}^{N}w_{\mathrm{spec},n}\textrm{\boldmath$h$}_{k}^{(n)}$. The trainable weight parameter vector $\textrm{\boldmath$w$}_{\text{spec}}\in\mathbb{R}^{N}$ is initialized such that $\sum_{n=1}^{N}w_{\text{spec},n}=1$. As a result, the aggregated feature $\tilde{\textrm{\boldmath$h$}}_{k}$ has the dimension $\mathbb{R}^{c\times f\times t}$, where $c,f$, and $t$ denote the number of features, the height of feature maps and the width of the feature maps obtained through CNNs, respectively. These aggregated features with different $k$ are then concatenated across several frames in the $t$ dimension and subsequently pooled in both $t$ and $f$ dimensions. A combination of average and max pooling is used in the time direction; a combination of attention [17] and max pooling was employed in the frequency direction. The final output of our spectrogram feature extractor is hereinafter denoted as $\textrm{\boldmath$h$}_{\text{spec}}$.

Following previous studies on automatic MOS prediction [2, 3], we utilize a pretrained SSL model to extract speech features from an input waveform. The raw waveform is first fed into the SSL model to extract hidden states from the each layer of the Transformer encoder $(\textrm{\boldmath$e$}_{1},\textrm{\boldmath$e$}_{2},\cdots,\textrm{\boldmath$e$}_{M})$. Then, the hidden states are aggregated using a weighted sum $\tilde{\textrm{\boldmath$e$}}=\sum_{m=1}^{M}w_{\text{SSL},m}\textrm{\boldmath$e$}_{m}$, where $M$ denotes the number of Transformer encoder layers. The trainable weight parameter vector $\textrm{\boldmath$w$}_{\text{SSL}}\in\mathbb{R}^{M}$ is initialized such that $\sum_{m=1}^{M}w_{\text{SSL},m}=1$. Finally, unlike in previous studies [2, 3], combination of attention [17] and max pooling along the sequence dimension are applied to the aggregated hidden state vectors for each time step. The final output of our SSL feature extractor is hereinafter denoted as $\textrm{\boldmath$h$}_{\text{SSL}}$.

Following the UTMOS system [3], we build our MOS prediction system using multiple MOS datasets for the model training with the data-domain encoding (i.e., conditioning the system on the dataset ID). This aims to address the biases in different MOS tests, possibly including the range-equalizing bias [6]. For the data-domain encoding, simple look-up embedding table is used for converting discrete dataset ID to continuous data-domain embedding $\textrm{\boldmath$h$}_{\text{domain}}$.

Note that this data-domain encoding cannot define IDs for unseen MOS datasets and thus does not necessarily work properly for the out-of-domain prediction. One can deal with this issue by, for example, predicting MOS for some seen data-domains and taking the average of multiple predicted scores [3]. Because the primal focus of this paper is the range-equalizing bias, we thoroughly investigate the domain gap between the training and test datasets in our ablation study (Section 4.6).

A simple fully connected layer is prepared and trained for predicting the MOS of input speech using the fusion of extracted spectrogram and SSL features denoted as $\textrm{\boldmath$h$}_{\text{spec}}$ and $\textrm{\boldmath$h$}_{\text{SSL}}$, respectively. The input is the concatenation of these two features and the data-domain embedding along the feature dimension:

where $\operatorname{FC}(\cdot)$ and $\operatorname{Concat}(\cdot)$ denote a fully connected layer and feature concatenation, respectively.

In this ablation study, we compared the following systems:

• •

  Ours: The proposed system using the feature fusion.

• •

  Ours w/o SSL: The proposed system using only the spectrogram feature extractor.

• •

  Ours w/o spec.: The proposed system using only the SSL feature extractor.

• •

  B01: SSL-MOS [2] trained on the original BVCC [12] samples and labels. This system was considered as baseline system in the VMC 2024 track 1.

• •

  UTMOS [3]: The opensourced MOS prediction system.

“Ours w/o SSL” was trained with a learning rate ranging from $1\times 10^{-3}$ to $1\times 10^{-7}$, a batch size of 10, and for 20 epochs. As explained in Section 3.4, “Ours w/o spec.” was built with the two-stage training. We first trained the $\operatorname{FC}$ layer and data-domain embedding for 20 epochs using the learning rate ranging from $1\times 10^{-3}$ to $1\times 10^{-7}$ and batch size of 32. Then, we fine-tuned all model parameters for 5 epochs using the learning rate ranging from $3\times 10^{-5}$ to $1\times 10^{-9}$ and batch size of 32. The system using the feature fusion, “Ours,” was built upon these two systems. Specifically, we utilized these two feature extractors trained through “Ours w/o spec.” and “Ours w/o SSL.” The following $\operatorname{FC}$ layer, and data-domain embedding were randomly initialized.

The stage 2 training was performed for 8 epochs using a learning rate ranging from $1\times 10^{-3}$ to $1\times 10^{-5}$ and a batch size of 16. The stage 3 training was iterated with 2 epochs using a learning rate ranging from $5\times 10^{-5}$ to $1\times 10^{-8}$ and a batch size of 8.

In this comparison, we used all datasets listed in Table 1 for the training and set the data-domain ID for the MOS prediction to “BVCC” in three our systems.

The results are shown in Table 2. For correlation-based metrics, we can see that all of our three systems consistently outperforms both two baseline models in all metrics. Furthermore, in correlation-based metrics, while there are little difference in scores between “Ours w/o SSL” and “Ours w/o spec.,” the fusion system, “Ours,” demonstrates a significant improvement in scores compared to these two systems. These results indicate that our systems are more effective for zoomed-in MOS prediction compared to the existing baseline systems, particularly in correlation-based metrics. It also suggests the effectiveness of fusing spectrogram and SSL features in these metrics.

One noteworthy observation is that “Ours w/o SSL” achieves the best MSE in all cases, but the worst in many cases in the correlation-based metrics. On the other hand, “Ours w/o spec.” scored the highest MSE, but outperforms “Ours w/o SSL” in many cases in the correlation-based metrics. From this perspective, we can infer that the spectrogram features derived from our image feature extractor are better at capturing fine differences in synthetic speech and predicting absolute MOS values, while SSL features are better at predicting rankings among multiple speech synthesis systems. In summary, these results suggest that the fusion of these features improves the prediction of absolute speech quality while further improving the correlation-based measures.

We also computed the evaluation metrics between the ground-truth MOS and human-annotated MOS (“BVCC MOS”), which was collected without considering the range-equalizing bias. The results from Table 2 demonstrate that the bias actually exists and “BVCC MOS” is not well correlated with the ground-truth MOS. In contrast, our fusion system shows better scores than “BVCC MOS” in all metrics except for system-level MSE. Considering that the prediction is made with the data-domain embedding of BVCC, these results suggest that our system has demonstrated robust prediction of MOS for unseen listening test settings.

In this experiment, we compared “Ours” in Section 4.4 with the following systems:

• •

  Ours w/o Stage 2: The proposed system without performing the stage 2 training.

• •

  Ours w/o Stage 1&2: The proposed system with performing only the stage 3 training.

The fine-tuning for “Ours w/o Stage 2” was performed for 20 epochs using a learning rate ranging from $1\times 10^{-4}$ to $1\times 10^{-7}$ and batch size of 8. The training for “Ours w/o Stage 1&2” ran 20 epochs using a learning rate ranging from $1\times 10^{-3}$ to $1\times 10^{-7}$ and batch size of 8. The training dataset and target domain-ID setting was the same as those used in Section 4.4.

The results are shown in Table 3. As the number of multi-stage learning stages is reduced and the two feature extractors are no longer pre-trained for the MOS prediction task, the behavior of the learned models can be seen to approach “Ours w/o SSL” (i.e., lower MSE and lower correlation-based metrics). This may be desirable in situations where we want to accurately predict the absolute MOS, but not when we want to compare different speech synthesis systems. In summary, these results suggest that the proposed multi-stage learning is essential for boosting the ability of the SSL features to capture differences between multiple synthetic speech samples.

This might be because the SSL and spectrogram were combined and trained before being optimized individually. Due to the different learning speeds of SSL feature extractor and the spectrogram feature extractor, the fully connected layer might have resulted in a model that emphasizes one over the other. Specifically, in this case, the spectrogram features might have been given more importance, leading the system to resemble “Ours w/o SSL.”

For predicting MOS, we used “Ours” built with the almost same experimental setting as described in Section 4.4. Here, we changed the datasets for the training and the data-domain ID for the inference. Specifically, we trained “Ours” with “All datasets” and that without {BVCC, BC, SOMOS, sarulab-data}. This experiment enabled us to examine which dataset was essential for improving the MOS prediction performance in the zoomed-in test situation. In addition, by examining the prediction results when changing the data-domain ID, we can verify which domain (i.e. dataset) was closer to the zoomed-in dataset used in the VMC2024 Track 1. Note that only the mean values are presented in the results for the BC datasets, even though the data-domain ID was prepared for each BC dataset.

The results are shown on Table 4(d). In terms of the training datasets, “All datasets” achieves the best scores. However, in some cases the scores improve by excluding BVCC or BC from the training data. In addition, excluding SOMOS or sarulab-data from the training data tends to degrade the MOS prediction performance significantly. These results suggest that when building MOS prediction systems to compare the performance of high-quality speech synthesis, it is crucial to exclude datasets that are likely to contain low-quality speech systems when training. They also indicate that using MOS datasets containing as many results as possible from evaluation of synthetic speech produced by state-of-the-art DNN-based speech synthesis.

Focusing on the difference among data-domain for the MOS prediction, the MSE is the lowest for sarulab-data (i.e., the 50% zoomed-in BVCC) and the highest for BVCC, which clearly shows the effect of range-equalizing bias [6]. However, this tendency is not observed when comparing the correlation-based metrics. These results suggest that the negative effects caused by the range-equalizing bias are dominant in the prediction of the absolute MOS.

Additionally, when comparing the scores of correlation-based metrics between datasets with a 25% zoomed-in rate and those with a 12% zoomed-in rate, it can be observed that the scores are better for the 25% zoomed-in rate datasets in almost all cases. This suggests that the quality of the speech data used for training was closer to that of the 25% zoomed-in rate datasets.