Deep MOS Predictor for Synthetic Speech Using Cluster-Based Modeling

A GST model is proposed in [15] for style-expressive end-to-end TTS. It consists of a reference encoder, style token layer, and sequence-to-sequence model. The reference encoder extracts reference embedding of the reference utterance to refer the style. The GSTs of the style token layer, which are shared by all training sequences, become the soft clusters of reference embedding through training. The goal of the style token layer is to calculate the style embedding, which represents the style of the reference utterance, as a weighted sum of the GSTs. The weights assigned to the GSTs are learned by a multi-head attention module [16]. The style token layer consisting of GSTs and the attention module is randomly initialized and then trained jointly with the whole GST model. Therefore, the GSTs can become useful soft clusters for style modeling, and the GST model can synthesize speech with a specific style of the reference utterance by extracting the style embedding from it.

Although Wang et al. [15] suggested the GST layer for TTS, they also showed that GSTs could be used as speaker classification features. There is also a report [17] that the GST layer helps to improve speech recognition performance. In this paper, we report for the first time that the GST layer also helps to improve the performance of the quality assessment task. Here, we refer to the GST as the GQT, as we mentioned in the previous section. Besides, the reference encoder, GQTs, and multi-head attention module are collectively called the GQT layer.

As the GSTs are targeted for a separate reference utterance, they need a separate reference encoder. Unlike GSTs, the GQTs are targeted for the same input utterance of MOS prediction. Therefore, we design our reference encoder to share the convolutional layers with MOSNet. A gated recurrent unit (GRU) [18] layer follows the convolutional layers and the last hidden state of the GRU layer serves as the reference embedding. Then, the quality embedding is calculated in the same way to calculate the style embedding in [15]. As a kind of skip connection, the quality embedding is added to all the frame-level feature vectors of the CNN. We use the resulting representation as an input of the BLSTM layer.

When we use a GQT layer for a similarity score prediction task, we apply the shared GQT layer to generate two quality embeddings from an utterance pair. Then we add each quality embedding to the corresponding CNN feature map before the concatenation of two CNN feature maps.

The Encoding Layer [19] was proposed for texture recognition by learning the inherent visual codewords directly from the loss function. The codewords are learned from the distribution of the CNN features. The Encoding Layer also acts as a pooling layer, which converts feature vectors of any size into a fixed-length representation. Given $N$ feature vectors, $X=\{x_{1},...,x_{N}\}$, and $K$ codewords, $C=\{c_{1},...,c_{K}\}$, the output representation of the Encoding Layer is $e=\{e_{1},...,e_{K}\}$, called residual encoding vector. The residual vector $r_{ik}$ is calculated by $r_{ik}=x_{i}-c_{k}$ and the assigning weight for $r_{ik}$ is given by

$$w_{ik}=\frac{\exp(-s_{k}\|r_{ik}\|^{2})}{\sum_{j=1}^{K}\exp(-s_{j}\|r_{ij}\|^{2})}\,,$$

(2)

where $s_{k}$ is a learnable smoothing factor for $c_{k}$. Then the residual encoding for the k-th codeword $c_{k}$ is calculated as follows:

$$e_{k}=\sum_{i=1}^{N}e_{ik}=\sum_{i=1}^{N}w_{ik}r_{ik}\,.$$

(3)

In the speaker recognition task, Cai et al. [20] and Jung et al. [21] used the Encoding Layer for aggregating the frame-level speaker features to generate speaker embedding and improved the performance. Motivated by this, we apply the Encoding Layer in the speech quality assessment model. However, we use the Encoding Layer on frame-level scores, not frame-level feature vectors. Moreover, we utilize both GAP layer and Encoding Layer together to combine information from both layers. In ground terrain recognition task, Xue et al. [22] showed it is better to use both the Encoding Layer and the GAP layer.

Specifically, we put an Encoding Layer parallel to the GAP layer. The outputs from the Encoding Layer and the GAP layer are the residual encoding vector and the average score, respectively. They are concatenated and used as an input of the following FC layer to predict the utterance-level score.

When we use an Encoding Layer for similarity score prediction, the Encoding Layer aggregates the frame-level similarity scores as in MOS prediction.

As in [14], we use the MOS and similarity evaluation results from the Voice Conversion Challenge (VCC) 2018 [23]. The challenge comprised two tasks: Hub task (parallel VC) and Spoke Task (non-parallel VC). The VCC 2018 dataset is based on the device and production speech (DAPS) dataset [24], which includes recordings of professional US English speakers. There are a total of eight source speakers and four target speakers. A total of 23 teams submitted systems to the Hub task, with 11 of them additionally participating in the Spoke task. There are a total of 38 evaluated systems, including the source speaker, the target speaker, and two baseline systems for the two tasks.

For MOS evaluation, 267 people rated the naturalness of 20,580 submitted utterances with a score ranging from 1 (“Completely unnatural”) to 5 (“Completely natural”). The corresponding number of evaluation results is 82,304, and the ground-truth MOS of each utterance was obtained by averaging all the MOS ratings of the utterance. Among the 20,580 $<$audio, ground-truth MOS$>$ pairs, we use 15,580, 3,000, and 2,000 pairs for training, validation, and testing, respectively. The MOS of each system is obtained by averaging all the MOS values of the utterances from the system.

The same 267 people also rated the similarity between two utterances with a score among 1 (“Same, absolutely sure”), 2 (“Same, not sure”), 3 (“Different, not sure”), and 4 (“Different, absolutely sure”). An utterance pair consists of an anchor utterance, which can be either converted speech or human speech, and a reference utterance, which is an utterance from either the source or the target speaker of the anchor speech with the same linguistic context. There are a total of 30,864 evaluation results, and the ground-truth similarity of each utterance pair is obtained from the average of the scores received. Among a total of 21,608 $<$audio pair, ground-truth similarity$>$ pairs, we use 17,286 for training, 2,161 for validation, and 2,161 for testing. We consider a pair $<$anchor system, reference system$>$ as one system pair, then the corresponding number of system pairs for similarity score prediction is 76.

We also use the MOS evaluation results of the VCC 2016 [25] to test the generalization ability of the models trained on the VCC 2018 training set. The VCC 2016 comprised only a parallel voice conversion task, and there are a total of 26,028 utterances from 20 systems. Each system has 1,600 utterance-level evaluation results without any description of the utterances. Therefore, we can report only system-level performance on the evaluation results of the VCC 2016.

We implement all the models using PyTorch and train them on a single NVIDIA GTX 1080 Ti GPU with four different random seeds. We set $\alpha$, the weighting factor for the frame-level MSE, to 0.8. When we use the GQT layer, we use 10 GQTs and 8 heads for a multi-head attention module. When we use the Encoding Layer, we use 10 codewords. We use a batch size of 16 for MOSNet + GQT + EL and 32 for the rest of the models. We use the Adam optimizer with a learning rate of 0.0001 and set a dropout rate to 0.3. We use the validation set to select the model with the lowest MSE during 200 epochs. We report the average MSE, linear correlation coefficient (LCC) [26], and Spearman’s rank correlation coefficient (SRCC) [27] of the models trained with four random seeds.

First, we discuss the results on MOS prediction with the VCC 2018 test set, shown in Table 2. MOSNet + GQT made improvements with all the metrics except the system-level MSE. To directly interpret the role of each GQT, we should adjust the weights of GQTs and observe the change in the predicted MOSs for various input utterances. However, this process is practically impossible since it requires listening to a lot of utterances and analyzing the factors that affect the MOS, which are expensive and subjective. Instead, from the MOS prediction results, we can infer that the GQTs become useful soft clusters for MOS evaluation.

Using the Encoding Layer improved all the metrics except the utterance-level SRCC as the model aggregated the frame-level scores using their distribution and learned the embeddings that are useful for the aggregation. It achieved the lowest MSE and highest LCC at both the utterance and system level. From the fact that MOSNet + EL shows better performance than MOSNet + GQT, we can say that considering the distribution of frame-level MOSs is more important than learning the quality embeddings for MOS evaluation.

When we combine MOSNet with both the GQT layer and Encoding Layer, the performance of MOSNet + GQT + EL is better than MOSNet + GQT but worse than MOSNet + EL. In other words, the Encoding Layer helps MOSNet + GQT, but the GQT layer does not help MOSNet + EL. As will be described in Section 4.5, we conjecture that GQTs prevent the embeddings of the utterances with similar scores from getting too far from each other, which results in that MOSNet + EL cannot separate the embeddings according to the frame-level scores as before.

With the test results using the VCC 2016 data, we can conclude that either the GQT layer or the Encoding Layer also improves the generalization ability of MOSNet. The Encoding Layer helps generalization through the sophisticated aggregation of frame-level scores. Moreover, the GQT layer directly improves the generalization ability of the model by learning the universal criteria for speech quality evaluation.

The performance of the similarity score prediction models is evaluated with the MSE, LCC, SRCC, and accuracy, as shown in Table 3. As we use a scalar for both the ground-truth and predicted similarity score, we regard the scores lower than 2.5 as the answer “Same” and the scores higher or equal to 2.5 as the answer “Different.” Then we calculate the accuracy of a model as the ratio of cases when both answers from the model and humans are the same. All the proposed models show improvements in all the metrics except the accuracy.

Note that, unlike in MOS prediction, using the GQT layer shows better results than using the Encoding Layer. We can infer that finding the criteria for voice similarity evaluation is more important than using the distribution of the frame-level similarity scores. Furthermore, SIMNet + GQT + EL achieved the best performance among the four models, which means that SIMNet + EL takes advantage of the GQT layer. Considering this result and the fact that we judge the voice similarity throughout the whole utterance rather than specific frames, we can infer that SIMNet + EL does not separate the embeddings as far as in MOSNet + EL, according to the frame-level scores.

Finally, we test how well our model, SIMNet + GQT + EL, approximates human assessment in terms of an evaluation method used in VCC 2018 [23]. As mentioned earlier, the utterance-level similarity score can be classified into “Same” or “Different.” According to the method, the similarity score of a system is the ratio of the utterances that received “Same” compared to the target speech. Then we obtain the MSE, LCC, and SRCC by comparing the scores using the answers of our model and human, which are 0.037, 0.714, and 0.696, respectively.

To discuss the effect of the GQT layer and Encoding Layer to the embeddings, we visualized the embeddings of four MOS prediction models using t-distributed stochastic neighbor embedding (t-SNE) [28] in Figure 1. We displayed a frame-level feature of CNN-BLSTM as a single dot. We consider two systems submitted by one team as the same system because they are based on similar algorithms, which results in a total of 26 systems. We use 390 random utterances from the VCC 2018 test set so that an average of 15 utterances per system exists. We evenly select half of 26 systems after sorting the systems according to the MOS. The color in Figure 1 indicates the system that generates the corresponding utterance. Note that each system has its own system-level MOS in that different systems have different MOSs and the utterance-level MOSs are similar within each system. The red dots are for the source speech, and the orange dots are for the voice conversion system with the highest MOS.

Compared to (a), (b) shows that the GQT layer prevents the embeddings of the same system from getting apart from each other. Comparing (a) and (c), we see that the Encoding Layer separates the embeddings according to the system. This shows that the Encoding Layer automatically learns the embeddings that are more useful for the aggregation, by utilizing the distribution of the frame-level scores. In (d), we see that the embeddings of the same systems get close to each other while embeddings of different systems get apart from each other. Specifically, + GQT + EL learns distinguishable embeddings between different systems that have higher MOSs, including S00, N10, and N17. However, the other systems having lower MOSs are less distinguishable from each other than in MOS + EL. We infer that this is the main reason for the degradation of performance.