Polyphone Disambiguation in Mandarin Chinese with Semi-Supervised Learning

Early studies [3] focus on building knowledge-based pipelines in which the pronunciations are determined by a sequence of pre-designed rules. Such measures typically segment sentences into words and eliminate any potential pronunciation ambiguity on words that can be matched in a dictionary, then an expert system is utilized in solving the remaining polyphones. Despite the low computational cost, extra information such as POS tag is usually mandatory for high-quality rule-based analysis. Furthermore, knowledge-based approaches are prone to fail on polyphonic characters when they are part of words with distinctive meanings, as shown in Figure 1.

Deep Neural Networks have risen to be the default method to solve problems in natural language and speech processing [4, 5]. Shan et al. [6] leverage extra POS tags and implement a bi-directional LSTM layer to model neighboring contexts of the target character. In [7], a convolutional seq2seq model proposed by [8] is trained on audio-generated texts contained polyphonic characters. Cai et al. [9] treat the Polyphone Disambiguation as a classification problem and generate combined character embedding using a bi-directional LSTM layer and Tencent Chinese Word2Vec embeddings[10].

In recent years, the introduction of BERT [11] and similar unsupervised pre-training methods [12, 13] take advantage of rich unlabeled text data and train a neural network on performing pre-text tasks such as cloze tests. Then the pre-trained model can be connected with simpler neural nets and get fine-tuned on downstream tasks. Dai et al. [14] are the first to apply a pre-trained Chinese Bert on the polyphone disambiguation problem.

These advancements are mainly contributed by the application of supervised learning on increasingly larger models and additional commercial datasets. Nonetheless, very few datasets [15] for polyphone disambiguation are made available to the public. The existing labeled dataset is also relatively small and unbalanced. In short, the lack of data constitutes an obstacle to the development of Polyphone Disambiguation.

In this paper, we propose Polyphone Pseudo Labels (Semi-PPL), a Semi-supervised Learning approach for task of Polyphone Disambiguation. The proposed system consists of two parts: the prediction network built upon a pre-trained language model and the SSL mechanism that is designed to assign pseudo labels. Unlike the previous study [14] that directly use a computationally expensive Bert, we adopt a tiny-Electra [1] to compute character context representations.

In Table 1, the base prediction model itself has demonstrated performance that surpasses all recent learning-based methods [11, 9, 15] on our evaluation set. Furthermore, we are the first to apply Semi-supervised Learning (SSL) on the Polyphone Disambiguation task to reduce the dependence on expensive manually labeled data. Pseudo-labeling [16, 17, 18, 19, 20, 21, 22, 23, 24, 25] is among the most successful SSL strategies in solving computer vision problems. Pseudo-labeling typically starts by acquiring two models [17]. One model serves as the teacher, assigning labels to unlabeled data that satisfy criteria and expand the labeled sets. The other one takes the role of the student and learns from the labeled training set combined with pseudo labeled data. We adapt pseudo-labeling to the current task by designing an entropy criterion for label assignment to polyphonic character embedding. Pseudo labels are only assigned to unlabeled texts when the base model is ’certain’, which is measured by information entropy of output probability distribution. We also propose a language consistency regularization mechanism that imposes output similarity between augmented versions of texts contained polyphonic characters. We publish a dataset that includes 300K labeled texts and an evaluation set with 1187 difficult sentences that are manually annotated.

In summary, the contribution of our work is fourfold:

1. 1.

   We propose an end-to-end Deep Neural Network structure that achieves state-of-the-art performance on polyphone disambiguation with a light weighted model. Our model greatly relieves the hardware requirement compared to other Bert based approach, making it possible to be deployed in the large-scale scenario.

2. 2.

   We introduce Semi-PPL, a Semi-Supervised learning paradigm to tackle the challenging Polyphone Disambiguation problem, boosting the performance significantly only using cheap unlabeled data.

3. 3.

   We design a novel consistency regularization mechanism specifically for character-level tasks on Chinese texts. As shown in the experiments, it greatly enhances the accuracy of the model on difficult short sentences.

4. 4.

   We publish a sizeable dataset for Mandarin Chinese polyphone disambiguation and provide a benchmark with recent learning-based approaches.

The model of our approach can be divided into two parts: a pre-trained Electra model and the prediction network as depicted in Figure 2. The pre-trained Electra transforms a sequence of Chinese characters into character-level embeddings incorporated with contextual information. The Convolution-BLSTM prediction net serves the purpose of further learning sentence-level knowledge. The output hidden vectors of the selected polyphonic characters are accepted as the input of a series of dense layers to produce the final probability distribution.

Multiple studies [16, 21, 22, 29] in the field of Computer Vision have proved the effectiveness of Semi-Supervised Learning (SSL). SSL is expected to enhance task performance by leveraging numerous unlabeled data while only a limited quantity of labeled data are available. The introduction of our proposed method is divided into two subsections. Section 2.2.1 explains the details of the consistency loss applied on input text data contained polyphonic characters, while Section 2.2.2 covers several viable options for pseudo label assignment.

Empirically, we find considerable unsolvable cases are among ten characters after applying a knowledge-based approach with segmentation and dictionary querying. Another problem we noticed is that the past methods perform poorly on short sentences that are frequently used in daily conversations. We prepare a dataset specifically for these ten polyphonic characters, which includes 326,000 sentences for training and a test set with 1,100 manually written and annotated texts that are deliberately made difficult for exposing model vulnerability. The number of occurrences of each character is even in both training and test sets. We plan to publish the dataset in hope of providing a thorough and convenient evaluation for future studies. As for unlabeled data, we gather 25,420,601 lines of text from Wikipedia. All texts in traditional Chinese are converted into Simplified Chinese.

Experiments are conducted under three configurations: 1) Base model 2) Base model with consistency loss (Base model + C), 3) Base model with consistency loss while enabling pseudo labeling (Base model + CP). In configuration 2 (Base model + C), the weight of consistency loss is set as 0.008 and the weight of classification loss as 1.

We train our base model on labeled data supervised by Cross-Entropy loss and consistency loss for 130 epochs with a batch size of 128. AdamW optimizer is applied with a learning rate of 1e-5. The output embedding size of tiny-Electra is 256. The hidden size of Conv1D and BLSTM layers is 256. We also apply a residue connection that concatenates the Electra embedding with the output of BLSTMs.

As for configuration 3 (Base model + CP), the training setting of the base model is the same as configuration 2 (Base model + C). After we train the base model, the weight of consistency loss is modified to 0.003 and the learning rate is set to 1e-6. The consistency loss on unlabeled data is then computed and combined with consistency supervision on labeled data. We begin assigning pseudo labels to unlabeled data with the incremental entropy threshold $t$ set as Tmin = 0.81 and Tmax = 0.85, with step=0.1. The ratio between unlabeled data and labeled data in each batch is 1:1. We only assign pseudo labels during odd epochs, while the even epochs are for solidifying the model on data appended from the previous epochs.

In this section, we compare our results with recent learning-based methods. To make a fair comparison, their model is trained on our dataset, and the weight of linear classification layers is shared among all 10 target characters. As depicted in Table 1, our base model outperforms all previous methods in the quantitative experiments. The proposed method even surpasses methods [14] using base Chinese Bert with BLSTM and linear layers. The model complexity, measured by the number of parameters (Params) and Memory Access Cost (MAC), is also significantly lowered compared to the predecessor using Bert. There is a performance boost in accuracy after the introduction of consistency regularization and pseudo labeling.