CINO: A Chinese Minority Pre-trained Language Model

Multilingual Pre-trained Language Models. Devlin et al. (2019) introduced the first multilingual pre-trained language model mBERT trained with Masked Language Modeling (MLM). Conneau and Lample (2019) proposed Translation Language Modeling (TLM) to train the multilingual model with cross-lingual supervision. Since then, various kinds of multilingual pre-training objectives have been proposed. Unicoder Huang et al. (2019) trains the model with the objectives including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual MLM. InfoXLM Chi et al. (2021) proposed a pre-training task based on contrastive learning from an information-theoretic perspective. Pan et al. (2021) also introduced an alignment method based on contrastive learning. Cao et al. (2020) proposed an explicit word-level alignment procedure. ERNIE-M Ouyang et al. (2021) integrates back-translation into the pre-training process. VECO Luo et al. (2021) uses a cross-attention module to build the interdependence between languages explicitly. In this work, we only use non-parallel data and an objective similar to MLM for pre-training CINO.

Non-English Pre-trained Language Models and Benchmarks. Many pre-trained models have been trained on English corpora, or corpora that are heavily biased toward English. To make NLP techniques accessible to people from different cultures, researchers have developed pre-trained models and benchmarks targeting different languages: FlauBERT and the FLUE benchmark for French Le et al. (2020), KLUE-BERT and the KLUE benchmark for Korean Park et al. (2021), IndoBERT and the IndoLEM benchmark for Indonesian Koto et al. (2020), and there are Chinese-BERT-wwm Cui et al. (2021) and Arabic BERT AraBERT Antoun et al. (2020). However, there are no pre-trained language models targeting Chinese ethnic minority languages.

There are 56 ethnic groups and more than 80 languages in China. Standard Chinese (Mandarin) is the official language, spoken mainly by ethnic Han Chinese, which accounts for more than 90% of the total population. Ethnic minorities have their own languages. According to the study in Moseley (2010), the ethnic minority languages Mongolian, Uyghur, Kazakh, Tibetan,Yi, and Korean are safe (five of them are covered by CINO), which are spoken by about 25 million people, while the rest are in unsafe or endangered status.

Besides the ethnic minority languages, there are dialects and varieties of Chinese across the country. In this work, we consider Yue Chinese (also known as Cantonese), a widely used group of varieties of Chinese in Southern China and have been carried by immigrants to Southeast Asia and many other parts of the world.

Some languages in Table 1 are spoken and widely used in more than one country, such as Korean, Mongolian and Kazakh. In this work, we named them as minority languages based on their status in China.

CINO is a multilingual transformer-based model with the same architecture as XLM-R. For the CINO-base, it has 12 layers, 768 hidden states, and 12 attention heads; for the CINO-large, it has 24 layers, 1024 hidden states, and 16 attention heads. The main differences between CINO and XLM-R are the word embeddings and the tokenizer. We start from the word embeddings and the tokenizer of XLM-R and adapt them for the minority languages by vocabulary extension and vocabulary pruning, as depicted in Figure 1.

Vocabulary Extension. The original XLM-R tokenizer does not recognize Tibetan scripts and Traditional Mongolian scripts, so we extend the XLM-R tokenizer and XLM-R word embeddings matrix with additional tokens.

We train sentence-piece tokenizers for Tibetan and Mongolian on their monolingual pre-training corpora respectively. Each of the tokenizers has a vocabulary size of 16,000. Then we merge the vocabularies from the Tibetan and Mongolian tokenizers into the original XLM-R tokenizer. The merged tokenizer has a vocabulary size of 274,701.

To extend the word embeddings, we resize the original word embeddings matrix of shape $V\times D$ to $V^{\prime}\times D$ by appending new rows, where $D$ is the hidden size, $V$ is the original vocabulary size, $V^{\prime}$ is the new vocabulary size. The new rows represent the word vectors of the new tokens from the merged tokenizer. They are initialized with a Gaussian distribution of mean 0.0 and variance 0.02.

Vocabulary Pruning. Next, we prune the word embeddings matrix to reduce the model size. We tokenize the pre-training corpora with the merged tokenizer, and remove all the tokens that have not appeared in the corpora from the merged tokenizer’s vocabulary and the word embeddings matrix. The above process discards 139,342 tokens.

Finally, we obtain the CINO model structure with a vocabulary size of 135,359, a model size of 728 MB for the base model, 1.7 GB for the large model, 68% and 79% size of XLM-R-base and XLM-R-large, respectively. A smaller vocabulary size leads to not only a memory-friendly model but also a faster model by reducing the cost of computing the log-softmax in the MLM task. The time cost of each iteration in pre-training is reduced by approximately 35% by reducing the vocabulary size from 270k to 140k.

We adopt the MLM objective for pre-training. In addition, we apply the following strategies for balancing training data and faster pre-training.

Data Collection and Annotation. WCM is based on the data from Wikipedia. It covers seven languages: Mongolian, Tibetan, Uyghur, Kazakh, Korean, Cantonese, and Standard Chinese. We build the dataset from the Wikipedia page dumps and the Wikipedia category dumps²²2https://dumps.wikimedia.org/other of the languages in question.

To annotate the data, we first generate a category graph for each language. Each node represents a category, and each edge stands for the affiliation between a pair of categories. By referring to the category system of Chinese Wikipedia, we choose ten categories for the classification task: Art, Geography, History, Nature, Science, Personage, Technology, Education, Economy, and Health. Then, we start from the categories of each page and backtrack along the routes in the category graph until reaching one of the ten target categories, and we set this category as the label of that page. Owing to some affiliation conflicts, like one subcategory belonging to two categories simultaneously, we reconstructed the graph by removing certain edges between the 10 target categories and their subcategories which are assessed as unreasonable by our human evaluation team.

Data Cleaning. After getting the labeled data, we apply several strategies to improve the quality of the datasets. We remove dirty data like large blocks of URLs and file paths. Then, the examples are filtered by their lengths (after being tokenized by the CINO tokenizer) by removing those examples shorter than 20 or longer than 1024 tokens.³³3We discard examples that are too long because long examples likely cover multiple topics while we assign a single label to each example.

Subsampling. Since there are both high-resource languages like Korean and low-resource languages like Uyghur, we down-sample the data in the high-resource languages and the high-resource categories to balance the numbers of examples among different languages and different categories. We fix the size of the training set (Chinese articles) to 32K and downsample the datasets of the languages with abundant articles to about $5\%\sim 20\%$ size of the training set. Similarly, we also downsampled some categories if they dominate in some languages. We did not apply the above process to Uyghur due to its extreme scarcity.

Finally, we obtain 63,137 examples. WCM contains the train/dev/test set for Standard Chinese and only test sets for other languages. The detailed distribution is listed in Appendix C.

Data Collection and Annotation. To collect the minority language examples, we crawl the news from the news websites in ethnic minority languages and record the category to which each news item belongs. To collect the Chinese news, we reuse the pre-existing dataset SogouCS News Wang et al. (2008) and CAIL 2018 Xiao et al. (2018). We select the appropriate categories and down-sample the two datasets to make the whole dataset more balanced.

After gathering the raw data from all the languages, we first merge the categories that have similar meanings (for example, we merge the categories Finance and Economy). Since the definition of news category may vary from website to website and language to language, we remove the categories that are not consistent in different languages by manually checking a sampled subset. We also remove the categories that do not appear in more than two languages. Finally, we obtain a dataset containing eight categories: Education, Sports, Health, Tourism, Legal, Economy, Culture, and Society.

Data Cleaning. The crawled news is much cleaner than the Wikipedia pages, and each document naturally belongs to only one category. Therefore we only perform length filtering by keeping the documents that contain more than 30 tokens after tokenization.

The dataset contains 56,764 examples in total. We split the dataset into a training set and a development set. The detailed distribution is listed in Appendix C.

To assess the quality of the datasets, we randomly sample 200 examples from WCM and 200 examples from CMNews and manually check whether the contents of the examples match their labels. The results are shown in Table 2. Matching Acc denotes how many examples match their labels under human evaluation. We find that $88.5\%$ of the sampled examples from WCM and $93.5\%$ of the sampled examples from CMNews are correctly labeled, which shows CMNews has less noise.

Pre-training Data. We randomly sample a subset dataset from the public base version of WuDaoCorpora Yuan et al. (2021) as the Standard Chinese corpus; the corpora of the minority languages are in-house data, consisting of short monolingual sentences. The total corpora size is 28 GB. The statistics of the pre-training corpora are listed in Appendix A.

Experiment Settings. CINO is trained with the fast MLM objective (4) with the masking probability is 0.2 and the max sequence length 256. We initialize the parameters of CINO with XLM-R. We use the AdamW optimizer Loshchilov and Hutter (2019) with the peak learning rate of 2e-4 for the base model and 1e-4 for the large model. The learning rate is scheduled with 10k and 5k warmup steps followed by a linear decay for the base and the large model respectively. The sampling hyperparameter $\alpha$ is set to $0.7$. We train the model with the batch size of 4,096 for 150k steps for the base model, and the batch size of 8,192 for 75k steps for the large model. The pre-training is performed on 16 NVIDIA A100 GPUs. The full pre-training hyperparameters are summarized in Appendix B.1.

How does CINO perform on the newly introduced languages? How does CINO perform on the languages pre-existing in XLM-R? Does CINO show multilingual and cross-lingual abilities? To answer these questions, we evaluate CINO on (1) Tibetan News Classification Corpus Qun et al. (2017) (TNCC); (2) Korean news topic classification Park et al. (2021) (YNAT); (3) WCM and CMNews. On TNCC and YNAT,⁴⁴4The splitting sizes of TNCC and YNAT are listed in Appendix C. we evaluate the in-language model performance, i.e., we train and evaluate the model on the same language. On WCM and CMNews, we evaluate the cross-lingual ability. We describe the details in Section 5.4.

For each task and each model, we run the experiment five times with different seeds and report the mean metrics. The fine-tuning hyperparameters of each experiment are listed in Appendix B.2.

Besides the common multilingual pre-trained models mBERT and XLM-R, we compare CINO models with the following baselines on some tasks.