Towards No.1 in CLUE Semantic Matching Challenge: Pre-trained Language Model Erlangshen with Propensity-Corrected Loss

With the development of data science applications, semantic matching plays a fundamental role in Natural Language Processing (NLP). These applications, such as information retrieval, often require semantic matching to combine data from multiple sources prior to further analysis Li et al. (2021). It is natural to think of semantic matching as a binary classification issue. Given a pair of sentences, the artificial intelligent system is required to determine if two sentences provide the same meaning.

To up the challenge of this problem, the CLUE team Xu et al. (2020) releases CLUE Semantic Matching Challenge. In detail, the QBQTC (QQ Browser Query Title Corpus) challenge dataset includes three labels, which are {$0$, irrelevant or not matching; $1$, some correlation; $2$, high relevant}, instead of only two. Furthermore, QBQTC dataset is designed as a learning-to-rank (LTR) dataset that integrates relevance, authority, content quality, timeliness and other dimensional annotations, which is widely used in QQ search engine business scenarios²²2https://browser.qq.com/. In additional, this challenge is included in CLUE benchmark, which is widely evaluated in Chinese NLP community.

To improve BERT Devlin et al. (2019) in Chinese tasks, we release a new PLM, Erlangshen. Note that we can not explain the details of Erlangshen, such as training implementation and detailed methods. This advanced Chinese PLM follows the private policy and company agreement of IDEA³³3https://idea.edu.cn/. Fortunately, we are allowed to describe the idea of the design and the basic structure. Although Erlangshen is kept under wraps, we open-source its weights in Huggingface⁴⁴4https://huggingface.co/IDEA-CCNL for accessible use in custom tasks.

After the necessary declarations, we introduce our basic ideas for constructing Erlangshen. Our Erlangshen employs the similar multi-layer transformer architecture as BERT. By following the pre-training and fine-tuning paradigm, Erlangshen is pre-trained with Masked Language Modeling (MLM) Devlin et al. (2019) and Sentence-Order Prediction (SOP) Cui et al. (2021) task. When considering the linguistic characteristics of Chinese, we apply Whole Word Masking (WWM) Cui et al. (2021) to process whole Chinese words instead of individual Chinese characters in WordPiece. Furthermore, since the original MLM strategy only randomly masks tokens, the language model can easily recover common tokens such as “I am”. To address this issue, we propose a Knowledge-based Dynamic Masking (KDM) method to mask semantically rich tokens such as entities and events. Besides understanding intra-sentence information, we apply SOP task to learn inter-sentence information instead of next sentence prediction task. The main reason is that SOP is much more effective than NSP Cui et al. (2021); Lan et al. (2020). After pre-training on large-scale unsupervised data, Erlangshen and its variants are released in Huggingface.

In fine-tuning phase, we add a classifier layer on the top of Erlangshen and predict its label. In detail, given a pair of sentences $S_{1}$ and $S_{2}$, we generate the input ($x_{inp}$) with special tokens in BERT Devlin et al. (2019) as follows:

$$x_{inp}={\tt[CLS]}S_{1}{\tt[SEP]}S_{2}{\tt[SEP]}$$

(1)

Then, we fed [CLS] as the global feature into the classifier for generating the final answer.

By observing the QBQTC dataset with three labels, it suffer data imbalance problem (Details in Section 4.1). Since the label $1$ occupies the majority of samples, we focus on the correlation between different categories. We introduce the Propensity-Corrected Loss (PCL) to encourage the model to predict the dominant categories, which corrects the predicting propensity.

We consider a label set {$0$, irrelevant or not matching; $1$, some correlation; $2$, high relevant}. Given a pair of sentences ($S_{1}$ and $S_{2}$), we argue that the model must take care of the “distance” of labels. For example, if it is a not-matching pair, the penalty for the model to misclassify it as label $2$ must be larger than as label $1$. Because label $1$ is closer to label $0$ in “distance” than label $2$. If the opposite occurs, a loss bonus (which reduces the original loss) is provided. Similarly, this case also works with misclassifying labels $1$ into labels $0$ or $2$. Note that the label $0$ represents irrelevant; the penalty for assigning the label $1$ to the label $0$ is also greater than assigning it to the label $2$.

In detail, considering simple implementation, we apply PCL based on all samples. Let $label\in\{label_{0},label_{1},label_{2}\}$ to be the ground truth set. Since PCL considers multiple conditions, we compute $PCL_{+}$ and $PCL_{-}$ as follow.

$$PCL_{+}=\varepsilon\frac{\sum_{i=1}^{N}Label_{0}^{i}}{\sum_{i=1}^{N}Label_{1}^{i}+\sum_{i=1}^{N}Label_{2}^{i}},$$

(2)

$$PCL_{-}=\varepsilon\frac{\sum_{i=1}^{N}Label_{0}^{i}+\sum_{i=1}^{N}Label_{2}^{i}}{\sum_{i=1}^{N}Label_{1}^{i}},$$

(3)

where $N$ is the sample number of the dataset and $\varepsilon$ is a scaling ratio.

Then, the PCL ($\mathcal{L}$) can be calculated as:

$$\mathcal{L}=\left\{\begin{matrix}LS-PCL_{-},&\text{Condition 1},\\ LS+PCL_{+},&\text{Condition 2},\\ LS,&\text{Condition 3},\end{matrix}\right.$$

(4)

where $LS$ represents the Label-smoothing loss. Condition 1 tends to encourage the model, e.g., the model misclassify label $1$ into label $2$, which is better than label $0$. Condition 2 tends to punish the model, e.g., the model misclassifies label $2$ into label $0$. Condition 3 considers the cases where no correction propensity is required.

Interestingly, PCL can be computed within a batch rather than across all samples. Therefore, as a future work, we will consider PCL in local level and global level.

We summarize QBQTC dataset splits in Table 1. We further investigate the average number of words for $S_{1}$ and $S_{2}$, average token number after organization and the label ratio. The dataset details are shown in Table 2.

In all our experiments, we apply the variant “Erlangshen-MegatronBert-1.3B-Similarity”⁵⁵5https://huggingface.co/IDEA-CCNL/Erlangshen-MegatronBert-1.3B-Similarity as “Erlangshen”. The detailed settings are followed the open-sourced code⁶⁶6https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/hf-ds/fengshen/examples/clue_sim. We tune the models in train and dev set, and evaluate them in test_public set for easy analysis.

After submitting our prediction files to the CLUE official website, the comparison with existing SOTA performance is listed in Table 3. Our proposed Erlangshen with PCL outperforms all models, which achieves $72.54$ points in F1 Score and $78.90$ points in Accuracy.

To verify the effectiveness of our Erlangshen and Propensity-Corrected Loss (PCL), we conduct ablation studies on Test_public set, as shown in Table 9.

We first verified the effectiveness of our PLM Erlangshen. Regardless of the loss employed, our model outperform widely used MacBERT and RoBERTa-wwm. For label smoothing loss, our Erlangshen improve $3.06$ points on F1 Score and $2.52$ points on Accuracy over MacBERT.

Then, we explore the effectiveness of the proposed PCL. Among the models, PCL helps all models to achieve a better performance than the original label smoothing one. For MacBERT, PCL can provide up to $2.60$ points improvement in F1 score, and up to $1.80$ points improvement in accuracy. For Erlangshen, PCL can benefit up to $2.61$ points improvement in F1 score, and up to $0.90$ points improvement in accuracy. However, Erlangshen obtains less gain from PCL than other models. Our powerful Erlangen might already learn imbalance information in the dataset, so it has learned some propensity by itself rather than loss’ help.