Can depth-adaptive BERT perform better on binary classification task of Chinese texts

As discussed in section 2, the test accuracy becomes fluctuating when the number of transformer layers larger than some certain values. We then compare the predictions of each model and capture something interesting. When the test accuracies of two models are similar, the predictions on the same instance are also close. In addition, all these 12 models provide consistent predictions on most of the instances, and divisions only arise over a few indistinguishable ones. While these divisions would become less and less with the test accuracies being closer.

Specifically, given a dataset $\mathbb{S}$ containing $n$ examples $\rm{x_{1}},\rm{x_{2}},\cdots,\rm{x_{\textit{n}}}$. We construct 12 truncated BERT-based classification models, namely $M_{1}$ to $M_{12}$ (here, the subscript $i$ of $M$ denotes the number of transformer layers), to make predictions on the $\mathbb{S}$. The test accuracy of a model $M_{i}$ is denoted as ${acc}_{i}(i=1,\cdots,12)$. And the calculation of $M_{i}$ on $\rm{x_{p}}(p=1,\cdots,\textit{n})$ is formulated as:

Where $\sigma$ is the sigmoid function, $g_{i}(\cdot)$ is the function of a fully-connected layer, $h_{i}(\cdot)$ is the function of the encoder, and $f_{i}(\cdot)$ is the function of the $i$-th transformer encoder layer. As mentioned above, when $acc_{i}$ is close to $acc_{j}$, we find $M_{i}(\rm{x_{p}})$ is also close to $M_{j}(\rm{x_{p}})(p=1,\cdots,\textit{n};\textit{i},\textit{j}=1,\cdots,12,\ and\ i<j)$. For classification task, we know $\sigma(g_{i}(\cdot))$ is a generalized linear transformation targeting at finding a hyperplane separating examples best without transforming the representations. Naturally, the prediction $M_{i}(\cdot)$ mainly depends on the transformation of the encoder $h_{i}(\cdot)$. The higher separability of the hidden states produced by the encoder is, the higher prediction probability $M_{i}(\cdot)$ and the higher test accuracy $acc_{i}$ of the model are. As a result, we could use the hidden states to help us decide where to truncate.

Inspired by the distribution of points in high dimensional space, we treat the output representations of transformer layers in encoder as coordinates of point, and apply the separability theory of point set to expressing the distribution of examples in our datasets. We truncate BERT targeting at producing representations of dataset with maximum seperability.

Here, we introduce three widely used separability measures, namely Class Scatter Matrices (CSM) Duda et al. (1973), Thornton’s Separability Index (SI) Thornton (2002), and Hypothesis Margin (HM) Gilad-Bachrach et al. (2004).

Class Scatter Matrices, CSM : is a fraction of trace of between-cluster scatter and within-cluster scatter. It evaluates the overall distribution of an entire dataset.

Where $tr(\cdot)$ denotes the trace operation: calculating the sum of one’s diagonal elements. $c$ and $n_{i}$ are the number of classes in a dataset and number of examples in class $i$ respectively. While $m$ and $m_{i}$ are the mean vectors of whole dataset and class $i$ separately. $\rm{x_{\textit{i},\textit{j}}}$ is the $j$-th instance in class $i$. The within-cluster scatter and between-cluster scatter are denoted as $S_{W}$ and $S_{B}$.

Separability Index, SI: calculates the average number of instances that share the same category label as their nearest neighbors. It is a measure evaluating the class overlap.

Where $\phi(\rm{x_{\textit{i}}})(\phi(\rm{x_{\textit{i}}})\in\{0,1\})$ is the label of $\rm{x_{\textit{i}}}$, $\rm{x}_{\textit{i}}^{\prime}$ is the nearest neighbor of $\rm{x}_{\textit{i}}$, and $n$ is the number of examples in a dataset.

Hypothesis Margin, HM: measures the sum of the distance between the hypothesis and the closest hypothesis that assigns the alternative label to the given instance.

Where nearhit($\rm{x}_{\textit{i}}$) and nearmiss($\rm{x}_{\textit{i}}$) denote the nearest instance to $\rm{x}_{\textit{i}}$ with the same and different label, respectively. And $n$ is the number of instances in a dataset. $\|\cdot\|$ denotes the Euclidean norm.

According to the definitions above mentioned, the larger these three separability values are, the more easily the examples are classified. We could apply our truncation method with the assistance of the separability measures.

As we all known, BERT is pre-trained on two general tasks, namely masked language model (MLM) and next sentence prediction (NSP). It means there is little disturbance of wrong prediction corresponding to specific task brought in. Making decisions on the basis of original pre-trained BERT is more objective.

What’s more, once the fine-tuning processing starts, the predicting error would radially propagate back to each layer of the entire network. Breaking the network during this stage leads to reconstruct new matching relations among whole net, which is far more time-consuming.

On account of these factors, we decide to truncate BERT before fine-tuning. If we can make a reliable decision about where to truncate before fine-tuning, resources like time and storage would be saved extraordinarily.

In this sub-section, we provide a general algorithm about truncating BERT (12-layer) properly for downstream tasks by means of separability measures (Algorithm 1).

Given a target binary classification task and the corresponding dataset, we can estimate where to truncate before fine-tuning without building up the network. Our three-step method is simple enough and time-saving. In particular, all the instances in the dataset should be vectorized with BERT embeddings in the first step. Secondly, the vectorized instances are sent into the pre-trained BERT model and propagate forward once without backward propagation. In this step, we will obtain 12 hidden representations from different transformer layers of the BERT for every instance. The hidden representations produced by the same transformer layer are packed into a common set. Then in the third step, we calculate separability values for these 12 representation sets. The number of transformer layers related to the maximum separability is the suggested number of layers in the truncated BERT. We truncate BERT according to that and append the task-specific layer on to form the final model.

In this way, we can truncate BERT adaptively before fine-tuning according to target tasks. Putting these truncated BERT into force could save as much as time and memory with little expense.

In this sub-section, we will give a brief introduction about 8 corpora used in this paper, including ChnSentiCorpHou, WaiMai, SST-2, CoLA, Nikon-JD, AFQMC, MRPC and RTE. The ChnSentiCorpHou is a sub-set of a series of Chinese sentiment analysis corpora organized by Tan et al. Tan and Zhang (2008). The WaiMai is a binary classification corpus about reviews of delivery service and takeaway food¹¹1https://github.com/SophonPlus/ChineseNlpCorpus/tree/master/datasets/waimai_10k. The SST-2 Socher et al. (2013), CoLA Warstadt et al. (2019), MRPC Dolan and Brockett (2005) and RTE Dagan et al. (2005); Haim et al. (2006); Giampiccolo et al. (2007); Bentivogli et al. (2009) come from the general language understanding evaluation (GLUE) benchmark Wang et al. (2018), and AFQMC²²2https://dc.cloud.alipay.com/indexn#/topic/intro?id=3 is selected from the Chinese language understanding evaluation (CLUE) benchmark Xu et al. (2020). The Nikon-JD corpus is organized by our team. More details of these datasets are provided in Appendix A. In Table 1, we list some critical information about these corpora. We select two Chinese and two English single sentence classification tasks. In addition, two Chinese and two English sentence pair classification tasks are also chosen. The assignments in this section vary from sentiment classification and acceptability analysis to semantic similarity and natural language inference. Different assignment is related to different size of dataset and difficulty degree.

We select the 12-layer BERT model (BERT_BASE) as our basic language model and apply the truncation method on these 8 datasets. For each single sentence task, we put a [CLS] and [SEP] at the beginning and the end of the given text separately. And for sentence pair task, we also concatenate two texts into a sequence with [SEP] token being the separator, besides the special pre-processing at beginning and end. Then send processed sequences into the BERT for following procedures.

Before constructing models and fine-tuning, we utilize whole dev sets of ChnSentiCorpHou, Nikon-JD, MRPC and RTE, and randomly sample about 50%, 50%, 10% and 10% examples from dev sets of SST-2, CoLA, WaiMai and AFQMC separately. Then we calculate CSM, SI and HM with these prepared instances for each task, and according to that truncate BERT model adaptively. Next, we construct classification models based on the truncated BERT models, then fine-tune and test them on above mentioned corpora. More experimental details are in Appendix B.

For comparison, we also set up some baselines:

BERT base: We construct binary classification models based on BERT_BASE Devlin et al. (2019) as the standard models.

Distil-Pretrain: We build up classification models based on the distilled language models (DistilBERT Sanh et al. (2019) and albert-chinese-tiny³³3https://huggingface.co/clue/albert_chinese_tiny). In this way, the knowledge distillation theory is used in pre-training period.

Distil-task-biLSTM: We fine-tune a BERT-based classification model as the teacher, and construct a student using the biLSTM as the encoder with the assistance of the attention mechanism. Then let the student imitates prediction of the teacher.

Distil-task-kimCNN: We construct a student according to the kimCNN Kim (2014). While the teacher models are same as those of Distil-task-biLSTM.

Pruning-AGP: We use the Automated Gradual Pruner (AGP) zhu2017prune to implement the task-specific magnitude pruning on our tasks. Here we refer to the implementation provided in the Distiller library zmora2019neural.

Random truncation: Random truncation means selecting a random number ranging from 0 to 11 as the number of layers that should be truncated. Here, we use the mean test accuracy of 12 various truncated BERT-based models to denote the random accuracy.

Note that all these baselines use the BERT embeddings, the batch size and max sequence length are same as those of our models (Table 6). Differently, during implementing task-specific distillation, we set the learning rate as 1e-3.

In order to evaluate whether our separability measures are appropriate or not and then choose a better one for truncating BERT, we firstly plot three measures with the test accuracies in Figure 2 (Here we only provide the results of CoLA and ChnSentiCorpHou, while others are shown in Appendix D). Obviously, we could easily find that the tendencies of these three measures are similar to the test accuracies to some extent. As revealed in the Equation 3-7, distance-based measures (CSM and HM) are more exact than the one based on number of instances (SI), while HM is easily influenced by some specific examples and loses effectiveness. Among these three measures, the CSM is more stable. To make a quantitative analysis, we use the Pearson’s correlation coefficient (PCC) to evaluate the relations. The average of PCC in Table 2 also indicates that the CSM is more relevant to test accuracy. As a result, we choose the CSM measure to help us truncate BERT.

As shown in Figure 2 and Appendix D, when the number of transformer layers are 7, 11, 12, 10, 11, 12, 5 and 9, the CSM values reach the maximum on ChnSentiCorpHou, WaiMai, SST-2, CoLA, Nikon-JD, AFQMC, MRPC and RTE, respectively. So we accept these suggestions and truncate BERT for these tasks. In this way, the test accuracies of our truncated model are 93.05%, 90.83%, 92.26%, 85.96%, 97.31%, 74.12%, 76.52% and 63.68%.

The test accuracy, processing time and scale of parameters are listed in Table 3, Table 4 and Table 5, respectively. Obviously, our method, Truncated-CSM, is capable of the 99% language understanding capabilities of the full model in average, and almost surpasses the other compressing methods on all these tasks except the MRPC. Also, about 20% of time and memory are saved both in fine-tuning and test with little additional cost, which is impossible for knowledge distillation and pruning. What’s more, our method is simple to implement and transfer to other tasks. On the contrary, we can draw conclusions from Table 4 and Table 5 that distillation methods are too complex to apply and consume far more time and storage from building up the teacher model to obtaining the student. The pruning methods are of the same circumstances in fine-tuning. The overhead of knowledge distillation and pruning methods in fine-tuning is awful and unbearable. However, we do not get satisfying predictions with these two kinds of ways.

According to these results, we confirm our method is effective and provide the following suggestions:

• •

  Only 1 layer of transformer encoder is enough to perform well on most simple tasks.

• •

  For some complex tasks, a few more layers (3-5 layers) could produce satisfying results.

• •

  More exact estimation could be made by randomly sampling some instances to calculate CSM value.