On the Use of BERT for Automated Essay Scoring:
Joint Learning of Multi-Scale Essay Representation

The AES task is defined as following:

Given an essay with $n$ words $X=\{x_{i}\}^{n}_{i=1}$, we need to output one score $y$ as a result of measuring the level of this essay.

Quadratic weighted Kappa (QWK) (Cohen, 1968) metric is commonly used to evaluate AES systems by researchers, which measures the agreement between the scoring results of two raters.

We obtain the multi-scale essay representation from three scales: token-scale, segment-scale and document-scale.

Token-scale and Document-scale Input We apply one pre-trained BERT (Devlin et al., 2019) model for token-scale and document-scale essay representations. The BERT tokenizer is used to split the essay into a token sequence $T_{1}=[t_{1},t_{2},......t_{n}]$, where $t_{i}$ is the $i$th token and $n$ is the number of the tokens in the essay. The token we mentioned in this paper all refer to WordPiece, which is obtained by the subword tokenization algorithm used for BERT. We construct a new sequence $T_{2}$ from $T_{1}$ as following. $L$ is set to 510, which is the max sequence length supported by BERT except the token $[CLS]$ and $[SEP]$.

$T_{2}=\left\{\begin{array}[]{lcl}$[CLS]$+[t_{1},t_{2},..,t_{L}]+$[SEP]$&&{n>L}\\ $[CLS]$+T_{1}+$[SEP]$&&{n=L}\\ $[CLS]$+T_{1}+$[PAD]$*(L-n)+$[SEP]$&&{n<L}\\ \end{array}\right.$

The final input representation are the sum of the token embeddings, the segmentation embeddings and the position embeddings. A detailed description can be found in the work of BERT (Devlin et al., 2019).

Document-scale The document-scale representation is obtained by the $[CLS]$ output of the BERT model. As the $[CLS]$ output aggregates the whole sequence representation, it attempts to extract the essay information from the most global granularity.

Token-scale As the BERT model is pre-trained by Masked Language Modeling (Devlin et al., 2019), the sequence outputs can capture the context information to represent each token. An essay often consists of hundreds of tokens, thus RNN is not the proper choice to combine all the token information due to the gradients vanishing problem. Instead, we utilize a max-pooling operation to all the sequence outputs and obtain the combined token-scale essay representation. Specifically, the max-pooling layer generates a $d$-dimensional vector ${W=[w_{1},w_{2},...,w_{j},...,w_{d}]}$ and the element $w_{j}$ is computed as below:

$w_{j}=max\{h_{1,j},h_{2,j},...,h_{n,j}\}$

where $d$ is the hidden size of the BERT model. As we use the pre-trained BERT model bert-base-uncased⁴⁴4https://huggingface.co/bert-base-uncased, the hidden size $d$ is 768. All the $n$ sequence outputs of the BERT model are annotated as $[h_{1},h_{2},...,h_{i},...,h_{n}]$, where $h_{i}$ is a $d$-dimensional vector $[h_{i,1},h_{i,2},...,h_{i,d}]$ representing the $i$th sequence output, and $h_{i,j}$ is the $j$th element in $h_{i}$.

Segment-scale Assuming the segment-scale value set is $K=[k_{1},k_{2},...k_{i},...,k_{S}]$, where $S$ is the number of segment scales we want to explore, and $k_{i}$ is the $i$th segment-scale in $K$. Given a token sequence $T_{1}=[t_{1},t_{2},......t_{n}]$ for an essay, we obtain the segment-scale essay representation corresponding to scale $k_{i}$ as follows:

1. 1.

   We define $n_{p}$ as the maximum number of tokens corresponding to each essay prompt $p$. We truncate the token sequence to $n_{p}$ tokens if the essay length is longer than $n_{p}$, otherwise we pad $[PAD]$ to the sequence to reach the length $n_{p}$.

2. 2.

   Divide the token sequence into $m=\lceil{n_{p}/k_{i}}\rceil$ segments and each segment is of length $k_{i}$ except for the last segment, which is similar to the work of  (Mulyar et al., 2019).

3. 3.

   Input each of the $m$ segment tokens into the BERT model, and get $m$ segment representation vectors from the $[CLS]$ output.

4. 4.

   Use an LSTM model to process the sequence of $m$ segment representations, followed by attention pooling operation on the hidden states of the LSTM output to obtain the segment-scale essay representation corresponding to scale $k_{i}$.

The LSTM cell units process the sequence of segment representations and generate the hidden states as follows:

where $s_{t}$ is the $t$th segment representation from BERT $[CLS]$ output and $h_{t}$ is the $t$th hidden state generated from LSTM. $Q_{i}$, $Q_{f}$, $Q_{c}$, $Q_{o}$, $U_{i}$, $U_{f}$, $U_{c}$ and $U_{o}$ are weight matrices, and $b_{i}$, $b_{f}$, $b_{c}$, and $b_{o}$ are bias vectors.

The attention pooling operation we use is similar to the work of (Dong et al., 2017), which is defined as follows:

$o$ is the segment-scale essay representation corresponding to the scale $k_{i}$. $\alpha_{t}$ is the attention weight for hidden state $h_{t}$. $Q_{a}$, $b_{a}$, $q_{a}$ are the weight matrix, bias and weight vector respectively.

The model architecture is depicted in Figure 1.

We apply one BERT model to obtain the document-scale and token-scale essay representation. The concatenation of them is input into a dense regression layer which predicts the score corresponding to the document-scale and token-scale. For each segment-scale $k$ with number of segments $m$, we apply another BERT model to get $m$ $CLS$ outputs, and apply an $LSTM$ model followed by an attention layer to get the segment-scale representation. We input the segment-scale representation into another dense regression layer to get the score corresponding to segment-scale $k$. The final score is obtained by adding the scores of all $S$ segment-scales and the score of the document-scale and token-scale, which is illustrated as below:

$y_{k}$ is the predicted score corresponding to segment-scale $k$. $y_{doc,tok}$ is the predicted score corresponding to the document-scale and token-scale. $\hat{W}_{seg}$ and $b_{seg}$ are weight matrix and bias for segment-scale respectively. $W_{doc,tok}$ and $b_{doc,tok}$ are weight matrix and bias for document and token-scales, $o_{k}$ is the segment-scale essay representation with the scale $k$. $w_{doc}$ is the document-scale essay representation. $W$ is the token-scale essay representation. $H_{doc,tok}$ is the concatenation of document-scale and token-scale essay representations.

We use three loss functions to train the model.

MSE measures the average value of square errors between predicted scores and labels, which is defined as below:

$MSE(y,\hat{y})=\frac{1}{N}\sum_{i}(y_{i}-\hat{y}_{i})^{2}$

where $y_{i}$ and $\hat{y}_{i}$ are the predicted score and the label for the $i$th essay respectively, $N$ is the number of the essays.

Similarity (SIM) measures whether two vectors are similar or dissimilar by using cosine function. A teacher takes into account the overall level distribution of all the students when rating an essay. Following such intuition, we introduce the SIM loss to the AES task. In each training step, we take the predicted scores of the essays in the batch as the predicted vector $y$, and the labels as the label vector $\hat{y}$. The SIM loss awards the similar vector pairs to make the model think more about the correlation among the batch of essays. The SIM loss is defined as below:

where $y_{i}$ and $\hat{y}_{i}$ are the predicted score and label for the $i$th essay respectively, $N$ is the number of the essays.

Margin Ranking (MR) measures the ranking orders for each essay pair in the batch. We intuitively introduce MR loss because the sorting property between essays is a key factor to scoring. For each batch of essays, we first enumerate all the essay pairs, and then compute the MR loss as follows. The MR loss attempts to make the model penalize wrong order.

$MR(y,\hat{y})=\frac{1}{\hat{N}}{\sum_{i,j}max(0,-r_{i,j}(y_{i}-y_{j})+b)}$ $r_{i,j}=\left\{\begin{array}[]{lcl}$1$&&{\hat{y}_{i}>\hat{y}_{j}}\\ $-1$&&{\hat{y}_{i}<\hat{y}_{j}}\\ $-$sgn(y_{i}-y_{j}$)$&&{\hat{y}_{i}=\hat{y}_{j}}\\ \end{array}\right.$

$y_{i}$ and $\hat{y}_{i}$ are the predicted score and label for the $i$th essay respectively. $\hat{N}$ is the number of the essay pairs. $b$ is a hyper parameter, which is set to 0 in our experiment. For each sample pair $(i,j)$, when the label $\hat{y}_{i}$ is larger than $\hat{y}_{j}$, the predicted result $y_{i}$ should be larger than $y_{j}$, otherwise, the pair contributes $y_{j}-y_{i}$ to the loss. When $\hat{y}_{i}$ is equal to $\hat{y}_{j}$, the loss is actually $\left|y_{i}-y_{j}\right|$.

The combined loss is described as below:

$Loss_{total}(y,\hat{y})=\alpha MSE(y,\hat{y})+\beta MR(y,\hat{y})+\gamma SIM(y,\hat{y})$.

$\alpha$, $\beta$, $\gamma$ are weight parameters which are tuned according to the performance on develop set.

ASAP data set is widely used in the AES task, which contains eight different prompts. A detailed description can be seen in Table 1. For each prompt, the WordPiece length indicates the smallest number which is bigger than the length of 90% of the essays in terms of WordPiece number. We evaluate the scoring performance using QWK on ASAP data set, which is the official metric in the ASAP competition. Following previous work, we adopt 5-fold cross validation with 60/20/20 split for train, develop and test sets.

CRP data set provides 2834 excerpts from several time periods and reading ease scores which range from -3.68 to 1.72. The average length of the excerpts is 175 and the WordPiece length is 252. We also use 5-fold cross validation with 60/20/20 split for train, develop and test sets on CRP data set. As the RMSE metric is used in the CRP competition, we also use it to evaluate our system in ease score prediction task.

The baseline models for comparison are described as follows.

EASE ⁵⁵5http://github.com/edx/ease is the best open-source system that participated in the ASAP competition and ranked the third place among 154 participants. EASE uses regression techniques with handcrafted features. Results of EASE with the settings of Support Vector Regression (SVR) and Bayesian Linear Ridge Regression (BLRR) are reported in (Phandi et al., 2015).

CNN+RNN Various deep neural networks based on CNN and RNN for AES are studied by (Taghipour and Ng, 2016). They combine CNN ensembles and LSTM ensembles over 10 runs and get the best result in their experiment.

Hierarchical LSTM-CNN-Attention (Dong et al., 2017) builds a hierarchical sentence-document model, which uses CNN to encode sentences and LSTM to encode texts. The attention mechanism is used to automatically determine the relative weights of words and sentences in generating sentence representations and text representations respectively. They obtain the state-of-the-art result among all neural models without pre-training.

SKIPFLOW (Tay et al., 2018) proposes to use SKIPFLOW mechanism to model the relationships between snapshots of the hidden representations of an LSTM. The work of  (Tay et al., 2018) also obtains the state-of-the-art result among all neural models without pre-training.

Dilated LSTM with Reinforcement Learning (Wang et al., 2018) proposes a method using a dilated LSTM network in a reinforcement learning framework. They attempt to directly optimize the model using the QWK metric which considers the rating schema.

HA-LSTM+SST+DAT and BERT+SST+DAT  (Cao et al., 2020) propose to use two self-supervised tasks and a domain adversarial training technique to optimize their training, which is the first work to use pre-trained language model to outperform LSTM based methods. They experiment with both hierarchical LSTM model and BERT in their work, which are $HA-LSTM+SST+DAT$ and $BERT+SST+DAT$ respectively.

BERT²  (Yang et al., 2020) combines regression and ranking to fine-tune BERT model which also outperforms LSTM based methods and even obtains the new state-of-the-art.

To compare with the baseline models and further study the effectiveness of multi-scale essay representations, losses and transfer learning, we conduct the following experiments.

Multi-scale Models. These models are optimized with MSE loss, and BERT-DOC represents essays with document-scale features based on BERT. BERT-TOK represents essays with token-scale features based on BERT. BERT-DOC-TOK represents essays with both document-scale and token-scale features based on BERT. BERT-DOC-TOK-SEG represents essays with document-scale, token-scale, and multiple segment-scale features based on BERT. Longformer (Beltagy et al., 2020) is an extension for transformers with an attention mechanism that scales linearly with sequence length, making it easy to process long documents. We conduct experiments to show that our multi-scale features also works with Longformer and can further improve the performance in long text tasks. Longformer-DOC-TOK-SEG uses document-scale, token-scale, and multiple segment-scale features to represent essays, but based on Longformer instead of BERT. Longformer-DOC represents essays with document-scale features based on Longformer.

Models with Transfer Learning. To transfer learn from the out-of-domain essays ⁶⁶6For each prompt, we use all the essays from other prompts in ASAP data set., we additionally employ a pre-training stage, which is similar to the work of (Song et al., 2020). In this stage, we scale all the labels of essays from out-of-domain data into range 0-1 and pre-train the model on them with MSE loss. After the pre-training stage, we continue to fine-tune the model on in-domain essays. Tran-BERT-MS has the same modules as BERT-DOC-TOK-SEG with pre-training on out-of-domain data. MS means multiple scale features.

Models with Multiple Losses. Based on Tran-BERT-MS model, we explore the performance of adding multiple loss functions. Tran-BERT-MS-ML additionally employs MR loss and SIM loss. ML means multiple losses. Tran-BERT-MS-ML-R incorporates R-Drop strategy (Liang et al., 2021) in training based on Tran-BERT-MS-ML model.

For the proposed model architecture which is depicted in Figure 1, the BERT model in the left part are shared by the document-scale and token-scale essay representations, and the other BERT model in the right part are shared by all segment-scale essay representations. We use the "bert-base-uncased" which includes 12 transformer layers and the hidden size is 768. In the training stage, we freeze all the layers in the BERT models except the last layer, which is more task related than other layers. The Longformer model used in our work is "longformer-base-4096". For the MR loss, we set $b$ to 0. The weights $\alpha$, $\beta$ and $\gamma$ are tuned according to the performance on develop set. We use Adam optimizer (Kingma and Ba, 2015) to fine-tune model parameters in an end-to-end fashion with learning rate of 6e-5, $\beta 1$=0.9, $\beta 2$=0.999, $L2$ weight decay of 0.005. The coefficient weight $\alpha$ in R-Drop is 9. We set the batch size to 32. We use dropout in the training stage and the drop rate is set to 0.1. We train all the models for 80 epochs, and select the best model according the performance on the develop set. We use a greedy search method to find the best combination of segment scales, which is shown in detail in Appendix A. Following (Cao et al., 2020), we perform the significance test for our models.

Table 2 shows the performance of baseline models and our proposed models with joint learning of multi-scale essay representation. Table 3 shows the results of our model and the state-of-the-art models on essays in prompt 1, 2 and 8, whose WordPiece length are longer than 510. We summarize some findings from the experiment results.

• •

  Our model 12 almost obtains the published state-of-the-art for neural approaches. For the prompts 1,2 and 8, whose WordPiece length are longer than 510, we improve the result from 0.761 to 0.772. As Longformer is good at encoding long text, we also use it to encode essays of prompt 1, 2 and 8 directly but the performance is poor compared to the methods in Table 3. The results demonstrate the effectiveness of the proposed framework for encoding and scoring essays. We further re-implement BERT² proposed by  (Yang et al., 2020), and our implementation of BERT² is not as well-performing as the published result. Though  (Uto et al., 2020) obtain a much better result(QWK 0.801), our method performs much better than their system with only neural features(QWK 0.730), which demonstrates the strong essay encoding ability of our neural approach.

• •

  Compared to the models 4 and 6, our model 11 uses multi-scale features to encode essays instead of LSTM based models, and we use the same regression loss to optimize the model. Our model simply changes the representation way and significantly improves the result from 0.764 to 0.782, which demonstrates the strong encoding ability armed by multi-scale representation for long text. Before that, the conventional way of using BERT can not surpass the performance of models 4 and 6.

Multi-scale Representation We further analyze the effectiveness of employing each scale essay representation to the joint learning process.

Table 4 and Table 5 show the performance of our models to represent essays on different feature scales, which are trained with MSE loss and without transfer learning. Table 4 shows the performance on ASAP data set while Table 5 shows the performance on CRP data set. The improvement of BERT-DOC-TOK-SEG over BERT-DOC, BERT-TOK, BERT-DOC-TOK are significant ($p$<0.0001) on CRP data set, and are significant ($p$<0.0001) in most cases on ASAP data set. Results on both table indicate the similar findings.

• •

  Combining the features from document-scale and token-scale, BERT-DOC-TOK outperforms the models BERT-DOC and BERT-TOK, which only use one scale features. This demonstrates that our proposed framework can benefit from multi-scale essay representation even with only two scales.

• •

  By additionally incorporating multiple segment-scale features, BERT-DOC-TOK-SEG performs much better than BERT-DOC-TOK. This demonstrates the effectiveness and generalization ability of our multi-scale essay representation on multiple tasks.

  Models	Average QWK
  Longformer-DOC	0.746
  Longformer-DOC-TOK-SEG	0.771

Reasons for Effectiveness of Multi-scale Representation Though the experiment shows the effectiveness of multi-scale representation, we further explore the reason. We could doubt that the effectiveness comes from supporting long sequences, not the multi-scale itself. As Longformer is good at dealing with long texts, we compare the results between Longformer-DOC and Longformer-DOC-TOK-SEG. The results of the significance test show that the improvement of Longformer-DOC-TOK-SEG over Longformer-DOC are significant ($p$<0.0001) in most cases. Performance of the two models are shown in Table 6, and we get the following findings.

• •

  Though Longformer-DOC supports long sequences encoding, it performs poor, which indicates us that supporting long sequence ability is not enough for a good essay scoring system.

• •

  Longformer-DOC-TOK-SEG outperforms Longformer-DOC significantly, which indicates the effectiveness of our model comes from encoding essays by multi-scale features, not only comes from the ability to deal with long texts.

These results are consistent with our intuition that our approach takes into account different level features of essays and predict the scores more accurately. We consider it caused by that multi-scale features are not effectively constructed in the representation layer of pre-trained model due to the lack of data for fine-tuning in the AES task. Therefore, we need to explicitly model the multi-scale information of the essay data and combine it with the powerful linguistic knowledge of pre-trained model.

Transfer Learning with Multiple Losses and R-Drop We further explore the effectiveness of pre-training with adding multiple loss functions and employing R-Drop. As is shown in table 7, by incorporating the pre-training stage which learns the knowledge from out-of-domain data, Tran-BERT-MS model improves the result from 0.782 to 0.788 compared to BERT-DOC-TOK-SEG model. The model Tran-BERT-MS-ML which jointly learns with multiple loss functions further improves the performance from 0.788 to 0.790. We consider it due to the reason that MR brings ranking information and SIM takes into account the overall score distribution information. Diverse losses bring different but positive influence on the optimization direction and act as an ensembler. By employing R-Drop, Tran-BERT-MS-ML-R improves the QWK slightly, which comes from the fact that R-Drop plays a regularization role.