What Makes a Star Teacher? A Hierarchical BERT Model for Evaluating Teacher’s Performance in Online Education

Based on traditional education literature, we extract two sets of course features in our data – teachers’ verbal cues and meta-information features.

We present the results for the correlation analysis in Table 2. In particular, we find some interesting patterns that may guide teachers in an online education setting to enhance students’ evaluations in the future. We expand the statistically significant results for negative ($-$) and positive ($+$) correlations with the instructor rating.

• •

  ($+$) Question ratio. This suggests that the question-asking strategy may enhance students’ learning experience, regardless of the lack of direct interaction with students in the online education setting. Because it can potentially arouse students’ thinking and attract students’ attention.

• •

  ($+$) Emotion appealing. This indicates that the entertainment value of a course is also crucial, and it is consistent with online consumer experience theory in the marketing literature (Woltman Elpers, Wedel, and Pieters 2003). In a traditional course in the physical classroom, the teacher and students can have direct interactions and eye-contact, and students are required to sit in the classroom until the class ends. However, in the online education setting, students can leave the course whenever they want and it becomes even more challenging for the teacher to engage students in a class. Teachers who tend to use an emotion appealing strategy could potentially enhance students’ experience and better engage students in the online class.

• •

  ($-$) Hedging. This refers to those teachers who tend to use more hedging words such as “a little”, “kind of”, and “more or less” might be less committed to the content of their speech. Therefore, a higher percentage of hedging words might be a negative signal of teaching quality.

• •

  ($+$) Course structure quality. This indicates that the structure quality of the online course might be an essential factor in students’ online learning. A well-structured course is an indicator of a teacher’s effort in preparing the course and its potential high quality.

• •

  ($+$) Course length. The length of a course is an indicator of the amount of content contained in the course. A longer course may enable the students to acquire more knowledge from the course and thus result in more positive evaluations.

It is worth noting that our study only reveals the correlations between these signals and the ratings, and this study aims to build a better rating prediction system. However, we caution that strong correlations do not necessarily imply causality.

We experiment with two prediction tasks – instructor rating and course rating prediction. We split 70%, 10%, and 20% of courses as training, validation, and test set. We minimize the MSE loss for training. We take pretrained BERT_base with 12 layers to encode local semantic features from each lecture. Pretrained model weights are obtained from Pytorch transformer repository³³3https://github.com/huggingface/PyTorch-transformers. We also update the BERT_base parameters during training. The global transformer layers are trained from scratch. Besides, we set the number of global transformer layers as 2 based on the preliminary experiments. We find the 2 layer global transformer layers work much better than 1 layer. Due to the resource limitation, we did not try three layers, and we leave this to future work. We set the number of the final linear layer as 1. All transformer-based models/layers have 768 hidden units. The model is trained on Nvidia V100 GPU. The optimizer is Adam (Kingma and Ba 2014) with learning rate of 3e-5, $\beta_{1}$ = 0.9, and $\beta_{2}$ = 0.998; we also applied learning rate warmup over the first 500 steps, and decay as in (Vaswani et al. 2017). We set the epochs as 50, batch size as 16. We apply dropout (with a probability of 0.1) before all linear layers.

Our proposed hierarchical course BERT model is compared with various baselines.

(1) Extracted course features only. Since extracted course features are part of our proposed hierarchical course BERT model. If the extracted course features have excellent prediction performance, there is no need to build an advanced model. The regression with extracted course features is conducted using seven machine learning algorithms to control for the variability, including linear regression, support vector machines with the radial based kernel (SVM), Multi-layer Perceptron (MLP), Random Forest, AdaBoost, GradientBoosting, and Bagging. For the ensemble method like AdaBoost and Bagging, we use a decision tree as a base component in the ensemble. We perform a grid search in $\{10,50,100\}$ to find the optimal number of components for ensemble methods on the validation set. For MLP, we perform a grid search to find the optimal number of hidden layers ($H_{l}$) and hidden size ($H_{s}$) ($H_{l}\in\{1,2,3\}$, $H_{s}\in\{20,50,80,100\}$). Adam optimizer is used to optimize the parameters.

(2) Word2Vec + extracted course features. The pre-trained word2vec model on the entire Google News dataset is adopted to obtain 300-dimensional word embedding. Besides, all words in the same course are added to represent the course. These semantic features are similar to BERT features without considering structure information. Moreover, the extracted course features are concatenated to the course word2vec embedding. We repeat the same machine learning algorithms and the hyperparameter tuning process as (1).

(3) Doc2Vec + extracted course features. The paragraph vector algorithm proposed by (Le and Mikolov 2014) is used to obtain 300-dimensional lecture embeddings. Using the Gensim implementation  (Rehurek and Sojka 2010), doc2vec models for 50 epochs are trained while words occurring less than 10 times in the respective training corpus are ignored. Doc2vec embeddings are trained over individuals lectures. Besides, we add the doc2vec embedding for lectures in the same course together as the course embedding features. Then, the extracted course features are concatenated to the course doc2vec embedding. We repeat the same machine learning algorithms and the hyperparameter tuning process as (1).

(4) Lecture BERT (LecBERT). In this baseline model, each lecture is regarded as a unit. This baseline does not have access to the course structure information. We assume lectures in the same course share the same rating. We assign the same rating score to the lectures in the same course for training. For each lecture, the lecture $[CLS]$ is obtained after the BERT model, and the linear layer directly is added after the BERT model to make lecture rating prediction. In the test stage, prediction for each lecture is first conducted; then, we take the average over all lectures in the same course to obtain the course rating. The pretrained BERT_base with 12 layers to encode semantic features from each lecture. We set the epochs as 5, batch size as 64. Other hyperparameters are same as proposed model.

We use the following two metrics to measure the prediction performance:

• •

  Root mean squared error (RMSE):

  $$\text{RMSE}=\sqrt{\frac{1}{N}\sum_{j=1}^{N}(y_{j}-\hat{y}_{j})^{2}}$$

  (6)

• •

  Mean Absolute Error (MAE):

  $$\text{MAE}=\frac{1}{N}\sum_{j=1}^{N}|y_{j}-\hat{y}_{j}|$$

  (7)

Besides, compared to course features only, adding additional semantic features (e.g., Word2vec or Doc2vec) can largely increase the performance for both instructor rating prediction task and course rating prediction task. For example, with Bagging, the group of Doc2vec + course reduces RMSE by 12.4% and MAE by 16.2% compared to the course features alone for instructor rating tasks. Also, adding Doc2vec embedding performs better than Word2Vec. This indicates (1) better semantic features and (2) combination of deep semantic features and extracted course features are two ways to get better performance. Our proposed model builds on these findings.

(1) StrucCourse BERT. The concatenated course features to the model are removed, and other specification from our proposed model is maintained, to study the importance of the sparse feature in addition to the dense features.

(2) Course BERT. For this model, the structure information is further removed from StrucCourse BERT (i.e., section embedding and lecture position embedding). However, we keep the global transformers to exchange the lecture information, suggesting that we drop the structure information while keeping the lectures information exchange. For this model, each lecture is first fed into the BERT model; then, the lecture $[CLS]$ tokens are directly put into transformers without adding the section embedding and lecture position embedding to explore the importance of course structure information.

(3) Lecture BERT. The lecture exchange information is further removed from Course BERT. Specifically, we remove the global transformer layers and treat each lecture as an independent.

The results are summarized in Figure 4. We find the course structure information is the most important element in our model. The course sparse features and global transformers provide additional gains. Some interesting findings are highlighted as follows.

• •

  Extracted course features can further improve the performance. Adding the extracted course features can further improve the performance. Compared to StrucCourse BERT, the model with additional extracted course features can provide additional gains. Regarding the instructor rating prediction task, our proposed model reduces RMSE by 7.1% and MAE by 19.7% compared to StrucCourse BERT. Regarding the course rating prediction task, our proposed model reduces RMSE by 12.5% and MAE by 17.9% compared to StrucCourse BERT.

• •

  Course structure information provides a significant gain. Adding section embedding and lecture position embedding to represent the natural course structure can significantly improve the performance. Compared to the Course BERT, StrucCourse BERT with additional section embedding and lecture position embedding reduces RMSE by 33.9% and MAE by 39.5% for the instructor rating prediction task. For the course rating prediction task, StrucCourse BERT reduces RMSE by 28.0% and MAE by 43.1% compared to the Course BERT.

• •

  Global transformers to exchanging the lecture information are important. Compared to Lecture BERT treating lectures as independent, the Course BERT using global transformers to exchange the lectures information within the same course exhibits better prediction performance. For example, regarding the instructor rating prediction task, the Course BERT reduces RMSE by 7.7% and MAE by 10.7% compared to Lecture BERT; regarding the course rating prediction task, the Course BERT reduces MAE by 12.5% compared to Lecture BERT.