Jointly Modeling Heterogeneous Student Behaviors and Interactions Among Multiple Prediction Tasks

Recurrent Neural Networks (RNNs) have been widely adopted to model sequence data and have achieved good performance in various domains such as NLP (Sutskever et al., 2014). There are many well-known variants of RNN models, such as LSTM (Hochreiter and Schmidhuber, 1997), GRU (Chung et al., 2014), bidirectional LSTM (Graves and Schmidhuber, 2005). Hidasi et al. (Hidasi et al., 2016) firstly introduced GRU to recommender systems. They leveraged GRU to model click behaviors of users. Zhu et al. (Zhu et al., 2017) proposed a variant of LSTM called Time-LSTM. They proposed that it is important to exploit the time information when modeling users’ behaviors. To achieve this goal, they equipped LSTM with time gates to model time intervals.

Academic performance prediction task is the most frequently studied among prediction tasks about students. In this paper, we choose Weighted Average Grade (WAG) which is on a 100-point scale to quantitatively describe the academic performance of a student in one semester. WAG can be seen as term GPA.

There are various factors that impact the academic performance in complex ways (Khan and Ghosh, 2021). Student demographic information and historical academic performance are widely explored by researchers (Shahiri et al., 2015; Khan and Ghosh, 2021). Some technologies such as online learning platforms are used in some courses nowadays. Students’ digital records collected by online learning platforms such as logs about video-watching behavior, time spent on specific questions, and test/quiz grades have been leveraged (Brinton and Chiang, 2015; Calvo-Flores et al., 2006; Romero et al., 2013; Lopez et al., 2012; Minaei-Bidgoli et al., 2003; Khan and Ghosh, 2021). A few studies find that effective teaching motivates the student to perform better (Khan and Ghosh, 2018, 2021). Besides, some studies explore other factors such as attitudes towards study (Osmanbegovic and Suljic, 2012). As for employing students’ behaviors on campus to predict academic performance, Wang et al. (Wang et al., 2015) found correlations between students’ cumulative GPAs and automatic sensing behavioral data obtained from smartphones. However, the passive sensing behavioral data they used is only collected from a small number of students and the collecting way is not universal enough. Yao et al. (Yao et al., 2017) studied the effect of social influence on predicting academic performance based on students’ multiple behaviors. The effect of students’ behaviors is very indirect. Zhang et al. (Zhang et al., 2018) extracted statistics features and relevance features from students’ multiple behaviors and used these features to predict academic performance.

Most methods used for academic performance prediction are based on traditional data mining techniques. Khan and Ghosh (Khan and Ghosh, 2018) regarded the student evaluation of teaching excellence and the number of student evaluations as input and used association rule mining to establish the relationship between teaching and academic performance. Feng et al. (Feng et al., 2009) did a stepwise linear regression to predict academic performance using the online measures as independent variables. Sweeney et al. (Sweeney et al., 2015) explored the factorization machine, a general-purpose matrix factorization algorithm based on (student, course) dyads. Xu et al. (Xu et al., 2017) leveraged an ensemble learning method to predict academic performance continuously in the program based on the evolving academic performances. Besides traditional methods, some researchers seek novel solutions with the help of deep learning. Wang and Liao (Wang and Liao, 2011) gathered information about gender, personality type, and anxiety level through questionnaires. Then they adopted a feed forward neural network to predict academic performance. LSTMs are also explored. Fei and Yeung (Fei and Yeung, 2015) leveraged an LSTM to model student weekly activities and predict academic performance in Massive Open Online Courses (MOOCs).

These studies will ignore the influence of profile information when modeling student daily behaviors and treat all days equally.

Book-borrowing predictions are meaningful and have been studied by many researchers. Cano et al. (Cano et al., 2018) analysed student book-borrowing behavior during the semester through clustering and association rule algorithms. They proposed that librarians could take better decisions (such as decide the quantity to offer per topic and human resources needed to satisfy the demand) for their users after knowing the analysis results.

Besides, most existing studies on book-borrowing prediction focus on predicting library circulation. Tian (Tian, 2011) regarded the time series of library circulation flow as the chaotic time series and leveraged support vector regressions to predict library circulation of coming months. They mentioned that when the number of enrollments in a college increases sharply, the importance of book-borrowing predictions comes out conspicuously. Kumar and Raj (Kumar and Raj, 2016) found that the number of borrowed books one month and twelve months earlier could estimate the number of borrowed books in a month with an autoregressive integrated moving average model. Wang et al. (Wang et al., 2012) leveraged a feed forward neural network to predict library circulation of next five days based on data of previous twenty days. These works proposed that they were of positive significance for library capacity planning, management of library books and staff, and library acquisition budget supporting. These works ignore the impact of individuals.

Existing studies on course-failing prediction mainly focus on classifying students into two categories: either pass or fail for a given course. Existing studies do not distinguish this task from academic performance prediction task (Yu et al., 2011). If there is only one course, these two tasks are the same. Once we knew one student’s mark, we know whether the student failed the course or not and vice-versa. If there is more than one course, these two tasks are different. For example, if student $s$ gets $80$ points in course $c_{1}$ ($3$ credits) and gets $50$ points in course $c_{2}$ ($3$ credits), the academic performance of student $s$ is $65$ which is above $60$ points. But, the number of failed courses of this student is $1$ (he/she fails course $c_{2}$).

Actually, both the two tasks could be equally important. According to college rules of student management, either a low Weighted Average Grade or a certain number of failed courses lead to the academic probation even the academic dismissal. Thus, to effectively predict those students who may violate college academic requirements, both the academic performance prediction and the course-failing prediction are needed. Besides, some students may be more concerned with their Weighted Average Grades, because they may apply for graduate studies. Knowing their Weighted Average Grades previously would help them to study harder. Nevertheless, some students may be more concerned with whether they fail or not, because if they failed a course, they would not be qualified for student loans. In this case, knowing whether they fail or not would help them to act accordingly.

Methods used in course-failing prediction include k-nearest neighbour methods (Tanner and Toivonen, 2010), ensemble methods (Yu et al., 2011), feed forward neural networks (Sukhbaatar et al., 2019) and so on. In detail, Tanner and Toivonen (Tanner and Toivonen, 2010) took student status features, student performance features, lesson requirement features, and customer demographic features as possible input of a k-nearest neighbor method to do course-failing prediction in an online course setting. Yu et al. (Yu et al., 2011) did feature engineering and used a random forest algorithm (i.e., an ensemble method) to predict whether the student will be correct on the first attempt for a step based on interaction logs generated in intelligent tutoring systems. Sukhbaatar et al. (Sukhbaatar et al., 2019) employed a feed forward neural network on the set of prediction factors extracted from the online learning activities of students to identify students at risk of failing in a course.

Multi-task learning (MTL) was first analyzed by Caruana (Caruana, 1997) in detail. Multi-task learning could improve learning efficiency and prediction accuracy for each task when compared to training a separate model for each task. One important reason is that multi-task learning allows sharing of statistical strength and transferring of knowledge between related tasks. Multi-task learning has been used successfully in many fields, such as computer vision (Dai et al., 2016; Ranjan et al., 2017), natural language processing (Collobert and Weston, 2008), urban computing (Lu et al., 2020).

Student $s$’ profile information $\bm{D}^{s}$ is represented in the form of one high-dimensional vector including many one-hot encoded vectors. To reduce the dimension and get a better representation, we use a dense embedding layer. The transformation is formalized as:

where $\bm{W}_{D}$ is the mapping matrix.

Thinking that the combination of entering the library and going back to the dormitory behaviors can reveal students’ academic efforts, we extract the daily behavior sequence of entering the library and the daily behavior sequence of going back to the dormitory from $\{r_{Lib}\}$ and $\{r_{Dorm}\}$ respectively.

More specifically, we divide one day into $24$ time slots by hour (i.e., $[00\!:\!00,01\!:\!00),[01\!:\!00,02\!:\!00),...,[23\!:\!00,24\!:\!00)$). According to $\{r_{Lib}\}$, almost 100% of entering the library records are generated in 07:00-23:00. So we use $16$ elements to record entering the library frequency in 07:00-23:00 for each day. In this way, we get the daily behavior sequence of entering the library: $\bm{B}_{1,1},\bm{B}_{1,2},...,\bm{B}_{1,x},...,\bm{B}_{1,X}$ ($x$ is the index. $X\!=\!63$ since we make predictions after the first half of the semester (63 days).).

One going back to the dormitory record is defined as the last record of entering the dormitory of the day. Based on $\{r_{Dorm}\}$, around 84% of going back to the dormitory records are generated in 18:00-24:00. So we use a vector with a length of $6$ to record the situation of going back to the dormitory in 18:00-24:00 for each day. In this way, we get the behavior sequence of going back to the dormitory: $\bm{B}_{2,1},\bm{B}_{2,2},...,\bm{B}_{2,x},...,\bm{B}_{2,X}$.

Similar behaviors of different kinds of students may mean differently. We propose a variant of LSTM called Profile-aware LSTM. We treat student profile information as a strong signal in the gates of Profile-aware LSTM (as Equation (2), (3) and (5) show). That is to say, what to extract, what to remember, and what to forward are affected by student profile information. The behavior sequence is the input of Profile-aware LSTM (as Equation (4) shows).

Profile-aware LSTM model is formulated as follows and the detailed structure is shown in Figure 3:

where $\bm{B}_{m,x}^{s}$ and $\bm{h}_{m,x}^{s}$ are one input element and the corresponding output of Profile-aware LSTM unit, i.e., hidden state at time $x$, respectively. $\bm{\bar{D}}^{s}$ is calculated with Equation (1). $m=1,2$. $1\!\leq\!x\!\leq\!X\!=\!63$. $\bm{W}$ terms denote weight matrices and $\bm{b}$ terms are bias vectors. $\sigma$ is the element-wise sigmoid function and $\odot$ is the element-wise product.

We use two Profile-aware LSTMs to model two kinds of behaviors respectively.

The hidden representation of all heterogeneous behaviors of the $x$-th day can be formalized as:

where $\oplus$ is the concatenation operation.

Behaviors of different days to the task will have different degrees of impact. Inspired by the success of the attention mechanism in machine translation (Bahdanau et al., 2015), we apply a novel soft-attention mechanism over Profile-aware LSTM to draw information from the sequence by different weights. In detail, we consider each vector $\bm{\hat{h}}_{x}^{s}$ as heterogeneous behaviors representation of the $x$-th day, and represent the sequence by a weighted sum of the vector representation of all days. The attention weight makes it possible to perform proper credit assignments to days according to their importance to the student. Mathematically, we compute soft-attention weight $\alpha_{x}^{s}$ with the following equations:

where $\bm{\bar{D}}^{s}$ is computed with Equation (1), $\bm{\hat{h}}_{x}^{s}$ is computed with Equation (7), $\bm{W}$ terms denote weight matrices, and $\bm{b}_{a}$ is the bias vector. Similar to Profile-aware LSTM, student profile information also contributes to the attention weights.

The advanced student behavior representation is generated using the following equation:

The student representation $\bm{R}^{s}$ is the concatenation of $\bm{\bar{D}}^{s}$ and $\bm{\bar{B}}^{s}$:

After obtaining the student representation, we concatenate it with task-related features. In detail, we have three tasks: PAP task, PNBB task, and PNFC task. For PAP task, the task-related information includes the historical WAG information and the involved course information; for PNBB task, the task-related information includes the historical number of borrowed books information; for PNFC task, the task-related information includes the historical number of failed courses and the involved course information as Figure 4 shows. Then we feed the concatenated features to stacked Multi-task Interaction Units. In what follows, we introduce how to handle the various kinds of task-related information and introduce the detailed structure of the Multi-task Interaction Unit.

We find that students get higher grades easily in some courses such as CS362 as Figure 5 shows. This means different courses have different levels of difficulty. So for each course, we extract descriptive statistics (i.e., minimum, maximum, median, first quartile, third quartile, mean, standard deviation) as features. These features can describe the properties of distribution from multiple aspects. The feature vector of course $c_{\theta}$ is represented as $\bm{e}_{1,\theta}$. Next, we aggregate feature vectors of all courses by leveraging course credit information:

Then we concatenate $\bm{R}^{s}$, $\bm{\hbar}_{1,T}^{s}$ with $\bm{v}_{1}^{s}$:

Then we concatenate $\bm{R}^{s}$ with $\bm{\hbar}_{2,T}^{s}$:

We also extract some features such as course failure rate and descriptive statistics. The feature vector of course $c_{\theta}$ is represented as $\bm{e}_{3,\theta}$. We merger information of every course by:

Then we concatenate $\bm{R}^{s}$, $\bm{\hbar}_{3,T}^{s}$ with $\bm{v}_{3}^{s}$:

Firstly, we utilize Fully Connected (FC) layers to make the inputs more task-specific and to let the inputs have the same dimensions, which can be written as:

where $\bm{W}$ terms and $\bm{b}$ terms are learnable parameters, and $\mathrm{PReLU}$ (He et al., 2015) is the nonlinear activation function. $\bm{\hat{R}}_{1}^{s,1}$, $\bm{\hat{R}}_{2}^{s,1}$, and $\bm{\hat{R}}_{3}^{s,1}$ are calculated with Equation (15), (17), and (20), respectively.

Then we utilize the co-attention mechanism to explicitly control the knowledge transfer. The co-attention mechanism was first proposed to jointly reasons about visual attention and question attention (Lu et al., 2016). The co-attention mechanism demonstrates the ability of capturing the complementary important information. With the co-attention mechanism, we can get the updated representation explicitly incorporating other tasks’ knowledge. Mathematically, we compute co-attention weights with the following equations:

where $\sigma$ is the sigmoid function. That is to say, we leverage dot product and sigmoid function to calculate the co-attention weights.

The updated representations are generated using the following equations:

After $L$ Multi-task Interaction Units, we can get the outputs of the last unit: $\bm{\hat{R}}_{1}^{s,L+1}$, $\bm{\hat{R}}_{2}^{s,L+1}$, and $\bm{\hat{R}}_{3}^{s,L+1}$. Then we adopt separate output layers to get the prediction results, which can be denoted as follows:

where $\bm{W}$ terms and $b$ terms are parameters.

We use the mean squared error (MSE) as the loss function for training the three tasks:

where $i$ denotes one training sample, $U$ is the number of training samples, and $\Phi$ terms are all trainable parameters. $y^{i}_{1,T+1}$, $y^{i}_{2,T+1}$, and $y^{i}_{3,T+1}$ denote the labels of the $i$-th sample.

The total loss is computed as the sum of the three individual losses:

where $\lambda$ terms are balance weights. Here, balance weights of 1 are implicitly used among the three tasks.

The training process is outlined in Algorithm 1. We adopt the adaptive moment estimation (Adam) (Kingma and Ba, 2015) as the optimizer.

In order to improve the generalization capability of our models, we adopt dropout (Srivastava et al., 2014) to prevent the potential overfitting problem.

The data were collected in a college with an enrollment of $10k$ undergraduate students. Dataset statistics are shown in Table 2.

To demonstrate the effectiveness of our proposed model, we compared our proposed model, i.e., DAPAMT, with various baselines.

• •

  HA: We give the prediction result by the average value of historical observations.

• •

  LSTM (Fei and Yeung, 2015): LSTM is widely used to model sequence data. We leverage LSTMs to model historical observation sequences.

• •

  BRR (Wang et al., 2015): Bayesian Ridge Regression (BRR) is a generalized linear model which has $L_{2}$ regularization.

• •

  SVR (Tian, 2011): Support Vector Regression (SVR) is a variant of support vector machine (SVM) for supporting regression tasks. SVR is a minimum-margin regression which could model the non-linear relation between features.

• •

  RF (Chen et al., 2019): Random Forest (RF) is an ensemble method with decision trees as base learners. It is based on the “bagging” idea.

• •

  GBDT: Gradient Boosting Decision Tree (GBDT) is another kind of ensemble method using decision trees as base learners. GBDT is a generalization of boosting to arbitrary differentiable loss functions.

• •

  MLP (Wang and Liao, 2011; Sukhbaatar et al., 2019): Multi-layer Perceptron (MLP) consists of multiple layers of nodes. Each layer is fully connected to the next layer in the network.

• •

  APAMT (Liu et al., 2020): Attentional Profile-Aware Multi-Task model (APAMT) is proposed in our preliminary work. It leverages the Profile-aware LSTM and the soft-attention mechanism to model the daily behavior sequence. Besides, it adopts a simpler multi-task learning framework compared with DAPAMT. Specifically, it implicitly models the interactions among tasks by setting shared layers and task-specific layers.

We adopt Mean Square Error (MSE) to quantify the distance between the predicted scores and the actual ones. MSE penalizes large errors more heavily than the non-quadratic metrics, and thus takes higher numerical values. Given our tasks, if a method performs very well for half the students and poorly for the other half, we still think it is not a good method. Hence we select MSE as the metric to compare methods. The smaller the values are, the better results the method has. The metric is defined as:

where $i$ denotes one testing instance, $U$ is the number of testing samples, $z^{i}$ is the actual value, and $\tilde{z}^{i}$ is the predicted value. For better understanding the metric and actual data, we analyze our dataset. The minimum WAG is $26.83$ and the maximum WAG is $97.82$ in our dataset. Thus the MSE range could be between $0$ and $5039.58$ on PAP task. The minimum number of borrowed books is $0$ and the maximum number is $137$ in our dataset. Thus the MSE range could be between $0$ and $18769$ on PNBB task. The minimum number of failed courses is $0$ and the maximum number is $10$ in our dataset. Thus the MSE range could be between $0$ and $100$ on PNFC task. We can know one method is more likely to get a small MSE on PNFC task and get a big MSE on PNBB task.

For baselines except APAMT, we extract features from heterogeneous student behaviors as Guan et al. (Guan et al., 2015) suggested. In addition, due to the uncertainty of the historical observation sequence, we extract some descriptive statistics (minimum, maximum and mean) as features. As for the data preprocessing, we represent the categorical features with one-hot encoding. We process numerical inputs with the Min-Max normalization to ensure they are within a suitable range. For WAG, the number of borrowed books and the number of failed courses, because we use $\tanh$ as the activation function in the output layer, we scale them into $[-1,1]$. In the evaluation, we re-scale the predicted values back to the normal values, compared with the actual values. For the other numerical inputs, we scale them into $[0,1]$. Around half of the data (Fall 2016) are used as the training set and the other half (Spring 2017) are used for testing.

The hyper-parameters of all models are tuned with a ten-fold cross-validation method on the training dataset. We only present the optimal settings of our model are as follows. The dense embedding layer has $30$ neurons. The dimensions of the hidden states in the Profile-aware LSTMs which handle entering the library and going back to the dormitory behaviors are set as $12$ and $4$ respectively. The dimension of the hidden state in the dynamic LSTM which handles the historical WAG/number of borrowed books/number of failed courses sequence is set as $5$. The FC layer of the Multi-task Interaction Unit has $100$ neurons. The activation function used in the FC layer is $\mathrm{PReLU}$. We adopt $4$ Multi-task Interaction Units in total. We apply dropout before the output layer with a dropout rate equal to $0.4$.

The two-tailed unpaired $t$-test is performed to detect significant differences between DAPAMT and the best baseline. There are two conditions that should be met: the two groups of samples should be normally distributed; the variances of the two groups are the same (this can be checked using Levene’s test).

To comprehensively evaluate our model, we conduct five experiments. First, we compare our proposed model with state-of-the-art methods to show the effectiveness of our model. Second, we do ablation studies to prove the effectiveness of the key components in DAPAMT such as the Profile-aware LSTM. Third, we provide the visualization of learned representations to show whether our model can learn general representations which could be transferred to some new tasks. Fourth, we randomly select some students to visualize soft-attention weights and co-attention weights. We do this experiment to see whether our model can provide interpretability and whether attention mechanisms are effective. Finally, we study how hyper-parameters in DAPAMT impact the performance.