Analyzing Student Strategies In Blended Courses Using Clickstream Data

Raw data from educational software systems often comes in the form of time-stamped student actions with an array of suitable identifiers. Evidence for correlations between activity log-based features and performance outcomes are plentiful. Many of the commonly discussed features revolve around simple summary statistics such as counts of certain types of actions, and have been shown to be indicative of students’ success particularly in MOOCs. Recent lines of research find links between general course completion in MOOCs and the number of watched videos [38, 9], the number of question answer attempts [9], and the time spent on assignments [4]. Similar results have been observed for blended courses but are much scarcer [39, 15]. In [15], the authors analyze sequences of transitions between different online platforms in two undergraduate level college courses. Their study finds that, although students are generally more likely to stay on the same platform in a study session, high achieving students transition more often and are more likely to use the discussion board. In many cases, the limited amount of data in blended courses is problematic and can lead to complications such as zero-inflated count variables.

A major shortcoming of count-based methods is their failure to leverage the sequential structure of students’ interactions with educational software systems. Both the order and the time difference between actions promise to carry valuable information that can be taken into consideration when relying on sequence based methods instead. In this work, we propose a pipeline for analyzing student online behavior based on session study sequences. While the order of actions is taken into account explicitly, time differences help us to derive reliable study sessions.

Sequence-based approaches to processing online student activity data group student actions into smaller sessions. In the case of click actions, these sequences are generally referred to as clickstreams. The goal when breaking a flow of actions into session clickstreams is to maintain some notion of interpretability, i.e. to devise meaningful study sessions. While this appears to be easy in some cases, it is generally non-trivial to find automated cut-offs rules that find sensible representations of study sessions for a large and diverse set of clickstreams at once.

Previous research suggests several different strategies to split clickstreams. The authors of [8] choose fixed duration time frames to group student actions from a several months long MOOC. The researchers decide for durations between one day and one month and show some success in the downstream prediction of student achievements with their choices. Similar fixed durations are used in [2]. Another popular splitting strategy is based on time-out thresholds where a new sub-session is started when no action was performed in a predefined time window [31, 5, 35, 43, 12]. The authors of other studies go one step further and combine the approaches by first, splitting at a fixed duration cut-off and second, at data-driven timeout thresholds of 15 minutes for ‘study sessions’ and 40 minutes for ‘browser sessions’ [15]. Similar data-driven approaches are pursued in [45, 39]. Other common heuristics include splitting at navigational criteria such as reloading of the course page [25].

On a high level, the problem of devising meaningful sub-sessions is closely related to the problem of time-at-task estimation in web-usage mining. Ideally, study sessions reflect time periods in which students interact with the material without any major breaks or distractions. There is a rich body of literature on time at task estimation that suggests that there is no one-fits-all solution to finding suitable time windows to split activity streams at [25, 6, 11]. Previous research suggests that the exact splitting heuristic can have a significant effect on overall model fit, model significance, and even interpretation of findings in the downstream modeling tasks [25]. In [25], the authors explore the effect of 15 different time-at-task estimation procedures on five different models of student performance. Overall, the authors conclude that there is no universally best method and recommend a mixture of existing methods including data-driven components. Following this suggestion, we employ a multi-step splitting procedure including navigational criteria, data-driven time-out thresholds, and separation of assessment weeks inspired by the procedure in [15].

Different methods have been proposed to process sequence-type student action data dependent on the amount of data, the length of sequences, and the goal at hand. Several lines of research rely on Markov chains and hidden Markov models which lend themselves well to visualization of sequences, but can make quantification of group differences in outcomes challenging [14, 13, 19]. Another commonly used class of methods is clustering of activity sequences [12, 22, 16]. Using data from three large MOOCs, the authors of [22] draw on simple $k$-means clustering of sequences of interactions with video lectures and assessments and observe four high-level student trajectories: completing assessments, auditing the course, disengaging after a while, and sampling content. In order to cluster the sequences, the authors rely on a numerical translation of student actions. The authors of [12] cluster and visualize students’ interactions with a college math environment, and instead rely on Levenshtein distance to measure the distance between sequences. Some works combine Markov models and clustering to account for the randomness introduced by the Markov models and report more robust results [40, 26, 23].

Although the described methods allow for a relatively easy grouping of sequences, interpretation of clusters can be non-obvious. One way to address this problem is to deliberately focus on finding relevant sub-parts of action sequences. Methods based on this goal can be summarized under the term pattern mining, and are both wide-spread and diverse. A relatively recent approach is given by differential pattern mining which focuses on automatically extracting patterns that are both above a certain threshold in frequency, and sufficiently different among groups of interest (e.g. high and low achieving students) [21, 20]. Other lines of research rely on more traditional data mining techniques [17, 34], or extraction of $n$-grams, i.e. sub-sequences of $n$ consecutive actions [8, 32, 44, 36]. The authors of [32] use a multi-step procedure to extract frequent $n$-grams that are subsequently used to identify different strategies in a collaborative interactive tabletop game. Part of our analysis is based on a similar approach to extract frequent behavioral patterns, and combines ideas of $n$-gram extraction and clustering to get more robust results.

A different class of promising methods is rooted in Natural Language Processing (NLP). Hybrid language models lend themselves well to the sequential structure of education data, and their use for student activity sequences has lead to some success in retrieving patterns and creating new visualizations. The underlying idea is that, given sufficiently fine-grained data, students’ sequential actions resemble words building sentences and can be attributed some ‘semantic meaning’. The NLP toolbox has not yet been explored fully, but some attempts to using language models for educational data are noteworthy and relevant for the context of our work. The authors of [44] use topical $n$-gram models to automatically extract ‘topics’ in the form of frequent patterns from clickstreams. In [36], the authors train a skip-gram neural network to receive a structure preserving vector embedding of the types of clicks student can make. After standard dimensionality reduction, the researchers are able to provide a new kind of visualization of students’ trajectories through the course. Since modern NLP models generally require large amounts of granular training data, work relying on these models has exclusively focused on MOOCs so far. In this study, we draw on Latent Dirichlet Allocation (LDA) in order to automatically extract frequent patterns and compare derived student strategies against the results of a more traditional $n$-gram pipeline. In some sense, LDA is similar to the ideas proposed by [44] but requires less training data which renders it particularly useful for blended courses. In addition, we use an adapted form of the skip-gram model proposed by [36] in order to explore the context of student actions in our data. To the best of our knowledge, this is one of the first works to employ NLP methods for analysis of blended courses.

The data this study builds on was collected through the educational software system Diderot. Diderot is a cloud-based course support system commonly used to assist undergraduate and graduate level college courses. The system spans a wide range of functionalities including sharing of lecture notes, a discussion board (called post office), in-class attendance polls, homework submission, and automatic code grading. This bandwidth usually renders the use of additional outside technological course support unnecessary. In turn, the student usage data collected from Diderot can give an almost comprehensive view on students’ course participation outside of face-to-face class times.

When it comes to sharing of lecture notes, Diderot takes a more granular and interactive approach as compared to traditional learning management systems. Content is split into small sub-entities (called atoms) which are displayed in a linear fashion following the outline of a chapter. Atoms are highly interactive and come with a variety of clickable icons that allow students to take notes, bookmark, follow, or like atoms, and, in particular, to ask questions concerning their content. Discussions about course material that are sparked in this way are visually attached to the respective atom, allowing other students to submit comments. This setup results in much richer data on interactions with lecture notes than we can expect from PDF formatted lecture material.

Most student usage data from Diderot is presented by individual, time-stamped click actions that come with various identifiers coding the exact type, user, and location of the interaction. In turn, the activity data can be broadly separated into navigation (e.g. Load course, Click link, Search), discussion (e.g. Go to post office, Create post), and behaviors (e.g. Like atom, Follow post).

Our data is drawn from the second half of a large sophomore-level computer science course taught at Carnegie Mellon University in spring 2019. Since data is not available for the first part of the course due to initial technical difficulties, we exclude all students who dropped the course throughout the semester. One additional student was excluded based on inflated click patterns which suggested an attempt at automatically scraping content. Along with the click data, we rely on performance information measured by homework and exam grades, as well as student-level lecture and recitation attendance logs. All data is collected through Diderot and matched based on anonymous student identifiers. A summary of the click data over the seven week observation period is displayed in Figure 1.

Types of clicks. At finest granularity, Diderot allows for several tens of thousands distinct click actions within a single course since every individual click is associated to a fully specified object and activity. However for the sake of analysis, we group clicks into different types where the appropriate level of granularity is non-obvious. We aggregate clicks based on the type of object they refer to as well as the activity performed. In order to maintain interpretability, this aggregation is performed separately in each sub-part of the course given by lecture notes, homework material, recitation notes, a library documentation (which is comprised of coding references), and practice exams. This leaves us with 37 different click types, the most common of which are summarized in Table 1.

Grades and types of assessment weeks. Performance outcomes are measured by percentage grades in five homeworks and two exams (a midterm and final exam) that fall into the observation period. This naturally divides the data into seven assessment weeks with a deadline for a homework problem set or exam at the end of each period. Deadlines are approximately evenly spaced with only one extended homework period of 11 days after the midterm exam (which also spans over a four day spring holiday), followed by a shorter homework period of only 5 days. We take interest in relating students’ study behavior to two distinct outcome variables: (1) The type of the assessment week, i.e. homework deadline or exam, and (2) the percentage grade students received in the respective assessment. As depicted in Figure 1, there are visible spikes of increased activity before the assessment week deadlines especially before the two exams. In addition, we note that the distribution of grades appears notably different between homeworks and exams which is confirmed by a two-sample Kolmogorov-Smirnov test ($p<0.001$). While the distribution of exam grades is approximately bell-shaped with heavy tails and a slight left-skew, i.e. more particularly high scores than particularly low scores, the homework grade distribution is left-skewed with additional modes at 0 and 100. This difference in distributions is unsurprising as exams are generally graded on a curve and cannot be skipped by students, while homeworks allow for more variability.

Class attendance. Attendance in lecture and recitation sessions was taken with Diderot polls. If a student participated in the poll, which was generally only open for a few minutes, it was assumed that they attended the session. We treat attendance in lectures and recitations separately and aggregate the binary information on an assessment week basis by taking the mean. In turn, student’s attendance scores lie between 0 and 1 with the exception of the final exam week which is not associated to any contact class time.

In raw form, each student is associated with a single clickstream which consists of ordered click actions over the whole observed time period. We employ a multi-step procedure to split this data into more meaningful study sessions. First, we divide the clickstreams based on assessment weeks. Second, we split the resulting sub-clickstreams each time a Load course action is recorded, and last, we choose a data-driven timeout threshold to further break up the resulting sequences.

In order to find a suitable timeout threshold, we employ a technique similar to [15] and examine the distribution of time differences in the sub-sequences. We find that the distribution of waiting times supports a wide range but is rapidly decaying. While 75 % of clicks are made within 2.81 minutes or less, a small subset of clicks has time differences of up to 7 days. Figure 1 shows kernel density estimates of log-transformed minutes until the next click within the sub-clickstreams obtained after the second step of our procedure. Different estimates are obtained for distinct categories of actions. While the logarithmic distribution of post-related and miscellaneous clicks is unimodal with the majority of follow-up clicks made within one minute, the distribution for clicks related to homework and lecture notes has an additional mode at about 5-10 minutes. This disparity is unsurprising given that most actions can be expected to be short, while reading through lecture notes or homeworks can be a more lengthy process. In order to preserve both types of sessions, we separate clickstreams at a 60 minutes threshold if the last action was loading of lecture notes or homework related content, and at 5 minutes otherwise. As a result, we obtain a total of total of 35,703 session clickstreams where each clickstream has between one and 115 clicks with mean of 3.98 clicks and standard deviation of 6.23; 75% of session clickstreams have at most 4 clicks.

We explore the contexts in which different types of clicks are made in order to gain some understanding of how students generally use the course support system. This is crucial since Diderot is a fully integrated interactive platform that allows the same type of click in contexts that can have different interpretations. Inspired by [36], we tackle this problem by devising a structure-preserving embedding of the click types into a real-valued vector space, i.e. each click type is mapped onto a vector such that click types that appear in the same contexts or are interchangeable are close to each other. This type of embedding can be obtained from a skip-gram model which is a common supervised two layer neural network model often used for language type data (see Figure 2).

Training data for the model is build by extracting pairs of neighboring click types from the session clickstreams. More concretely, each input click is paired with each click appearing within some index in the same clickstream. Both the window size and the number of hidden units are important hyperparameters. Since most of our clicks are short and we seek an embedding of only 37 clicktypes, we explore small values for both parameters, i.e. window sizes in $\{1,2\}$ and embedding sizes in $\{3,4\}$. After this small grid search, we only retain the model with the lowest average training loss in the last 2000 training steps. In order to speed up training, we rely on mean noise-contrastive estimation (NCE) loss where 8 negative classes are sampled for every batch instead of computing the entire softmax output. All models are trained over a maximum of 300,000 training steps with SGD with learning rate 1 and a batch size of 512. Training is terminated early when the average loss over 2000 training steps does not change considerably for 5 consecutive non-overlapping 2000-step periods. Because training the model is only the surrogate task in order to obtain the embedding, we train on all available data which comprises 206,514 or 363,260 pairs dependent on the window size.

Frequent patterns counts as features. In order to explore what role the extracted strategies play in homework solving versus exam preparation and whether they drive success, we build two prediction models based on patterns counts from the clustered $n$-gram method. For this, a representative pattern of 3 clicks is chosen for each of the devised strategy clusters, and its occurrences in each of the session clickstreams is counted by comparing against each 3-gram derived from the clickstream. Since we cannot expect the chosen pattern to accurately represent the whole cluster, we allow a Jaro-Winkler distance up to 0.2 when comparing the sub-sequences. This procedure allows matching of click sequences with only one replacement ($1-jw(abc,abd)\approx 0.18$), one transposition ($1-jw(abc,acb)\approx 0.10$), or one replacement and one trasposition ($1-jw(abc,adb)\approx 0.20$). In order to build student and assessment week based prediction models, we aggregate pattern counts along assessment weeks and individual students by simple addition. Similar methods have been employed by [8, 27, 41, 10].

Predicting assessment type. A random forest classifier is trained to predict the assessment type, i.e. homework or exam, from frequent pattern counts, the number of clicks, and the number of session clickstreams a student has within a given week. In practice, it is unlikely that we would need to predict the assessment type as it is usually known. However when paired with careful analysis of feature importance and partial dependence, such model can yield valuable insights into the most important differences in student behavior between homework and exam weeks. We use 80 % of the 1,148 student-week combinations for training and hold back 20 % as test set. Hyperparameters including the maximum tree depth, the maximum number of features to consider at splits, the minimum number of samples per leaf, and the number of trees are chosen by a grid search over a range of values, where models are trained with 5-fold cross validation on the training set. Our model draws on Gini impurity to measure the quality of splits, and we evaluate feature importance based on the mean decrease in impurity (MDI) associated with splitting at a given feature when predicting $Y$. For a set of fitted trees $\mathcal{T}=\{T_{1},\ldots,T_{N}\}$, the MDI of a feature $X_{m}$ is defined as

where $p(t)$ is the proportion of samples that reaches node $t$, $v(s_{t})$ is the variable used to split $s_{t}$, and $\Delta i(s_{t},t)$ is the decrease of impurity generated by the split.

Predicting grade outcomes. Similar to the assessment type prediction model, we train a random forest regressor to predict students’ grade outcomes based on strategy counts, the number of clicks, the number of session clickstreams, and attendance information. The additional consideration of lecture and recitation attendance requires us to remove all observations from finals week, since no face-to-face class time has taken place in the last week of the course. Since this constitutes half of all exam observations in the data and grade distributions of homeworks and exams are significantly different ($p<0.001$), we limit our prediction model to homework grade prediction entirely. Of the 820 homework samples, 80 % are used for training and 20 % for testing. A grid search of hyperparamters with 5-fold cross-validation on the training set is performed, and feature importance is measured analogous to Equation 1 with the MSE as impurity measure.

We extract strategy features for assessment week level prediction models by matching session clickstreams against the extracted frequent patterns. The results are summed up for each student-week combination and thus roughly represent how often a given student has used a strategy in a given assessment week. After this aggregation, 91.03 % show at least one occurence of one of the patterns. We generally expect not all student click behavior to follow the extracted strategies or stringent strategies at all. Thus, it is unsurprising that some of the student week combinations do not involve any of the patterns.

We train a random forest classifier to predict the assessment type on pattern counts, the number of clicks and the number of sessions within a given week. A total of 80 % of the data is used for hyperparameter tuning and training, while 20 % is withheld for testing. The model reaches a classification accuracy of 93.68 % on the training data which constitutes an evident improvement over the naive majority class prediction (71.90 % of the training data have the label homework). Based on a permutation test, we find that the model performs better than random on the training set ($p<0.01$). A total of 100 permutations of labels were used for this evaluation. Accuracy on the test set is 93.91 % which suggests sufficient generalization ability of the prediction model.

The prediction model results suggest that students use the educational support system differently and employ the different strategies at different rates when preparing for exams as compared to doing homeworks. We examine feature importance in the model in order to gain more insights into these differences. Figure 4 depicts the mean decrease in impurity (MDI) for splits at the different covariates, as well as partial dependence of the predictions on the most important features. We see that predictions are mainly driven by pattern counts of the strategies look at lecture notes (MDI = 0.395), in-depth review of lecture notes (MDI = 0.259), look at practice exams (MDI = 0.140), and active homework engagement (MDI = 0.094). Partial dependence plots show that while increased counts in the strategies related to lecture notes and practice exam engagement increase the probability that the model predicts an exam week, higher counts in the active homework engagement strategy increase the models likelihood of predicting an upcoming homework deadline. These results suggest that students approach to learning is driven by the kind of performance assessment they are given. It appears that the increased activity in exam weeks (see Figure 1) is largely based on increased engagement with lecture notes and practice exams, while interactions with the homework related content is generally less pronounced.

We train a random forest regression model to predict homework grades on a individual week and student level. Features include students’ strategy counts, the number of clicks, the number of sessions, and the mean attendance in both lectures and recitations. Training is conducted on 80 % of available data while 20 % are withheld for testing. After hyperparameter tuning with 5-fold cross validation, the prediction model realizes a MSE of 0.046 on the training data set. A permutation test based on 100 permutations of labels shows a significant improvement over random performance with this model ($p<0.01$). On the test set, the model attains a prediction MSE of 0.054 which suggests sufficient generalization ability.

Figure 5 explores the importance of the different features for predictions and displays partial dependence relations for the most important covariates. Since we use MSE as impurity measure, the mean decrease in impurity (MDI) for a given feature effectively corresponds to the mean decrease in variance we receive by splitting at the feature. We see that, in fact, the most relevant features appear to be the number of clickstream sessions (MDI = 0.311), the number of clicks (MDI = 0.221), lecture attendance (MDI = 0.123), and recitation attendance (MDI = 0.112). Partial dependence plots reveal that increases in any of the above features increase the predicted homework score percentage by a relatively large margin of up to 20 percentage points. Conversely, strategy counts appear to be less relevant for grade predictions with some exceptions. Most notably, the predicted grade rises with the number of times students actively engaged in homeworks (MDI = 0.077).

Overall, our results show some success in prediction of homework grade outcomes. The extracted features, including some of the pattern counts, add valuable information to the prediction model. In particular, students who come back to Diderot more often and thus use an increased number of study sessions to solve their homeworks, and students who generally interact with the system at high rates are predicted to have better grade outcomes. In addition to time at task, the mere attendance in lectures and recitations increases students’ grade outcome predictions. In fact, students in the our data set who attended at least one lecture in a given assessment week on average received a homework percentage grade of 76.58 %, while students who skipped lectures on average scored 55.86 %. For recitation attendance this corresponds to 74.34 % and 49.20 % respectively.

Both of the discussed prediction models provide valuable insights for instructors and educational system design. The tree-based ensemble methods are particularly suitable for initial modeling and processing of features on different scales. Their main advantage over many other models is the relatively straightforward explainability of predictions given partial dependence plots and measures of feature importance which renders them a useful approach to high stakes at-risk prediction.