Detecting Disengagement in Virtual Learning as an Anomaly using Temporal Convolutional Network Autoencoder

Various affect and behavioral features, corresponding to the affective and behavioral states of the student in the video, are extracted from the consecutive video frames.

Affect features: Altuwairqi et al. altuwairqi2021new linked engagement to affective states. They defined different levels of engagement according to different values of valence and arousal in the circumplex model of affect altuwairqi2021new . Abedi et al. abedi2021affect showed that sequences of continuous values of valence and arousal extracted from consecutive video frames are significant indicators of affective engagement. In the circumplex model of affect altuwairqi2021new , different combinations of values of valence and arousal correspond to different affect states. Positive values of valence and arousal with low fluctuations over time correspond to affective engagement, and any deviation from these values indicates affective disengagement. The pretrained EmoFAN toisoul2021estimation on the AffectNet dataset (mollahosseini2017affectnet, ), a deep neural network to analyze facial affects in face images, is used to extract valence and arousal from consecutive video frames (refer to abedi2021affect for more detail).

Behavioral features: Woolf et al. woolf2009affect defined different characteristics of behavioral engagement as on-task behavior against off-task behavior corresponding to behavioral disengagement. In many cases, a student engaged in a virtual learning material focusedly looks at the computer screen, i.e., the student’s head pose and eye gaze direction are perpendicular to the computer screen with low fluctuations. There are low fluctuations in yaw, pitch, and roll of the student’s head while engaged with the learning material. Accordingly, inspired by previous research abedi2021affect ; liao2021deep ; thomas2018predicting ; kaur2018prediction ; niu2018automatic , eye location, head pose, and eye gaze direction in consecutive video frames are considered as one set of behavioral features. Ranti et al. ranti2020blink demonstrated that eye blinks are withdrawn at precise moments in time so as to minimize the loss of visual information that occurs during a blink. High and irregular blink rate indicates disengagement niu2018automatic ; ranti2020blink . The intensity of facial action unit AU45, indicating how closed the eyes are abedi2021affect , will be used as another behavioral feature.

According to the above explanations, the following 11 features are extracted from consecutive video frames, 2-element affect features: valence and arousal (2 features), and 9-element behavioral features: eye-closure intensity (1 feature), x and y components of eye gaze direction w.r.t. camera (2 features), x, y, and z components of head location w.r.t. camera (3 features); pitch, yaw, and roll of the head (3 features).

To illustrate how the extracted behavioral and affect features are capable of differentiating between engaged and disengaged samples, some of the extracted features from the consecutive 300 frames of the videos in Figures 1 (c) and (d) are drawn in Figure 3. The difference between features extracted from the video in Figure 1 (c), an engaged sample, and Figure 1 (d), a disengaged sample, is observable in Figure 3. These differentiating features and anomalies in features of the disengaged sample over time help AEs detect disengagement.

In addition to using the above 11-element frame-level features, being extracted from single frames of video, 37-element segment-level features are also extracted from video segments. Each video segment consists of multiple consecutive video frames. The mean and standard deviation of valence and arousal values over consecutive video frames (4 features), the blink rate, derived by counting the number of peaks above a certain threshold divided by the number of frames in the AU45 intensity time series extracted from the input video (1 feature), the mean and standard deviation of the velocity and acceleration of x and y components of eye gaze direction (8 features), the mean and standard deviation of the velocity and acceleration of x, y, and z components of eye gaze direction (12 features), and the mean and standard deviation of the velocity and acceleration of head’s pitch, yaw, and roll (12 features). As will be described in Section 4, the frame-level, and segment-level features are used to extract features from short, and long videos, respectively.

Thill et al. thill2021temporal introduced TCN-AE for unsupervised anomaly detection in time series. As depicted in Figure 4, a TCN-AE consists of an encoder for compressing (encoding) the input $n$-dimensional feature vector of length $T$ along the time and feature axes. The TCN-AE decoder then attempts to decode the compressed representation and reconstruct the original input feature vector. The architecture of the TCNs in the encoder (TCN1) and decoder (TCN2) are identical and are the vanilla TCN with dilated convolutions bai2018empirical . However, the weights of the TCNs are updated independently during training. The Conv1 and Conv2 are trainable $1\times 1$ convolution layers. The average-pooling bottleneck layer down-samples the feature map along the time axis. The upsampling layer then up-samples and restores the length of the original feature vector. The up-sampled feature vector is passed through the TCN2 and Conv2 to reconstruct the original feature vector with dimension $n$ and length $T$.

The Mean Squared Error (MSE) is used as the loss function in the training phase. The TCN-AE neural network is trained only by the normal (engaged) samples and is forced to minimize the reconstruction error for these samples. In the test phase, the MSE between the input feature vector and its reconstructed version by the trained TCN-AE is calculated. High values of MSE are expected for the anomalous (disengaged) samples that significantly differ from the normal (engaged) samples on whom the TCN-AE is trained, see Figure 2. As will be explained in Section 4, in addition to the TCN-AE, LSTM- and feedforward-AEs are also examined for disengagement detection in virtual learning.

The performance of the proposed method is evaluated on two publicly available video-only student engagement datasets, DAiSEE gupta2016daisee and EmotiW kaur2018prediction .

The frame-level behavioral features, described in Section 3.1, are extracted by the OpenFace baltrusaitis2018openface . The OpenFace also outputs the extracted face regions from video frames. The extracted face regions of size 256 × 256 are fed to the pretrained EmoFAN toisoul2021estimation on AffectNet mollahosseini2017affectnet for extracting affect features (see Section 3.1). The frame-level features are extracted from 10-second video samples (with 300 frames) of the DAiSEE dataset to be fed to the AEs. The extracted feature vector from each frame is considered as one time step of the temporal AE models. As the EmotiW dataset contains videos of around 5-minute length, following the previous works on this dataset thomas2018predicting ; liao2021deep ; abedi2021affect , the videos are divided into 10-second segments with $50\%$ overlap, and segment-level features, described in Section 3.1, are extracted from each segment. The segment-level feature vector extracted from each video segment is considered as one time step of the temporal AE models.

For the TCN-AE (described in Section 3.2), the parameters of the TCN1 and TCN2, giving the best results, are as follows, 8, $h=24$, 8, and 0.05 for the number of levels, number of hidden units, kernel size, and dropout bai2018empirical . The Conv1 and Conv2 are $1\times 1$ convolutional layers with $h=24$ input channels and $n$ output channels. The average-pooling bottleneck layer has a kernel of size $d=4$. The interpolation layer up-samples the feature maps with a factor of $d=4$. $n$ is the dimensionality or the number of features in the feature vector. In frame-level features, the number of features n is 9, and 11 when using only behavioral, and both behavioral and affect features, respectively. In segment-level features, the number of features n is 33, and 37 when using only behavioral, and both behavioral and affect features, respectively. To compare the perfomance of the TCN-AE with the TCN binary classifier, one TCN with the same architecture as TCN1, trailed by a $24\times 1$ fully-connected layer and a Sigmoid activation function is also implemented. The binary cross-entropy loss is used as the loss function for the TCN binary classifier.

In addition to the TCN-AE, LSTM- and feedforward AEs are also implemented as follows. The encoder part of the LSTM AE contains two LSTMs. The first LSTM accepts the n-element feature vector and the number of neurons in its hidden layer is h. The second LSTM’s input dimension is h and the number of neurons in its hidden layer is b. b is the dimension of the encoded version of input, i.e., the output of encoder. The decoder part of the LSTM AE contains two LSTMs. The first LSTM’s input, and output dimensions are b, and h, while the second LSTM’s input, and output dimensions are h, and n, respectively. The n-dimensional output of the final LSTM is the reconstruction of the input. To compare the performance of AEs with a binary classifier, the encoder part of the LSTM AE trailed by a fully connected layer and a Sigmoid has also been implemented. The binary cross-entropy loss is used as the loss function for the binary classifier. The feedforward AE contains two fully-connected layers as encoder and two fully-connected layers as decoder of size, n × (2 × b), (2 × b) × b, b × (2 × b), and (2 × b) × n. The values of b, and h are 64, and 128 achieving the best results.

To compare the performance of the above feature-based AEs with end-to-end models, a 3D-Convolutional Neural Network (CNN) AE is also implemented. The encoder part of the 3D-CNN AE contains three 3D convolution, ReLU, and 3D max-pooling blocks. The decoder part contains three 3D transposed convolution, ReLU, and 3D max-pooling blocks to reconstruct the input. For comparison, several end-to-end binary classifiers are also implemented. See the references in Table 1 for more details. In all the models, the Adam optimizer with decaying learning rate is used for optimization.

The majority of the existing engagement measurement approaches solve multi-class classification or regression problems, according to Table 1. To compare the binary classification version of those methods with the proposed method, the final fully-connected layer of their proposed neural network architecture is altered to have one output neuron (for binary classification), and their loss function is changed to binary cross-entropy loss.

Table 4 shows the AUC ROC and AUC PR of different models for disengagement detection of the test samples in the DAiSEE dataset. Table 5 shows the AUC ROC and AUC PR of different models for disengagement detection of the validation samples in the EmotiW1 and EmotiW2 settings of the EmotiW dataset. As the main focus of this paper is studying the performance of AEs in disengagement detection as an anomaly detection problem, for each AE model, the result of its binary classification counterpart is also reported. The results of using two feature vectors of behavioral features only and behavioral and affect features together are reported for each model. The higher AUC ROC and AUC PR of the AE models, including feedforward-, LSTM-, and TCN-AEs, compared to their equivalent binary classifiers (described in Section 4.2) show the effectiveness of detecting disengagement in an anomaly detection setting. In almost all the models, after adding the affect features to the behavioral features, the performance improves. It shows the effectiveness of the affect features as indicators of engagement (or disengagement). The superiority of the temporal LSTM- and TCN- models, analyzing sequences of feature vectors, over non-temporal feedforward models, shows the importance of analyzing the temporal changes in the affect and behavioral states for disengagement detection. In all the DAiSEE, EmotiW1, and EmotiW2, TCN-AE significantly outperforms LSTM-AE. This is because of the superiority of TCN in modeling sequences of larger length and retaining memory of history compared to the LSTM.

For the DAiSEE in Table 4, the results of end-to-end binary classification models are also reported for comparison (see abedi2021improving for details of the end-to-end models). Due to the longer length of videos, these models are not applicable to EmotiW. While the performance of 3D CNN AE is better than 3D CNN binary classifier and other end-to-end models, the performance of the end-to-end models is much lower than feature-based models. Therefore, the behavioral and affect handcrafted features are more successful in differentiation between engagement and disengagement.

The baseline value for AUC PR (the performance of a random binary classifier) is the ratio of the number of positive samples to the total number of samples khan2022unsupervised . Referring to Table 2 and Table 3, this baseline value is 88 / 1784 = 0.0493, 4 / 48 = 0.0833, and 14 / 48 = 0.2917 for DAiSEE, EmotiW1, and EmotiW2, respectively. As can be seen in Table 4 and 5, all the developed models outperform the random binary classifier by a large margin. Also, the AUC ROC of the developed models, specially AEs, is much higher than the AUC ROC of a random classifier (0.5).

Due to the imbalanced distribution of samples in the datasets 4.1, we reimplement the best binary classifiers (using both behavioral and affect features and TCN) using a weighted binary cross-entropy loss function abedi2021improving ; paszke2019pytorch . According to Tables 4 and 5, the binary classifiers with weighted loss functions (indicated by WL) outperform their non-weighted counterparts; however, their performance remains inferior to that of AEs.

Comparing the performance of the TCN-AE and TCN binary classifier using behavioral and affect features for the DAiSEE, EmotiW1, and EmotiW2 datasets in Table 4 and Table 5, the percentages of improvement in AUC ROC are 11.37%, 10.27%, and 6.78%, respectively, and the percentages of improvement in AUC PR are 20.62%, 34.01%, and 12.39%, respectively. Therefore, the average improvement of the TCN-AE, compared to the TCN binary classifier, is $9$% on AUC ROC and $22$% on AUC PR.

The confusion matrices in Table 6 are the results of thresholding reconstruction errors and output class probabilities of TCN-AE and TCN binary classifier using behavioral and affect features on the DAiSEE, EmotiW1, and EmotiW2 datasets. As the TCN-AE is trained in an unsupervised setting and only on the engaged (normal) samples, the disengaged (anomalous) samples cannot be used for finding an optimal threshold for thresholding reconstruction errors thill2021temporal ; khan2017detecting . Therefore, the instructions in khan2017detecting are followed to find an optimal threshold using only engaged (normal) samples of the training set. The numbers in the confusion matrices are the number of correctly and incorrectly classified samples. In all the DAiSEE, EmotiW1, and EmotiW2 datasets, TCN-AE outperforms TCN binary classifier in detecting disengaged samples.