Multiple Instance Learning for Cheating Detection and Localization in Online Examinations

As one of the most challenging problems in computer vision, video anomaly detection has been extensively studied for many years. Generally, reconstruction model and predictive methods are used to verify the normality of the model [24, 12, 25, 26, 27, 28, 29, 30, 31, 32, 33, 21]. Reconstruction methods, such as Generative Adversarial Networks (GAN) [26, 27], and Autoencoder [28, 29, 30] attempt to distinguish abnormal events by redrawing the input. In particular, the memory module [31] is introduced to record the prototype of the normal data, which represents the different patterns of the normal video frames for unsupervised anomaly detection. Predictive methods attempt to encode normal events through statistical models. Xu et al. [32] used Support Vector Machines (SVM) to detect anomalies after extracting the features. Ruff et al. [33] utilized Convolutional Neural Networks (CNNs) to map the normal data as the center of the sphere and exclude the abnormal samples outside it. Tian et al. [21] proposed Multiscale Temporal Network (MTN) to capture the local and global temporal dependencies between video clips, and achieved state-of-the-art results. Inspired by the temporal dependencies of MTN networks, we introduce a framework to capture dependencies between clips: we first optimize the feature encoder in a fine-grained manner by introducing MIL. Then, in order to capture spatio-temporal dependencies between video clips, the spatio-temporal graph module is applied.

The methods of online proctoring can be divided into three types: online human proctoring, semi-automatic proctoring and fully automatic proctoring. Online human proctoring means that remote proctors supervise candidates throughout the online exam process. However, this is labor-intensive and the cost increases with the number of students. To eliminate the human cost, fully automatic supervised methods have been proposed, which usually exploit machine learning techniques to identify cheating behaviors. However, existing methods of fully automated proctoring suffer from several problems, including the ”black-box” nature of deep learning algorithms and unreliable decision-making caused by biased training datasets. Therefore, it is almost impossible to rely entirely on fully automated methods to determine whether a student is cheating during an online exam. To solve the problems caused by fully automated proctoring methods, semi-automated proctoring was introduced, which allows humans to complete the final decision with the aid of machine learning methods [34, 35, 36]. A representative work is the Massive Open Online Processor proposed by [35]. Their method is the first to use machine learning techniques to detect suspected student cheating and send the detections to teachers for further review. However, it requires each candidate to use multiple devices for online exams, which increases exam costs. Costagliola et al. [34] proposed a visual analysis system to assist proctors in invigilating an exam, but it is limited to identifying the cheating behavior that a student is looking at another student’s screen, which is not enough in online exams. Atoum et al. [36] developed a multimedia analysis system to detect various cheating behaviors during online exams. But their approach requires two cameras which is unrealistic contemporarily. Li et al. [37] facilitate the supervision of online exams by analyzing video recordings of exams and mouse movement data from each student. In addition, the angles of the head posture have also become the key point to determine whether the examinee is cheating [38, 39, 40]. But most researchers tend to focus on a few features, without fully considering head posture, gaze angle, body posture, background, and other features.

Different from the methods above, we use the weakly supervised method to identify the above three types of cheating behaviors (not include the types of cheating behavior that can be detected by the system level, such as obtaining page monitoring, and voice access for direct detection). In addition, a variety of behavioral features, including head posture, gaze angle, body posture, background, and facial appearance, are considered comprehensively for cheating detection. In summary, our method is able to efficiently detect candidate cheating behaviors while ensuring real-time performance.

In recent years, MIL has shown remarkable performance in the field of video anomaly detection. MIL is a weakly supervised learning method which treats the video as a package, and uses the splitted clips of each video as an instance [33, 13]. With the help of a specific feature aggregation function, video-level annotations can be used to supervise learning in the instance level. MIL serves as an anomaly classifier, using the features extracted from the action classifier to detect anomalies, and deep MIL ranking loss to calculate anomaly scores. On this basis, Wan et al. [41] introduced dynamic loss and center-guided regularization. Similarly, Zhu et al. [42] proposed an attention-based MIL model, and introduced an optical flow based autoencoder for feature encoding.

Different from them, we use MIL as a label generator for training, and learn discriminative features in the video frames by adding a feature encoder. In particular, Feng et al. [43] used a similar approach as ours to build the model. In contrast, we propose the spatio-temporal graph module in the feature encoder to capture the spatio-temporal dependency information between clips. Further, in the second stage, we use the multi-modal features, which allow the spatio-temporal graph module to comprehensively consider the relationships between the feature of clips.

The GCN [44, 45, 46, 47, 48] has been extensively studied on the problem of processing graph-structured data. In the field of video anomaly detection, GCN is often used to mine various relationships between features [16, 17]. Zhong et al. [16] applied GCN for the first time to clean the label noise in anomaly detection datasets, and exploited two relationships of feature, namely feature similarity and temporal consistency, to correct label noise. Similarly, Chang et al. [17] proposed a contrastive attention module to address the problem of abnormal data imbalance, which mainly consists of a temporal graph convolutional layer to utilize video temporal context.

In the process of cheating behaviors detection, the feature similarity graph can effectively capture the long-term relationship between objects/regions and promote the propagation of supervised signals of clip similarity [16]. In addition, the cheating behaviors of candidates may appear at close time points, which conforms to the characteristic of temporal consistency. Therefore, we establish a spatio-temporal graph in the graph module of the feature encoder to obtain the spatio-temporal relationship between clips, thereby improving the prediction performance of the encoder.

According to the characteristics of multiple instance learning, each instance in the negative bag is negative, while the instances in each positive bag are unidentified. Therefore, we need to use the label generator to generate clip-level anomaly scores for the instances in the positive bags as their labels, and differentiate the anomaly scores between positive and negative instances as accurately as possible. It should be noted that a clip is usually composed of several frames, and the number of frames in a clip could be adjusted according to the experimental results. In the feature extraction stage, we deploy a classical feature encoder, i.e. C3D [49] pretrained on Sport-1M [50] or I3D pretrained on Kinetics-400 [51] to extract clip-level features required by the label generator. However, if the video of indeterminate length is simply fused into 32 segment features as the input of the multilayer perceptron (MLP) [10], the prediction effect of the final model is often not ideal. In order to accommodate the variation in the duration of untrimmed video and class imbalance, we introduce a continuous sampling strategy [43]:

After the feature extractor extracts features $\left\{f_{i}\right\}^{N}_{i=1}$ from the original video $V$$=$$\{v_{i}\}^{N}_{i=1}$, we uniformly sample $L$ sub-bags $U=\left\{f_{l,t}\right\}^{L,T}_{l=1,t=1}$ from these clips. Specifically, there are $T$ consecutive clips in each sub-bag, as shown in Figure 2. The $T$ is a hyperparameter that needs to be tuned. In detail, in order to make the anomalous score of the anomalous video clips higher than the normal video clips, we adopt deep MIL ranking loss [10] to train MLP. The sampled features are entered into the label generator to predict the corresponding anomaly scores $\left\{s_{l,t}\right\}^{L,T}_{l=1,t=1}$. By performing averaging pooling, the anomaly score $S_{l}$ for each sub-bag is derived:

Then, the anomaly scores of all instances in the positive bags marked as $S^{a}=\left\{s_{i}^{a}\right\}^{N}_{i=1}$ are min-max normalized to form clip-level labels:

where ${y}^{a}_{i}$ between [0,1] is a clip-level label. Finally, the generated labels $Y^{a}=\left\{y_{i}^{a}\right\}^{N}_{i=1}$ are mixed with the clip-level labels $Y^{n}$ to train the feature encoder.

As shown in Figure 3, we take the feature extractor (i.e. C3D/I3D) as the backbone and introduce two modules: self-guided attention module and spatio-temporal graph module. Compared to a feature encoder with self-guided attention [43], spatio-temporal graph module can help the encoder to capture spatio-temporal contextual anomalies in the features. Furthermore, in order to fully consider the candidate’s behavior in the proctoring scenario, we use OpenFace 2.0 [52] to extract the multi-modal features of student, and comprehensively analyze these features in spatio-temporal graph module.

Deep MIL Ranking Loss: In the process of generating labels with the label generator, we hope that the instances in the positive bag have higher anomaly scores than instances in the negative bag. A simple method is using the ranking loss function to encourage positive instances to get high anomaly scores. However, this method requires pre-arrangement of clip-level annotations while MIL only requires video-level ones. Therefore, a MIL ranking objective function [10] is proposed. On this basis, we use a sub-bag as a fundamental unit to be compatible with the continuous sampling strategy:

where $max$ is the function to find the maximum anomaly score of all sub-bags in each bag. After getting anomaly scores, only the sub-bag with the highest scores in the positive and negative bags are ranked. That is because the positive sub-bag with the highest score is most likely to include the abnormal clips, and the sub-bag with the highest score in the negative bag contains normal clips that are most likely to be misjudged as abnormal. The purpose of our MIL ranking function is to separate positive and negative sub-bags as far as possible in terms of anomaly scores. Therefore, our ranking loss is as follows:

In real-life scenarios, abnormal events generally last for a short time. Therefore, the anomaly scores of instances in the positive bag are generally sparse, which indicates that there may be abnormal events in only a few sub-bags. Secondly, anomalous events tend to depend on the context, and anomaly scores should change smoothly between instances. Therefore, we minimize the difference in scores between adjacent sub-bags to highlight context-based abnormal clips. Finally, by introducing sparsity and smoothness constraints to the sub-bag scores, the deep MIL ranking loss is obtained:

where ① is used as a smooth term, and ② as a sparse term. $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters used to balance the sorting loss.

Classification Loss: After gaining the labels $Y^{a}$ generated by the label generator, we apply the corresponding abnormal video $V^{a}$, and further combine with $V^{n}$ and its annotations $Y^{n}$ to train our feature encoder. We utilize the cross-entropy as the loss function of the feature encoder. Since only a few clips of anomalous videos are anomalous, there exists a class-imbalance problem at training time. We introduce class-reweighting to cross-entropy loss $L_{w}$:

where $w_{0}$ and $w_{1}$ are class weights for abnormal and normal class. We set up $w_{0}=1.2$ and $w_{1}=0.8$ [43].

We conduct experiments on three real-world surveillance video datasets of different scales: UCF-Crime, ShanghaiTech, and Online Exam Proctoring (OEP).

UCF-Crime [10] is a large-scale real surveillance video dataset. It contains 13 kinds of abnormal categories, a total of 1,900 untrimmed long videos, including 950 abnormal videos and 950 normal videos in the same scene. A total of 1610 videos are adopted for training and 290 videos are utilized for testing.

ShanghaiTech [9] is a medium-scale campus surveillance video dataset containing 437 videos. It includes 130 abnormal events in 13 scenarios. This dataset was first proposed for unsupervised learning, so all abnormal videos are in the testing dataset. In order to adapt to weakly supervised learning, these videos are reorganized into 238 training videos and 199 testing videos [16].

OEP [36] is an online examination monitoring dataset, composed of 24 different examinees’ surveillance videos. Among them, examinees in the first 15 videos were asked to cheat as much as possible. In order to capture the real exam scene, the last 9 examinees were required to take the real exam. These two requirements enhance the authenticity of the data while enriching the types of cheating methods.

Since the OEP paper uses dual-camera detection, it is able to detect more situations. We remove cheating videos that use second cameras to detect and that don’t fit the closed-book exam scenario. Finally, 69 normal videos and 65 abnormal videos are obtained, including $Absence$, $AnotherPerson$, and $Device$. In detail, the $AnotherPerson$ event covers two scenarios: an assistant appears on the screen and the candidate turns his head to communicate with the Off-screen assistant. The case where the candidate covers his mouth to communicate with the Off-screen helper is not considered, because the cheating behavior would be caught in real-time by the browser’s voice monitor. In the closed-book exam, the necessary drawing boards and calculators are given in the electronic form. Therefore, $Device$ event takes into account any prohibited items (which may be carried by the candidate or passed through an assistant), whether they appear on or off the screen. For $Absence$, candidates who leave the monitoring range are sensitively captured. Finally, our model is trained through a random combination of short videos.

Evaluation Metrics: According to previous works[2, 10, 41], we exploit the area under the curve (AUC) of the frame-level receiver operating characteristics (ROC) as the main metric. Higher AUC means stronger detection ability and better performance under different recognition thresholds.

Feature Extractor: In order to verify the universality of our model, we apply two mainstream feature extractors: C3D [49] is a 3D convolutional network encoder. We use the pre-trained model on the dataset Sports-1M [50], and input the features of the layer fc7 into the label generator. Compared to C3D, I3D [51] has a two-stream architecture. We first extract the output $(2\times 7\times 7\times 1024)$ of the mixed_5c layer in the I3D network, and then use an average pooling 3D with a convolutional kernel $(2\times 7\times 7)$ to obtain the 1024-dimensional features as the input of the label generator.

Label Generator is a 3-layer MLP with unit numbers of 512, 32, and 1. The dropout probability between each layer is 0.6. In the training process, we regard 16 frames as a clip, and set the hyperparameters: $L=32$, $T=8$, and $\lambda_{1}=\lambda_{2}=8\times 10^{-5}$. Finally, we use the Adagrad optimizer to train the generator with a learning rate of 0.01.

Feature Encoder: The self-guided attention module consists of 2 encoding units, namely $M_{1}$, $M_{2}$. Let $C$ be the channel number of $F_{4}$ and $2K$ be the channel number of $F^{*}$. $M_{1}$ consists of a $3\times 3\times 3\times C$ 3DConv layer with the stride of 2, a $1\times 1\times 1\times 2K$ and a $1\times 1\times 1\times 1$ 3DConv layer. The first two layers are activated by the ReLU function, and the last layer is activated by the Sigmoid; $M_{2}$ consists of a $3\times 3\times 3\times C$ 3DConv layer with the stride of 2 and two $1\times 1\times 1\times 2K$ 3DConv layers. It is worth noting that $C$ is determined by the number of channels of the feature map derived from the base backbone (C3D/I3D); $K$ is a hyperparameter. Through the hyperparameters analysis in Section V, it is concluded that the performance of the model reaches best when $K=15$.

In the spatio-temporal graph module, the output dimensions of the first two fully connected layers are 512 and 128, at the 60% dropout rate. Then, there are two graph convolutional layers applied to each graph: a hidden layer with 32-unit activated by ReLU, and the last 2-unit output layer. After multiple experiments, it was found that the training parameters had minimal impact on performance, so we generally set $base\_learning\_rate=0.0001$, $weight\_decay=0.0005$ and train 300 epochs.

Compared with several existing works in weakly supervised learning, we evaluate our model CHEESE in terms of accuracy, effectiveness, and universality. In order to adapt MIL to ShanghaiTech and OEP, we re-standardized the datasets. In addition, the applicable scenarios of UCF and ShanghaiTech are different from OEP, and crowded scenes cannot use Openface to extract features, so we removed the feature fusion process in the experiment shown in Table II and directly put the 128-dimensional feature into two graphs for learning.

When it comes to the UCF dataset (as shown in Table II), The AUC of CHEESE reaches 80.56%, which is comparable to the performance of other models. These results verify the effectiveness of our proposed method. In addition, compared with the works on the ShanghaiTech dataset, the AUC of CHEESE increases to 90.79%, which verifies the adaptability of our method on different datasets. Moreover, in both datasets, our method is second only to the method proposed by Feng et al. [43] in performance. After analysis, we believe that the excessive noise present in the crowded scene affects the ability of the spatio-temporal graph module to capture the dependence. For OEP datasets, by combining multi-dimensional features in the proctoring scenario, CHEESE can effectively help GCN establish dependencies in the video.

To verify the accuracy and robustness of CHEESE on the OEP dataset, we compare related state-of-the-art unsupervised and weakly supervised learning methods [31, 3, 10, 43, 16]. In Table III, we can find that the AUC of CHEESE reaches 87.58%, which confirms the accuracy and effectiveness of CHEESE. It is worth mentioning that the method of Ionescu et al. [3] outperforms the unsupervised method proposed by Park et al. [31] on the OEP dataset. It is concluded that the object detection method extracts key information for cheating behavior detection [3], which proves that extracting some clues beforehand, such as body posture, head posture, etc., can lead to better performance. Moreover, our method outperforms the method proposed in [43] on the OEP dataset. They also used pseudo-labels to optimize the encoder, which shows that the spatio-temporal graph module can effectively improve prediction accuracy. These results validate that our proposed method is more applicable to the proctoring domain than baselines.

Due to the different scenarios, we cannot utilize Openface to extract the facial features of crowded scenes in the UCF and ShanghaiTech datasets. Therefore, the features and dimensions of the convolutional layers imported into the spatio-temporal graph module are also different. To control variates and take full advantage of multi-modal features, we conduct the following ablation studies on OEP datasets.

We report the runtime of our method for the OEP dataset used C3D in Table V with GPU. The results show that current experimental configuration can support real-time videos below 25 FPS, and higher frame rates can lead to issues such as latency or partial frame loss, which can lead to performance degradation. However, the experimental results are not the upper limit of the model’s processing speed. In scenarios with high real-time requirements, the processing time of the model can be further reduced by improving the GPU configuration.

Compared to the excellent performance of C3D in real-time processing (processing speed up to 313 FPS [49]), our method spends a lot of time on the CHEESE^MF, which uses Openface 2.0 for face recognition and feature extraction. In other words, the time consumption of CHEESE depends mainly on the runtime of Openface 2.0. Baltrusaitis et al. [52] stated that the approach they proposed was applicable to real-time environments, which illustrates the feasibility of our approach.

We visualize the qualitative results of CHEESE with 4 videos on OEP dataset. As shown in Figure 6, by learning multi-modal features, CHEESE is able to pinpoint anomalous events and predict anomaly scores very close to zero on normal videos, which demonstrates the effectiveness of our model.

In order to verify the spatial interpretability of the feature encoder, we also visualize the spatial activation map via Grad-CAM [54] on $A^{*}$. As shown in Figure 7, heat maps of three types of cheating events are collected. When the Another Person event occurs, the attention of CHEESE will turn to the external assistant. Similarly, when devices such as mobile phones, and calculators appear on the screen, the model will continue to focus on the location of these prohibited items. In addition, we try to detect Another Person event where the external assistant is not on the screen. In the fourth row of the picture, the student is talking with people outside the screen. By observing the heat map, it can be found that the model will focus on the examinee’s eyes and areas surrounding them, and at the same time, it will be judged as anomaly. Above all, CHEESE can sensitively locate the decisive positions that help determine whether the current frame is abnormal, which shows the effectiveness of our proposed method.