Supervised Contrastive Learning for Detecting Anomalous Driving Behaviours
from Multimodal Videos

Driver monitoring systems target to identify distracted or dangerous driving behaviours that could cause potential traffic accidents. It can assess a driver’s alertness, attentiveness, and focus and generates warning signals if distracted driving behaviour is observed. However, distracted driving does not comprise of a unique set of behaviours and could result due to negligence, carelessness, fatigue, or other unknown reasons. Most of the time a driver may be driving safely, yet a minor distraction could be fatal. Therefore, due to the diversity and less occurrence of these distracted behaviours, we term them as ’anomalous’ driving behaviours. This should not be confused with traditional ’anomaly detection’ problems where the anomalous events are not available during training phase. In our problem formulation, some annotated anomalous driving behaviours are available during training phase. From a machine learning perspective, this represents an imbalanced class classification problem with lots of normal or safe driving data and few anomalous driving samples. Moreover, the distribution of anomalous driving behaviours could be different during training and test phases due to variations in people exhibiting different types of anomalous driving behaviours. Hence, new anomalous driving behaviours may occur during testing that were not observed in the training phase.

For handling imbalanced classes (in videos), supervised 3D Convolutional Neural Network (3DCNN) or its variants with weighted classes could be the default choice [3, 4]. However, more recently contrastive learning (CL) has shown state-of-the-art results in both self-supervised [5] and supervised classification problems [6]. CL [5] enables to learn effective visual representations by contrasting positive and negative pairs; thus, enabling them to work in imbalanced dataset situations. CL architecture is generally realized through an encoder (2D or 3D CNN or its variants), followed by projection head (feed forward layers). Supervised CL [6, 7, 8, 9, 10] leverages the class labels present in the data to achieve state-of-the-art results in classification problems. In the denominator of a typical CL loss function, the exponential of dot product of negative pairs is summed. If the labelled negative pairs are closer to the positive pairs, due to negative samples being near to positive samples (e.g.,. mislabeling), then this can lead to difficulty in the overall optimization of the loss function. This problem could be mitigated by scaling the sum term such that the summed similarity of negative pairs remain far to align with the loss function. To circumvent this potential problem, we proposed to average the sum of negative pairs instead.

The projection head is generally discarded in self-supervised [5] and supervised CL approaches [6] with the assumption that the representation learned at the encoder stage captures a generic visual representation. According to Bommes et al. [10], it is possible to detect anomalies in images by maintaining the projection head. We assert that for the video-based supervised classification task, the visual representation learned at the projection head stage can be more helpful for the (imbalanced) classification task.

We validated the above mentioned modifications on a driver anomaly detection (DAD) dataset containing nine combinations of four data modalities – top and front fitted depth and Infrared Radiation (IR) cameras inside a driver simulator [11], comprising of a training set ($25$ participants, $650$ minutes) and test set ($6$ participants, $133$ minutes) with different normal and anomalous driving behaviours captured through these cameras. The majority of the anomalous behaviours in the test set are not present in the training set. Our two-fold contribution is, (i) the proposed contrastive loss function and (ii) utilizing the entire network, including the projection head, in the test phase that improved results on six out of nine camera modalities by $4.23\%$ to $8.91\%$.

Anomaly detection problems mostly refer to situations where the data for the anomalous class (or positive class) is not available during the training phase either due to its rarity, difficulty in collection, and health or safety hazards [12]. However, in many situations, some labelled data may be available for the positive class, albeit with a large skewed data distribution. Such problems can be handled in a supervised machine learning setup using different strategies, such as weighted loss functions and data augmentation. In the context of video-based anomaly detection, there exist many approaches based on autoencoders [13, 14] and adversarial learning [15, 16, 17]. Supervised approaches for video classification are mostly based on 3D-CNN [18] and/or combined with sequential models [19, 20]. However, these models may be difficult to train on highly skewed data, especially in cases where the distribution of positive or anomalous classes differs across the training and test set.

Contrastive learning [21] approaches have been evolving around the idea of contrasting positive pairs against negative pairs, maximizing the ‘similarity’ among positive pairs while minimizing the ‘similarity’ between positive-and-negative pairs. The contrastive loss equation is also closely connected to N-pair loss, triplet loss, as the latter being a special case of generalized contrastive loss where the number of positives and negatives are each one [6].

Chen et al. [5] added a projection head consisting of several feed-forward layers in front of an encoder that improved the quality of learned representation. They showed that models with unsupervised pre-training with CL achieved better than their supervised counterparts. Khosla et al. [6] extended this idea for supervised CL to leverage labels present in the data. Their results achieved better results with Resnet-200 on the ImageNet dataset. They also showed that supervised CL outperformed cross-entropy on other datasets and two ResNet variants. This approach was shown to be robust to data corruptions, and stable to the choice of optimizers and data augmentations. CL has been used in anomaly detection with success. Zheng et al. [22] applied self-supervised CL on graph neural networks for anomaly detection and outperformed state-of-the-art methods on several datasets.

Kopuklu et al. [11] introduced the driver anomaly dataset containing $783$ minutes of overall video data collected from $31$ individuals, using top and down fitted depth and IR cameras. They applied a supervised CL approach (the neural network architecture was the same as Chen et al. [5]) that achieved better results than cross-entropy loss on detecting anomalous driving behaviours. The work presented in this paper presents modifications on Kopuklu et al. [11] in terms of a new loss function and including projection head during test phase to obtain improved results on six out of nine camera modalities and support them with statistical testing.

Anomalous driving behaviours involve various types of distraction, such as looking/moving away head from the road, setting the radio/GPS, texting or calling on a cell phone, to name a few. To capture different types of driving distractions, several driving datasets are available. CVRR-HANDS 3D [23] and DriverMHG [4] are hand focused datasets that are correlated with a driver’s ability to drive. In addition, datasets such as DrivFace [24], DriveAHead [25] and DD-Pose [26] provides information on drivers’ faces for paying attention and head pose annotations of yaw, pitch and roll angles. Drive&Act [27] provides distraction related anomalous driving data for $5$ near-IR cameras. There are many other driver monitoring datasets that provide face, head, hands, and body actions to detect normal and distracted driving behaviours, e.g., DMD [28], and Pandora [29]. A major limitation of these datasets is that the normal and anomalous driving samples are present in both the train and test set; thus indirectly simplifying the complexity of the problem. Anomalous behaviours vary across people, the distribution of actions available during training may not be available during testing. The generalization capabilities of these datasets in real-world driving situations remain unknown.

The network architecture for supervised CL framework consists of the following three components:

• •

  Encoding layers, $f_{\theta}$(.) – A 3D-CNN that extracts vector representations ($h_{i}$) of input video clips ($x_{i}$), $h_{i}=f_{\theta}(x_{i})$.

• •

  Projection layers, $g_{\beta}$(.) – comprises of fully connected layers to transform latent representation from the encoding layers to another latent representation, s.t. $v_{i}=g_{\beta}(h_{i})$. The contrastive loss is defined on the l2-normalized versions of this representations.

• •

  Contrastive loss – calculated from the summation of the similarities between vector representations of each normal video (anchor) and all other normal video clips (normal pairs) and the similarities between each anchor and all the anomalous video clips (anomalous pairs).

In consecutive training iterations, each mini-batch contains $K$ normal and $M$ anomalous video clips. The encoding and projection layers create visual representations, $v_{ni}$, and $v_{ai}$, $i\in\{1,\ldots,K+M\}$ from normal, and anomalous training clips, respectively. Each mini-batch contains $K(K-1)$ normal pairs and $KM$ anomalous pairs, which are input to the contrastive loss function. According to Equations 1 and 2, the contrastive loss calculates the similarity between normal and anomalous pairs by calculating the dot product between the latent representations $(v_{ni}^{T}v_{nj}$ and $v_{ni}^{T}v_{am}$, scaled by a temperature parameter $\tau$ and exponential function. The numerator comprises of the similarity of normal pair and the denominator contains the similarity between normal pair and an aggregate of the similarities of all the anomalous pairs. Minimization of this objective function results in joint training of the encoding and projection layers to maximize the similarities between normal samples and minimize the similarities between normal and anomalous samples. However, the summation of similarities in anomalous pairs could be problematic in cases where anomalous samples are similar to normal samples, either due to nature of the data, mislabeling or noise. In such cases, the overall sum of the negative pairs could become larger, which is undesirable for the task and makes optimization harder. In order to overcome this issue, we propose to scale the sum of anomalous pairs s.t. the overall similarity of aggregated anomalous pairs remains smaller (modification #1). Scaling the summation of anomalous pairs with a very small number could lead to learning trivial representation. Therefore, we propose to use the average of summation of negative pairs (see Equation 1). Other possibilities for scaling could be explored by treating it as a hyper parameter.

After training with contrastive loss, projection layers are normally discarded in test phase [5, 11, 6]. The rationale for this choice comes from the work of Chen et al. [5] in the context of self-supervised contrastive learning that the latent representation, $h_{i}$, learned at the encoding stage will be more general and act as a pre-trained model for further classification task. However, we argue that for the supervised learning task, the representations learned at the projection layers ($v_{i}$) can be more useful by virtue of the contrastive loss learnt at the projection layers and aligns with the supervised learning task of keeping normal and anomalous classes far apart (modification #2).

In the DAD dataset, the entire anomalous clips were given one label. A closer inspection suggested that these clips may not contain anomalous actions for the entire duration. Thus, two members of the research team manually re-labelled only the ‘anomalous’ frames in those clips and discarded rest of the frames. While manually re-labeling, we categorize a frame as normal driving behaviour if the participant has both the hands on the wheel and if the angle between the front facing direction and their current gaze is approximately smaller than $60$ degrees, as determined by visual inspection. Any frame that does not meet the above criteria is considered as anomalous driving behaviour. The normal driving clips were not re-labelled to keep variations of typical driving behaviours in those clips. This additional step is indeed specific to this dataset; however, the intent was to improve the quality of the training data. The labels in the test set remain unchanged for a fair comparison. We expect manual labeling to perform better because there will be less overlap with the normal driving in anomalous clips; thus, preventing the model from learning conflicting information.

During the test phase, the jointly trained network is used for obtaining latent visual representation for each test sample. A template for the normal driving, ($v_{n}$) [11], can be calculated from the latent representations obtained at the projection head of all the normal clips:

For a test clip, $t_{i}$, cosine similarity can be calculated between its latent representation at the projection head and the normal template as follows:

Tables II and III show the AUC ROC and AUC PR results. The rows in the tables show the camera modality and the columns show different combinations of loss (original and proposed), projection head (absent or present), and labeling (original or manual). The highlighted cells indicate the highest AUC for that modality for a given combination of loss, projection head, and labeling. The baseline results for each modality corresponds to the combination of original loss, no projection head, and original labeling. Each cell in the second column of Tables II contains two numbers, the first number is the AUC we calculated after modifying the problem in [11] (as described in section V-C), and the second number is the original AUC reported in [11]. We observe that in terms of AUC ROC, out of nine modalities, the proposed loss with the presence of projection head and manual labeling performed better in six of them (improved performance from $4.23\%$ for Fusion(DIR) to $8.91\%$ for Front(IR)). In the rows where the last column’s result is the best, the percentage increase is calculated by comparing this number with the baseline (the first number in the first column). In two cases, original loss with no projection head and manual labeling performed better (Front (D) and Front (DIR)). However, eight of the nine best performing models had manual labeling, further highlighting the role of good quality of labels in supervised classification problems. Some other findings are as follows:

• •

  Fusion of camera modalities (D and IR) for each top and front view improved the performance in comparison to individual camera and modality type irrespective of loss type and its various combinations.

• •

  Major performance improvement was achieved when top and front views were fused for each of the depth and IR cameras – Fusion (D) (AUC= $0.9609$) and Fusion (IR) (AUC=$0.9552$). Combining different views can help in capturing complementary information which improved the performance.

• •

  The best performance was achieved when both the camera modalities and views were combined (AUC=$0.9738$). Thus, apart from top and front views, combining depth and IR cameras further boost the performance in detecting anomalous driving behaviours.

In terms of AUC PR (see Table III), eight out of nine highest numbers are for proposed loss and distributed across different combinations of projection head and manual labeling. This further ascertains that modifying the loss function is important in a supervised contrastive learning task, and adding projection head and manual labeling further improves the performance.

We presented a new supervised contrastive loss function along with considerations for including projection head and refining labels to improve the detection of anomalous driving behaviours from videos. We support our results through rigorous statistical testing. We found that combining video clips from different views improves the performance drastically, with the best results achieved by combining top and frontal views and depth and IR cameras.