We are using the Distracted Driving dataset provided by State Farm, which contains images of drivers, distracted and non-distracted, taken by constant-placed 2D dashboard cameras of dimension $640$ x $480$ pixels in RGB [16]. The dataset contains $22,424$ labelled unique images of drivers that fall into one of the following $10$ distracted driver classifications (as described in Introduction): “safe driving, “texting right”, “talking on phone right”, “texting left”, “talking on phone left”, “operating radio”, “drinking”, “reaching behind”, “hair and makeup”, and “talking to passenger”. The data used to label these images are in CSV form that lists the driver subject number with their distraction ”class” and unique image identification number. The class distribution of the images is mostly even, but two of the classes (“safe driving” and “reaching behind”) vary about $10\%$ from the median number of images per class and one of the classes (“hair and makeup”) varies about $20\%$ from the median, which can affect the class predictions that our model prioritizes.

Our problem is technically challenging due to the fact that there are only $26$ unique drivers in the dataset, so we need to be careful not to overfit to the drivers in our training data. In addition, the images of drivers in different distracted poses were taken from frames of videos collected by State Farm, thus many of the images are similar, which further exacerbates the overfitting problem.

For our baseline model, we utilized ResNet-50 with pretrained weights. In order to enhance model performance, we used preprocessing steps that resized the image to $256\times 256$ spatial dimensions, center cropped to $224\times 224$ spatial dimensions normalized, and normalized with a fixed mean and standard deviation (specifically a mean of $[0.485,0.456,0.406]$ and a standard deviation of $[0.229,0.224,0.225]$). We let our ResNet-50 baseline model perform the feature extraction.

As part of our experimentation, we performed several other preprocessing and image processing steps on our train and test data in order to prevent the model from overfitting on certain features of the driver (i.e the drivers face) and help the model focus on the important parts of the image (i.e the driver’s posture). As described in the following sections, we used OpenCV to identify an eye in the image and blurred out a region relative to the eye that approximately covered the face of the driver in order to combat overfitting to the driver’s facial features and also performed skin segmentation, highlighting the driver’s skin while blacking out the rest of the image in order to draw the models attention to the driver’s posture. Afterwards, we ensembled our methods, including classical data augmentation techniques, to see if we could use them in conjunction build a strong classifier.

In our initial approach, we decided to split our data into $80\%$ train data and $20\%$ test data, resulting in approximately $18,000$ train images and approximately $4,500$ test images. When we ran our pre-trained ResNet-50 model with additional fully connected layers for our 10-class classification problem, we got results that seemed too good, and we realized that this was likely because our train and test set were incredibly similar due to the aforementioned limited number of drivers and limitations of video data being treated as individual images. Because of this, the model was overfitting on both the test and train data (despite not seeing the test data during train time) but would likely not generalize well to any new drivers and environments. Thus, we decided to take a new approach with the the train and test split.

We chose $5$ drivers at random from the $26$ (because this is approximately $20\%$ of the data) and included all of their images in the test data. We used data from the other $21$ drivers for the training set, thus ensuring minimal overlap between the two datasets. Using this approach, our baseline results were more realistic and would likely generalize well to new drivers.

ResNet has been a state-of-the-art convolutional model since it won the ImageNet classification challenge in 2015. This model presents residual modules as a solution to the problem of optimizing deeper networks, thereby enabling researchers to take advantage of the greater representation power of deep networks while not trading off on training error. These residual modules learn a residual mapping rather than directly trying to fit the desired mapping underlying the data.

The ResNet model includes stacks of residual blocks, consisting of two $3$ x $3$ convolutional layers to learn the residual mapping, as shown in figure 3. The full architecture is shown in the appendix. As described in the Related Works section, we chose to use ResNet-50 for our model as it performed well on similar challenges and converged quickly. Thus, we are using a pretrained ResNet-50 as our baseline model with a fully connected layer of hidden size $512$, a ReLU nonlinearity layer, a dropout layer (with probability $0.2$), and two fully connected layers of output size $10$ (for the $10$ distracted driver classes). For our baseline, we trained these last layers of the model using our driver-separated training set with the ResNet-50 specific preprocessing detailed in Data Preprocessing. We trained the model for 25 epochs using an Adam optimizer and achieved decent results, as specified later. For the loss function, we use cross-entropy loss with multi-class softmax where M is the number of classes ($10$ for our problem):

Data augmentation is a popular technique for improving model generalization and preventing overfitting by allowing the validation error to continue decreasing with the training error. This is because augmented data represents a more comprehensive input space, through which the differences between the training and any possible testing sets are more likely to be minimized [14]. Since overfitting and generalization are the two main challenges that we face with our dataset, we decided to experiment with two on-the-fly data augmentation techniques—specifically, random rotation and brightness adjustment.

Since all the drivers in the dataset were facing to the right, we used random rotation augmentation to produce examples of our driver rotated slightly between $0$ and $45$ degrees in both directions. In order to further increase our dataset size, we decided to add a second augmentation introducing a small, random amount of color jitter (characterized by brightness, saturation, contrast, and hue adjustments) to add a non-spatial based form of noise into our dataset. It is important to note that we do this data augmentation “online” or on the fly, performing the transforms using the torch transforms library each time the train and test data is loaded into our model.

The input to our random rotation and brightness adjustment transforms is the images from the State Farm Distracted Driving Dataset, resized and cropped to fit the ResNet-50 standard size of $224$ x $224$ x $3$ and with the appropriate color normalization, as explained in 3.2. We use the Pytorch transforms library in order to perform this random rotation and color jitter. The output of this procedure is two sets of the transformed input images. We then concatenated these transformed image sets with our original input images and shuffled, giving an augmented set of three times the size of our original training dataset. Numerically, using these two augmentation techniques in tandem, we produced a training set of $55,032$ images from the original training set of $18,344$ images.

In order to further combat the challenge of overfitting to the drivers and generalizing to different kinds of images, we need to encourage the model to consider the driver’s posture so that it does not memorize patterns in the appearance of the 21 drivers’ faces. As described by T. Muraki et al, facial blurring via a Gaussian blur is an effective way of anonymizing people for privacy purposes [8]. The same technique, called Random Blur, can be applied to our problem to blur our the driver’s face in order to discourage the model from overfitting to the individual drivers in our dataset. We took inspiration from the random blur techniques and experimented with facial blurring as an augmentation technique on both the training and test sets.

In order to perform facial blurring, for each train and test image, we identified the position of the driver’s eyes via a pretrained eye detector from haarcodes in OpenCV [9]. From the eye size and location, we then extrapolated the coordinates to find the approximate area corresponding to the driver’s face and blurred out the facial region by smoothing over the image in that approximate area using a Gaussian Blur with appropriate variance. We then used these images in combination with the original input images to assemble new Facial Blur augmented training and test sets for a ResNet-50 model with pretrained weights.

The inputs to our facial blurring script were the images from the State Farm distracted driving dataset resized and cropped to fit the Resnet-50 standards as explained in Data Preprocessing. The outputs of this blurring script are all the same images from the train and test sets with the drivers’ faces blurred. Unlike the classical augmentation method, we decided to perform this technique offline since it is is more complex and time consuming. Using this technique, we were able to double the size of our training set while improving the generalization of our model.

After performing our first set of experiments and analyzing the saliency maps from our models, we noticed that the model was focusing on the posture, particularly arm and hand position of the driver, for many of the successfully classified images. Thus, we hypothesized that if we could encourage the model to focus more on the driver’s posture we would be able to improve the performance of our classifier and help the model generalize better to new images of drivers.

Skin segmentation is a technique that has been widely used in many biometric applications, face recognition, and hand gesture recognition. It is the act of separating skin and non-skin pixels in an arbitrary image and has been shown to be effective for posture detection and hand gesture analysis, as discussed in Related Works. There are many different methods of performing skin segmentation. In this study, we focus on the method presented by R. Rahmat et al, which combines the lighting channels of the HSV, YCbCr and Normalized RGB color spaces to obtain high accuracy of skin detection [11]. This method enables us to find a simple, effective, and efficient way to identify the skin of a driver and black out the rest of the image to teach our model to focus on a driver posture.

We implemented a version of this combination of “chrominance” technique based on Will Brennan’s Skin Detector, [2] which uses masking and thresholding of RGB (red, green, blue values) and HSV (hue, saturation, value) through the OpenCV library in order to leave the colored the portion of the image which contains skin (falling within the range of RGB and HSV defined by the user) intact and blacks out the rest of the image. In order to detect the range of colors which we would consider “skin”, we wrote a python script to allow us to toggle the RGB and HSV values of our input images to find a range of RGB and HSV values that would, on most images, black out the background and leave the skin intact. It is important to note that there is some bias in this technique as it does not work as well on persons of color and is dependent on the lighting in the image, but as there is little ethnic diversity in our dataset and the images are lighted homogeneously, we were able to find an RGB and HSV color range that fit many images.

The inputs for our skin segmentation script were the original set of images from the distracted driving dataset and the ranges of RGB and HSV values, which we would use as the ranges for skin segmentation. The outputs are used as input images to the model and show the original image with only the skin of the driver remaining in color and the background blacked out. As with the facial blurring set, we produced these images offline as the generation was complex and time consuming. We then used these images to augment our training set by resizing and cropping them to fit the ResNet-50 standard size of $224$ x $224$ x $3$ and the appropriate color normalization, and then concatenating them with our original input images and shuffling. Through this technique we were able to double the size of our dataset while also teaching our model to focus on the driver’s posture and less on any background noise.

Finally, through our experimentation, we found that we had developed several different powerful pre-processing augmentation techniques. In order to round out our ablation study, we decided to explore combining these techniques to see if a specific combination of pre-processing augmentation steps could boost our model’s performance. As such, we experimented with different combinations of the techniques by concatenating together the appropriate datasets we had generated for facial blurring and/or skin segmentation and generating the classically augmented images on-the-fly. We found that by combining a number of techniques, we were able to achieve high scores on our multi-class classification task, as described further in the Results and Discussion section.

We tuned a few main hyperparameters such as learning rate, mini-batch size, and probability of dropout in the final dropout layer (introduced as part of the final classification layers of the model). We started the learning rate and betas of the Adam optimizer with the default recommended values ($10^{-3}$ for the learning rate and $[0.9,0.99]$ for the betas), and tuned the learning rate to be $3*10^{-3}$ because it allowed for faster training with slightly improved results. We used a mini-batch size of $64$ images to fit the memory requirements of our CPU and GPU and allow for faster convergence while also being large enough to have minimal noise in the calculated gradients. Finally, we chose the dropout probability to be 20% (meaning there is an 80% chance of retaining a given unit) based on the recommendations of the paper that introduced dropout as a regularization technique [15].

We used F1 score to evaluate our best model as well as accuracy, precision, and recall as metrics to gain a quick understanding of where our model tended to fail, which then informed the further quantitative and qualitative metrics we examined.

F1 score is a measure of model performance that gives a summary of the precision and recall metrics and is calculated as follows:

Precision gives a measure of how many of the images predicted to be in a given class $c$ actually belong to that class, and is calculated as follows:

.

Recall gives a measure of how many of the images that belong to a given class $c$ are actually predicted to be in that class, and is calculated as follows:

.

The results of our experiments are detailed in the table in Figure 7. We also generated the Confusion Matrices in Figure 8 and the Appendix to provide more insight into our model’s behavior. Please refer to the Methods section for further explanation of our experiments.

The Saliency maps for our different model are shown below in Figure 9.

We generated the saliency maps shown in Figure 9 in order to see what parts of the image the model attended to, how that may have affected the model’s behavior, and how we could address the issues with our different experiments. Considering the saliency map of an image correctly classified as safe driving, we observe that the model was primarily focusing on the arms and posture of the driver.

Seeing that this was a successful signal for the model, we decided to amplify it, at first by blurring the faces of the driver and adding those images as a supplement to the training images in order to teach the model to treat the faces as noise and not overfit to them so that the model would focus more on posture of the driver when identifying distraction. Considering the saliency map for the output of the blur augmented images model, we see a higher concentration of red on the arms of model and key chest and neck points, which shows that the model focused more on the body of the driver as desired. This is likely one the reasons that the model which used the blur augmented images performed even better than the baseline.

Along the same line of reasoning, we decided to augment the input images to the model with skin segmented images, where only the drivers’ posture was highlighted and the rest of the image was blacked out as shown in Figure 6. As shown in the results table, we actually found that augmenting with skin segmented images alone led the model to do slightly worse than the baseline. Looking at the saliency map, this is likely because the model was trying to make sense of the blacked out portions of the images while also focusing more on the arms and posture of the driver. Thus, the model may have overfit to these blacked out regions, which we had intended to be ignored, thereby causing the model to perform worse than the baseline.

Therefore, the model did the best using a combination of all the different pre-processing augmentation techniques since the saliency maps show that the model focused very clearly on the driver’s posture while ignoring the background and paying less attention to the face. In turn, the model did not overfit to the train images, which we hypothesize is because the classically augmented images provided the model with more of a variety to train on. Looking at the saliency map, it is clear to see that with a combination of all our data pre-processing augmentation techniques, the model learns to focus on the most salient features of the image and ignore noise, thus leading it to perform very well.