Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety

The use of hard hats in construction is an instance where appropriate safety covering can prevent injury or death and may be enhanced by IoT-style worksite safety monitoring. Annually, there are more than 800 casualties of construction workers as a result of workplace accidents [1]. In 2020, the private construction industry reported 1,008 deaths, marking the highest number of fatalities among all private industries [2]. Many of these fatal injuries can be lessened or avoided with proper wearing of hard hats.

Hard hats provide essential protection against head injuries caused by falling objects, electrical hazards, and collisions, which are prevalent risks in construction and industrial settings. Given the increased risk of fatal injuries among construction workers, it is crucial to enforce stringent regulatory safety measures within the workplace.

OSHA’s Section 1926.100 requires protective helmets for employees in areas with possible head injury risks [3]. Despite regulations mandating hard hat use, compliance is inconsistent, leading to preventable injuries. Having a guideline supported by camera detection may help workers be more aware of when and where to wear hard hats. By properly detecting workers’ hard hat status through visual surveillance, sites can raise awareness of the issue and assist with the enforcement of hard hat-wearing.

The question we explore in this research is to what degree foundation model approaches are ready for use with real-world data in these safety perception domains and where their strengths and weaknesses may lie. Therefore, in this research, we make research contributions of (1) the evaluation of zero-shot methods using foundation vision-language models (VLMs) of hard-hat detection and association and (2) the creation of an evaluation benchmark dataset for hard hat detection, titled Hardhat Safety Detection Dataset, combining and filtering annotations of the following image source datasets: Hard hat workers dataset[4] and the SHEL5k Dataset[5].

We begin by presenting a framework for application of the zero-shot detection method in our hardhat detection process using a cascaded detection method. Our experimental results with 5,210 images show that the OWLv2 model achieves an average precision of 0.6493 on hardhat detection. We also conduct a case study on several failed detections to analyze the limitations of using OWLv2 for hardhat detection, and discuss methods of improving the model for real-world robust detection.

Traditional machine learning object detection algorithms historically rely on manual annotations and specialized algorithms, which can be time-consuming and resource-intensive, especially as the models are limited to learning from provided datasets. Moreover, these methods often lack the flexibility to adapt to new environments or variations in hardhat designs [6].

In contrast, foundation vision-language models, or VLMs, with their ability to generalize from text descriptions and visual features, offer a more adaptable solution. VLMs are a type of artificial intelligence that integrates both visual and textual data to perform tasks requiring a deep understanding of these two modalities, and have been useful in a variety of real-world safety applications [7, 8, 9, 10, 11]. These models are designed to analyze images and interpret text, enabling them to carry out tasks such as image captioning, visual question answering, and multimodal reasoning.

Due to being pre-trained on a much larger magnitude of data, these models can approach unseen tasks with higher accuracy. Our research builds on the Vision Transformer for Open-World Localization (OWL-ViT) and the OWLv2 family of models found in [12] and [13], and is further detailed in [14].

We propose to evaluate foundational VLMs’ zero-shot learning capability for their readiness for use in the detection of hard hats within real-world data. Other researchers have applied various methods towards the same task of hard hat detection. Xie et al. [15] proposed the CAHD algorithm, a convolutional neural network based hard-hat detection algorithm. This algorithm achieved a mAP of 54.6% on the ImageNet Dataset[16]. However, the ImageNet dataset only shows objects in central focus, unlike real-world images where the object can be in the background, which we choose to address.

Accurately detecting and enforcing the use of protective gear like hard hats is a task that remains challenging despite advancements in object detection models. The inconsistencies and incomplete annotations in existing datasets, such as the Hard Hat Workers Dataset [4], further complicate this task, leading to unreliable performance metrics and hindering the development of effective safety solutions. Our methodology aims to address these challenges by releasing a new dataset that addresses the dataset issues and detailing our cascaded detection strategy using the OWLv2 model.

Using the cascaded object and attribution detection algorithm detailed in the previous section, we performed hard hat detection on the combined dataset, with further implementation details described in this section. The evaluation metric we utilize for our task is the average precision (AP), as well as precision and recall across thresholds.

We performed detections within the 5,210 images in the combined dataset. Our detections were conducted across several thresholds, ranging from 0.05 to 0.5, to calculate the precision-recall curve. With this threshold sweep, we calculated the AP.

The threshold for OWLv2 is the minimum confidence threshold to use to filter out predicted boxes. It is the value that determines the minimum level of confidence required for OWLv2’s detection to be accepted as a valid positive detection. OWLv2 calculates confidence through logits per image.

The intersection over union (IoU) is defined as the ratio of the area of overlap between the predicted bounding box and the ground truth bounding box to the area of their union. We use this as our evaluation metric by observing if the IoU is greater or less than 0.5 to determine whether detection is a true positive or false positive. The IoU is calculated by: $IoU=\frac{A\cap B}{A\cup B}$ or $IoU=\frac{Area\>of\>Overlap}{Area\>of\>Union}$, where $A$ stands for the predicted bounding box and $B$ stands for the ground truth bounding box.

To evaluate our cascaded methodology of a hierarchical multi-stage detection and observe its effectiveness, we additionally tested a nested detection approach and a direct detection approach of hard hats. The nested detection was performed by detecting persons and then detecting hard hats within the person’s bounding box. However, we note that the direct detection approach does not address the association task of the hard hat to a person, detecting hard hats on the ground. As with the complete cascaded detection, we performed a hyperparameter sweep of the threshold for both processes to compare the APs of hard hat detection.

Additionally, for our data, when using the cascaded approach, we treated all people (whether wearing a helmet or not) as one single class and did not focus on detecting annotations of faces or isolated helmets since we only concerned ourselves with helmets associated with a person’s head. However, when testing nested detection and direct detection, as described in Section IV, we used the isolated helmet class rather than the head with helmets.

The definition of ground truth varies for different categories. In person detection, all instances of persons are considered as ground truth, and for head prediction, every head without a helmet is the ground truth. However, for helmets, we include all helmets, even those on the ground, when computing the metrics. This causes errors in false negatives for the nested detection approach as it can only detect helmets on a person. Re-annotating the dataset is needed to resolve this issue.

As shown in Table II and Figure 7, the removal of cascading levels within the detection strategy improves performance. This is due to the fact that as we continue to crop the image multiple times, the image quality gets progressively worse, meaning with more stages of detection and cropping, the more inaccurate the final stage of detection will be. Additionally, the final stage, the hard hat detection, depends on the upper levels of detection, meaning that if the head or person is detected inaccurately or not detected at all, the hard hat has 0 chance of being detected, further reducing accuracy.

Therefore, adding layers to the detection approach adds more variables of error and reduces accuracy, proving that a direct detection approach is the most accurate. However, to address the task of associating a hardhat with a person, having at least the nested detection approach is necessary.

Additionally, the head detection within the cascaded detection was shown to have very poor performance. This may extend the problem described above. As helmet detection doesn’t have the best accuracies due to the reduction of quality by cropping, the rest of the detected heads are classified as heads without helmets. This means that there are many false positives for heads without helmets and many false negatives for helmets, resulting in a decrease in the precision of the head detection and the recall of the helmets.

One nuance of removing the head detection step is that people who are holding helmets will be classified as helmet-wearing, even though they are not wearing it. When observing the data, we notice that possible sources of error include incomplete annotations, obstruction, and similar-looking objects, as shown in Figure 8. To mitigate these issues, capturing and analyzing multiple images of the same scene while improving annotations could help reduce errors and improve accuracy.

We have demonstrated the potential of using foundation vision-language models for zero-shot detection of hard hats to enhance safety on construction sites. By creating the Hardhat Safety Detection Dataset and using the OWLv2 model in a cascaded detection approach, we found that direct detection of hard hats yields higher accuracy compared to multi-stage cascaded methods due to image quality degradation and compounding errors. Despite these challenges, associating detected hard hats with individuals is crucial for practical safety, emphasizing the need for robust annotations in datasets.

Future research should focus on several key areas to enhance the efficacy and reliability of hard hat detection. Improving and expanding datasets with comprehensive annotations is essential to address missing instances and to develop separate annotations for helmet-person association and detection. Additionally, developing techniques to avoid quality reduction in the cascaded detection approach and exploring state-of-the-art models and hybrid approaches can offer significant improvements. Furthermore, reducing false positives and negatives remains a critical challenge, and techniques such as multi-frame analysis, context-aware detection, and incorporating additional sensory data should be explored to enhance detection reliability.

Future research can significantly improve occupational safety by addressing these areas. The advancement of VLMs holds great promise for creating safer work environments.