STN PLAD: A Dataset for Multi-Size Power Line Assets Detection in High-Resolution UAV Images

A common problem found in the literature when using Deep Learning to detect power line objects is finding data. There are not enough publicly available datasets to feed detectors based on deep learning methods, or they do not cover enough power line components [1, 16]. Many similar works use private datasets, generally provided by the companies or government agencies financing the projects, which tend not to publish them [17, 18, 1, 19, 20]. Nevertheless, in the literature search presented by Liu et al. [16] and in the work of Abdelfattah et al. [15], a few publicly available datasets were found but with many limitations. Table I summarizes the main public image datasets related to power lines assets for the object detection task. The last line shows the proposed dataset for comparison purposes.

As evidenced in Table I, there is a minimal amount of public datasets related to power line asset detection. They target distinct tasks that can be detection, classification, or segmentation. For instance, the one in [21] is specifically related to conductor wires in low-resolution images for binary classification. Zhang et al. [22] propose two datasets of binarized masks of conductor wires of power lines in urban and mountain scenarios, respectively. Abdelfattah et al. [15] propose a dataset containing pixel-wise annotation (a.k.a. instance segmentation) of both transmission towers and power lines. However, the main competing datasets are CPLID [13], and the one from Tomaszewski et al. [14] as they are the only ones that use bounding box annotations.

CPLID [13] is a dataset related only to a specific type of insulator with a specific shape and size. Although it also has annotations of defects in some of those insulators, it lacks a diversity of data since there is only one type of power line asset. The defective samples are also limited because all of them are from data augmentation, i.e., a single faulty insulator was cropped from an image and then pasted into a limited set of backgrounds, like seen in Figure 2(a). On the other hand, STN PLAD provides asset variability in diverse scenarios.

The dataset in Tomaszewski et al. [14] has even more limitations. They mainly target data quantity (they reported 2630 images) rather than data variability, as they video recorded a ceramic long rod insulator hanging on an apparatus built by them and then extracted some of the frames using a stationary camera. These images only contain one of nine different backgrounds that do not correspond to real-world power line scenarios. An image from one of these nine variations is shown in Figure 2(b). From the perspective of deep learning techniques, this dataset has a very limited variability of scenes. In summary, although this dataset has a reasonable amount of data, the images taken in the same scenario have a high degree of similarity. Thus, the dataset ends up not being independent and identically distributed (IID) [23, 24], an essential dataset property. In practical terms, it is not feasible to use it in most techniques based on deep learning since it poses little to no challenge to these techniques. In STN PLAD, the images are collected by a drone in the field, providing several real-world scenarios with multiple objects (size, appearance, position, orientation, self-occlusion, background).

A few works address the issue of detecting objects related to power lines in high-resolution images. The first one, Zhang et al. [19], is a study on object detection in high-resolution images captured through Unmanned Aerial Vehicles (UAVs), using deep learning techniques. In their work, the authors propose the MOHR dataset. This private dataset has over ten thousand high-resolution UAV images with five classes: car, truck, building, collapse, and flood damage. The UAV altitudes are high, ranging from 200 to 400 meters, making the objects look quite small. The authors apply six general-purpose object detectors, SSD [25] and Faster R-CNN [26] included, to the MOHR dataset. The results suggest that detecting small object instances in high-resolution UAV images remains challenging since they perform poorly. The best mean Average Precision (mAP) achieved was 43.94%, yielded by RFCN-DF [27]. Those results reinforce the relevance of this task.

The works of Kong et al. [28] and Zhu et al. [29] share some of their authors and contents, indicating that one is an incremental improvement over the other. The former proposes a technique to detect small objects in high-resolution images. The technique, based on Faster R-CNN, is tested on a private dataset of 3700 high-resolution images. However, the proposed approach here is limited and prone to issues since only small objects inside the context of a large one are attainable. For instance, dampers are usually small independent objects far from larger ones. Another issue is the low Average Precision (AP) of some classes, such as the tower plate (73.2%), which the authors justify by saying they are too small.

Finally, Zhu et al. [29] attempts to improve the efficiency of their previous work by merging the two stages in order to share early convolutional layers. They also use a private dataset with high-resolution images with six classes: electric tower, vibration damper, spacer, insulator, bird’s nest, and tower plate. Despite the efficiency of this method, the same issues and limitations that existed before regarding object detection are maintained. The only difference is that the objects are not so small as their previous work, which is one of the main factors that positively impact the mAP.

These last two works, [28] and [29], are not reproducible since the datasets are private, and they are not open-source.

In the context of power line inspection, SSD is one of the suggested techniques of two recent reviews [16, 1] to target the problem of detecting assets on power transmission towers. Both reviews have the same context as this work, focusing on inspecting power line assets from UAV images. Moreover, they analyze Deep Learning techniques applied to solve problems in the area. Some of the mentioned problems are assets detection, assets segmentation, assets fault identification.

The parameters of the SSD used are the same as in the original work by Liu et al. [25], such as the backbone, VGG16, and all the parameters and dimensions for the convolutional layers. In the original work, two different input layers were proposed, $300\times 300$ and $512\times 512$. In our experiment, the latter was used since it achieved better accuracy than the former, according to the original results. Also, the images used for this experiment have a higher resolution. This high-resolution implies that the larger the size of the input layer, the less the resizing effect will affect the input image quality. Finally, weights pre-trained with the COCO Dataset [31] were used.

The objective of including the Faster R-CNN [26] in the tests was to use a recent technique of object detection to obtain results close to the current state of the art. Faster R-CNN-based networks are also suggested to detect and inspect power transmission towers according to the same reviews mentioned in subsection IV-A. They are also well consolidated, have performed well in object detection competitions, and are used by similar works [16, 1].

The network used for this experiment was the Feature Pyramid Network (FPN) Faster R-CNN [32]. FPN aims to improve the detection of small objects, as it uses multi-scale feature maps and higher resolution layers to build new semantically rich layers. Thus, information from the initial layers is used. These layers are traditionally less condensed, but even so, they already have a high semantic level. ResNet-101 was used as a backbone, which had the best detection result in its publication [32]. Also, the input resolution was chosen in order to decrease the image resizing impact. The input image is resized to $2736\times 1824$, representing a downscaling factor of $4$ when compared to the original size, which is much less than the downscaling factor of approximately $19$ used in the SSD experiment. All other parameters were kept as the original.

After observing the results related to the SSD and Faster R-CNN methods, it was noticed that a simple pipeline modification could enhance the overall performance of power line assets detection in STN PLAD. This approach takes advantage of the images’ high resolution, where information is lost after resizing. In the Multi-Size Power line Asset Detection (MS-PAD) workflow, represented in Figure 4, two independent networks are trained separately. The SSD was chosen because it performed better, as it will be shown in the next section.

The first of these two networks uses the original pipeline that resizes its input, but this time trained without the Stockbridge damper class. The damper asset was excluded because it has a much smaller size, being harder to identify after image resizing. This strategy can be applied to other assets not contemplated in this work that are small. An important note is that, although the tower plate is also small, it still had enough features after resizing that made it highly recognizable.

The second network is responsible for detecting small objects, in this case, the Stockbridge damper class. It goes through a different process, where the initial image is split following a grid. The image is divided into 16 smaller ones with a fixed resolution of $1368\times 769$ or $1368\times 912$, depending on the original image, which can be $5472\times 3078$ (3:2) or $5472\times 3648$ (16:9). This $4\times 4$ division is constant since the drone pilots followed a data collection protocol, in which the drone had to stay at similar distances from the transmission towers, as described in section III.

In summary, only the Stockbridge damper class is submitted to the second step of MS-PAD, which divides the high-resolution image in a $4\times 4$ grid. This choice is based on the average area of the classes of Table II and the AP in the original image-resizing approach. The Stockbridge damper has the lowest average area and was not well detected using the previously mentioned methods, indicating the need to receive extra attention compared to other classes.

For this test, both detectors were trained once and for the same period, about two days. The mAP results of using the MS-PAD approach for SSD and Faster R-CNN were 90.2% and 88.6%, respectively, showing that SSD has a slight advantage over Faster R-CNN. These methods were also applied to the two main dataset competitors of the proposed STN PLAD. In CPLID [13], SSD and Faster R-CNN achieved 98.17% and 98.31% mAP, respectively. Regarding Tomaszewski et al. [14], both detectors reached 100% mAP.

Figure 5 and Figure 6 show the visual results regarding the detected objects by the SSD and the Faster R-CNN methods, respectively, using the original approach, which only resizes the input image. In the images, the bounding boxes’ colors are connected with the dataset classes: blue is for the Insulators; yellow is for the Spacers; green is for the Stockbridge dampers; red is for the Tower plate; white is for the Transmission tower. It is possible to see in both figures how small the Stockbridge dampers are related to other objects. These images also illustrates failure cases, like the middle insulator in Figure 5 and the transmission tower in Figure 6.

For this experiment, $k=5$, so five splits with randomly selected samples were used, in which each split is used twice: one time for the original image-resizing approach and another time for MS-PAD. The total amount of iterations for each training session is fixed at 20,000. The obtained results regarding AP for each split are shown in Table III. In addition, Table IV shows the average results from Table III reached by each approach side-by-side, in a direct comparison.

Figure 7 and Figure 8 present the qualitative performance of MS-PAD for big and small objects, respectively. The color codes previously mentioned are maintained for these images.

The proposed STN PLAD is the first public power line assets dataset with multiple objects in real-world scenarios. It contains five classes of entirely different objects with multiple instances each, in varied real backgrounds. The data collection protocol allows for a balance in data quantity and variability since the captured images vary in illumination, backgrounds, and weather conditions. Also, the drone position is not fixed in order to obtain objects data from different perspectives. Another STN PLAD challenging characteristic is that there are many objects per image (18.1, on average) compared to the related public datasets, which commonly have a small instance per image rate (1 and 1.9, on average). Thanks to this process, STN PLAD poses a reasonable challenge to recent deep learning techniques, as observed in the section V, in which the best of the tested approaches achieved an 89.2% mAP, leaving considerable room for improvement.

Although the proposed STN PLAD provides new grounds in the power line area and stimulates the development of power line asset detection methods, it still has limitations. The main one is related to its total amount of images. That prevents some data-hungry object detectors from performing successfully since they would require a more extensive dataset. Another disadvantage is that the images were only collected from one private transmission line. Even though different transmission lines tend to be similar, it would be better to have images of several transmission lines in other places to reduce the bias of background, environment, and electrical assets appearance. Finally, the images belong to a power line in Brazil, which may not apply to other countries.

This discussion is related to the experiment in subsection V-A. According to the proposed methodology, when performing this experiment, it was expected that the results related to the Faster R-CNN would surpass the results from the SSD network considering the applied metrics. However, it can be observed in the results reported in subsection V-A that it did not occur. This result was obtained due to the limitations of the used data. Deep learning techniques benefit from the use of large amounts of data. According to Ng [34] [35] when the number of data limits a deep learning technique, shallower techniques can obtain comparable or even better results than deeper techniques. The used Faster R-CNN is much deeper and has more trainable parameters than the used SSD network. Therefore, for a limited amount of data, the learning of the used Faster R-CNN is limited.

The other results in this comparison were related to the SSD and Faster R-CNN performance in the competing datasets. In [14], both methods achieved 100% mAP, as expected due to the reasons presented in subsection II-A. In [13], the original SSD and Faster R-CNN obtained performances above 98% mAP. The high mAP values obtained by both object detectors in both competing datasets showed how well-resolved their challenges already are.

It is possible to see in Table III the mAP values reached by MS-PAD are higher than all mAP values of the Original approach in at least ten percentage points ($k=5$) and at most 17.5 percentage points ($k=2$).Table IV shows a direct comparison, in which the values for each approach are an average of the five splits showed in Table III. The best values for each class AP and mAP are in bold. MS-PAD yields the best AP result in three out of the five total classes, and there is a gap of 13.7 percentage points between mAPs. That gap is primarily due to the Stockbridge damper AP improvement, which grew 62.4 percentage points using MS-PAD.

It is noteworthy that the performance of large objects changes when comparing the Original and the MS-PAD. That may happen because during the MS-PAD resize branch training, one less class is considered (Stockbridge damper). That directly influences how the network learns since there is a different amount of objects and classes, directly impacting the final performance of large assets. Also, it is important to note that there is no guarantee that the performance impact will be positive or negative when training with one less class.