CLOI: AN AUTOMATED BENCHMARK FRAMEWORK FOR GENERATING GEOMETRIC DIGITAL TWINS OF INDUSTRIAL FACILITIES

In our previous work [Agapaki et al. (2018)], we identified the most frequent and laborious to model object types, which are cylindrical objects (straight pipes, electrical conduit and circular hollow sections), valves, elbows, I-beams, angles, channels and flanges. Cylinders require 80% of the total modelling time of the ten most important object types in EdgeWise [ClearEdge (2019)] and represent 45.5% of the total number of objects in an industrial plant on average. EdgeWise was selected compared to other state-of-the-art software, because it is the only commercially available tool that attempts to automatically extract cylinders from the point cloud of an industrial plant without significant user assistance. EdgeWise has significantly accelerated 3D modelling of industrial plants according to the findings discussed above. However, it has some limitations, which can be summarized as follows:

1. 1.

   Structural elements (I-beams, angles, channels) should be manually modelled and their location in the point cloud is roughly defined based on the modeler’s discretion.

2. 2.

   Segmentation of cylinders has been partially achieved with detection rates being 75% recall and 62% precision on average [Agapaki et al. (2018)]. The same metrics for cylindrical objects labelled as pipes are 58% and 47% respectively. It is also important to note that EdgeWise erroneously includes points that do not belong to a geometric shape. This is due to fitting errors, which occur since primitive shapes are perfect shapes, whereas the scanned, physical objects are imperfect (e.g. a cylindrical pipe may be bent).

3. 3.

   EdgeWise is not designed to output geometric shapes in an open and generic format. As such, modelers cannot easily exchange data between different operational-phase gDT platforms due to data inconsistency between them.

Therefore, the evaluation of EdgeWise uncovered (a) the substantial performance of this software in detecting cylinders with its pitfalls, (b) the inability of the software to (i) further classify cylinders into conduit or pipes or CHSs and (ii) detect and further classify I-beams, channels, flanges, valves and angles in spite of their high frequency in an industrial facility.

This performance of EdgeWise has substantial room for improvement and this paper intends to address the above-mentioned limitations in order to automatically generate gDTs of industrial facilities and assist the tedious current practice. We propose a geometric twinning framework for existing industrial facilities and bench-mark it with the current state of practice. In the following section, the state-of-the-art research methods related to the above-mentioned limitations are presented. We then outline the framework in the proposed solution, which is followed by the experiments and results. The conclusions are then derived in the last section.

Class segmentation methods applied on industrial shapes have been widely investigated. We categorize them into three groups: (a) attribute based methods, (b) machine learning and (c) deep learning methods. A comprehensive review of class segmentation methods based on hand-crafted features is provided by [Agapaki and Nahangi (2020)] and some of the most important methods are explained in the paragraphs that follow.

3D instance segmentation is based on 3D geometric class segmentation networks. These methods can be grouped into shape-based (top-down) or shape-free (bottom-up). Our readers can refer to [Agapaki and Brilakis (2020b)] for a comprehensive literature review of each of these methods. We elaborate on the state-of-the-art literature on shape-free methods, since these are more suitable for the generation of gDTs from TLS industrial data [Agapaki and Brilakis (2020b)].

Shape-free methods are based on deep learning networks, which aggregate features per point and output instance labels per point given a similarity matrix between pairs of points [Wang et al. (2018b), Wang et al. (2019)] or embedding another network measuring point-wise distances [Pham et al. (2019)]. PointNET [Qi et al. (2017b)] and PointNET++ [Qi et al. (2017a)] is the backbone network for these methods, meaning that they achieve class segmentation as well. Although these networks take into consideration the local neighbourhoods of points, they cannot explicitly define the boundaries of complex industrial shapes. Object boundaries can be taken into account by considering the class and instance segmentation labels. The readers can refer to [Xie et al. (2019)] for a detailed review of all the instance segmentation methods.

The proposed framework consists of two major parts. Specifically, these parts are (1) class segmentation and (2) instance level segmentation that intend to answer the research questions as outlined in the Background section and aim to outperform the existing state of practice and research in the industrial modelling space.

We propose a novel hybrid framework which develops deep learning networks and leverages their detected outputs with industrial engineering knowledge, in order to automatically extract labelled point clusters corresponding to industrial shape components without generating surface primitives (class point clusters) and then to efficiently detect individual industrial shapes from the labelled point clusters (instance point clusters).

Real-world industrial environments are more challenging than buildings that have been extensively studied and scanned in previous research efforts as mentioned in the Background section. Industrial components do not comply with a universal colour scheme, rather colours depend on each manufacturer’s specifications [Agapaki and Brilakis (2020a)]. Industrial spaces are typically large and unstructured with shapes that may span across their whole length/width and they are heterogeneous spaces where there are usually no direct contextual rules in separate systems (piping, structural, electrical) and only the components that belong to the same system are internally connected with strong context. For example, the relative location of a cylinder with respect to an I-beam in a factory does not imply that the locations of these objects should comply to specific spatial rules. We propose a 3D-slicing facility window method, CLOI-NET-class based networks and CLOI-Instance graph-connectivity algorithms to tackle these challenges. The 3D windows are used to segment the TLS dataset in non-overlapping parts, so that a portion of these windows will be used for training. These windows should be non-overlapping, so that the training and test set are disjoint. These algorithms are the core foundation of the methods built upon them to enhance the segmentation and detection results. The proposed algorithms can deal with the challenges outlined above and can accurately detect the majority of CLOI industrial shapes.

Most of the CLOI shapes match 1 to 1 to a component class, (i.e. the shape is unique to this component), but for cylinders the shape is not unique. So the method focuses on segmenting the CLOI shapes, and by default, equivalently segments their component classes except for cylinders. Segmentation of the subcategories of cylindrical shapes (i.e. pipes, circular hollow sections, handrails, electrical conduit) is beyond the scope of this research. The proposed framework is not applicable for connections of steel members (welding and bolting). The proposed algorithms address scale variance (The algorithms are scale invariant, since we feed them with objects at different scales (from a few centimeters to some meters.) of industrial objects and intra-class variations. For instance, there are many types of valves as expressed above, which are grouped in one class and the proposed algorithms should be able to segment valves of all the above mentioned categories.

We illustrate the developed hybrid framework in Figure 2. It consists of two major processes: Process 1, class segmentation of CLOI industrial point clusters, and Process 2, instance segmentation of CLOI industrial shapes from point clusters.

The proposed framework starts with a raw, laser-scanned, PCD of an existing industrial facility (data format: points in .pcd, .txt, .las, .xyz). External noise such as vegetation, adjacent buildings is removed using commercial software as explained in [Agapaki and Brilakis (2020a)]. The industrial PCD contains CLOI-shapes and any other industrial shapes inside a factory (data format: points in .pcd, .txt, .las, .xyz). The first step of the framework is to automatically split the PCD facility in 3D windows and the 3D windows in “3D blocks”. Then, the 3D blocks are aligned in the global coordinate system. As such, the outputs of this step are 3D block PCDs (data format: points in .pcd, .txt, .las or .xyz). Then, we manually annotate industrial facilities to generate a benchmark dataset and the outputs of this step are class and instance segmentation labels and points. It is important to note that this is an essential offline step needed for training purposes and serves as the ground truth for the validation of the framework.

Next, we propose a three-step class segmentation method (Process 1) to segment the CLOI point clusters from the 3D blocks. The final outputs of this process are seven industrial shapes, namely cylinders, elbows, channels, I-beams, angles, flanges and valves, in the form of labelled point clusters (data format: points in .pcd, .txt, .las, .xyz). Then, we suggest an optimal manual annotation (if the users select it) to remove the erroneous point clusters maintained from Process 1 followed by proposing an efficient instance segmentation method (Process 2) through which the seven CLOI classes (in point cluster format) can be directly segmented to individual shapes. The final outputs of this process are point data corresponding to the points, class and instance labels per point. We elaborate on each process in the following sections.

We validate Process 1 on the CLOI benchmark dataset [Agapaki et al. (2019)], which is composed of four laser scanned industrial facilities. The original number of laser scanned points, the number of instances, the area and the manual labor hours to manually annotate (with class and instance labels) each facility are documented in Figure 3.

The methods of Process 1 bypass the stage of surface generation altogether and directly output segmented and labelled point clusters. The 3D window parsing method breaks down the whole industrial facility into subset windows for more efficient processing. The key insight behind Process 1 is to formulate a high dimensional feature space to automatically assign labels per point so that the target point clusters can be quickly located in the point cloud.

The inputs of the method are the spatial coordinates of TLS points and the outputs are labelled, segmented point clusters with confidence levels of the predictions. Here we define segmented point clusters as all the points that belong to one class i.e. all cylinder points is one class point cluster. The method consists of three major steps: Step 1 partitions each facility into smaller spaces using a 3D sliding window/block approach and prepares the data for training, Step 2 predicts a class label per point using a modified version (SFR) of a geometric deep learning network for point cloud segmentation (PointNET++) with the goal to accurately segment the CLOI shapes. In Step 2, the user has two options on how to train the network, either training with no data from the test facility or manually annotating data of the test facility and including those for training. The latter is based on the assumption that, inevitably, any class segmentation algorithm will have errors, which will have to be manually corrected eventually. Therefore the goal is to minimize the total manual annotation time. Step 3 refines the predicted class labels by improving class level predictions with stronger contextual relationships.

The success of the proposed pipeline is measured not by maximising the point-wise accuracy of the method, rather by minimising the cost that it incurs to the modelers when using it. This novel method leverages the advances in point cloud deep learning segmentation, contextual shape specific attributes and active learning in order to accurately predict point-wise class labels with no significant difference in performance for diverse industrial environments. A critical part of this method’s novel design is the stage-wise annotation, which permits both human-annotated and automatically annotated points to influence the system’s view of what needs the most human attention next. Details of our methodology, named CLOI-NET-Class, can be found in [Agapaki and Brilakis (2020a)].

The inputs of Process 2 are the predicted point clusters from the CLOI-NET-Class method for the evaluation of the proposed framework. The same 3D block generation method from Process 1 is used for segmenting the input data. The outputs of this process are point-wise instance labels (individual point clusters of CLOI shapes).

Process 2 consists of two major steps: Step 1 predicts an instance label per point by using a graph-based method, namely Breadth First Search (BFS) that was originally introduced by [Bauer and Wössner (1972)]. Step 2 is a boundary segmentation method that is used to enhance the instance segmentation results of Step 1. An assumption of the method is that the initial TLS industrial data is partitioned in 3D non-overlapping sliding windows with overlapping 3D blocks. The outputs of Step 1 are connected components based on connectivity relationships in order to segment the instances as output. The boundary segmentation method in Step 2 outputs binary labels on whether a point is a boundary point or not. These instance point clusters present industrial shapes at Level of Detail (LOD) 300.

The novelty of Process 2 is two-fold:

1. 1.

   the efficiency of the BFS algorithm by applying it on the entire PCD and connectivity between points

2. 2.

   the intelligence of the boundary segmentation method to account for boundary points and robustly process points in small regions.

Readers can refer to [Agapaki and Brilakis (2020b)] for details of the CLOI-Ins instance segmentation process.

The author generated the first dataset of class labelled point clusters of industrial facilities, CLOI, [Agapaki et al. (2019)] to validate Processes 1 and 2. CLOI consists of 10 classes that cover a wide range of industrial scenes (both indoor and outdoor). The TLS datasets of four laser scanned industrial facilities are used for the generation of CLOI as shown in Figure 3. One facility is a warehouse, one is a petrochemical plant, one an oil refinery and the fourth a processing unit. These facilities are anonymized since rights are reserved by AVEVA Group Plc. and British Petroleum. All datasets were obtained using static terrestrial laser scanners. This research provides the (to the best of our knowledge) hitherto largest collection of terrestrial laser scans of industrial facilities with point-level (a) class and (b) instance ground truth annotations. (a) refers to one of the ten CLOI classes and (b) is an index number that refers to a specific individual shape and is not further used in this work. In total, it consists of 12,497 shapes and 7.1 billion points with their class and instance labels for each point. To this end, this research provides CLOI, the largest annotated dataset based on already existing datasets [Agapaki and Brilakis (2020a)] and the only dataset of industrial environments that is captured with more than one sensors. This means that processing CLOI point cloud data is independent of the data capturing system that was used to generate the data. CLOI is also the only dataset available for processing PCDs of industrial environments. Detailed statistics and scanner specifications of the data can be found in [Agapaki et al. (2019)].

Two research platforms were developed for the framework validation; one capable of high computing for training deep neural networks and one for visualisations of large scale TLS industrial datasets. Training of the CLOI-NET-Class method was performed on Google Cloud instances. We implemented the deep learning class segmentation experiments on Tensorflow 2.0 as a proof of concept prototype and ran experiments on Google Cloud (Deep Learning VM image) with NVIDIA Tesla P100 GPUs. Visualizations of point clouds and segmentation results were implemented on the CLOI platform which is based on the Potree Viewer (http://potree.org/) in JavaScript. Potree is built upon ThreeJS and allows for rendering of large point clouds in a WebGL web browser [Schuetz (2016), Devaux et al. (2012)]. We created the user interface to select the TLS dataset of a CLOI facility, then segment the CLOI classes and validate with the ground truth class labels. The user can also select a point and only view the points associated with that CLOI class. Further details about the implementation of Process 1 and Process 2 can be found in [Agapaki and Brilakis (2020a)] and [Agapaki and Brilakis (2020b)] respectively.

The CLOI dataset was generated by manually annotating the four industrial facilities. The Ground Truth (GT) datasets are the desired outputs to compare against those generated by the proposed methodology and also used for training. The following GT datasets were created for the CLOI dataset validation.

GT class: A given industrial facility, TLS scanned, point cloud input is segmented into the eight CLOI classes. Each individual point was assigned a class point-wise label. Figure 3 shows each CLOI facility coloured with one of the eight class labels and the manual annotation time involved to generate the GT per facility. The number of shapes (instances), original number of 3D points and the area per facility are also provided. One can distinguish that even if a small facility area is scanned, the density of the scans may be so high that the number of points is much higher compared to a sparsely scanned facility. For instance, the oil refinery is only $300m^{2}$, making it the smallest facility of the dataset, but it has the largest number of surveyed 3D points.

GT instance: A given point cloud input is assigned to an individual instance point cluster.

GT boundary: A given point is classified as a “boundary” point if there is more than one instance in a neighbourhood of radius $4cm$ around it. The data structure used to define the neighbourhoods around each point is a kDTree.

The performance of the framework was evaluated based on:

1. 1.

   the performance of the CLOI-NET-Class segmentation network

2. 2.

   the performance of the CLOI-Instance segmentation network.

The inputs of the proposed framework are the class segmented clusters of Process 1. The class segmentation experiments showed average accuracy and mIoU of 79.8% and 44.65% when all the CLOI facilities are included for training except the one of interest to segment that is tested. CLOI-NET-Class has been proven to be consistent, reliable and without significant bias when tested on all the CLOI facilities. The author validated the theoretical active learning model as outlined in [Agapaki and Brilakis (2020a)]. Results showed that the total cost annotation function and the validation accuracy follow the theoretical model and the optimal data pre-annotation percentage that minimized the total annotation cost is between 20$\pm$10%. The CLOI-NET-Class performance following the active learning approach had on average 15% higher accuracy than the passive learning approach.

The performance of Process 2 (CLOI-Ins segmentation) was 73% mPrec and 71% mRec on all CLOI facilities using the ground truth class labels as inputs [Agapaki and Brilakis (2020b)]. For the evaluation of the framework, we compared the state-of-the-art instance segmentation networks (SGPN [Wang et al. (2018b), Wang et al. (2019)]), the BFS algorithm and the proposed CLOI framework in Table 1. The results illustrated in Table 1 show that SGPN has very low performance on the oil refinery data with the ASIS network performing better in all efficiency metrics. The oil refinery is used to compare the state-of-the-art deep learning instance segmentation networks, the BFS algorithm and the CLOI framework methodology. For the application of the BFS algorithm, the minimum instance size was selected for the predicted CLOI class point clusters based on performance. Therefore, the author conducted experiments to determine the minimum instance size based on the performance in terms of precision and recall on the CLOI datasets. The results in Figure 4 illustrate that the optimal trade-off between precision and recall is for minimum instance size 200 points instead of the minimum instance size of 20 points that was computed based on the ground truth class segmentation labels [Agapaki and Brilakis (2020b)]. This is attributed to noisy predicted class labels compared to the ground truth class labels used to evaluate Process 2 independently. There is an exception for the minimum instance size ($\mu$) and the minimum neighbourhood size ($\epsilon$) for the case of cylinders. The results indicate to set the instance size at 50 points and the minimum neighbourhood size ($\epsilon$) at 3cm (instead of 4cm) only for cylinder instance point clusters due to the observation that cylinders have higher class segmentation label predictions and the CLOI-Instance methodology benefits from that. We also observe in Table 1 a 10% increase in precision due to the class boundary constraint on the BFS algorithm for a minimum neighbourhood of 4cm.

The author then tested the performance of the same methods per CLOI shape in Table 2. We present these results for the oil refinery dataset as an example for comparison of the best performing existing instance segmentation methods and the proposed CLOI framework. The illustrated results in Table 1 and 2 demonstrate that the CLOI-Instance methodology clearly outperforms the current state-of-the-art research.

Another important note is that the CLOI framework results are calculated assuming that the users pre-annotate X% of the test facility with X% being the value from Table 3 depending on the facility. These percentages are based on the active learning curves of [Agapaki and Brilakis (2020a)].

Then, we present the precision and recall per CLOI class and the average precision and recall curves in Figure 5 as a reference. The results for the other three facilities are included in the Appendix. It is evident that for all datasets the recall metric of all the CLOI classes outperforms the precision metric for all the IoU threshold values. The greater difference between the mean precision and mean recall is for the oil refinery (Figure 5(c)), which is attributed to the high complexity of this dataset. This leads to reduced performance for all classes. Although the CLOI-Instance proposed methodology has promising results compared to the state-of-the-art methods for the instance segmentation task, the results demonstrate that the predicted class labels significantly reduce the precision and recall metrics compared to the same results presented given the ground truth class labels [Agapaki and Brilakis (2020b)].

The CLOI framework performance of cylinders is relatively high across the CLOI facilities given their high class segmentation performance [Agapaki and Brilakis (2020a)] for all the IoU threshold values. We remind the reader that the cylinder class segmentation performance was 81.25% precision, 81.75% recall and 68.25% IoU on average. There are though some cases where the cylinder instance point clusters are over- or under-segmented. These cases are the Cyl cases presented in [Agapaki and Brilakis (2020b)]. The results of the CLOI framework show an additional pain point. This is the uncertainty of the CLOI-NET-Class segmentation on predicting the class labels of the points. This leads to erroneous instance label predictions and mostly impacts the CLOI classes that have low class segmentation performance (the reader can refer to [Agapaki and Brilakis (2020a)] for a detailed discussion).

Another achievement of the CLOI framework is that it correctly segments sub-instances of an instance point cluster that has the “other” class label and even outperforms the manual instance segmentation in cases where a ground truth instance is under-segmented (Figure 6(a) and Figure 6(b)). This particularly applies for instances close to the floor or roof of a facility. The superior performance of the CLOI framework is attributed to the connectivity information that the BFS algorithm uses to segment instances. Another case where the CLOI framework outperforms the manual instance segmentation is for sequences of pipe components that have different radii. An example of that is Figure 6(c) where the CLOI framework correctly segmented the cylinder from a pump and a flange with steel rods.

We then recommend to use the 25% IoU threshold that gives slightly improved results (50% mPrec and 35.3% mRec for all the CLOI facilities). The CLOI shapes that have significantly higher metrics are those with higher class segmentation results as explained above. These are cylinders (53.6% mPrec and 44% mRec), elbows (66.8% mPrec) and I-beams (63% mPrec and 64.3% mRec).

One of the main goals of Process 2 was to prove that the CLOI-Instance method requires competitively less manual segmentation time compared to the current practice. We validated this hypothesis for the overall framework given that the class segmentation labels are predicted from the CLOI-NET-Class method (Process 1). We use the percentage of CLOI shapes that the CLOI-Instance method correctly predicts as a proxy to approximate the number of manual labour hours that are still needed in order to achieve an accurate gDT generation. The results are summarized for each CLOI dataset in Table 5. A comparison of the manual instance segmentation time for the CLOI benchmark dataset generation and the CLOI overall framework segmentation time is presented in Figure 7. The total number of man hours needed when deploying the overall framework is calculated as follows. The number of manually segmented CLOI shapes is computed as the product of the number of shapes that are missed by the framework ($1-recall$) and the average time it takes a modeller to manually segment a given shape. An assumption for the simplification of the calculation here that each CLOI shape takes the same time regardless of its complexity. The results illustrate that 35% of the manual labour hours are saved on average. The oil refinery dataset is one of the most complex CLOI datasets and this is reflected in reduced savings in labour hours for instance segmentation. It is noteworthy that for all CLOI facilities, the cylinder CLOI shapes have relatively low recall ($\approx 40\%$) which is attributed to the large number of conduit that are clustered together in one instance.

We evaluated in [Agapaki et al. (2018)] the state-of-the-art commercial software that semi-automatically segments cylinders from TLS industrial datasets, however a direct comparison cannot be made since the total number of cylinders considered in that evaluation does not match the number of cylinders in the CLOI dataset. However, the number of cylinders correctly detected by EdgeWise can be compared with the number of cylinders segmented by the proposed framework. The results in Table 6 demonstrate that the proposed framework correctly segments more cylinders than those detected by EdgeWise. The proposed framework is designed to better segment conduits and even with the discussed limitations, Table 6 illustrates its superiority to EdgeWise which is mostly in the correctly predicted conduits that EdgeWise does not identify.

The performance of the proposed framework is then compared directly with EdgeWise assuming that the modeling of CLOI shapes will be manually performed in EdgeWise. Therefore, the average modeling labour time per object is taken from [Agapaki et al. (2018)] and multiplied with the number of objects that are not automatically segmented. The output in labour hours in shown in Figure 8 and compared with the manual labour hours for the objects that EdgeWise cannot automatically detect (a fraction of cylinders and the rest of CLOI shapes). Figure 8 shows that 21% and 39% more time savings are achieved when the proposed framework is utilized for the warehouse and petrochemical plant respectively.

The warehouse and the petrochemical plant datasets are then used as a proxy to estimate the average percentage of labour hour reduction of the CLOI framework compared to EdgeWise per CLOI class. The average percentage per class is shown in Table 7. An assumption was made that the modeling time of all cylindrical shapes is the same, since our framework detects cylinders and not their sub-classes, i.e. pipes. Then, the CLOI framework is directly compared with EdgeWise for the petrochemical plant with 240,687 objects that was used for manual modeling in [Agapaki et al. (2018)]. The same assumptions are used here for consistency of the results. The results in Figure 9 reveal that 12 person-months are needed when using the CLOI framework instead of the 17 person months that are needed when using EdgeWise. In particular, CLOI saves 10% more man-hours for cylinder modeling, which is translated in 773 labour hours saved. Although there is still time required for manual cylinder extraction, the proposed framework clearly outperforms the commercial software EdgeWise.

Some or all data, models, or code used during the study were provided by a third party. Direct requests for these materials may be made to the provider as indicated in the Acknowledgements.

We thank our colleague Graham Miatt, who has provided insight, expertise and data that greatly assisted this research. We also express our gratitude to Bob Flint from BP International Centre for Business and Technology (ICBT), who provided data for evaluation. The research leading to these results has received funding from the Engineering and Physical Sciences Research Council (EPSRC) and the US National Academy of Engineering (NAE). AVEVA Group Plc. and BP International Centre for Business and Technology (ICBT) partially sponsor this research under grant agreements RG83104 and RG90532 respectively. We gratefully acknowledge the collaboration of all academic and industrial project partners. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the institutes mentioned above.