Symbol Spotting on Digital Architectural Floor Plans Using a Deep Learning-based Framework

This paper proposes a DL-based framework for spotting symbols in digital real-world architectural floor plans. Our contributions are two-fold.

1. 1.

   We leverage recent advances in DL by adapting a YOLO-based [4] object detection network to the problem of symbol spotting in real-world architectural floor plans. We propose a training strategy based on tiles, which allows us to circumvent many issues particular to DL object detection networks, including the size of the objects of interest relative to the size of the images, aspect ratios, and data augmentation.

2. 2.

   Our proposed DL-based symbol spotting framework successfully addresses the main issues of traditional on-the-fly symbol spotting, namely graphical notation variability, occlusion and clutter, and variable graphical complexity of symbols.

The remainder of the paper is structured as follows: Sec. 2 reviews related works, Sec. 3 details our symbol spotting approach, Sec. 4 discusses experimental results, and Sec. 5 presents concluding remarks.

From a library of proprietary digital architectural drawings, designed by 10 architectural firms, we selected 115 different units showing various levels of difficulty in terms of density of visual information, occlusion, and clutter. Architects typically share floor plans in the PDF format. We converted the PDFs into 150 DPI images, and annotated 12 architectural symbol classes, such as bathroom sinks, windows, and entry doors (see Fig. 4). We do not make the dataset of real-world architectural plans public due to intellectual property issues, but are working towards securing the necessary permissions for a future release.

We face several problems when dealing with architectural floor plan images in the context of DL systems. First, the average floor plan size is $5400\times 3600$ pixels, whereas individual symbols are very small (e.g. $70\times 80$ pixels for a bathroom sink). As a result, symbols tend to disappear in the output feature map of CNNs, preventing them from being detected. In addition, floor plan images have diverse aspect ratios and resizing them to a fixed size, as required by CNN architectures, dramatically changes the symbol morphology and thus decreases the classification performance.

We propose a tiling strategy to tackle the above problems, which uses a scale parameter $\alpha$ and stride size $S$. First, all the $[\alpha M\times\alpha M]$ overlapping tiles that have a starting point at least $S$ pixels apart are extracted from the floor plan images. $[M\times M]$ is the required input size of the utilized CNN, which is usually less than $[256\times 256]$ [35]. Tiles that do not encompass at least one complete symbol are automatically discarded from the training dataset. The tile size must be selected so that tiles are larger than symbols. Also, selecting larger tiles can boost their contextual information, as in architectural plans, the occurrences of some symbols might be spatially correlated. For instance, we can expect to see a bathroom sink symbol in the vicinity of a toilet symbol. At the same time, the tiles must be small enough so that the symbols still appear in the deeper layers and output of the CNNs. Tiles are also useful for data augmentation. Fig. 5 shows neighbouring tiles containing a range symbol captured at different locations within the tiles. The tile size in the figure is $[224\times 224]$ (i.e. $\alpha=1$ and $M=224$, required by ResNet50 [41]) and $S=50$ pixels.

Single shot object detection architectures based on image grids (such as YOLO) seem appropriate and accurate enough to localize architectural symbol boundaries, compared to more complex and heavier two-stage classification architectures (such as Faster R-CNN [37]), due to the following considerations. Floor plan images differ from natural scene images (for which most CNN-based object detection systems were developed) on several aspects. Floor plans are typically grey-level with a small number of possible symbol classes, compared to colourful natural scene images with a large number of possible object classes. Additionally, floor plans have a simpler background and chances of overlap between symbols is low (this does not apply of course to other parts of the image such as textual information and measurements, which may have a considerable overlap with symbols). Here, we use YOLOv2 [4] as a single shot object detection architecture. We selected YOLOv2 instead of YOLO [35] because of its higher localization accuracy and recall performance. The improvements of YOLOv3 [36] consist mainly in a prediction across three scales and a better feature extraction network, but at the cost of a slower and heavier system. As architectural symbols have similar sizes and simpler structures compared to objects in natural scenes, YOLOv3 cannot offer a noticeable improvement over YOLOv2. In YOLOv2, the input image is divided into non-overlapping grids. Each grid can only predict one object. A backbone CNN (e.g. Darknet19) extracts features and for each grid, a set of prior anchors are evaluated based on a loss function which penalizes localization, confidence and classification errors.

In the training phase, we use the tile dataset described in Sec. 3.1) to train the network. In the inference phase, the input image is first broken down into tiles. Each tile is then passed through the network and symbols are detected. Fig. 3 shows the inference process. The detected symbols in the overlapping tiles are shown in the bottom-right image. As a given instance of a symbol typically appears in several tiles, it is detected multiple times. To refine and concatenate the results, we perform a non-maximum suppression step as follows. For overlapping detections, we compare all pairs of bounding boxes. If their overlap is larger than a threshold (a percentage of the size of the smaller bounding box), the bounding box with the highest classification score is retained. In cases of close scores, the larger bounding box is selected and the smaller one is removed. The bottom-left image in Fig. 3 shows the final results for a $10\%$ threshold.

From the 115 units of the dataset (see Sec. 3.1), we used 90 units for extracting tiles. The remaining 25 units are used as a test set for evaluating the framework on entire floor plans. Given $S=50$ and a tile size of $[227\times 227]$, the 90 units generated 4707 tiles containing at least one complete symbol. We randomly selected 80% of those 4707 tiles for training the network, with the remaining 20% tiles used for validation. During training, we employed the Adam optimizer [42] with a mini-batch size of 30, a fixed learning rate of $10^{-4}$, and data augmentation with horizontal and vertical flipping and rotation and scale changes. Moreover, 10 prior anchors were calculated from the size of the symbols. We experimented with three different backbones, the original Darknet19 [4], as well as ResNet50 [41] and Xception [43].

Table 1 (first three rows) shows the performance on the tile validation set, whereas Table 2 shows the performance per symbol class and the global performance for the test set of 25 entire floor plans. In the tables, $mAP$, $AP_{50}$ and $AP_{75}$ represent the mean average precision and the average precision for IoUs equal to 50% and 75%, respectively. The IoU (Intersection over Union) is obtained as follows:

where $A$ and $B$ are the bounding boxes of the detected symbol and the ground truth symbol. From Table 1, we can see that the ResNet50 backbone significantly outperforms Darknet19 and Xception, with Xception having the lowest performance. Looking specifically at the $AP_{50}$ metric, as an IoU of 50% is acceptable in symbol spotting, the average precision is very high. From Table 2, again focusing on $AP_{50}$, we can see that our method performs strongly for most symbols, with some yielding 100% precision. The lowest score is obtained for the window symbol, which is a particularly difficult case due to its triviality and varying aspect ratio. Incorporating contextual information on walls could help improve the window detection results.

Fig. 6 shows examples of detected entry doors, using the ResNet50 backbone, with the bounding boxes and detection scores (max = 1) highlighted. This figure showcases the efficiency of our DL-based symbol spotting system compared to the traditional methods. Our system successfully addressed occlusion and boundary degradation, which can highly affect the raster-to-vector conversion and thus the structural representation of symbols in methods such as [25, 23, 12], rotation, which is one of the weaknesses of the pixel-based methods such as [7, 44], and graphical notation variability. Furthermore, as entry doors have a limited number of primitives, some of them cannot survive the vectorization step required by vector-based methods. They also include very small closed regions that can easily make the symbol unrecognizable by methods that employ closed regions as constituent primitives, such as [21, 19, 27].

Fig. 7 shows symbol spotting results for four units with different designs and layouts, using the ResNet50 backbone. Qualitatively speaking, the results are excellent, and we see that our approach works well even in the presence of high levels of noise, occlusion and image degradation. Considering the varied sources of the plans, we can also conclude that our method successfully bridges the semantic gap related to intra-class graphical notation variability.

We also provide an evaluation on the public synthetic images of the Systems Evaluation SYnthetic Documents (SESYD)¹¹1http://mathieu.delalandre.free.fr/projects/sesyd/ dataset, which is the standard dataset in the field. Its synthetic floor plan collection includes 1000 floor plan images (some of which have very similar unit layouts), containing up to 16 different query symbol classes, with only one graphical notation per symbol class. For training the system, we randomly picked 50 floor plan images and extracted the tile images. Since images are large and the floor plans are sparser than real-world floor plans, we used $[680\times 680]$ tiles with $S=100$ to include more contextual information around each symbol. This yielded 11,753 images divided into subsets of 9402 and 2351 tiles for training and validation purposes, respectively. To test our system on entire images, we used the selection from the GREC Symbol Recognition and Spotting contest [45]. This contest set contains 20 images from the original dataset of 1000 images (ideal) and three degraded versions (60 images). Noise levels #1 to #3 in [45] simulate thinner and thicker lines than the original image lines, and add global noise to the image, respectively. All of our results on SESYD are obtained with the ResNet50 backbone, as it yields a better performance on the real-world dataset.

Table 1 (last row) shows the results on the validation tile set, and Table 3 shows the performance per symbol class and the global performance for the test set of 80 entire floor plans. Looking at $AP_{50}$ in both tables, our framework yields a very high precision rate, with 100.00% for many of the symbol classes. Again, the window classes (window1 and window2) are the most problematic ones, and would benefit from additional contextual information.

Table 4 compares our results with other published symbols spotting approaches. In this context, the evaluation metrics in the literature differ from the ones commonly used for assessing object detection networks, and are computed instance-wise and pixel-wise. For the instance-wise metrics, detected symbols that have some overlap with the ground truth are all counted as positive detections, and precision, recall and F-score values are calculated accordingly. Pixel-wise metrics, based on relevant and non-relevant retrieved pixels, refine the localization assessment [47]. In Table 4, $P$, $R$ and $F$ stand for precision, recall and F-score, respectively. The ‘Queries’ column indicates how many of the 16 symbols in the dataset are employed in the evaluation. The winning method in [45] was [46]. Our method significantly outperforms all other methods; the one method with comparable performance ($5^{th}$ row) was evaluated on a limited subset of the symbols only.

Fig. 8 shows examples of spotted symbols on a SESYD layout for the noise #3 degraded version. All symbol instances are correctly detected except for one (bottom right). Although SESYD does not include graphical notation variability, occlusion nor clutter as the real-world dataset does, it does allow us to conclude that our approach is able to deal with various degradation and noise levels.