Translational Symmetry-Aware Facade Parsing for 3D Building Reconstruction

Most of conventional facade parsing methods relied on the predefined procedural grammars, which aim at finding the best parametric setting of a facade. It is easy for them to incorporate the architectural rules into the inference process while they are usually computationally demanding and tend to fail in cluttered environments [12, 13].

Zhao et al. [14] assume most of vertical lines belong to buildings, where the input image are parsed into building, sky and ground. Then, they further refine the extracted shape of each facade unit. Wendel et al. [15] and Recky et al. [16] try to find the repetitive patterns in facades to do the parsing. Koutsourakis et al. [17] build the models through a set of basic shapes with the parametric rules. A tree representation of variable depth and complexity is used to account for the elaborate and varying architectural styles. They employ Markov Random Field (MRF) to optimize the parameters of procedural grammars for a building facade. Ripperda and Brenner [18] also employ a tree representation, where the reversible jump Markov Chain Monte Carlo is used in the tree construction. Teboul et al. [19] formulate the shape grammars using reinforcement learning, where the promising results can be achieved if buildings conform to these grammars.

Mathias et al. [20] propose a three-layer approach to facade parsing. They hand-crafted the prior knowledge into each layer. With the first layer, a recursive neural network (RNN) is trained to label facades at super-pixel level. In the middle layer, they introduce the knowledge about distinct facade elements. Then, they combine the output of RNN with object detectors, and treat the merging procedure as a 2D Markov Random Field over the pixels. Cohen et al. [21] formulate the facade parsing as a sequential optimization problem instead of the usual classification task or grammar learning, where dynamic programming is used to solve the optimization problem.

There are a few research efforts having been devoted to tackling the facade parsing problem using deep learning techniques. DeepFacade [22, 3] suggest a reflective symmetric loss as a constraint to train the neural networks. In [22], Liu et al. propose a variance-based loss, where the centers of horizontal and vertical line segments should lie on the same straight line for symmetric objects. Later, they employ a two-stage object detector [3], in which the predicted bounding boxes are treated as the symmetry indicator. Schmitz and Mayer [23] train CNN on images patches to parse the facade. John Femiani et al. [24] propose three different network architectures to perform multi-label facade image segmentation, where each one has a unique feature.

Our approach is also related to image segmentation, scene parsing and object detection. Long et al. [25] are the first to train an end-to-end deep convolutional neural network for general image segmentation task. Chen et al. [26][27][28] employ the dilated convolution instead of plain convolution. This approach avoids the use of a deconvolution layer, and make the network easier to train. CRF post-processing [26] can be applied to refine the results. Encoder-decoder models [29], feature pyramids [30][31] and skip-connection [32] are also important techniques in semantic segmentation. Abdulnabi et al. [33] propose a multi-modal recurrent neural networks (RNNs) for indoor scene labeling. They train two RNNs at the same time, and connect the two networks through an information transfer layer. The recent object detection techniques can be grouped into two stage approach and single stage method. Two stage approaches [34, 35, 36] first employ a region proposal network to find the potential bounding boxes, and then predict their labels via a classifier. Single stage methods [7, 37] do not rely on the region proposal networks. [38] combine the object detection and semantic segmentation. Zhou et al. [5, 6] represent the objects as points with the extendable attributes. For example, they represent bounding boxes as the width and height from the center point, or the corner location of a bounding box.

Usually, the 3D buildings are represented by either the point cloud or procedural models. The former relies on the geometric information while the latter focuses on the semantics.

A mature pipeline for reconstructing 3D urban scenes from images is generally based on structure-from-motion techniques [39][40][41][42]. Their output are typically point clouds or meshes without the labels like windows and doors, however, the semantically reconstructed 3D models are needed in real-world applications. Martinovic et al. [12] employ CRF to predict the labels of point clouds and meshes in order to semantically classify the facade elements.

Muller et al. [11, 9] propose a high quality procedural modeling method, which is able to generate the high quality 3D building models with the predefined shape grammars. However, the parsing capability of conventional approaches is limited comparing to the deep learning-based methods. Nishida et al. [4] propose an interactive tool to extract the building grammar from a single image, where the user needs to mark out the silhouette. Convolutional neural networks are used to extract different components of a facade from large-scale building mass to fine-scale windows and doors geometry. Their method generates visually satisfied results, which mainly depends on a window detector with the predefined shape components allowing some deviations. In this paper, our presented method serves as a general facade parser that aims to recovering the accurate layout of building facades.

The conventional two-stage methods [34] first compute potential regions using a region proposal network, and then extract the features from neural network. Moreover, each region proposal is further classified. Therefore, two-stage methods essentially formulate the detection as a region proposal classification problem.

In contrast to the two stage scheme [22, 3], we propose a novel anchor-free neural network architecture by fusing the semantic segmentation and detector within the same framework. Moreover, anchor-free approach is more like semantic segmentation rather than two-stage proposal classification, which can not only reduce the computational cost but also enjoys the merits of better convergence.

In this paper, we choose FRRN [43] as the backbone network, since it does not require to be pre-trained on a large dataset like ImageNet [44]. Thus, we can learn the neural network model from scratch solely using the facade dataset while still obtaining the promising results. The FRRN is a typical encoder-decoder architecture network, where the encoder part downscales the feature map 16 times with respect to the original image $I\in R^{W\times H\times 3}$. Moreover, the consecutive transposed convolution is employed to upsample the feature map.

Our network has two branches, as shown in Fig. 2. One branch is for semantic segmentation, and another is for anchor-free detection. We aim to predicting a semantic segmentation response map with the keypoint detection map. The former is a full sized map as the original image while the latter is down-scaled with a stride of 4. Thus, the training loss is made of five different terms as follows:

where $L_{ce}$ is the segmentation loss, $L_{det}$ is the focal loss for detection, $L_{wh}$ is the size loss, $L_{off}$ is the local offset loss, and $L_{corner}$ is the corner position loss. $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are the weights for each part of the loss, respectively. Since all the losses are normalized in our implementation, we set $\lambda_{1}=\lambda_{2}=\lambda_{3}=\lambda_{4}=1$, empirically.

For the semantic segmentation branch, we use the cross-entropy loss function [25]:

where $\mathbf{x}$ is the input image array, and $\mathbf{y}$ is the probability distribution of the category label of the image. $M$ is the number of classes of the dataset, and $N$ is the number of pixels in the image. $i$ is pixel index. $y_{i,c}$ is the true probability distribution, and $p_{i,c}$ is the prediction.

The semantic segmentation branch is an encoder-decoder architecture that restores the predicted heatmap to the original image size, instead of downsampling by a factor of 4 as the detection branch. This is because semantic segmentation requires a finer heatmap in order to obtain the better results. If the predicted heatmap is downsampled, then the enlarged final results will be not accurate enough.

Anchor-free methods [5] detect objects by keypoint map, where each pixel represents the potential of being the center for an object. This is more like a segmentation response map rather than a classification task setting. Therefore, it is natural to fuse segmentation with anchor-free object detection.

For the detection branch, we employ the base structure of CenterNet [5]. We represent each object in a facade as the center point, whose size is an attribute of each point. Besides predicting the size, we employ an auxiliary corner point detection in the training process. The output stride downsamples the output prediction by a factor $R=4$. Such scheme is able to reduce computational cost while retaining the high accuracy.

The training objective is a focal loss [30]:

where $N$ is the number of instances in image. $\hat{Y}_{ij}\in[0,1]$ denote the predicted probability for the position $(i,j)$. $\alpha$ and $\beta$ are the hyper-parameters.

The object size is predicted by estimating the width and height of the bounding box. $(x_{1},y_{1},x_{2},y_{2})$ is denoted as the bounding box of an object with the size of $(w,h)$, where $w=x_{2}-x_{1}$ and $h=y_{2}-y_{1}$. Therefore, we employ an L1 regressor to predict the size for each object:

During the inference, discretization error usually occurs due to the offset drifting. To tackle this issue, we predict a local offset $\hat{O}\in\mathcal{R}^{\frac{W}{R}\times\frac{H}{R}\times 2}$ for each center point with the following loss function:

where $p\in\mathcal{R}^{2}$ denotes the center point of ground truth, and $\tilde{p}$ is computed with $\lfloor\frac{p}{R}\rfloor$.

We can add corner points as the additional attribute of an object. Although we only need to predict the location and size of the window in a parametric setting, estimating the corner location still greatly improves the training process. Such scheme provides the extra supervision information for the network during training, which leads to better convergence and promising prediction results. Moreover, corners are not needed in the inference stage, which do not incur the extra computational cost in prediction.

As the previous offset head [5], we only need to add a few channels to represent each corner of the object, where four corners are estimated for each subject with a feature map $\hat{C}\in\mathcal{R}^{\frac{W}{R}\times\frac{H}{R}\times 8}$. Each point takes two channels, and there are four corners for each object center point. Thus, we have 8 additional channels in total. Then, the corner loss can be derived as follows:

where $\tilde{q}=\lfloor\frac{q}{R}\rfloor$. The loss only occurs at each corner $q$, and other locations are ignored in our implementation.

In general, translational symmetry exists almost in every facade of modern urban buildings. This means that the property of an object stays invariant under translation. For a building facade, the same kind of objects are with the similar appearance and size. In particular, objects of the same category share the same shape and size, whose centers usually fall on a straight line. Typically, translational symmetry in facades have a direction of either horizontal or vertical. A facade is translational symmetric in horizontal means that the elements are same in the horizontal direction. It is also true for the vertical direction. Some facades are translationally symmetric in both directions, while others are only symmetric in one direction. This makes it necessary for us to decide in which direction the building is symmetric. Thus, we need to know how symmetric a facade is.

We define a quantitative metric to evaluate the symmetry of a facade. For building facades, translational symmetry relies on two aspects. Firstly, the objects are translational symmetric, whose center coordinates have equal distances in the symmetric orientation and share the same value in the orthogonal direction. Secondly, the sizes are invariant in the symmetric direction. Based on these assumptions, the translational symmetry metric $T$ is composed of two terms:

where $T_{c}$ represents the score of how symmetric the centers are, and $T_{s}$ is the score of how symmetric the sizes are. Since $T$ has an orientation, we denote $T^{h}$ as the measure for horizontal direction and $T^{v}$ for vertical one.

We take horizontal translational symmetry as an example, which should satisfy two conditions: 1) their vertical coordinates need to be as close as possible; 2) the distances between horizontal coordinates should be the same. Therefore, we derive the following formulation for $T_{c}$:

For $T_{s}$, all objects may have the same size. Thus, the bigger the variance of sizes, the less the symmetry is. We formulate $T_{s}$ as below:

where $\mathbf{s}_{i}=(w_{i},h_{i})$ represents the width and height of object $i$, and $N$ is the number of objects.

Based on the above formulation, we find that the lower $T$ leads to the higher degree of translational symmetry. We aim at solving the following minimization problem:

where $\mathbf{x}=\{x_{1},x_{2},\dots,x_{N}\}$ are the horizontal coordinates of all the objects, $\mathbf{y}=\{y_{1},y_{2},\dots,y_{N}\}$ denote all the vertical coordinates, and $\mathbf{S}=\{\mathbf{s}_{1},\mathbf{s}_{2},\dots,\mathbf{s}_{N}\}$ represents all the sizes. Our goal is to find a configuration to minimize $T$.

The straightforward solution is to set $x_{i}=\hat{x},y_{i}=\hat{y},\mathbf{s}_{i}=\hat{\mathbf{s}}$, however, which has some issues. As the facade may not always be symmetry in both directions, we have to estimate the symmetric direction first. Assuming that a facade has symmetry in one direction and use it as the refinement target. If $T^{h}<T^{v}$, we apply the horizontal refinement. Otherwise, we refine the vertical direction. Another problem is that the windows may have slightly different sizes even in the symmetric direction. Practically, we introduce a weighted sum to allow one object to keep the size of its initially predicted result.

where $\tilde{T}=\sigma(T)\in[0,1]$ is the result of $T$ being scaled to $[0,1]$, and $\sigma$ is the sigmoid function. $x_{new}$ is the new center locations, and $\mathbf{s}_{new}$ are sizes of objects after refinement. $x_{i}$ and $\mathbf{s}_{i}$ are the corresponding values before refinement, $\hat{x}$ and $\hat{\mathbf{s}}$ are the mean.

To facilitate the fair comparison against the state-of-the-art methods, we evaluate our presented method on three typical facade datasets, including the Ecole Central Paris (ECP) dataset [46], the RueMonge dataset[12], and the ArtDeco dataset [47].

The ECP dataset is a popular 2D facade parsing testbed with a total number of 104 annotated building images. All annotations are pixelwise-labeled semantic segmentation maps. This dataset has 8 classes, including window, wall, balcony, door, shop, sky, chimney and roof. All the images in ECP dataset contain the rectified and cropped facades of Haussmannian style buildings in Paris. The original annotation labels the images using a Haussmannian-style grammar. This often results in the imprecise or even wrong annotations. Therefore, we employ the labels provided by [1], where the annotations can better fit the ground truth. Some methods [23] only focus on windows and walls, which do not use full annotations. In this paper, we make full use of all the labels.

The Ruemonge dataset has 428 facade images in total, where only a portion of them are annotated. The training set has 113 images, and the testing split contains 202 images. For a single building, it may have several pictures from slightly different perspective. This is because the images are taken from a camera moving down a street. Thus, overlapping occurs frequently. Differently from ECP dataset, Ruemonge dataset has 3D point clouds and mesh annotations. Previous method [12] has explored both 2D and 3D information, where the 3D method runs much faster than the 2D approach with the higher accuracy. Our method is a pure 2D approach.

The ArtDeco dataset consists of 80 rectified facade images with similar style. In contrast to other datasets, some of images have occlusions due to vegetation. This makes it ideal for testing the inference of architectural elements in the presence of large occlusions, as it provides the hand-annotated ground truth for the labels behind vegetation.

We train our network from scratch on each dataset. Our proposed framework has single neural network with multiple heads. Such scheme is much easier to train comparing against DeepFacade [3], where they have to train two networks simultaneously. Usually, our training process is able to converge within 140 epochs. We conducted our experiments on a PC with 4-GPU at a initial learning rate of 2.5e-4.

As the facade datasets only provide the semantic segmentation labeling, i.e., heatmap for the images, we have to generate object detection annotations from these labels. We employ the automated labeling method in order to avoid introducing new supervision in the experiments, which may not be fair to other methods for comparisons.

We manage to generate the bounding boxes for windows, balconies and doors by extracting each facade object instance from the semantic segmentation map. We find that all pixels in the same object are connected, and all the different connected regions are objects. Thus, we compute the convex hull of the object region, and then find the minimal bounding box containing this convex hull. Therefore, we can easily generate the bounding boxes for each facade objects from the semantic segmentation map.

For the four corner locations of each object, we find them through a sorting algorithm. If we set the top left corner of the entire image as the origin, then the top left corner of object is the point having the least distance to the origin in an object. Similarly, the other three points can be found by setting the origin to the top-right, bottom-right and bottom-left corners of the image. Thus, we can directly extract the corner locations from the segmentation map without resorting to the manual labeling.

We conduct the quantitative experiments on three datasets to demonstrate the efficacy of our method.

The proposed neural network architecture consists of several components. Moreover, we are interested in examining the efficacy of translational symmetry refinement. To this end, we conduct a series of ablation study to analyze how each part will affect the final results.

Our network consists of two branches, namely semantic segmentation and object detection. It is interested to study how each head of our network affects the accuracy of final results. We conducted the ablation study on the RueMonge dataset. Specifically, we train our network with three different settings, semantic segmentation head only, detection head only and both heads simultaneously.

Table V shows the IOUs scores for each setting. The ‘Segmentation’ column is the result obtained by training with the semantic segmentation head only. The ‘Detection’ column is the result training with only anchor-free detection results. The ‘Fusion’ column is the full network training with both heads. Since the detectors are only trained with the classes of window, balcony and door, we only report the accuracy for these three classes in the ‘Detection’ column.

It can be seen that semantic segmentation alone can not yield the optimal results on the RueMonge dataset. The IOUs of the window, door and balcony are slightly lower than the results of training with the detection head. When training them together, the fusion method achieves the best accuracy. This demonstrates the efficacy of our method, and the anchor-free detection head indeed improves the accuracy.

To study the effectiveness of translational symmetry-based refinement, we conduct ablation study on the ECP dataset with two different settings. One directly outputs of the neural network while another refines the inference results. Table IV shows that the refinement module generally increase the IOU score at around 2 percent. This demonstrates the effectiveness of our proposed method.

In the following, we show the qualitative results of our method on different testbeds.

Fig. 4 shows the visual results of our presented method on the ArtDeco dataset, which consists of occlusion caused by vegetation. As in the upper-left group, there are trees in front of buildings. It can be observed that our method has predicted the rough position of windows behind the trees. Although the guess is not perfectly accurate, it is an acceptable prediction. After the translational symmetry refinement, the overall visual result has improved significantly. Occlusions may affect the translational symmetric score, and the refinement process tends to keep the original size of the objects. In case of no occlusions, the sizes are more evenly distributed than the occluded cases.

Fig. 5 shows the visual results on the ECP dataset. It can be seen that the sizes of windows, balconies are more unified than before the refinement, which is more visual pleasant and closer to the ground truth. As the facades in this dataset are more translational symmetric than the ArtDeco, we can obtain better results.

The above experimental results demonstrate that our method not only obtains the high parsing accuracy, but also show the promising visual predictions.

Table VI shows the testing time for different methods, where each entry is the total time of predicting all images in the testing dataset. Martinovic et al. [12] require the highest computational cost, since it involves the complex CRF inference stage. Their 2D method costs 379 minutes for testing, and the 2D + 3D approach needs 470 minutes. Deep learning-based methods are relatively fast in inference. DeepFacade [3] processes the whole testing set within 30s while our presented method only needs 17s. This is because our anchor-free framework just computes the feed-forward network pass once rather than twice for the two stage method in DeepFacade.

As for the training time, DeepFacade [3] has to train two different kinds of networks while our proposed approach only needs to train a single network. This means that our network costs half the time of DeepFacade during training if both methods use the same backbone network. In practice, the training of our presented neural network can be done within two hours due to the small scale of the dataset.

The above experiment indicates that our proposed approach is highly efficient.