GlobalMapper: Arbitrary-Shaped Urban Layout Generation

A urban layout is defined by a network of roads that forms city blocks and within each city block a set of buildings. In Fig. 3, each city block $B$ is represented by a graph $G$ of the multiple buildings within the block and by a set of block shape features $m$.

Buildings. We represent the building layout of an arbitrary city block $B$ using a stubby grid graph $G$ (e.g., the graph has many more columns than rows and each vertex typically has 2 or 3 incident edges). We define the graph as $G=\{V,E\}$, where $V$ is a set of node vertices $V=\{v_{i}\}$, $i\in[1,N]$ that each vertex $v_{i}$ describes each building, and $E$ is a set of edges $E=\{e_{ij}\}$, $i,j\in[1,N]$ between spatially adjacent nodes/buildings. $N$ is defined as the maximum building count in one block. We empirically set $N=120$ to handle $>$99% percent of real city blocks in our dataset.

Each vertex $v_{i}$ stores a building’s geometric and semantic feature information. In particular, the per-vertex (or building) features are the following:

• •

  $e_{i}$ is the binary existence flag of this building (e.g., a value of one implies the building exists, else zero).

• •

  $(x_{i},y_{i})$ is the physical position of building center.

• •

  $(w_{i},h_{i})$ is the width and height of the building’s oriented 2D bounding box (the width dimension is parallel to the main axis),

• •

  $s_{i}$ is an integer representing the building shape type (described below), and

• •

  $a_{i}$ is the occupancy ratio of the building area to the oriented 2D bounding box area.

The aforementioned features $\{s_{i},a_{i}\}$ represent each building by a shape type and an occupancy ratio. In a preliminary analysis, we found that buildings in our test areas can be fit with IoU=$95.78\%$ and Hausdorff Distance $=1.36m$ (i.e., the average of maximum distance between the contour of the actual building and our best-fit polygonal building shape is $1.36m.$) using one of four fundamental parameterized shape types: Rectanglar, L-shape, U-shape and X-shape. During the preprocessing phase, we use Powell optimization to determine the best-fitting building shape type and calculate an occupancy ratio (see Supplemental for more details). Both variables enable our later-described synthesis step (Sec. 3.3) to generate a synthetic building footprint.

Blocks. The block shape features $m$ are composed of a binary mask $k$ and the mask scale $l$. We typically use a mask resolution of 64x64 pixels. The longest side of each block’s oriented bounding box is rotated to horizontal and scaled to fit within the mask (see Fig. 3).

In this section, we first describe the spatial transform of an arbitrary urban layout into a canonical form, and then describe the shape encoder setup which enables using block shapes as a condition of the decoder setup (Fig. 2). We also discuss our variational model structure and training scheme.

Canonical Spatial Transform. In order to support a wide variety of urban layouts at arbitrary scales, we introduce a transformation parameterized by block shape feature $m$. It maps each building graph $G$ to a canonical representation $G^{\prime}$. Prior urban procedural modeling work [54] indicates that urban layouts follow typical conventions which we exploit to define the canonical representation. In particular, using the contour of block mask $k$ we compute a block’s 2D skeleton and modify its endpoints to obtain an extended skeleton as a compact polyline structure forming a main axis (Blue line in Fig. 3). Then, for each building in $G$ we compute its normalized position ($x^{\prime}_{i},y^{\prime}_{i}$ in Fig. 3) relative to one of the main axis polyline segments, and to one of the minor axis perpendicular to a main axis line segment. The length of main axis and minor axis is utilized to normalize building’s position and size, ultimately producing the canonical $G^{\prime}$. This spatial transform provides a regularized spatial representation of building layout regardless of block shape as compared to direct scaling by most former relative methods [62, 26, 17, 2]. Additional details are in Supplemental.

Shape Encoder. To obtain a latent representation of block shape feature $m$, a CNN-based autoencoder is trained to reconstruct the binary mask $k$. The block scale $l$ is stacked with the binary mask as the second channel of the input image. The encoder model has 4 layers of convolutional layers and the corresponding decoder model is the mirror image of this. The model is able to reconstruct a binary mask with IoU $>$ 0.98. During our variational training, the latent representation $m^{\prime}$ encoded by the shape encoder can be utilized as a conditional prior of the decoder.

Conditional Variational Training. As a conditional VAE, our model aims to represent the conditional data distribution of $p(G^{\prime}|m^{\prime})$, which is intractable to compute. By variational learning, the model approximates the distribution by maximizing its evidence lower bound (ELBO). The objective function is as follows:

where the approximate posterior $q_{\phi}(z|G^{\prime},m^{\prime})$ is parameterized by $\phi$, and the decoder $p_{\theta}(G^{\prime}|z,m^{\prime})$ is a deep neural network parameterized by $\theta$. The prior distribution $p(z)$ is a standard Gaussian distribution in our training. Particularly, our model should be able to encode and reconstruct $G^{\prime}$ conditioned by $m^{\prime}$.

Graph Attention. We use a graph attention network [57] (GAT) as the backbone of our encoder and decoder. Graph attention networks perform weighted multi-layer messaging passing between connected nodes. In particular, the edges of our 2D grid graph topology enable message passing between buildings next-to and in-back-of other buildings. We denote node features in the canonical graph as $f_{i}$. Our formulation can be represented by:

We collect all individual node feature vectors into feature matrix $F^{t}=\{f_{i}^{t}\},i\in[1,N],t\in[1,T]$ and apply equation Eq. 2 a total of $T$ times. Then, the feature matrices from all $T$ iterations are aggregated as $[F^{t},F^{1},....,F^{T}]$ to form a 512 dimensional normal distribution that is sampled by the $z$ of a variational reparameterization.

The decoder $p_{\theta}(G^{\prime}|z,m^{\prime})$ is conditioned by $m^{\prime}$. As Fig. 2 indicates, we concatenate $m^{\prime}$ with $z$ as the input of our decoder. Initially, a sampled latent space vector $z$ is pushed through a MLP to produce an initial feature matrix $\hat{F^{0}}=\{\hat{f}_{i}^{0}\},i\in[1,N]$, which consists of single feature vector for each node vertex/building. Then, a similar multi-layer messaging passing, using Eq. 2, is performed $T$ times yielding $\hat{F}^{T}$. Finally, each vector component $\hat{f}_{i}^{T}$ within $\hat{F}^{T}$ is decoded by independent MLPs for predictions of each building feature element in $\hat{G^{\prime}}$ (e.g., $\{e_{i},x_{i},y_{i},w_{i},h_{i},s_{i},a_{i}\}$). Predicted features are calculated by reconstruction loss. The weighted sum of each loss component is back-propagated to graph attention networks for weight updating.

Inverse Spatial Transform. With block shape feature $m$, the spatial information of $\hat{G^{\prime}}$ (e.g., building position, width, and height) is inversely transformed back to its original scale and location.

In this phase, we generate building shapes within each generated city block. Our goal is to produce building footprints that are similar (but not necessarily identical) to the original building coverages, positions, and shapes. Further as shown in Sec. 4.4, we can produce such an output using only a small fraction of the real-world data.

Our method synthesizes the aforementioned four building shape types (Rectangular, L-shape, U-shape, X-shape) using a parameterized function (see Supplemental for details). For the buildings of a city block, the shape parameters of each generated building type $s_{i}$ are randomly perturbed within a predefined range until a configuration best satisfying the generated occupancy value $a_{i}$ is found within a maximum number of iterations. Then, the generated building footprint is rotated so that its width dimension is again parallel to the block’s main axis.

Datasets. We collected datasets from 28 large cities across North America (Chicago, D.C., New York City, Atlanta, Dallas, Los Angeles, Miami, Seattle, Boston, Providence, Baltimore, San Diego, San Francisco, Portland, Milwaukee, Minneapolis, Austin, New Orleans, Phoenix, Denver, Toronto, Vancouver, Pittsburgh, Tampa, San Jose, Norfolk, Austin, and Houston). The total number of city blocks is 119,236, containing 2,513,697 buildings, which is 187-times bigger than the training set of [62]. We split the data 80% for training, and 20% for validation. We obtained the datasets by downloading 2D layouts from OpenStreetMap (OSM) [42]. Minimal clean-up was done to simplify parallel multi-lane roads to single edges and to remove self-looping edges.

Training Details. We performed an end-to-end training of our pipeline. After various experiments, we set learning rate $lr=0.001$, and each training typically converges in $10^{5}$ iterations (9 hours on a single NVIDIA A5000 GPU). We determined the best configuration of loss functions to be using $L2$ loss for all geometry features $\{x_{i},y_{i},w_{i},h_{i}\}$, and cross entropy loss for categorized features $\{e_{i},s_{i}\}$, and KL-divergence loss for $z$. The relative loss weight for geometry features, categorized features, and KL-divergence loss is 4.0, 1.0, 0.5, respectively. We found additional regularization losses (i.e., a loss penalizing overlaps [62]) do not further reduce errors. The average inference time is 5.81ms per city block by a single A5000 GPU.

We compared our approach to 4 related layout generation methods: LayoutVAE [26], Gupta et al. [17], VTN [2], and BlockPlanner [62]. For first two methods, we utilized the codes provided by the author’s repository of  [17]. We provided the correct bounding box count for LayoutVAE and only retrained its BBoxVAE portion to predict bounding box geometry. For [17], we transferred our dataset to COCO format and retrained the network using default settings in repository. During generation, we setup their top-k sampling as k=5 to produce diverse layout outputs. For VTN [2], we modified the codes of  [17] by adding a variational training framework. For BlockPlanner [62], we only received partial model structure source code from the authors. We re-implemented the rest of the method to the best of our ability, and retrained it on our dataset by the suggested settings in their paper. Additionally, comparisons to diffusion models [25, 20] are provided in Supplemental.

Quantitative Results. We generated 1000 urban layouts by ours and by each existing method, and compared to the same amount of real urban layouts. Our method is conditioned on the road networks from real urban layouts. The LayoutVAE is conditioned on the building count we provided. VTN is conditioned on the corresponding real building layouts. Gupta et al. [17] generation is fully random. The set-to-set comparison is evaluated by 6 quantitative metrics. Overlap index [31] is the percentage of total overlapping area among generated building layouts within the urban block. The Out-Block index is the percentage of generated building layout area that is out of the urban block contour. Wasserstein distance (WD) indicates the similarity between two distributions. Specifically, we computed the WD between the distributions of building counts and bounding box geometry (location and size, centered around the origin). Lower score indicates generated urban layouts provide better similarity to the real world distribution of building counts and geometry arrangement. We also computed FID score [23] to evaluate diversity and quality of the generated urban layouts.

Inspired by DocSim index [46], we design a similarity metric LayoutSim (L-Sim) to evaluate the geometry similarity between pairs of urban layouts. We first rotate each of the two urban blocks to make the long side of its oriented bounding box horizontal, and translate to its center location without changing scale. Then we assign a matching score for all pairs of buildings from two different urban layouts. Pairs of buildings that are approximately overlapping and with similar sizes will have higher scores. Then we compute the maximum weight matching among all possible building pairs by Hungarian method [29]. The average matching score is the L-Sim between two urban layouts. The formula for matching score between building $b_{1}$ and $b_{2}$ is as followed:

The multiplier $MinArea(b_{1},b_{2})$ returns the minimum area of two buildings. Exponent factor $\|b_{1}-b_{2}\|_{2}$ is the $L2$ norm between the positions of two buildings. Matching large buildings will tend to increase the matching score. Large position difference will decrease the value of the matching score. We set $c=0.02$ to scale the relatively large position difference value in the real world.

As Tab. 1 shows, our method clearly outperforms existing methods in 5 metrics and achieves the second best in WD of building counts. Note that we provide building count (the label set as described in original paper [26]) to LayoutVAE. So it is unfair to us regarding the WD distance on building count. We don’t report the value in the table.

Qualitative Results. Fig. 4 shows urban layouts generated by our method and other methods (LayoutVAE [26], BlockPlanner [62], and VTN [2]). For improved fairness, we provide ground truth building shape types and occupancy ratio values to our competitors since they did not consider building shapes. Nonetheless, the shown visual quality indicates that our method is still better able to capture arbitrary block shape and provide good generated output. In particular, Blockplanner [62] struggles to produce densely distributed urban layouts as claimed. It might be due to two reasons. First, the model is retrained on a larger and diverse dataset (119,236 blocks, 28 cities) compared to the original paper (637 blocks, 1 city); second, our urban layout generation task differs to its original land lot generation task (i.e., most buildings are not supposed to touch each other as in their original case). VTN [2] produces rectilinear building layouts regardless of block shapes. Strictly-rectilinear layouts are common in document layouts, but far from realistic in urban layouts.

User Study. We conducted a user study for perceptual realism using our method and the above three prior methods. The study was performed in a two-alternative forced choice (2AFC) manner. We generated 18 comparisons, each containing a layout generated by our method and the same layout generated by one of the three prior methods (thus 6 layout pairs for each prior method). The urban locations do not overlap across the three sets. During the study, we presented two urban layouts side-by-side from different methods. Users were asked to choose the one that looks more realistic to the best of their ability. We also included 2 random duplicate questions for quality checking. Users that answered differently to the same question were discarded.

In total 62 survey results were received. We discarded replies that were finished too fast (less than 2 minutes), that always answered the same choice, and that did not pass the quality check. Replies from the remaining 50 valid users are summarized. Our method was preferred by 97.3% of answers over LayoutVAE [26], 86.7% over BlockPlanner [62], and 88.0% over VTN [2].

We performed a comprehensive ablation study of adding and removing components of our solution to our method and to prior methods. While the previous comparisons show the superiority of our full method to prior methods as published, we further analyzed the effect of providing our canonical spatial transformation (CST) and also shape encoder (SE) to our competitors. In particular, CST was added to all competitors, and the vector produced by SE was also concatenated to the input vector of Gupta et al. [17] and to the latent bottleneck vector of VTN [2] and BlockPlanner [62] before decoding. For ablations on our model structure, we experimented with alternative graph convolutional layers (e.g. GraphSAGE [18], transformer-based [51], and GCN [28]) with depth $T=3$. In addition, we varied attention depth ($T\in{1,2,3}$) in our model structure.

We performed one-to-one comparisons using 1000 validation urban layouts, which is a comparison process different to the set-to-set comparison in Sec. 4.2. This more restrictive comparison task comprehensively evaluates robustness of overlap, position, and coverage rate. Position error is the average percent of building position shifting over the block length. Coverage error is the difference of total building area coverage rate compared to the ground truth rate.

Ablation results in Tab. 2 illustrate that CST and SE both help robustness to arbitrary block shapes. It especially helps reduce Out-Block index and position error. This indicates our novelty has the potential to broadly benefit layout generation and improve adaptability to arbitrary canvas shapes. Moreover, even when both mechanisms are added to all competitors, our method still performs clearly better. Self-ablation results show that graph attention networks outperform other alternative convolutional layers, and the deeper structure improves model performance.

Given a pre-trained model, we are able to generate diverse and realistic urban layouts for cities around the world. Apart from the random/conditional generation showed in Fig. 1 and several more in Supplemental, we present several interesting generation options including sparse prior generation, semantic manipulation, and interpolation.

Sparse Priors Generation. Our approach supports conditioned generation given sparse building layouts as prior. In particular, given a road network and a few initial blocks of building layouts, we can generate more blocks with a similar style. Fig. 5 uses only a random 5% of prior building layouts to generate a city: the latent vector of an empty block is calculated as the distance-weighted sum of the nearest $k$ blocks given as prior ($k=5$ works well). Normal noise is also added to the latent vector for diversity.

This generation ability is useful to provide data equity for many/most cities globally, where complete layout information is not available. To evaluate this, we showcased a well-modeled city. Fig. 5 models Chicago in which we trained building height information as additional building features so that we can generate 3D building masses (still from only 5% of prior). Then, we extracted a 3D urban morphology suitable for an urban weather forecasting system, Urban WRF [9]. The model generated by our method from only 5% prior yields comparable local weather forecasting results as that provided by the ground truth. Our average per-pixel wind-speed prediction error is 0.23m/s (details in Supplemental). Additional results are available in [45].

Semantic Manipulation. A semantic editing of building layouts can be performed by exploiting plausible disentanglements. For example, we labeled building row numbers (e.g., from 1 to 4) of 20K test urban layouts. We found latent vectors for each row-label group form reasonable clusters. If we translate a latent vector of a 1-row building layout towards that of the 2-row building layout cluster center, the layout will gradually have another row of buildings. Results are provided in Supplemental for up to 4 rows of buildings.

Interpolation. Our method can generate urban layouts by interpolating latent space vectors between two layouts. In Supplemental, we show results from linearly interpolating either building layout latent vectors or block shape latent vectors from one to another. The intermediate layouts correspond to an intuitive style interpolation.