Deep Generative Framework for Interactive 3D Terrain Authoring and Manipulation

Let $T_{inp}$ be our (hand drawn) input sketches representing a rough topographic map. We assume that these inputs belong to some parametric distribution $p_{\theta}(T_{inp}|z)$ and each terrain is generated by some random process, involving an unobserved random variable $z$. Calculating $p_{\theta}(T_{inp}|z)$ is hard as $p(T_{inp})$ is intractable. Hence, we use the VAE to approximate another distribution $q(z)$ such that it is similar to learns the distribution parameters $\mu$ and $\sigma$, from which the latent vector $z$ is sampled. We use a re-parameterization trick to sample $z$ as $z=\mu+\sigma*\epsilon$, where $\epsilon$ is sampled from a Standard Normal distribution. This sampled vector is fed to the decoder which predicts $T_{rec}$, that is the reconstruction of original input $T_{inp}$.

$$L_{BCE}(T_{inp},T_{rec})=-[T_{inp}\log{T_{rec}}+(1-T_{inp})\log{(1-T_{rec})}]$$

(1)

$$\begin{split}L_{recons}=L_{BCE}(T_{rec}^{g},T_{inp}^{g})+\alpha[L_{BCE}(T_{rec}^{r},T_{inp}^{r})\\ +L_{BCE}(T_{rec}^{b},T_{inp}^{b})]\end{split}$$

(2)

We propose a novel auto-regressive reconstruction loss $L_{recons}$ between VAE input $T_{inp}$ and output $T_{rec}$ by modifying the traditional Binary Cross Entropy (BCE) loss (Equation 1) to emphasize the ridge/valley lines in the topographic map. Here, we represent the input topographic map $T_{inp}$ as RGB image where the Green channel ($T_{inp}^{g}$) is dedicated to represent level-sets while the Red ($T_{inp}^{r}$) and Blue ($T_{inp}^{b}$) channels are used to represent ridge and valley lines, respectively. A similar representation exist for output (reconstructed) topographic map ($T_{rec}$). We propose to give higher weightage to the loss on red and blue channels so as to give more importance to red and blue channels representing sparse ridge and valley lines/strokes in the topographic map sketches. This is desired in our joint representation of topographic map as ridge/valley lines/strokes are relatively sparse (occupying fewer pixels) in comparison to dense level-sets maps (green channel). This helps the model learn sharp features of river valleys and hill ridges. Additionally, a traditional KL divergence loss $L_{KL}$ (Equation 3) ensure that the probability distribution of latent vector $z$ follows a Standard Normal distribution.

$$L_{KL}=\sum_{i=1}^{d}{[\sigma^{2}+\mu^{2}-\log{\sigma^{2}}]}$$

(3)

Thus, the final VAE loss $L_{VAE}$ (Equation 4) is a combination of reconstruction $L_{recons}$ (Equation 2) and KL divergence loss $L_{KL}$ (Equation 3). The $\gamma$ parameter in Equation 4 is the weighting of the latent loss $L_{KL}$ which is set to 0.65, $\alpha$ is weight of red and blue channels in reconstruction loss which is set to 5, $\mu$ and $\sigma^{2}$ are the mean and variance learnt by the network.

$$L_{VAE}=L_{recons}+\gamma*L_{KL}$$

(4)

The VAE output $T_{rec}$ is a topographic map, which is further fed to our second stage for generating the plausible DEM for completion of terrain authoring task. However, apart from traditional terrain authoring task, the key importance of our multi-stage framework is that the VAE based learning of latent space also enables performing interpolation of terrains as well as generating novel variants of a given terrain, as explained below.

The aim of this stage is to generate the DEM given user topographic map sketch or in our case generated sketch from previous stage. Similar to TSynthNet [10], we adapt Pix2pix [11], a conditional GAN model for this task. Traditionally, GAN [8] are based on the idea of making a given noise distribution to match the desired data distribution. The conditional GANs [15] conditions both generator and discriminator on some auxiliary information to have better control over the generated output. The generator network $\mathcal{G}$ attempts to learn this mapping while the discriminator $\mathcal{D}$ learns to discriminate the generated data from ground truth data. Thus, the overall network is trained such that both generator and discriminator reach a Nash equilibrium by playing a two player minmax game while optimizing the value functions.

Let’s represent ground truth DEM as $D_{gt}$ and $D_{gen}$ as generated/predicted DEM by the Pix2pix model for the input topographic map sketch $T_{rec}$. Let $f$ be the feature embedding learnt by the generator encoder, the value function can be defined as:

$$\begin{split}V(\mathcal{D},\mathcal{G})=\mathbb{E}_{D_{gt}}[\;\log\mathcal{D}(D_{gt})\;]\\ +\mathbb{E}_{f}[\;\log(1-\mathcal{D}(\mathcal{G}(f)))\;]\end{split}$$

(5)

Our model uses hand drawn input topographic sketch maps as an auxiliary information in order to produce plausible DEMs. We use L1 loss for reconstruction from the generator i.e., $L1(\mathcal{G})$. So the final loss for cGAN training becomes:

$$L=\underset{\mathcal{G}}{m}in\;\underset{\mathcal{D}}{m}ax\;[V(\mathcal{D},\mathcal{G})+L1(\mathcal{G})]$$

(6)

We use popular DEM dataset used by other relevant works in the literature e.g.,  [3, 14] which is part of publicly available high-resolution DEMs of mountain ranges named Pyrenees [1] and Tyrol [2], respectively. DEM patches with a resolution of 2m/pixel has been used as ground truth elevation maps. Original DEM tiles were split into 200x200 pixels. Instead of using the whole dataset of 22000 image patches for training and 11000 image patches for testing, we randomly sample 3000 image patches for training and 878 image patches for testing. More details about the dataset can be referred from [14]. We prepare the training dataset by extracting the topographic map input sketches from DEMs (please refer to supplementary material for details).

Our VAE model (in stage 1) is a 12 layer network with 6 layers in encoder and 6 layers in decoder. The latent space dimension is set to 128. All the layers of encoder consists of a 3x3 convolution with stride of 2 and padding of 1, followed by Batch Normalization and using Leaky ReLU non-linearity. Due to stride of 2, the image resolution reduces by two after each layer, and we double the number of channels in each layer starting from 32 in first and 1024 in the final layer of encoder. We then flatten the output of encoder and pass it through fully connected layer to get 128 dimension mean and log variance. This mean and log variance is then used to sample and then the decoder is used to reconstruct the input. Each decoder layer consists of Transposed Convolution with stride of 2 and input and output padding of 1, followed by batch normalization and leaky ReLU as activation. Decoder also consists of 6 layers, so we reach the same resolution as input. At the end we have a convolution layer with 3x3 kernel, stride of 1, padding of 1 and output channels 3. Adam optimizer was used to update the parameters with learning rate of $0.001$ and an exponential scheduler with gamma set to $0.95$ while training the VAE with a batch size of $64$ for $100$ epochs.

Our conditional GAN generator (in stage 2) is a U-net inspired Pix2pix architecture [11]. This model was trained using Adam optimizer with learning rate of $0.0002$, $\beta_{1}$ set to $0.5$ and $\beta_{2}$ as $0.999$ for both Generator and Discriminator. We use a batch size of $32$ here.

We used 3000 terrain patches (of 256$\times$256) for training and 878 terrain patches (of 256$\times$256) for testing both stage 1 and 2 models of our method. All our experiments were performed on a single Nvidia GTX 1080Ti. The network implementations were done using Pytorch.

We demonstrate the result of the proposed pipeline in Figure 4. The figure shows the comparison between rendered ground truth and predicted DEM draped with corresponding satellite images. This shows that the model is able to generate realistic terrains close to the ground truth. Some more examples of diverse terrains generated are shown in Figure 6.

Generating Terrain Variants: We utilise the latent space created by VAE to generate different samples from the same input. Different terrains generated from the latent space encoding of the same input topographic map are shown in Figure 5. The red circled regions shows the variation in the generated terrains as compared to the original terrain. We can observe the generated terrains have realistic but slightly different topographical features from that of input terrain. This provides the user the flexibility to generate multiple terrain DEMs and use them for large scale generation of virtual terrain maps.

Terrain Interpolation: The latent space can also be used for automated fusion of topographic features across two terrains. Figure 3 shows an example interpolation of two input terrains in the latent space for different values of $\gamma$ parameter. This enables the user to generate new virtual terrain maps by combining a given pair of input terrain maps.

Please refer to supplementary video for 3D visualization of qualitative results.

We perform quantitative analysis and compare our model with TSynthNet [10]. Additionally, we also compare our method with two variants namely, “Only VAE” and “VAE+cGAN without modified VAE loss”. We attempted to learn the DEM generation using only VAE in the former model. The latter is an ablative variant of our model without modified VAE loss. We train all these models with same train/test split on terrain dataset (see subsection 3.1) and reported Mean Squared Error (MSE) as the evaluation metric.

We provide quantitative evaluation results in Table 1 . We can observe that we obtained superior performance with MSE of $28.4024$ from our model (VAE+cGAN with modified VAE loss). The ablative comparison with VAE+cGAN without modified VAE loss (i.e., MSE value of $73.67$) show that our proposed modified VAE loss that account for ridge/valley lines significantly improves the performance of our method. Only VAE model (i.e., single stage VAE based DEM generation) also performs inferior (with MSE of $57.265$) to our model justifying need of two stage framework. In comparison to TSynthNet [10], our model yield lower MSE value. It is important to note that apart from achieving lower MSE than TSynthNet on terrain generation task, our method also enables terrain interpolation and variant generation using the learnt VAE latent space. Thus, our model yield superior quantitative performance as compared to other methods/variants.

We also performed two additional ablation experiments to justify our proposed method. In the first experiment, we retain the same architecture as our method but train it differently. Here we first train the VAE of first stage but during training of cGAN in the second stage, we let the gradients back propagate all way to decoder (while VAE encoder weights are kept frozen). However, we achieved an MSE of $80.67$ quiet inferior to our model. In the second experiment we initially train the stage 1 VAE to learn the latent space. In the second stage we again learn another VAE but with borrowing and freezing the encoder weights from first stage and train only the decoder for generating DEMs. Thus, it is still a two stage pipeline with two VAEs. We obtained an MSE of $35.67$ with this variant. These ablative studies justifies choice of our proposed two stage framework and learning strategy.

We performed a detailed user study involving $6$ users. In order to test the degree to which our model can generate realistic terrains, we presented users (students trained in geographical information system course) with set of generated (reconstructed) and ground truth terrain pairs overlaid with satellite images, similar to ones shown in Figure 6 and asked them to differentiate between the two. The outcome was that users were unable to decisively differentiate and choose the real terrain only $50\%$ of the time. Total $83.3\%$ of the users agreed that the terrains generated are very realistic while $16.7\%$ said that it is fairly realistic.

In the second experiment, we provide the user with a simple interface to draw input sketches. The designed interface is simple with controls such as Clear, Save, Exit. We provide the option to vary brush thickness so that the dense level-sets can be drawn with only few strokes. The user interface and the DEM generated for a hand draw user input is shown in Figure 7. The input can also be interactively edited to get desired output. Figure 8 shows visualization of generated terrains during an interactive terrain editing session. We asked the users several questions regarding the interface and the application such as 1) whether the input is intuitive, 2) if it is easy to express one’s intent, 3) if the terrains generated follow the sketches etc. $33.3\%$ users said that the input is very intuitive while $66.7\%$ users agreed that it is fairly intuitive. $50\%$ of the user strongly agree that the generated terrain follow the input sketches, while the remaining $50\%$ fairly agree. All the users strongly agree that the system is fast and reactive. When asked to rate on a scale of 1 to 5, on how easy it was to express one’s intent, $50\%$ of the users gave a rating of 5, $16.7\%$ gave a rating of 3 and $33.3\%$ gave a rating of 2. We observe that the users were able to generate DEMs with ease after a couple of attempts.