CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis

Satellite-to-street view synthesis is a challenging task that has been extensively studied. To mitigate the difficulties posed by the large differences across views, many studies explored additional semantic priors to enhance the structure of street-view synthesis results Zhai et al. (2017); Regmi & Borji (2018); Tang et al. (2019); Wu et al. (2022). Zhai et al. Zhai et al. (2017) is a pioneer in this domain that infers the street-view semantic map from the satellite semantic map via a learnable linear transformation. Tang et al. Tang et al. (2019) utilized both the satellite image and the semantic map of street-view image as input to synthesize the target street-view image via image-to-image translation. Although providing a strong structure prior of street-view images, the semantic map is not always available in the actual cross-view synthesis scenarios.

Another group of studies proposed satellite-to-street synthesis methods without using additional semantic information of street-view images, which explored various cross-view projection or transformation methods to provide geometry guidance specifically for panoramic image synthesis Lu et al. (2020); Toker et al. (2021); Shi et al. (2022); Qian et al. (2023). In Lu et al. Lu et al. (2020), a geo-transformation method was proposed for leveraging the height map of satellite view to produce the additional building geometry condition to facilitate street-view panorama synthesis. Toker et al. Toker et al. (2021) applied a polar transformation method proposed by Shi et al. (2019) to cross-view image synthesis and designed a multi-tasks framework in which image synthesis and retrieval are considered jointly. Shi et al. Shi et al. (2022) employed a learnable geographic projection module to learn the geometry relation between the satellite and ground views to facilitate street-view panorama synthesis. Inspired by the success of neural radiance field (NeRF) Mildenhall et al. (2020), Qian et al. Qian et al. (2023) proposed a Sat2Density that can learn a faithful 3D density field as the geometry guidance for panorama synthesis.

In summary, existing studies on satellite-to-street view synthesis are based on generative adversarial networks, with the main aim of improving the structure of synthetic image via semantic or geometric guidance, generating street-view images with low quality and unrealistic textures. By contrast, our study proposes a novel cross-view synthesis method based on Stable Diffusion models Rombach et al. (2022), which designs a cross-view control guided denoising process with a novel cross-view attention module as well as structure and texture controls, generating street-view panoramas with much better perceptual quality and more realistic textures across various scenes.

In recent computer vision studies, diffusion models Ho et al. (2020) have exhibited remarkable performance in many content creation applications, such as image-to-image translation Saharia et al. (2022a); Li et al. (2023a), text-to-image generation Balaji et al. (2022); Ramesh et al. (2022); Saharia et al. (2022b); Zhang et al. (2024), image enhancement Saharia et al. (2022c); Whang et al. (2022); Gao et al. (2023); Wang et al. (2024), content editing Avrahami et al. (2022); Couairon et al. (2023), and 3D shape generation Luo & Hu (2021); Zeng et al. (2022); Liang et al. (2024); Li et al. (2024b), etc. For traditional denoising diffusion models, images are generated by progressively denoising from random Gaussian noise. For instance, Song et al. Song et al. (2021) proposed denoising diffusion implicit models (DDIM) that reduce the number of denoising steps using an alternative non-Markovian formulation. In latent diffusion models (LDM) Rombach et al. (2022), a variational autoencoder Kingma & Welling (2014) is trained for compressing natural images to a latent space, where the diffusion process will be performed in later stages.

Recently, an increasing number of diffusion models have been proposed for controllable image synthesis Gal et al. (2022); Zhang et al. (2023a); Huang et al. (2023b); Zhao et al. (2023); Ruiz et al. (2023). ControlNet Zhang et al. (2023a) leverages both text and a variety of visual conditions (e.g., sketch, depth map, and human pose) to generate impressive controllable images, which also avoids the need to re-train the entire large model by fine-tuning pre-trained diffusion models and zero-initialized convolution layers. Composer Huang et al. (2023b) integrates global text description with various local controls to train the model from scratch on datasets with billions of samples. Uni-ControlNet Zhao et al. (2023) enables composable control with various conditions using a single model and achieves zero-shot learning on previously unseen tasks. However, these methods utilize similar-view image inputs to control the structure and texture of the synthesis results, resulting in inapplicability to cross-view synthesis tasks.

In addition, several studies have proposed diffusion models for novel view synthesis tasks. For instance, MVDiffusion Tang et al. (2023) proposes a cross-view attention module to generate consistent indoor panoramic images, and Tseng et al. Tseng et al. (2023) utilizes epipolar geometry as a constraint prior to synthesize a consistent video of novel views from a single image. MagicDrive Gao et al. (2024) proposes a street view generation framework that leverages diverse 3D geometry controls (i.e., camera poses, road maps, and 3D bounding boxes) and textual descriptions. However, existing novel view synthesis methods rely on the continuity of image views or camera pose information, which cannot be satisfied in satellite-to-street cross-view settings. Several recent studies have aimed at cross-view synthesis task via diffusion models. Sat2Scene Li et al. (2024c) proposes a novel 3D reconstruction architecture that leverages diffusion models on sparse 3D representations to directly generate 3D urban scenes from satellite imagery. Streetscapes Deng et al. (2024) proposes an autoregressive video diffusion framework and introduces a novel temporal interpolation approach, generating long-range consistent street-view images based on map and height data. However, the task settings of these studies are different from the satellite-to-street view synthesis, and their methods fail to utilize satellite image information to generate realistic street-view textures.

Although diffusion models have achieved promising performance in numerous computer vision applications, few studies have been designed for the challenging satellite-to-street view synthesis task. In this work, we extend the application scenarios of diffusion models to satellite-to-street view synthesis. With both structure and texture controls from the satellite image, our cross-view guided denoising process enables the diffusion model to generate more realistic street-view panoramas.

Diffusion models are generative models that can generate samples from a Gaussian distribution to match target data distribution by a gradual denoising process Ho et al. (2020). In the forward process, diffusion models gradually add Gaussian noises to a ground truth image $x_{0}$ according to a predetermined schedule $\beta_{1},\beta_{2},\dots,\beta_{T}$:

$$q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I)$$

(1)

where $x_{t}$ is a noised sample with noise level $t$. The reverse process involves a series of denoising steps, where noise is progressively removed by employing a neural network $\epsilon_{\phi}$ with parameters $\phi$. This neural network predicts the noise $\epsilon$ present in a noisy image $x_{t}$ at step $t$. The simplified version of the loss function for training the diffusion model is formulated as follows:

$$L_{simple}(\phi,x)=E_{t,\epsilon}\left[\left\|\epsilon_{\phi}(x_{t},t)-\epsilon\right\|^{2}\right]$$

(2)

where $t$ is uniformly sampled from the set $\{1,\ldots,T\}$, and $x_{t-1}$ can be reconstructed from $x_{t}$ by removing the predicted noise:

$$x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\phi}(x_{t},t)\right)+\sqrt{\beta_{t}}\epsilon$$

(3)

where $\alpha_{t}=1-\beta_{t}$, $\bar{\alpha}_{t}$ is the cumulative sum of $\alpha_{t}$ and $\epsilon\sim\mathcal{N}(0,I)$.

To precisely control the generation of panoramas in cross-view scenarios, it is essential to establish structural and textural information from a street-view perspective based on satellite imagery. Specifically, we start by constructing three-dimensional voxels as intermediaries from the depth estimation results of satellite images. The structural control information is derived from projecting these 3D voxels onto the street-view panorama to obtain scene structure estimates. On the other hand, texture control is achieved through a weight matrix derived from the cross-view mapping relationship based on 3D voxels, representing the response regions on the street view image to different features of the satellite image.

Based on satellite scene estimation, we obtain binary maps to serve as structural controls for the street-view images. Utilizing cross-view mapping, we derive weight matrices to act as texture controls for the street-view images. Based on the characteristics of the structural and textural control information, we design an enhanced cross-view attention module to integrate both types of information, guiding the subsequent denoising process.

In our enhanced cross-view attention module, let $Q\in\mathbb{R}^{h_{p}\times w_{p}}$ denote the Query feature from the panoramic binary map $S_{pano}$, $K\in\mathbb{R}^{h_{s}\times w_{s}}$ denote the Key feature from the input satellite image $I_{sate}$, and $V\in\mathbb{R}^{h_{s}\times w_{s}}$ denote the Value feature, which contain detailed texture information from the satellite image. Here, $h_{p}\times w_{p}$ and $h_{s}\times w_{s}$ represent the resolution of the panorama and satellite feature map, respectively. Moreover, $E_{\text{sate}}$ and $E_{\text{pano}}$ denote the satellite and panoramic encoders. $W_{q}$, $W_{k}$ and $W_{v}$ are projection matrices. The definitions of $Q,K,V$ are as follows:

$$Q=W_{q}(E_{\text{pano}}(S_{\text{pano}})),\quad K=W_{k}(E_{\text{sate}}(I_{\text{sate}})),\quad V=W_{v}(E_{\text{sate}}(I_{\text{sate}})),$$

(9)

The process begins with the computation of an affinity matrix $A\in\mathbb{R}^{h_{p}w_{p}\times h_{s}w_{s}}$, reflecting the interaction between $Q$ and $K$. Following this, the weight matrix derived from the previous module is down-sampled to $M\in\mathbb{R}^{h_{p}w_{p}\times h_{s}w_{s}}$ and applied to each pixel within the satellite image to emphasize relevant features. This selective enhancement is crucial for the subsequent fusion of the detailed texture information from the satellite image into the panoramic feature $F_{\text{pano}}\in\mathbb{R}^{h_{p}\times w_{p}}$. The enhanced cross-view attention mechanism is formulated as follows:

$$\text{z}=\text{softmax}(A\odot M)\cdot V$$

(10)

In these expressions, $\odot$ denotes element-wise multiplication, where the weight matrix $M$ is applied to the affinity matrix $(A)$ to obtain the reweighted affinity matrix $(A^{\prime})$, emphasizing the connection between the most relevant pixels.

The output z, generated at each step, is ingeniously reincorporated into the network as a pivotal conditional element. By employing z as a dynamic conditional catalyst within our cross-view diffusion architecture, we ensure that each step of the denoising process is informed by the evolving latent representation, thereby enabling a controlled and gradual transition from $z_{t}$ to $z_{0}$. This process is meticulously orchestrated by a cross-view control guided denoising process, which integrates structural and textural knowledge extracted from $I_{\text{sate}}$ into the refinement of the final latent feature $z$, subsequently decoded through Stable Diffusion’s latent space decoder $\mathcal{D}$ to achieve the generated street-view panorama $I_{\text{pano}}$.

Cross-modal satellite-to-ground synthesis requires measuring both the consistency and realism of generated images. Traditional metrics like SSIM Wang et al. (2004) and FID Heusel et al. (2017) generally focus on single dimensions of similarity or realism, providing incomplete evaluations. Inspired by the use of large multimodal models for synthetic image scoring Cho et al. (2023); Huang et al. (2023a); Wu et al. (2024); Zhang et al. (2023b), we design a new evaluation process based on GPT-4o, as shown in Figure 3. This approach enables comprehensive and interpretable assessments of synthesized street-view images, aligning more closely with human judgment standards.

Firstly, we design three key evaluation dimensions for cross-view synthesized images: Consistency, Visual and Structural Realism, and Perceptual Quality. We adopted a rating scheme, establishing a 5-level rating system with scores ranging from 1 (poor) to 5 (excellent).

Consistency: This dimension evaluates the alignment of the content in synthesized images with real street-view images, including the structure and texture of buildings, the layout of roads, and the similarity of other significant landmarks, measuring the content consistency of the synthesized street-view images.

Visual Realism: This evaluates the visual effect and structural reasonableness of the generated images, including the realism of color, shape, and texture, as well as the the structural integrity, assessing whether they look like real street-view images.

Perceptual Quality: This evaluates the overall perceptual quality of the generated images, including aspects such as image clarity, noise level, and visual comfort, measuring the quality of the generated images.

To achieve more effective GPT scoring, we employed Chain-of-Thought (CoT) and In-Context Learning (ICL) strategies Alayrac et al. (2022); Zhang et al. (2023c); Brown et al. (2020); Peng et al. (2024) to enhance its stability and effectiveness. Firstly, we provided GPT-4o with a small number of effective human-scored examples from multiple users, enabling the model to effectively learn human scoring patterns. Secondly, by enabling the large model to explain the reasoning behind its scores, we have introduced an element of internal reflection to the evaluation process. Additionally, we used GPT-4o to act as Evaluator A and Inspector B. After receiving the initial scores and reasons from Evaluator A, Inspector B will assess the reasonableness of these scores and make the final scoring decision. If the scores are deemed reasonable, they will be retained; otherwise, Inspector B will provide new scoring results and justifications.

To validate the effectiveness of GPT-based scoring, we invited ten human users to perform the same scoring task and measured the consistency between their scores and those generated by GPT. We provided thorough training to the users to ensure they fully understood the satellite-to-street view generation task. The users’ scoring tasks and schemes were aligned with the GPT scoring. We ensured that each generated image was scored by at least two human users. Due to the large volume of cross-view datasets and the cost of both user and GPT scoring, we randomly sampled 1000 images from the evaluation sets of each dataset for assessment. In addition to our method, we selected the best comparative results from GAN and diffusion methods for evaluation. A total of 9000 images were used for user scoring, and we measured the agreement between these scores and the GPT scores.

In our experiments, we used three popular cross-view datasets to evaluate the synthesis results, i.e., CVUSA Zhai et al. (2017), CVACT Liu & Li (2019) and OmniCity Li et al. (2023c). These three datasets encompass rural, suburban, and urban scenes, providing a robust benchmark for comprehensively evaluating the performance of satellite-to-street view synthesis. Furthermore, in addition to the original satellite imagery and building height data provided by OmniCity, we supplemented multimodal data including text, maps, and multi-temporal satellite imagery, providing data support for street-view synthesis tasks using additional multimodal data sources.

CVUSA Zhai et al. (2017) is a standard large-scale cross-view benchmark, primarily featuring rural scenes such as roads, grasslands, and forests. This dataset comprises centrally aligned satellite and street-view images collected from various locations across the United States, which is randomly split into training and test sets in an 8:2 ratio.

CVACT Liu & Li (2019) is a widely used cross-view dataset that includes satellite and street-view images from Canberra, Australia. This dataset mainly consists of suburban scenes with relatively low buildings and open views. Unlike CVUSA dataset, the training and test sets of CVACT dataset are divided by region.

OmniCity Li et al. (2023c) is an urban cross-view dataset that includes street-view and satellite images from New York, USA. The primary scenes in OmniCity consist of dense urban buildings, and street-view images that are heavily obstructed by trees or vehicles will be filtered out. OmniCity is divided into training and test data by region.

Additionally, the orientation towards the north in both street view and satellite imagery is a critical attribute for cross-view datasets. In all three datasets, the north direction in satellite images is at the top of the image. In CVUSA Zhai et al. (2017) and CVACT Liu & Li (2019), the north direction in street-view images is in the center column, while in OmniCity Li et al. (2023c), it is in the first column.

We implement CrossViewDiff based on the ControlNet Zhang et al. (2023a) framework, incorporating the pre-trained Stable Diffusion Rombach et al. (2022) v1.5 model. The diffusion decoder is configured in an unlocked state and the classifier-free guidance Ho & Salimans (2022) scale is established at 9.0. For the final inference sampling, we adopt $T=50$ as the sampling step, consistent with the DDIM Song et al. (2021) strategy. The entire training process is performed on eight NVIDIA A100 GPUs, with a batch size of 128, spanning a total of 100 epochs. Our depth estimation method employs Marigold Ke et al. (2024) and is fine-tuned on the OmniCity dataset, which provides elevation information. We conduct our experiments at a resolution of $1024\times 256$ on the CVUSA Zhai et al. (2017) and $1024\times 512$ on OmniCity Li et al. (2023c) and CVACT Liu & Li (2019).

We compared our method with several state-of-the-art cross-view synthesis methods on the three datasets, including GAN-based methods such as Sate2Ground Lu et al. (2020), CDTE Toker et al. (2021), S2SP Shi et al. (2022), and Sat2Density Qian et al. (2023), as well as diffusion models for image transformation control like ControlNet Zhang et al. (2023a) and Instruct pix2pix (Instr-p2p) Brooks et al. (2023). For Sat2Density Qian et al. (2023), we follow their original setup, i.e., the lighting hints are determined based on the average values of the sky histograms obtained from random selections. For diffusion-based methods (ControlNet and instr-p2p), we use a pre-trained model consistent with that of CrossViewDiff and maintain the same sample steps. Note that all comparison methods are conducted according to their optimal experimental settings.

Following previous studies Lu et al. (2020); Toker et al. (2021); Regmi & Borji (2018), we used common metrics such as SSIM Wang et al. (2004), SD, and PSNR to evaluate the content consistency of synthesized images, and FID Heusel et al. (2017) and KID Bińkowski et al. (2018) to assess image realism. Furthermore, in Section 4.3.2, we use GPT-4o to evaluate the synthesized street view images across three dimensions: consistency, visual realism, and perceptual quality.

In our ablation study, we first assessed the effectiveness of our structure and texture control modules. As shown in the first two rows of each dataset of Table 5, using structural information derived from satellites as input proved effective, achieving improvements across multiple metrics such as SSIM Wang et al. (2004), FID Heusel et al. (2017), and KID Bińkowski et al. (2018). The last two rows of each dataset show the results of using direct Cross-Attention to incorporate global textures (w/o CVTM) and our Cross-View Texture Mapping (w/ CVTM) methods. Compared to the direct incorporation global textures, the approach guided by cross-view mapping relationships effectively assigns local textures from corresponding satellite regions to the appropriate locations in street-view images. Figure 8 presents qualitative ablation results on CVUSA Zhai et al. (2017) and OmniCity Li et al. (2023c), where structural control contributes to consistent content distribution, and texture control enhances the consistency of generated textures in buildings and forests.

Additionally, as the intermediaries for constructing both structural and textural controls, the 3D voxels derived from satellite depth estimation results significantly impact the accuracy of cross-view controls. Therefore, the precision of satellite depth estimation directly influences the effectiveness of these controls. To simulate depth estimation inaccuracies, we apply scaling factors (0.9 and 1.1) to the depth estimation results before generating street-view images, as detailed in Table 6. The experimental results indicate that while our method relies on depth estimation, the stability of the model’s output remains high, with minimal fluctuation in performance metrics.

In this section, we provide more experimental results of real-wolrd application scenarios using additional data sources. In addition to the satellite images, other inputs such as textual data, building height data, and public map data (e.g. OpenStreetMap¹¹1https://www.openstreetmap.org/) can also be used for generating street-view images. In this study, we explored the synthesis of street-view images using multiple data sources on the OmniCity Li et al. (2023c) dataset and analyzed their impacts. Based on OmniCity street-view images, we generate corresponding text prompts of street-views images using the CLIP Radford et al. (2021) model, and supplement the corresponding historical satellite imagery and OSM map data based on the street view capture locations.

As shown in Figure 9, textual data can provide some global information about the scene , but its lack of detail and specificity results in visually unrealistic images. OSM (OpenStreetMap) data offers semantic features of different areas, such as roads, buildings, and parks. These semantic features aid in generating street-view images with consistent semantic content. However, when using only OSM data, the structure and texture of the synthesized street view images still show a certain gap compared to real images. Building height data provides the outlines of buildings, and street-view images synthesized using this data show consistent building contours but lack texture and detail. Combining OSM and building height data for street view synthesis perform well in terms of semantics and structure. However, there are still deficiencies in texture details, such as building colors. Combining satellite imagery and building height data yields street-view images that are optimal in both structure and texture, visually closest to real street views. Table 7 provide the quantitative results obtained from different types of input data. Due to the rich texture information in satellite images, our CrossViewDiff achieved SSIM Wang et al. (2004) and FID Heusel et al. (2017) scores of 0.361 and 37.89, respectively, representing improvements of 4.6% and 17.6% compared to the results synthesized using OSM and building height data as inputs.

Next, we explored the results of synthesizing street-view images using satellite imagery data from different years. As shown in Figure 10, significant changes in terrain features over time can also be observed in our synthesized street-view images, such as the transformation of parking lots or vacant lots into buildings within the areas highlighted in red. Given the relatively recent widespread adoption of street-view imaging compared to the earlier availability of remote sensing satellite imagery, our effective satellite-to-street-view synthesis method unveils historical scenes from earlier times, offering practical application value.

Despite the above advantages, street-view images generated by CrossViewDiff still have several limitations. Although we fused features rich in structural and textural information based on satellite image, the gap between the two viewpoints is still large, and Stable Diffusion is more capable of creating additional details that do not actually exist. Figure 11 provides some typical failure cases obtained by CrossViewDiff. For satellite and street-view images that were not taken at the same season, even though the synthetic street-view image is consistent with the satellite’s features, it may not be consistent with the ground truth. Besides, in less constrained regions of the image such as the sky, the synthesis result is somewhat different from GT and has a certain amount of color shifting, resulting in the relatively low PSNR to some extent. Moreover, due to the presence of moving objects such as pedestrians and vehicles in the scene, achieving consistency in cross-view synthesis results remains challenging.