Digital Twin Buildings: 3D Modeling, GIS Integration, and Visual Descriptions Using Gaussian Splatting, ChatGPT/Deepseek, and Google Maps Platform

Large Language Models (LLMs) are neural networks, typically Transformer-based [1], pre-trained on extensive, diverse text/image corpora, typically sourced from web crawls. These models, designed for Natural Language Processing (NLP), typically interpret text-based prompts and generate text-based outputs. Certain models, such as ”DeepseekV3/R1” and their variants [2, 3], support object character recognition (OCR, i.e., reading text from images). Models like ”ChatGPT-4o” [4] and its variants additionally support full interpretation and analysis of image content.

LLMs have achieved widespread adoption since 2023. Beyond basic image and text interpretation, these models recently exhibited expert-level problem-solving in various scientific and engineering domains [5, 6].

Due to their large size, LLMs often face hardware constraints for local deployment. While popular LLM providers such as OpenAI and Deepseek, provide web browser interfaces for their models, they also offer Application Programming Interfaces (APIs). These APIs enable client-side software or code to query LLMs hosted on OpenAI or Deepseek servers, facilitating large-scale data processing without requiring human-in-the-loop manipulations via browser interfaces. Unlike traditional local deep learning, which necessitates GPUs for both training and inference, API-based LLM querying requires minimal local hardware and can be deloyed on devices such as mobile phones.

Google Map Platform is a cloud-based mapping service and a part of Google Cloud. Its API allows the client device to connect to various cloud-based GIS, mapping, and remote sensing services hosted on the Google Cloud servers.

The services utilized in this research include remote sensing image retrieval, map retrieval, elevation data retrieval, geocoding/reverse geocoding, and building polygon retrieval. However, Google Maps Platform also offers other APIs for urban and environmental research, including real-time traffic data, solar potential data, air quality data, and plant pollen data, in addition to the full suite of commonly used Google Maps navigation and mapping tools.

Although less known in the remote sensing and GIS community than its sister application Google Earth Engine, Google Map Platform has been used in a variety of GIS research including navigation, object tracking, city modeling, image and map retrieval, geospatial data analysis for commercial and industrial applications [7, 8, 9, 10]. It is also used as part of many commercial software for cloud-based mapping integration.

Google Earth Studio [11] is a web-based animation tool that leverages Google Earth’s satellite imagery and 3D terrain data. The tool is especially useful for creating geospatial visualizations, as it is integrated with Google Earth’s geographic data. It allows for the retrieval of images from user-specified camera poses at user-specified locations. In this research, we use Google Earth Studio to retrieve 360 $\degree$ multi-view remote sensing images of a building from its address, postal code, place name, or geographic coordinates following [12, 13].

We use the mesh extraction procedure we introduced in December 2024 [13]. For conciseness, the process is briefly described here, and is not benchmarked. We refer the readers to [13] for the original implementation details and benchmark comparisons. We also refer the readers to [14] for background and theory on Gaussian Splatting.

Gaussian Building Mesh (GBM) [13] is an end-to-end 3D building mesh extraction pipeline we recently proposed, leveraging Google Earth Studio [11], Segment Anything Model-2 (SAM2)[15] and GroundingDINO [16], and a modified [17] Gaussian Splatting [14] model. The pipeline enables the extraction of a 3D building mesh from inputs such as the building’s name, address, postal code, or geographical coordinates. Since the GBM uses Gaussian Splatting (GS) as its 3D representation, it also allows for the synthesis of new photorealistic 2D images of the building under different viewpoints.

We use the Python client binding for Google Maps Platform Services APIs to create an integration tool to automatically retrieve the GIS and mapping information of a building. For these image analysis experiments, the data is retrieved with four API calls. The first is a Google Maps Platform Geocoding/Reverse Geocoding API call which retrieves the complete address information including geographic coordinates, entrance(s) coordinates, and building polygon mask vertex coordinates. Then, a Google Maps Platform Elevation API call is used to retrieve the ground elevation using the building’s coordinates as input. Additional API calls to other Cloud Services can also be performed at this step. Finally, two API calls are made using the Google Maps Platform Static Maps API to retrieve map(s) and satellite/aerial image(s) at the desired zoom level. This process is illustrated in Figure 2. The aerial/satellite image(s) are then used as one of the inputs to our Multi-Agent LLM Module.

Our Google Map Platform Integration can easily be modified to retrieve additional data from the cloud-based mapping service by adding parallel API calls below the Geocoding/Reverse Geocoding API call. For example, if we wish to analyze real-time traffic data, we can simply perform API calls to the Traffic API.

The motivation of this module is to create a multi-agent LLM system to analyze the data retrieved from Google Cloud Platform Services integration. In this paper, we restrict the scope of this paper to the multi-agent content analysis of multi-view/scale images.

For this analysis, the primary goal is to retrieve and store keywords describing the architectural style, function, landscape, and architectural elements of the building using keywords. With the idea that an accurate set of keywords will allow an agent to reconstruct a text-based description of the image without actually seeing the image, we also retrieve the caption using keywords as a secondary objective. This process is illustrated in Fig. 3.

From a building’s address, place name, postal code, or geographic coordinates, we retrieve multi-view off-nadir images of the building of interest using Google Earth Studio (or use the ones previously retrieved in the GBM module). We also retrieve top-down view aerial/satellite image(s) at different scales using the building using Google Map Platform Integration. For each image, we initiate a GPT4o/GPT4o-mini agent and prompt it to analyze the image and retrieve a set of keywords for each image. We then initiate two agents, one to aggregate the keywords from all the images of the building, and one to turn the aggregated keywords into a human-readable caption description.

Although metrics such as BLEU and CIDEr are commonly used to evaluate captioning performance, they require supervised datasets with ground truth captions. However, our images lack ground truth captions. Therefore, we use the CLIP (Contrastive Language–Image Pretraining) score [18]. A CLIP-trained Transformer is used to embed both the caption and image into a shared image-language latent space. Given the text embedding of the caption, $\mathbf{t}$ and the image embedding of the corresponding image, $\mathbf{i}$, the CLIP score is given by

We also use perplexity to roughly assess LLM image-to-keyword extraction confidence. Perplexity is a measure of the model’s confidence in its response, and is given by

where $\log P(x_{i})$ is the log-probability of the token $x_{i}$ as assessed by the model during text-token autoregression.

We chose seven different buildings to test our framework. These include well-known landmarks, commercial, residential, and institutional buildings. We extract 31 multi-view images in a 360$\degree$ view pose around the building of interest, which we then use in conjunction with our GBM module to create the 3D colored mesh of the building. Then we subsample six images, one every 70$\degree$, as inputs to the Multi-Agent LLM module. We also use the Google Map Platform integration to retrieve two aerial/satellite image(s), one at Google Maps zoom level 18, and one at Google Maps zoom level 19 as inputs to the Multi-Agent LLM module.

In preliminary experiments, we noticed a relatively large variation in final CLIP-score across different attempts even while using the same model and prompt, since the LLMs’ outputs are not deterministic (even when using LLM Temperature = 0). As such, we perform two experiments. We want to understand the performance of the module when using different models as LLM agents, and we want to understand the distribution of scores across different attempts for both the keyword extraction step, and the captioning step.

Fig. 4 shows the level of detail does not significantly affect the LLM agents’ confidence in their own predictions. Perhaps surprisingly, the much smaller gpt-4o-mini is more confident in its own responses on average. We note this does not necessarily denote keyword accuracy since it is possible the larger model considers many equivalent alternate visual descriptions, lowering its confidence in any individual description. Visual inspection of captions and images shows that across all four configurations, caption-to-image agreement is high. Although it is possible that the smaller model is enough for the visual captioning task, we nonetheless used the larger model with high visual quality for multi-view/multi-scale keyword extraction to test the keyword-to-caption step.

Deepseek-reasoner, the Deepseek-R1 model has the poorest captioning score. Additionally, this model sometimes failed at the task, shown as outliers in Fig. 5. This is perhaps because image captioning is an autoregressive text generation task and not a reasoning task. Consequently, we decided not to test OpenAI’s reasoning model, the GPT-o1. Its performance on reasoning tasks is comparable to Deepseek-R1 according to [3], yet it is more expensive by a factor of 30-100. Deepseek-chat. The Deepseek-V3 model has on average the best captioning performance, offering a very good best price-performance ratio with performance comparable to GPT-4o, but prices similar to GPT-4o Mini (see Table I).

Our future research aims to leverage the multi-agent LLM tool for geospatial data analysis, integrating various data sources from Google Cloud Platform services, including Google Maps Platform APIs and Google Earth Engine. Additionally, benchmarking the reasoning capabilities of large language models, such as GPT-o1/o3 and Deepseek-R1, for remote sensing and GIS tasks could yield valuable insights.