TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning

Predicting depth from monocular RGB images allows single-camera perception systems to recover 3D environment structure [9, 10]. However, dense depth estimation from monocular images may be challenging, especially for aerial images, where the depth variation is small compared to the absolute depth values. In contrast, the depth of sparse visual keypoints may be obtained efficiently and accurately using triangulation [11] between tracked feature points from Kanade-Lucas-Tomasi tracking [12] or visual feature matching [13].

Depth completion is the task of reconstructing a dense depth image from an RGB image with given sparse depth estimates. Ma et al. [3, 14] develop a deep network for depth completion that passes the sparse depth and RGB image inputs through convolution layers, ResNet encoder layers, transposed convolution decoder layers, and a 1x1 convolution filter. The model is trained either with supervision from ground-truth depth images or via photometric error self-supervision from calibrated RGB image pairs. Instead of consuming sparse depth images directly, Chen et al. [15] pre-process sparse depth images by generating a Euclidean distance transform of the keypoint locations and a nearest-neighbor depth fill map. The authors propose a multi-scale deep network that treats depth completion as residual prediction with respect to the nearest-neighbor depth fill maps. We borrow the idea of densify the 2D sparse depth inputs for our model design. Chen et al. [4] design a 2D convolution branch to process stacked RGB and sparse depth images and a 3D convolution branch to process point clouds and fuse the outputs of the two branches so that the 2D and 3D features are combined. Our approach also combines both the 2D and 3D feature extraction phases but in the form of mesh. CodeVIO [16] uses a Conditional Variational Autoencoder (CVAE) to encode RGB and sparse depth inputs into a latent depth code and decode a dense depth image from the latent depth code. The sparse depth measurements are used to perform incremental depth code updates, allowing the depth reconstruction to be coupled with visual odometry estimation in the MSCKF filter [17]. Our method is loosely integrated with a feature-based VIO/SLAM algorithm.

We approach depth completion using mesh reconstruction from RGB images and sparse depth measurements. While both dense depth completion and mesh reconstruction are challenging, a mesh model is more memory efficient than a depth image. For example, a $512\times 512$ dense depth image requires $260,000$ parameters. In contrast, our approach can represent the same region using only 1024 vertices.

Online terrain mapping requires efficient storage and updates of a 3-D surface model. However, storing dense depth information from aerial images needs significant memory and subsequent model reconstruction for robot objectives like motion planning or environment exploration. An explicit surface representation using a polygonal mesh can be quite memory and computationally efficient. Compared to sparse point cloud models, mesh surfaces are continuous, allowing direct integration in motion planning algorithms as well as intuitive visualization for human operators. FLaME [18] performs variational optimization over a time-varying Delaunay graph to obtain an inverse-depth mesh of the environment, using sparse depth measurements from a VIO algorithm. Rosinol et al. [19] extend FLaME to optimize the mesh over dense depth image measurements in real-time using a parallel implementation. Rosinol et al. [20] detect vertical and horizontal planes to regularize mesh vertices, and optimize the mesh vertices and the camera poses using a factor graph. Voxblox [21] incrementally builds a voxel-based truncated signed distance field and can reconstruct a mesh as a post-processing step using the Marching Cubes algorithm [22]. Terrain Fusion [23] performs real-time terrain mapping by generating digital surface model (DSM) meshes at selected keyframes. The local meshes are converted into grid-maps and merged using multi-band fusion.

Recently, learning methods have emerged as a promising approach for mesh reconstruction from limited or no 3D information. Bloesch et al. [24] propose a learning method to regress the image coordinates and depth of mesh vertices in a decoupled manner. This allows an in-plane 2D mesh to capture the image structure. Pixel2Mesh [25] treats a mesh as a graph and applies graph convolution [26] for vertex feature extraction and graph unpooling to subdivide the mesh for refinement. Using differentiable mesh rendering [27, 28], the 3D mesh structure of an object can be learned from 2D images [29, 30, 31]. Mesh R-CNN [32] simultaneously detects objects and reconstructs their 3D mesh shape. A coarse voxel representation is predicted first and then converted into a mesh for refinement. Recent works [33, 34] can generate mesh reconstructions of complete scenes, including object and human meshes and their poses, from a single RGB image.

In contrast with many mesh reconstruction approaches, our method uses both visual and semantic features to refine the mesh geometry and generates mesh models with per-vertex semantic category distributions.

3D reconstruction from an image sequence is a fundamental problem in robotics and computer vision. Multi-view stereo (MVS) [35] aims to estimate the depth at one frame using several different frames. Classical MVS methods perform patch matching with photometric and geometric consistency [36]. These methods generalize well although the performance can be affected by low texture, lighting variation, and occlusion. Recently, learning-based methods that fuse multi-view learned features across frames for depth recovery have achieved excellent performance. NeuralRecon [37] reconstructs and fuses sparse TSDF volumes for each frame incrementally using 3D sparse convolutions and Gated Recurrent Units (GRUs). VoRTX [38] learns to fuse multi-view frames using a transformer model and projective occupancy. SimpleRecon [39] leverages relative poses among frames to build a cost volume and uses a multi-layer perceptron (MLP) to reduce the volume and avoid costly 3D convolution. Learning-based MVS [40, 41, 42, 43] can tackle challenges like severe occlusion but generally labeled data is needed for training. Recently, SLAM systems that integrate a learning-based MVS model to obtain dense 3D reconstruction in real-time have been proposed [44, 45]. Besides explicit 3D representations, implicit representations that model 3D structures as level sets of distance or radiance functions have shown impressive performance recently. Neural Radiance Fields (NeRF) [46] represent the color and density field of the scene through the weights of the neural network. For any given camera pose, an RGB and depth image can be generated through volume rendering over the NeRF, providing photo-realistic novel view synthesis. Another advantage of NeRF models is that they can be trained from posed RGB images without depth supervision. However, for any new 3D scene a NeRF needs to be trained from scratch which may take a long time. Numerous recent works have extended the NeRF model [47, 48, 49, 50] to enable MVS pre-training, capture high-frequency details, and apply to unbounded outdoor environments. Instant-NGP [51] and Nerfstudio [52] incorporate many of the recent NeRF model advances and offer open-source implementations.

Semantic 3D reconstruction aims to estimate both the geometric structure and semantic content of an environment from visual observations. Extending 2D semantic segmentation and depth prediction to a 3D multi-view consistent semantic model provides information that is critical for environmental monitoring tasks, such as observing vegetation recovery after a wildfire or controlling fuel build-up for fire prevention [53], or for terrain traversablility estimation in ground-aerial robot teaming [54]. Semantic information also allows human operators to specify tasks for mobile robots in terms of objects and concepts in the environment model. Semantic segmentation on the 2D images can be back-projected onto 3D space and multi-view information can be fused to annotate the 3D structure [55, 56, 57]. Besides, semantic segmentation can be directly performed on the 3D point cloud [58, 59] or the mesh [60]. Instead of treating the semantic annotation as the post-processing step after geometric reconstruction, researchers also investigate on how to jointly optimize geometric and semantic accuracy. Häne et al. [61] formulate a joint segmentation and dense reconstruction problem on voxels and show that appearance likelihoods and class-specific geometric priors help each other. Cherabier et al. [62] leverage variational energy minimization method for regularization to capture complex dependencies between the semantic labels and the 3D geometry. Guo et al. [63] jointly optimize the geometry and semantics by predicting the implicit neural representations of the signed distance, color and semantic field.

Our approach is most closely related to the concurrent work on semantic mesh mapping [64]. Our formulation utilizes sparse depths obtained from keyframe-based SLAM and emphasizes the interaction of geometric reconstruction and semantic segmentation in 3D mesh reconstruction.

Outdoor terrain structure can be viewed as a 2.5-D surface with height variation. Hence, we initialize a flat mesh surface and change the surface elevation based on the sparse depth measurements. The flat mesh is initialized with regular-grid vertices ($n=1024$ in our experiments) over the image plane, and the edges and the faces connecting the vertices. See Fig. 4 for an illustration. Subsequently, our mesh reconstruction approach only optimizes the mesh vertices and keeps the edge and face topology fixed. The initialized mesh $\mathcal{M}^{\text{int}}=(\mathbf{V}^{*},\mathbf{0})$ is used as an input to the mesh refinement stage, described in Sec. V-B, V-C. Since we do not update ${\cal E},{\cal F}$, we will omit them for simplicity.

We constrain the mesh vertex deformation to the $z$-axis to change the vertex heights only. The coordinates of the $i$-th vertex of the flat mesh, $\mathbf{v}_{i}=[v^{x}_{i},v^{y}_{i},1]$, are divided by a scalar inverse depth $\lambda_{i}$ to obtain the $i$-th vertex coordinates $[v^{x}_{i}/\lambda_{i},v^{y}_{i}/\lambda_{i},1/\lambda_{i}]$ of the initialized mesh. We concatenate $\lambda_{i}$ to obtain a vector $\boldsymbol{\lambda}\in\mathbb{R}^{n}$ of all vertex inverse depths.

Any point $\mathbf{p}$ on the mesh surface that lies in a specific triangle can be represented as a convex combination $\mathbf{p}=b_{i}\mathbf{v}_{i}+b_{j}\mathbf{v}_{j}+b_{k}\mathbf{v}_{k}$ of the triangle vertices $\mathbf{v}_{i},\mathbf{v}_{j},\mathbf{v}_{k}$ with weights $b_{i},b_{j},b_{k}\in[0,1]$ such that $b_{i}+b_{j}+b_{k}=1$. The vector $[b_{i},b_{j},b_{k}]^{\top}$ is called the barycentric coordinates of $\mathbf{p}$. We use barycentric coordinates to relate the sparse depth measurements $\mathbf{D}^{s}$ to the vertex inverse depths $\boldsymbol{\lambda}$, which is equivalent to a linear interpolation.

Let the valid measurements in the sparse depth image $\mathbf{D}^{s}$ be $\{(i,j),\mathbf{D}^{s}_{ij}\}$, where $(i,j)$ are the pixel coordinates and $\mathbf{D}^{s}_{ij}$ are the corresponding depth measurements. Each pixel $(i,j)$ falls within one triangle of the flat 2D mesh (see Fig. 4). Let $\mathbf{b}_{ij}\in\mathbb{R}^{n}$ be the barycentric coordinates of pixel $(i,j)$, where at most three elements of $\mathbf{b}_{ij}$, corresponding to the three triangle vertices, are non-zero. The inverse depth $1/\mathbf{D}^{s}_{ij}$ is related to the vertex inverse depths $\boldsymbol{\lambda}$ through the barycentric coordinates [71], $\mathbf{b}_{ij}^{\top}\boldsymbol{\lambda}=1/\mathbf{D}^{s}_{ij}$. Stacking these equations for all valid pixels $(i,j)$ in $\mathbf{D}^{s}$, we obtain:

where $\boldsymbol{\rho}$ is a vector of the valid inverse depth measurements in $\mathbf{D}^{s}$ with elements $1/\mathbf{D}^{s}_{ij}$. Using Laplacian regularization as in (5), we formulate a least-squares problem in $\boldsymbol{\lambda}$:

The problem in (11) has a closed-form solution:

The regularization term, not only makes the initialized mesh smoother, but also guarantees that the solution exists even when the number of sparse depth measurements is smaller than the number of mesh vertices. Since the 2D mesh projection and $\mathbf{L}_{n}$ are pre-defined, the problem can be solved very efficiently, e.g., in less than $0.1$ sec for a mesh with $1024$ vertices. Given $\boldsymbol{\lambda}^{*}$, we obtain an initialized mesh $\mathcal{M}^{\text{int}}$ with each vertex coordinate as $[v^{x}_{i}/\lambda_{i}^{*},v^{y}_{i}/\lambda_{i}^{*},1/\lambda_{i}^{*}]$.

Initialization using the sparse depth measurements only provides a reasonable mesh reconstruction but many details are missing. In the geometric refinement stage, we use a learning approach to extract features from both the 2D image and 3D initial mesh and regress mesh vertex spatial coordinate residuals. The ground-truth depth maps are used for supervision.

The photometric image information is useful for mesh refinement since man-made objects have sharp vertical surfaces, while natural terrain has noisy but limited depth variation. The sparse depth measurements also provide information about areas with large intensity variation. Inspired by Mesh R-CNN [32], we design a network that extracts features from the 2D image, associates them with the 3D vertices of the initial mesh, and uses them to refine the vertex spatial coordinates. Our network has 3 stages: feature extraction, vertex-image feature association, and vertex graph convolution.

To further enrich the environment representation, we introduce semantic information in the mesh reconstruction. By assigning per-vertex semantic features $\mathbf{C}$, we can interpolate the semantic information over the whole mesh using barycentric coordinates (see Fig. 6). To obtain a 2D semantic segmentation image from the mesh, we use the differentiable semantic renderer $\rho_{\mathbf{S}}$ introduced in (7). Both the vertex spatial coordinates $\mathbf{V}$ and the semantic features $\mathbf{C}$ can affect the rendered 2D semantic segmentation image $\rho_{\mathbf{S}}(\mathcal{M})$. Hence, by optimizing the semantic loss in (7), we can refine both the semantic features and the geometric structure of the mesh.

We first obtain 2D semantic segmentation features $\phi_{\text{deep}}(\mathbf{I};\boldsymbol{\theta}_{2Dsem})$ from the RGB image $\mathbf{I}$ using the DeepLabv3 model [73] with parameters $\boldsymbol{\theta}_{2Dsem}$. Then, we associate the mesh vertices to the 2D semantic feature map to get initial mesh vertex semantic features:

Fig. 6 illustrates the mesh vertex association with respect to the 2D semantic segmentation features. In the semantic refinement stage, we regress a semantic residual $\Delta\mathbf{C}$ for the semantic features. We use 3 layers of graph convolution $g^{\mathbf{C}}_{1}$,$g^{\mathbf{C}}_{2}$,$g^{\mathbf{C}}_{3}$. We also use the $\mathbf{V}_{g_{\text{in}}}$ extracted from ResNet in (15) as an input to the first graph convolution layer. Additionally, we concatenate the initial mesh vertex semantic features $\mathbf{C}$ in (17) to the graph convolution input:

where $\Delta\mathbf{C}\in\mathbb{R}^{n\times s}$ is the matrix of semantic residuals and $\mathbf{W}^{\mathbf{C}}_{1},\mathbf{W}^{\mathbf{C}}_{2}$ are two matrices as linear layers. The trainable parameters for vertex semantic graph convolution are $\boldsymbol{\theta}_{3D\mathbf{C}}=[\mathbf{W}^{\mathbf{C}}_{1};\mathbf{W}^{\mathbf{C}}_{2};\boldsymbol{\theta}_{g\mathbf{C}1};\boldsymbol{\theta}_{g\mathbf{C}2};\boldsymbol{\theta}_{g\mathbf{C}3}]$.

Now we can perform the joint geometric and semantic refinement. All trainable parameters for the 3D graph convolution are $\boldsymbol{\theta}_{3D}=[\boldsymbol{\theta}_{3D\mathbf{V}};\boldsymbol{\theta}_{3D\mathbf{C}}]$. An illustration of the joint geometric and semantic refinement is provided in Fig. 7. For the initial mesh ${\cal M}(\mathbf{V},\mathbf{0})$, we first estimate the geometric residuals $\Delta\mathbf{V}$ (16) from Sec.V-B. On the geometrically refined mesh ${\cal M}(\mathbf{V}+\Delta\mathbf{V},\mathbf{0})$, we initialize the semantic features as in (17) to get ${\cal M}(\mathbf{V}+\Delta\mathbf{V},\mathbf{C})$. Then we estimate the semantic residuals $\Delta\mathbf{C}$ (18). The final joint geometric and semantic refined mesh is $\mathcal{M}^{\text{ref}}=(\mathbf{V}+\Delta\mathbf{V},\mathbf{C}+\Delta\mathbf{C})$.

Given semantically annotated meshes obtained by our model from each camera view, a global mesh of the whole environment can be obtained by transforming each local mesh to the global frame using the camera pose trajectory and merging it into a combined global mesh. We design an approach to incrementally merge local meshes into a global mesh. Given a new local mesh obtained from a camera view with a known pose and the current global mesh, we update the global mesh by merging information from the local mesh.

First, we refine the global mesh vertices. We transform the global mesh to the local camera frame and project it onto the 2D image plane. If the resulting 2D global mesh covers an area over a certain threshold (e.g., 70%), we regard the local frame as duplicate and proceed to the next frame. Otherwise, we determine the overlapping parts between the 2D projections of the global and local meshes. We choose the vertices of the overlapped global mesh as a source point cloud and sample a point cloud from the overlapped local mesh vertices as a target point cloud. We perform non-rigid point cloud registration between the source and target point cloud using the Coherent Point Drift (CPD) algorithm [74]. Through this non-rigid transformation, we deform and refine the global mesh geometry based on the local mesh information.

Second, we introduce new vertices and faces into the global mesh from the non-overlapping region of the local mesh. We remove the faces of the local mesh that overlap with the global mesh projection on the image plane. Through this step, we decouple the global mesh and the local mesh because their 2D projections do not intersect with each other any longer. We perform 2D constrained Delaunay triangulation [75] over the global and local mesh projections, keeping the edges of existing triangles in tact. Through this step, we connect the global mesh and the local mesh to obtain a new global mesh, which is lifted back to 3D using the vertex depth values and the known camera pose.

Fig. 8 and Fig. 9 demonstrate the mesh merging process, which results in a single consistent global mesh and removes artifacts such as double layers in naïve mesh merging.

Our mesh reconstruction approach requires ground-truth depth and semantic segmentation data for supervised training, which are generally not available and challenging to obtain from RGB aerial images. We used photo-realistic point cloud models covering several km² reconstructed from real aerial images in WHU MVS/Stereo and SensatUrban dataset to render RGB, depth, and semantic segmentation images. This provides accurate depth and semantic supervision data, while keeping the RGB images realistic, which is important for real-world applications of our model. We also use data from the synthetic STPLS3D dataset, which has more variation in the scene layout and the texture. We divide the large point cloud models in each dataset into different regions and generate different image sequences over them. Each camera trajectory follows a sweeping grid-pattern, which is common in drone flight planning (see Fig. 10). The camera trajectories are chosen to ensure enough image overlap for tracking and sparse depth reconstruction. RGBD images with resolution $512\times 512$ are rendered along the trajectory from the ground-truth point cloud using PyTorch3D [66]. We keep the RGB aerial image resolution at around $0.2$ meter/pixel. When semantic labels are available in the point cloud model, we also render semantic segmentation images with the same size as the RGBD images. All the data sequences are available in the Supplementary Material.

During training, we use 1000 sparse depth measurements per image and generate a mesh model with 576/1024/2025 vertices. For the WHU dataset, the weights of the loss function in (9) are set to $[w_{2},w_{3},w_{\mathbf{V}},w_{{\cal E}},w_{\mathbf{S}},w_{\mathbf{C}}]=[3,1,0.5,0.01,0,0]$. For the SensatUrban and the STPLS3D dataset, the weights are set to $[w_{2},w_{3},w_{\mathbf{V}},w_{{\cal E}},w_{\mathbf{S}},w_{\mathbf{C}}]=[5,1,0.5,0.01,5,0.5]$ for the joint geometric-semantic training (Sec. V-C). For the geometric training (Sec. V-B), the last two weights are set to be $0$. The loss weights are decided through evaluation on the validation set. We use the 2D loss $\ell_{2}$ in (2) as the metric to choose the best model on the validation set. The Chamfer distance $d$ in the $\ell_{3}$ loss (3) is computed using 10000 samples.

For the WHU experiments in Sec. VI-C, we use ResNet-18 for the 2D feature extraction. For the remaining ones including the generalization experiments, we use ResNet-34. The ResNet is initialized with the pretrained weights on ImageNet-1K. The ResNet and GCN parameters of our model are optimized jointly during the mesh refinement training using the Adam optimizer [79] with initial learning rate of $0.0005$ for $100$ epochs. For the semantic reconstruction task, we first train a DeepLabv3 model with ResNet-50 backbone [73] alone for 2D semantic segmentation on the SensatUrban training set. The ResNet-50 is initialized with the pretrained weights on ImageNet-1K. We use the Cross Entropy loss for training and set the class weight as [ground, vegetation, building, traffic road] = [$1,2,3,3$]. During the mesh semantic refinement step, we use the Dice loss in (7) and keep the same per-class weights. We use three graph convolution stages for the WHU dataset and two graph convolution stages for the SensatUrban and the STPLS3D dataset. For the joint geometric-semantic training, we concatenate two graph convolution stages, where the first stage predicts the geometric residual only and the second stage predicts both the geometric and the semantic residuals. All trainable parameters of the model in (1) are $\boldsymbol{\theta}=[\boldsymbol{\theta}_{2D};\boldsymbol{\theta}_{2Dsem};\boldsymbol{\theta}_{3D}]$, including the parameters of the 2D features extraction, 2D semantic segmentation, and 3D graph convolution models.

Our experiments report the $\ell_{2}$ error in (2) and the $\ell_{3}$ error in (3) for the reconstructed meshes. The $\ell_{2}$ error emphasizes the accuracy of the projected depth, while $\ell_{3}$ emphasizes the regions of large depth variation.

For comparison, we define a baseline method that triangulates the sparse depth measurements directly to build a mesh. The baseline method performs Delaunay triangulation on the 2D image plane over the depth measurements and projects the flat mesh to 3D using the measured vertex depths. We refer to the baseline method as sparse-depth-triangulation (SD-tri). SD-tri defines vertices at all sparse depth measurements (500, 1000, or 2000) and, hence, may produce meshes with different number of vertices compared to other models.

First, we perform geometric reconstruction on the WHU dataset. The quantitative results from the comparison are reported in Table I. All models are trained with 1000 sparse depth measurements and directly generalize to different numbers of sparse depth measurements. We compared three options for the 2D inputs provided to the mesh refinement stage: an RGB image only (RGB, 3-channels), an RGB image plus rendered depth from the initial mesh (RGB+RD, 4-channels), and an RGB image plus rendered depth from the initial mesh plus Euclidean distance transform (EDT) obtained from of the sparse depth measurements (RGB+RD+EDT, 5-channels). The model using RGB-only does not perform as well as the other two. The RGB+RD+EDT model has the best performance according to the $\ell_{2}$ error metric. The RGB+RD method has similar performance in the $\ell_{2}$ metric and smaller $\ell_{3}$ error compared to RGB+RD+EDT. The RGB+RD model is used to generate our qualitative results in Fig. 11, 12, 13 with 1024-vertex meshes because it offers good performance according to both error metrics.

At the bottom of Table I, we evaluate the mesh reconstruction accuracy with noisy sparse depth measurements obtained from OpenSfM. The measurements are noisy due to feature matching errors, local minima during bundle adjustment, and the simple projective camera model used for optimization. The average per image errors of the 500/1000/2000 sparse depth measurements were 1.011/1.017/1.023 meters, respectively. The baseline SD-tri method performs well in a noiseless setting but degenerates drastically when noise from the SfM feature reconstruction is introduced. In contrast, our model is more robust to noise due to two factors. First, our mesh initialization and refinement stages both include explicit mesh regularization terms (in (5) and (6)). Second, the image features extracted during the mesh refinement process help distinguish among different terrains and structures. The latter is clear from the improved accuracy of the refined, compared to the initialized, meshes. We also report the performance using a mesh with only 576 vertices. When the depth measurements are noisy, the 576-vertex mesh has lower $\ell_{2}$ loss compared with the baseline method with similar number of vertices. It even has lower $\ell_{3}$ loss compared with meshes with more vertices generated from the baseline method.

Qualitative results are presented in Fig. 11 and 12. Compared with SD-tri and initialized meshes, the refined meshes have smoother boundaries on the side surfaces of the buildings. The guidance from the image features allows the refined meshes to fit the 3D structure better. Fig. 13 shows a global mesh reconstruction obtained by transforming and merging 12 camera-view mesh reconstructions. The local meshes are transformed to global frame using the camera keyframe poses and no post-processing is used to merge them into a single global mesh.

On the SemsatUrban dataset, we first perform geometric reconstruction with the same settings as in the WHU dataset. We train three models with different numbers of mesh vertices: $576=24^{2}$, $1024=32^{2}$ and $2025=45^{2}$. The quantitative results are reported in Table II. As the number of sparse depth measurements increases, the baseline SD-tri method has better accuracy because the number of mesh vertices also increases. Our initialized meshes with fewer vertices are comparable with the SD-tri mesh, and the refined meshes are much better, especially according to the 3D metric $\ell_{3}$. This shows that the joint 2D-3D loss in (9) enables our model to capture 3D structure details. Comparing the number of input depth measurements, we find that around 2000 measurements on the $512\times 512$ image provide the best performance, while more do not noticeably improve the results. Regarding the number of mesh vertices, all three mesh sizes perform well. While we can see that the 1024-vertex mesh is generally better than 576-vertex mesh, the 2025-vertex mesh does not show an advantage over the 1024-vertex mesh. This indicates that good accuracy can be achieved with a light-weight storage-efficient mesh model.

We choose the 1024-vertex mesh to perform joint geometric-semantic reconstruction using 1000 sparse depth measurements. To evaluate the semantic reconstruction, we render a 2D semantic image from the mesh reconstruction and calculate the per-class Intersection over Union (IoU). For comparison, we report the IoU of the DeepLabv3 2D semantic segmentation model (named 2D Seg), the direct projection of the 2D semantic segmentation image onto the initial mesh as in (17) (named Geo Init) and the semantic segmentation projection onto the geometrically-refined mesh (named Geo Refine). Only 2D Seg is using a dense semantic image while the other methods store semantic features on the mesh vertices and interpolate through the semantic mesh renderer. As we can see in Table III, our semantic residual refinement model improves the semantic segmentation performance compared to the direct projection of the 2D semantic segmentation image. Our approach also outperforms 2D Seg on most of the categories (ground, building, traffic Road) even thought it is using only 0.4% of the points to store the semantic information (1024 mesh vertices vs $512\times 512$ segmentation image). Using a compact mesh with few vertices improves the computation and memory efficiency but restricts the mesh from modeling small regions (e.g., small objects like cars, street furniture). Our approach can capture additional semantic categories if the number of mesh vertices is increased, e.g., to $64^{2}=4096$ or more, and a semantic segmentation network capable of segmenting small regions is used. Achieving this does not require any changes to the model architecture but only retraining the model parameters.

Further, we investigate whether the semantic mesh refinement affects the geometric reconstruction quality. In Table IV, we can see that our joint geometric-semantic mesh reconstruction achieves better geometric accuracy compared with purely geometric training. This can be explained by the fact that the semantic category information serves as regularization for the geometric properties. The results show that the geometric and semantic information help each other. More qualitative results for single-image reconstruction are provided in Fig. 14 and 15. Compared with SD-tri and initialized mesh, the refined mesh achieves higher reconstruction accuracy. The semantic refinement can improve the 2D semantic segmentation results. We can see that some noisy classification labels are removed after the refinement.

To evaluate our global mesh-merging method based on CPD and Delaunay triangulation, we compare it with a simple stacking method that transforms the same set of local meshes to the global frame and simply treats them as a single global mesh. For each scene, about 10% of the frames are used to generate a global mesh. The global mesh is rendered with respect to all frames, including ones that were not used for its construction, to compute the error. Therefore, the error is larger than the per-frame prediction evaluation. The quantitative results are reported in Table V. The stacked global mesh has many double layers in regions where the local meshes overlap while our method successfully merges the local meshes into a consistent global mesh, which improves the reconstruction accuracy, especially according to the 3D Chamfer error metric. Fig. 16, shows the reconstruction of three global metric-semantic meshes using data from the SensatUrban dataset [58].

We also evaluated our model on the synthetic STPLS3D dataset. We use a 1024-vertex mesh to perform geometric-only and joint geometric-semantic reconstruction using 1000 sparse depth measurements per image. The quantitative results are shown in Table VI. Our joint geometric-semantic mesh reconstruction method out-performs geometric-only mesh reconstruction, which verifies the effectiveness of fusing both geometric and semantic information.

In this section, we evaluate the generalization ability of our model, trained on one dataset and applied to another. To align the datasets, we regenerated the RGB images and the sparse depth measurements on the WHU dataset to follow the same camera intrinsic parameters and trajectory patterns in the SensatUrban and the STPLS3D dataset so that there are 660 frames for each trajectory and ORB-SLAM3 is used to estimate sparse depth measurements and camera keyframe poses. It is challenging to achieve zero-shot generalization, so we also include a finetuning step. During finetuning, we only use 10% of the target domain training set and train for 30 epochs. A validation set (10% of the target domain validation set) is used to choose the best model with the 2D loss $\ell_{2}$ as the metric. Usually, models trained on larger datasets show better generalization ability. We choose to use a model trained on WHU to generalize to SensatUrban and a model trained on STPLS3D to generalize to both WHU and SensatUrban. The average per image sparse depth errors in meters for 1000 depth samples were 1.771 on STPLS3D, 2.460 on WHU, and 1.597 on SensatUrban. The sparse depth measurements are generated through ORB-SLAM3.

First, we evaluate how the model trained on STPLS3D generalizes to WHU. The geometric error is reported in Table VII. The WHU dataset is more challenging due to the presence of denser and taller ($>30$m) buildings. The camera intrinsics and the flight pattern and height are different compared to the data generated in Sec. VI-C so the numbers in Table VII are not directly comparable with Table I. Zero-shot generalization does not work for WHU, which is understandable given the large domain gap. The STPLS3D synthetic dataset uses scene layouts extracted from a U.S. Geological Survey (USGS) which covers cities in the United States, while the WHU dataset is collected in a Chinese city. After finetuning with only 10% of the original WHU training set, our model generalizes well to WHU, and even outperforms the model trained purely on WHU.

Next, we evaluate how models trained on WHU and STPLS3D generalize to SensatUrban. The results are presented in Table VIII and Table IX. We report only geometric error for the model trained on WHU. We can see that zero-shot generalization from WHU to SensatUrban improves the initialized meshes, while a finetuned model performs even better. We train a geometric-only model and a metric-semantic model on STPLS3D. In terms of geometric loss, both STPLS3D models generalize well to SensatUrban and their performance after finetuning is close to that of a model trained on SensatUrban. However, the metric-semantic model is slightly worse than the pure geometric model. In terms of semantic segmentation performance, zero-shot generalization does not perform well and especially fails on the traffic road category. After finetuning, the metric-semantic model can largely close the gap between itself and the SensatUrban model. Given that the RGB images from the synthetic scenes in STPLS3D have very different appearance, it is understandable that the semantic model that heavily relies on the RGB image might be harder to generalize compared to the geometric-only model.

These experiments demonstrate promising generalization ability of our mesh reconstruction method, using limited data to finetune or even without finetuning in some cases. The model generalizes better in terms of geometric reconstruction than in terms of semantic classification. Nevertheless, it is exciting to see that a model trained on a synthetic dataset (STPLS3D) can generalize well to real data. This makes it possible to achieve good performance by training a model with inexpensive synthetic data that comes with free ground-truth labels and finetuning on a small set from the target domain.

In this section, we compare our approach with depth completion, multi-view stereo, and NeRF techniques on the SensatUrban dataset [58] with 1000 sparse depth measurements. The qualitative results are shown in Fig. 17. The quantitative results are shown in Table X. To evaluate the reconstruction quality of the different methods comprehensively, we report multiple 2D and 3D reconstruction accuracy metrics [37]. The threshold distance for 3D precision and recall is set to 0.5m due to the large scale of the reconstructed mesh.

Depth completion methods take a sparse depth image and other inputs, such as an RGB image, and recover a dense depth image. While there are many depth completion algorithms, few focus on the aerial image domain. We compare against the Sparse-to-Dense method [3], an end-to-end deep learning regression model, because it considers a similar problem setting and uses similar feature extraction as our approach. The Sparse-to-Dense model consists of a ResNet feature extraction encoder and per-pixel depth regression decoder. We trained both our model and Sparse-to-Dense with a ResNet-34 feature extractor, thus focusing the comparison on the performance of the pure 2D learning and per-pixel depth regression of Sparse-to-Dense versus the joint 2D-3D learning for mesh refinement of our method. The results in Fig. 17 and Table X show that the Sparse-to-Dense model is not as accurate as our method on the SensatUrban dataset, and it is beneficial to utilize our joint 2D-3D learning technique. At least on this dataset, it is challenging for Sparse-to-Dense to regress an accurate dense depth map, while, using the same 2D feature extraction network, our mesh initialization and refinement method performs better at reconstructing the 3D scene.

We also compare our method with COLMAP [36], a multi-view stereo technique that recovers 3D structure from a series of calibrated images using pixelwise view selection for depth and normal estimation. In this comparison, we used ground-truth camera poses and skip the SfM step. For each frame, we manually select neighboring frames within a radius of $50$ m for multi-view stereo matching. We use the reconstructed dense depth images to obtain a global point cloud, generate a global mesh, and crop the mesh at each camera pose to obtain local meshes associated with each frame. For the meshing step, we compared Poisson reconstruction [80] and Delaunay mesh reconstruction [81] and found that due to point cloud noise the Delaunay reconstruction performs better. The COLMAP is the dense depth estimation result, while the COLMAP Mesh is the subsequent meshing result. For COLMAP, we convert the dense depth images to a point cloud and sample 10000 points to compute the $\ell_{3}$ error. For COLMAP Mesh, we render the local mesh to get a rendered depth image to compute the $\ell_{2}$ error. The results are shown in Fig. 17 and Table X. The COLMAP method generally has better depth reconstruction measured by $\ell_{2}$ error, while our method has lower $\ell_{3}$ error because of the implicit regularization in our mesh reconstruction. The 3D error $\ell_{3}$ can be large when outliers appear in the reconstruction. COLMAP achieves better reconstruction at the cost of heavy computation for the MVS step. It takes around 4 seconds per frame to recover a dense depth image using GPU, while meshing requires additional time. Our method is much faster, with $0.07$ s per frame on a desktop with GeForce RTX 2080 Ti GPU and $0.45$ s per frame on a Jetson AGX Xavier edge computing platform. When it comes to online mesh reconstruction on a resource-constrained platform, our method offers an advantage over MVS.

Finally, we compare with Nerfacto [52], a NeRF model that combines components from recent NeRF papers to achieve a balance between speed and quality. As a NeRF model, Nerfactor takes posed RGB images and constructs an implicit 3D scene represented by a deep neural network. We used ground-truth camera poses for each image and trained separate Nerfacto models for each test image sequences. One in ten frames was chosen as an evaluation image during training. Notice here no depth images are used for the NeRF training. We observed that depth images rendered directly from the trained Nerfacto model have inconsistent depth across frames. Therefore, we exported a mesh model using the Poisson surface reconstruction [80] implemented in Nerfstudio [52]. Nerfacto exports meshes with different vertex density. A dense mesh has about 20000 vertices for each camera view, while a sparse mesh has about 1000, which is similar to our method’s mesh vertex density. To compute the 2D metrics in Table X, we rendered depth images from the mesh. The evalution results for Nerfacto are shown in Fig. 17 and Table X. Qualitatively, Nerfacto has similar reconstruction accuracy with our method but the quantitative metrics indicate that it is not as good as our method. Furthermore, Nerfacto needs to be trained for each novel environment and the training can hardly meet real-time requirement, while our method can make the inference faster.

Sec. VI-C compared the effect of using RGB, rendered depth, and Euclidean distance transform as inputs for the mesh reconstruction model. This section reports additional ablation studies on the SensatUrban dataset. We use a 1024-vertex mesh model and 1000 sparse depth measurements for training and testing. We evaluate the effects of mesh initialization, types of 2D input data, and number of graph convolution stages on the geometric mesh reconstruction accuracy. We also evaluate the performance effect of joint metric-semantic training and the choice of a semantic loss function.

The reconstructed mesh model is a more efficient representation than a dense depth image. A dense depth image requires $512\times 512\approx 0.26M$ parameters, and a semantic image also requires the same number of parameters. Our mesh model with fixed face topology only needs storage of the 3D vertex coordinates and the semantic labels. With 1024 vertices, our semantic mesh model requires only $2\%$ of the depth and semantic image parameters to obtain a high-fidelity reconstruction of a camera view. Our model has about 21M parameters (ResNet and GCN) and takes about 3GB GPU memory during inference. We report the inference time of our model on different computation platforms in Table XII. The results show that our mesh reconstruction algorithm can achieve 2 Hz on an embedded NVIDIA Jetson AGX Xavier computer, making it applicable for real-time deployment onboard a robot system. Regarding timing the baseline algorithm for sparse-depth triangulation, we evaluated its run-time frequency to be at around 20 Hz on the same platform.

Our 3D metric-semantic mesh reconstruction algorithm can run efficiently on an embedded computer but as a result the number mesh vertices used for reconstruction is limited, which in turn affects the geometric reconstruction accuracy. Furthermore, local meshes are generated using only a single camera frame without multi-view constraints, making it challenging to achieve consistent mesh merging into a global model. Potential avenues for future work that may improve the reconstruction quality include adaptively increasing the mesh vertices depending on the image feature distribution, considering techniques like deformable convolution [84] for associating the 3D mesh vertices with the 2D image features, utilizing sparse depth measurement uncertainty (e.g., keypoint covariances provided by SLAM) for weighted interpolation during the image features to vertex association, and improving the global mesh merging approach with multi-view constraints.