M2fNet: Multi-modal Forest Monitoring Network on Large-scale
Virtual Dataset

Precision forestry relies on accurate measurements of forest biometrics (e.g., diameter, height, form, etc.). Conventional field-based forest inventories are limited in spatial coverage and can be prohibitively time- and labor-intensive [2]. Recent studies have sought to automate the acquisition and quantification of key forest biometrics through remote sensing techniques.

Airborne laser scanning (ALS) has been extensively used for area-based estimation of aboveground biomass (AGB) using regression models on metrics like canopy height [29, 31]. While verified against field measurements, ALS coverage can remain spatially sparse. Satellite imaging provides complementary wide-area spectral data, which can predict AGB via empirical model fitting [9, 1]. However, resolution limits the diagnosis of fine-grained forest properties. To capture intra-stand details, terrestrial laser scanning (TLS), structure-from-motion (SfM), and neural radiance fields (NeRF) are alternative approaches to construct 3D point clouds of sample plots [15, 23, 18, 16]. Though robust, plot-based sampling suffers from generalization and edge artifacts [22]. Meanwhile, unmanned aerial vehicles (UAVs) enable rapid, flexible surveys, but remain constrained by flight endurance and line-of-sight restrictions [25, 28].

Our work contributes a large-scale virtual forest data by generating dense, perfectly annotated forest data, which can be used for training on various deep learning models. We demonstrate applications from individual tree delineation to property prediction using diverse synthesized sensing modalities.

Reliable detection and segmentation of individual trees from imagery is a fundamental task in forest monitoring. Previous work in tree crown segmentation relies on hand-crafted features and morphological processing on color aerial imagery [4, 24]. With the advent of deep convolutional neural networks (CNNs), data-driven approaches have demonstrated superior performance.

For tree detection, recent methods fine-tune object detection networks such as Faster R-CNN using RGB imagery to generate 2D bounding boxes around tree crowns [27, 20]. Light detection and ranging (LiDAR) point clouds have also been used to detect individual trees represented as 3D bounding cylinders [5]. Meanwhile, semantic segmentation formulations can produce pixel-wise masks that label tree trunks and canopy extensions in RGB-D data [13, 19].

However, existing datasets for training and evaluating such models are limited in diversity, scalability, and annotation quality [12, 30]. Here, we introduce a large-scale photorealistic synthetic forest rendering engine to automatically generate dense ground-truth data for tree detection and segmentation tasks. We demonstrate the value of our simulation-to-reality paradigm on benchmark models.

In order to precisely segment the regions of interest of the tree trunks from the high-quality tree models collected, Autodesk Maya was utilized to separate each tree model in FBX or OBJ format into its constituent trunk, branch, and leaf components. Only the trunk portions were rendered for the segmentation camera. Individual tree models of different species were populated within the designated terrain map with randomly assigned trunk densities and locations. Within this same terrain, a set of camera trajectories were defined, and the multi-sensor camera was moved along these trajectories to obtain synchronized multimodal collections (e.g., RGB, depth, mask) at each timestamp. This ensured proper alignment of the multimodal image sets. As a post-processing step to generate the COCO annotation files, OpenCV was employed to automatically detect object contours and bounding boxes, which were then exported as the final outputs. As a result, 50,000 video frames containing 3,000 for testing were obtained. Figure 1 gives some example data from the generated dataset.

The Figure 2 showcases our project setup based on Unreal Engine 4 for automated data generation within a forest environment. The virtual landscape features adjustable elements like trees, stones, and grass. A camera is in place to capture the scene, which is key for creating diverse datasets. The interface indicates that lighting needs to be rebuilt as pending update to enhance realism. This setup facilitates the generation of varied scenarios for simulation purposes.

The Figure 3 outlines a three-step data generation process using Unreal Engine. In the Model Preparation phase, raw models are imported into Unreal Engine for setup. Scene Generation follows, utilizing a random generation algorithm and a moving camera within the environment to create diverse scenarios. Finally, the Data Generation step outputs various data types, such as RGB images, masks, depth maps, and LiDAR. This structured approach enables the creation of a comprehensive dataset for simulation and analysis.

The Figure 4 displays a selection process for tree models in a simulation environment, showcasing the variety available for scene generation. The blueprint, BP_Tree, is designed to randomly select from an array of tree species, ensuring randomness and diversity in the virtual landscape. This collection of different tree models allows for a more realistic and varied representation of a forest ecosystem within the simulation.

An overview of the proposed multi-modal forest monitoring network is shown in Fig. LABEL:fig:intro_fig. The proposed M2fNet uses two independent encoders, which are hierarchical vision transformers with shifted windows (Swin) [17], to extract RGB and depth features. Considering that RGB and depth images have different emphasis on feature representation, both of which will be helpful for instance-level segmentation and detection. As the encoding stage increases, the features that can be extracted become more advanced. We then pass the features from different domains using a fusion module, which consists of a concatenation, a layer normalization, and a convolution layer. The output of the RGB encoder is added to the output of the fusion module to improve the segmentation performance on small trees and fine tree boundaries.

The fused features are then fed into a pixel decoder to transform the low-resolution features into high-resolution feature embeddings. Similar to Mask2former [3], we employ a query-based Transformer decoder with masked attention blocks to learn the semantic-level features from the interested tree objects. Specially, the tree query $Q_{query}\in\mathbb{R}^{N\times d}$ is updated by collecting the local tree features and context information within the foreground regions through self-attention. To deal with small and distant tree instances, high-resolution features from the pixel decoder are fed into the Transformer decoder layer. In this way, after obtaining the aggregated output query $Q_{out}$ and the high-resolution feature $Z$ from the pixel decoder, we predict the tree masks $\mu$ by multiplying the outputs from the two residual structures and adding simple linear layers for the different prediction heads. The predicted mask can be formulated as: $\mathcal{M}=MLP(Z\cdot Q_{out})$, where $MLP$ is the multi-layer projection.

The loss is a combination of the binary cross-entropy loss and the dice loss for mask prediction and the smooth L1 loss for box regression. The total loss can be provided as:

To quantify the efficacy of the proposed M2fNet architecture using the created simulation data, we employ standard evaluation metrics including mean average precision (mAP) at an IoU threshold of 50%. As evident in the preliminary results depicted in Figure 5, employing our dataset for training precipitates considerable improvements in real-world segmentation and detection performance, especially for tree trunk masks. Furthermore, the multi-modal M2fNet outperforms networks relying solely on RGB imagery as input. Collectively, these outcomes underscore the potential for advancing machine learning algorithms, including low-level vision and multimodal frameworks, in unstructured forest environments and other analogous scenarios, by capitalizing on synthetic, controllable datasets.

In addition to the quantitative evaluation described in the previous section, we also conducted a user study aimed at investigating whether the simulated forestry scenes are realistic enough to support as a strong and useful pre-training for subsequent forestry education and research. While user study for measuring simulation system has been approached in VR training systems in the past [8, 11], to the best of our knowledge so far has rarely been used in the evaluation of forestry simulator. More specifically, a different level of simulated forest scenes ranging from the most fake to the most realistic (the real-world captured forestry) has been included, covering a broad range of situations to be investigated in our user study. 15 participants with different backgrounds (computer graphics, design, engineering, forestry) are invited in the study to evaluate the two questions 1 to 5, where the questionnaire for measuring the fidelity of our simulated data can be found in Table 1.