Multimodal Collaboration Networks for Geospatial Vehicle Detection in Dense, Occluded, and Large-Scale Events

Convolutional Neural Networks (CNNs) feature a unique architecture of local weight sharing, significantly benefiting image processing and other areas by enabling the extraction of highly discriminative object features from input images. Currently, CNNs are the widely used method in the field of remote sensing [18, 19]. Unimodal feature hierarchical enhancement (Uni-Enh) involves dual-stream feature learning via CNNs and features hierarchical enhancement of each stram to more effectively capture intramodal relationships.

Firstly, we introduce a dual-stream CNN-based network for feature learning. Each stream in the network is composed of several CNN blocks, with each block consisting of a $3\times 3$ convolutional layer, Batch Normalization (BN), and Leaky ReLU activation. We define ${\boldsymbol{X}}\in{\mathcal{R}^{{d}\times M\times N}}$ as the CNN features. To distinguish, $\mathbf{X}_{RGB}\in\mathcal{R}^{{d_{RGB}}\times M\times N}$ and $\mathbf{X}_{H}\in\mathcal{R}^{{d_{H}}\times M\times N}$ are used to represent the feature maps of the RGB image and the height map (H), respectively. ${d_{RGB}}$ and ${d_{H}}$ represent the number of RGB stream channels and the number of height map stream channels, respectively. $M\times N$ denotes the feature map size. The output of the $l$-th layer of MuDet is denoted as $\boldsymbol{X}^{(l)}$.

where $\boldsymbol{X}^{(l)}$ denotes the feature maps of the $l$-th layer. $f$ is a nonlinear activation function. $N$ indicates the layer number of CNN. $\boldsymbol{W}^{(l)}$ and ${\boldsymbol{b}^{(l)}}$ are the learned weights and biases of the $l$-th layer, respectively.

Then, we develop a hierarchical enhancement strategy to amplify the distinctive features of each stream and integrate the outcomes into a cross-attention mechanism, thereby aiming to more precisely capture intramodal relationships.

For the RGB stream, we measure the grayscale values of the RGB image $\boldsymbol{X}_{RGB}^{(0)}$ and employ gamma transformation with diverse coefficients to refine details across both low and high grayscale spectrums. Thus, the input of the RGB stream is

where $A$ is a constant and $\gamma$ represents the gamma transformation function.

For the height map stream ${\boldsymbol{X}_{H}^{(0)}}$, we employ grayscale slicing to emphasize the height information of foreground objects while masking the height values of background objects, leveraging expert prior knowledge.

where $H_{1}$ and $H_{2}$ are the experiential thresholds for background objects. $\boldsymbol{X}_{H}^{(0)}{(i,j)}$ represents the height map value at a specific position $(i,j)$. The values ${I_{0},I_{1}}$ denote distinct slicing thresholds. The constants $C_{\min}$ and $C_{\max}$ signify the minimum and maximum height values, respectively.

The concept of cross-attention, as first introduced in the transformer architecture for language processing due to its potent semantic feature extraction and long-range feature capture capabilities [20]. It asymmetrically combines two independent sequences of embeddings, each with the same dimensions. Here, the two sequences correspond to the features of the two modalities. Given the feature map $\boldsymbol{Z}_{RGB}=\boldsymbol{X}_{RGB}^{(N_{RGB})}$ and $\boldsymbol{Z}_{H}=\boldsymbol{X}_{H}^{(N_{H})}$ of two modalities, the cross-attention mechanism is defined as follows:

where $\boldsymbol{Q}=g_{1}(\boldsymbol{Z}_{RGB})$, $\boldsymbol{K}=g_{2}(\boldsymbol{Z}_{H})$, and $\boldsymbol{V}=g_{3}(\boldsymbol{Z}_{RGB})$, with $g_{1}$, $g_{2}$, and $g_{3}$ being $1\times 1$ convolutions. Thus, $\boldsymbol{Z}_{mix}$ is also represented as follows:

Overall, the Uni-Enh block and the Mul-Lea block hierarchically enhance vehicle differentiation within each modality and interactively learn features between two heterogeneous modalities, respectively. They dynamically improve the completeness of information fusion both within and across modalities, resulting in enhanced multimodal feature discriminability and separability.

To enhance the distinction and detection of vehicles in large-scale events, we designed a hard-easy discriminative pattern. This pattern begins by calculating the confidence value of features within each modality, followed by constructing and thresholding easy-to-predict and hard-to-predict masks to accurately detect vehicles. This pattern ensures precise supervision of each modality, facilitating a more effective vehicle location. Fig. 3 shows an illustration of the hard-easy discriminative pattern. More specifically, we define the confidence value as $\boldsymbol{Conf}$,

,

where $h_{RGB}(.)$ and $h_{H}(.)$ represent the $1\times 1$ convlutions.

To accurately differentiate between hard and easy vehicles, we define a threshold $\theta$. If the vehicle confidence predicted by both the RGB stream and height map stream exceeds the threshold $\theta$, the vehicle predicted at position $(i,j)$ is classified as an easy-to-predict sample, and a mask $Mask_{easy}$ is given.

If the object confidence predicted by either the RGB branch or the height map branch is greater than the threshold $\theta$, and the other is less than $\theta$, indicating that not both modal features can detect the object, then the object predicted at position $(i,j)$ is considered a hard-to-predict sample. Thus, two masks $Mask_{RGB}$ and $Mask_{H}$ are given,

Finally, all detected vehicles can be formulated as follows:

The hard-easy discriminability strategy streamlines the differentiation between hard and easy vehicles through soft thresholding with the hard-easy mask, thereby significantly improving the separation of dense and occluded vehicles.

In this section, we employ distinct loss functions to supervise hard and easy vehicles separately.

For the easy-to-predict vehicles, we employed two loss functions: an object classification loss, denoted as $L_{cls}$, and an Oriented Bounding Box (OBB) regression loss, represented as $L_{reg}$. Specifically, the classification loss is formulated using focal loss [21], that is

where $\hat{p}$ represents the predicted probability, and $p={0,1}$ denotes the true label. In alignment with the paper [21], we set the hyperparameter $\gamma$ to 2.

To refine the regression results, the regression loss $L_{reg}$ employs the definition provided in Ref [13], utilizing OBB for more precise object localization.

where $\mathbf{l}$ and ${{\mathbf{\hat{l}}}}$ represent the distances from the sampling point to the horizontal bounding box (HBB) boundaries. $\mathbf{s}$ and ${\mathbf{\hat{s}}}$ denote the distances between the HBB vertices and the OBB vertices. $r$ and ${\hat{r}}$ are the ratios of the HBB to the OBB in terms of area. $IoU()$ signifies the Intersection over Union (IoU) between two HBBs. Define the ground truth distances ${\boldsymbol{l}}$ is composed of ${l_{1}}$, ${l_{2}}$, ${l_{3}}$, ${l_{4}}$, and the predicted distances ${\boldsymbol{\hat{l}}}$ is composed of ${\hat{l}_{1}}$, ${\hat{l}_{2}}$, ${\hat{l}_{3}}$, ${\hat{l}_{4}}$. The area of ground truth HBB is $area{}=\left({{l_{1}}+{l_{3}}}\right)\times\left({{l_{2}}+{l_{4}}}\right)$ and the area of predicted HBB is ${\widehat{area}}=\left({{{\hat{l}}_{1}}+\hat{l}}\right)\times\left({{{\hat{l}}_{2}}+{{\hat{l}}_{4}}}\right)$. Then, the Overlapping area is represented as

The area of the circumscribed HBB of the two HBBs above is represented as

The area of the union region of the two HBBs above is represented as ${U_{x,y}}=are{a_{x,y}}+{\widehat{area}_{x,y}}-area_{x,y}^{overlap}$. Thus,

Thus, the loss of easy samples is

where $N$ represents the number of samples. Figure 4 provides a detailed representation of the OBB.

For the hard vehicles, the total loss is

Ultimately, by individually supervising and learning vehicles of varying difficulty levels, the model’s optimization direction can be intentionally adjusted to further improve the detection performance of dense and occluded vehicles. Thus, the total loss is represented as

In this section, we label and present two multimodal vehicle detection benchmark datasets for remote sensing imagery. These datasets are distinguished from existing vehicle detection datasets by four unique features: 1) They are expressly crafted for multimodal vehicle detection in the context of large-scale events, with RGB information modal and height maps modal. Each dataset has varied resolutions and is collected using varied platforms. 2) They encompass densely packed and irregularly arranged objects, including a variety of vehicle styles as well as tents and branches. 3) They feature a wide range of occlusions, such as those caused by tents and branches. 4) They increase the complexity of vehicle detection due to the presence of distorted vehicles and the varied distribution of vehicles across large-scale areas.

The data annotation method employed utilizes the oriented bounding box (OBB) format, represented as $(x_{c},y_{c},w,h,\theta)$, where $(x_{c},y_{c})$ denotes the center coordinates, $w$ and $h$ specify the width and height of the bounding box, and $\theta$ represents the rotation angle relative to the horizontal axis of the standard bounding box. The detailed descriptions of the two multimodal datasets are as follows:

2) 4K-SAI-LCS MVD Dataset:

The 4K Stereo Aerial Imagery of a Large Camping Site (4K-SAI-LCS) dataset is a subset of aerial imagery acquired from a large-scale site [22, 23]. This dataset encompasses an expanse of $1.0\times 1.5$ km. Utilizing the German Aerospace Center’s advanced optical 4K camera system, a total of $114$ images were captured $54$ images with a leftward orientation and $60$ images with a rightward orientation. These images were obtained at altitudes of $600$m and $650$m above ground level, respectively. Image pre-orientation is performed by using the open source SRTM (The Shuttle Radar Topography Mission) data and measured GPS positions of the image projection centers. Precise image orientation is then accomplished by bundle adjustment using automatically extracted SIFT (Scale-Invariant Feature Transform)- tie points [24]. Afterward, the 3D point cloud is calculated using semi-global matching [25, 26, 27].

To preserve the original rich textures and the sharp boundaries of the vehicles in RGB images, instead of generating true orthophoto (TOP) and DSM images, in this paper, the multimodel dataset consists of the original RGB images and height maps. Unlike DSMs, height maps use the original image coordinates instead of geo-coordinates, facilitating a more direct correspondence with the RGB imagery. To further improve the point density, we use each test region with 4-6 overlapping images. Point clouds, created from different views, are merged and filtered. This process ensures that, after projection, there is a one-to-one relationship between each pixel in the 2D height map and its corresponding pixel in the RGB image. Both the images and height maps have ground sampling distances of $11$ cm, and each scene has a resolution of $5,184\times 3,456$.

We have annotated the 4K-SAI-LCS MVD dataset in the oriented bounding box (OBB) format using the LabelMe toolbox. This newly labeled dataset is now employed for multimodal occluded and dense vehicle detection. Fig. 5. I provide an example annotation image. The designated control zone of the festival scene encompasses a spacious parking lot and tent area. As a result, the primary objects depicted in the scene images include vehicles, tents, roads, and sanitation facilities, effectively representing the campground environment. The dataset presents significant challenges due to the dense and irregular parking arrangements of vehicles, which fall into diverse subcategories, including cars, transport vehicles, transport trailers, recreational vehicles, and camping trailers. This diversity leads to substantial intra-class variation. Moreover, the visual resemblance between vehicles and tents significantly complicates the task of vehicle detection, thereby increasing the complexity of the dataset and placing higher demands on the algorithms designed for detection. Fig. 5. II presents some examples of vehicles with varying degrees of occlusion and density.

1) ISPRS Potsdam City MVD Dataset:¹¹1http://www2.isprs.org/commissions/comm3/wg4/2d-sem-label-potsdam.html. The original Potsdam dataset was constructed for the “semantic segmentation competition” by the ISPRS III/4 working group and was first published in the ISPRS 2D semantic labeling contest. Potsdam is a typical historical city and this dataset includes $38$ different regions with true orthophoto (TOP) and digital surface models (DSMs). The TOP images were captured using Trimble INPHO OrthoVista, while the DSMs, detailing the absolute elevation values for each pixel, were produced via dense image-matching techniques utilizing Trimble INPHO 5.3 software. Both images feature a ground sampling distance of $5$ cm and a resolution of $6,000\times 6,000$ pixels.

Different from existing segmentation labels, we have re-annotated the Potsdam dataset in the OBB format for images that encompass both VIS and height map data. It features vehicles located in extensive building complexes, narrow lanes, and densely populated residential zones. The designated parking areas display notable overlaps and occlusions, posing challenges in distinguishing black vehicles, particularly those obscured by foliage. Fig. 5. II presents some examples of vehicles with varying degrees of occlusion and density.

3) Dataset Statistic: Table I lists detailed instance counts for the two multimodal vehicle datasets. Given the varying scenes of image acquisition, the vehicle density in the 4K-SAI-LCS dataset is higher than in the ISPRS dataset. The 4K-SAI-LCS dataset contains over 300,000 instances, while the ISPRS dataset comprises approximately 5,000 instances. The presence of differently dense targets also poses a significant challenge for detecting densely packed vehicles. Fig. 6 displays a curve graph comparing vehicle area to the number of vehicles in both datasets. This graph highlights the diversity and balanced distribution of vehicle objects within the two datasets.

In the experiment, data preprocessing and augmentation were applied to all images to prevent overfitting. For the 4K-SAI-LCS dataset, input images were cropped to three sizes: $640\times 640$, $800\times 800$, and $1,024\times 1,024$ pixels, each with a $200$-pixel overlap to maximize object information capture. The training and testing sets were equally divided, maintaining a $1:1$ ratio. To balance the volumes of the two multimodal datasets, $12$ original images from the ISPRS Potsdam City Dataset, featuring aligned VIS and DSM scenarios, were selected. These input images were cropped to $800\times 800$ pixels, with a $400$-pixel overlap for optimal object information capture. The ratio of the training set to the testing set for the ISPRS dataset was adjusted to $784:392$ to align with the experimental requirements.”

The initial and final learning rates are set at $1.5\times 10^{-4}$ and $1\times 10^{-6}$, respectively. We employ the Stochastic Gradient Descent (SGD) optimization strategy, with a weight decay of $5\times 10^{-4}$ and momentum of $0.9$. The training process is designed to run for a maximum of $200$ epochs, with a confidence threshold set at $0.2$ and a Non-Maximum Suppression (NMS) threshold of $0.45$. Given the distinct information content of height maps and RGB data, we utilize different backbone networks for each data stream: ResNet18 for height maps and Darknet53 for RGB data. The proposed MuDet architecture is implemented using the PyTorch framework on an NVIDIA GeForce RTX 3090 GPU.

Three common object detection evaluation criteria are utilized for quantitative analysis, including Precision (P), Recall (R), and Average Precision ($AP$).

where $TP$, $FP$, and $FN$ represents true positive, false positive objects, and false negative objects, respectively. Generally, higher values of these metrics indicate superior detection performance.

$AP$ is a global indicator, enabling fair comparison across different detection methods. In our experiments, $AP0.5$ refers to the Average Precision (AP) calculated at an Intersection over Union (IoU) threshold of 0.5.

where $k$ represents the threshold. $P(k)$ denotes the precision at the $k$-th threshold. is the precision at the $k$-th threshold. $\Delta R(k)=R(k)-R(k-1)$ calculates the change in recall between consecutive $k-1$-th and $k$-th thresholds.

In the experiment, six commonly used methods for vehicle detection in multimodal remote sensing (RS) images were selected for both quantitative and qualitative comparisons. These methods include You only look once (YOLOv3)[28], RetinaNet[21], Fully Convolutional One-Stage object detector (FCOS)[29], General Gaussian Heatmap Label Assignment (GGHL)[13], Representative Points (RepPoints)[31], and YOLOv8 [30]. Darknet53 was employed as the backbone network across all methods, complemented by a multi-scale feature pyramid network (FPN) for upsampling and fusion to ensure a balanced comparison.

In this section, we evaluated the contribution of the proposed MuDet through two ablation analysis experiments. Specifically, 1) the contribution of the modality increment; 2) the contribution of the fusion increment.

Table. IV lists the quantitative detection accuracy of the six methods for the 4K-SAI-LCS dataset in terms of AP. Overall, multimodal data significantly outperforms single modality data in terms of AP value. Specifically, RetinaNet, due to its focal loss design, reduces the weight of easily detected objects, achieving a detection performance approximately 2% higher than YOLOv3. FCOS, GGHL, and YOLOv8, all anchor-free methods, show distinct advantages over anchor-based methods. FCOS enhances detection precision by predicting object presence at each pixel, thus avoiding the complex anchor matching process. GGHL employs a Gaussian heatmap distribution technique to improve the learning capability of objects of different sizes in various positions. YOLOv8 has been redesigned with a regression-based loss function and sample matching strategies, among other modules, offering significant improvements in both speed and accuracy, approximately 6% higher than YOLOv3. Unlike traditional anchor-based methods, Reppoints models each object’s unique shape and contour without relying on predefined anchor box sizes or ratios, achieving optimal performance in single-modality object detection that may be influenced by background or semantically irrelevant foreground information. The proposed MuDet increases detection accuracy across different network backbones, including YOLOv3, GGHL, and YOLOv8, by over 5% compared to their corresponding single modality. MuDet introduces single-modality feature enhancement, multimodal cross-fusion, and a strategy for balancing samples of varying difficulty, effectively segregating vehicles, and eliminating the impact of background information.

Fig. 8 shows the visualization results of the MuDet on selected example images, demonstrating effective separability for dense and occluded vehicles, such as RVs with open doors and tents mounted on cars. However, a small fraction of white recreational vehicles (RVs) were missed due to the absence of open-door samples in the labeled 4K-SAI-LCS vehicle dataset. Additionally, we have to admit that the 4K-SAI-LCS dataset presents significant challenges for vehicle detection. Notably, some vehicles are difficult to distinguish, with their similarity to tents sometimes exceeding 80%.

Given the substantial impact that varying confidence thresholds can have on model performance, it is crucial to analyze their effects. Fig. 9 (a) illustrates the Precision-Recall (PR) curve for all comparative methods and the proposed MuDet across different thresholds. It can be shown that the proposed method not only achieves the highest precision for a given level of recall but also has the largest area under the PR curve, reaffirming the efficacy and superiority of MuDet. However, the recall metric still warrants enhancement. This likely stems from the significant intra-class variance and subtle inter-class variance. The challenge in achieving precise fine-grained separation becomes apparent when merging all patterned vehicles into a broad ”vehicle” category, especially when distinct types, such as cars and trucks, are amalgamated into a single category, hindering precise differentiation.

Table V lists a quantitative performance analysis of the ISPRS Potsdam city dataset, while Fig. 10 shows the visualization results for sample images using our methods.

Overall, the detection performance on the Potsdam city dataset is consistent with that on the 4K-SAI-LCS dataset, further demonstrating that MCo-Net improves the detection performance of dense and occluded vehicle objects. MuDet achieves a significant improvement over unimodal YOLOv3 and multimodal YOLOv3, with increases of 10% and 8%, respectively. Relying solely on single-modality data raises the likelihood of inaccurate detections or missed vehicles. Especially dark vehicles are obscured by tree branches or closely match the color of the branches. Furthermore, compared to competitive multimodal methods based on YOLOv8 and Reppoint, MuDet achieves improvements of up to 3% and 1.5%, respectively, further verifying the method’s robustness.

Fig. 9 (b) shows the PR curve for all comparison methods and the proposed MuDet under dynamic thresholds. MuDet obtains the largest area under the PR curve, similar to the performance on the 4K-SAI-LCS dataset, further demonstrating MuDet’s effectiveness and superiority. However, at equivalent levels of accuracy, this dataset exhibits a lower recall rate compared to the 4K-SAI-LCS dataset. This discrepancy is likely attributed to varying occlusion levels from tree branches and vehicle deformation, which not only diminishes the contrast between background and foreground but also amplifies the intra-class variation of vehicle targets, leading to a reduced recall rate.

Fig. 10 shows some detection results on the ISPRS Potsdam city dataset. While this dataset has a lower vehicle density compared to the 4K-SAI-LCS dataset, it also features other challenges, such as black vehicles obscured by tree branches and vehicles with deformations. For these vehicles, MuDet achieved commendable detection results. However, there remains a considerable opportunity for further refinement, particularly in improving the detection of vehicles with significant deformations or those extensively occluded by tree branches.