UVCPNet: A UAV-Vehicle Collaborative Perception Network for 3D Object Detection

Research on aerial-ground collaboration is still in its preliminary stages, with most studies concentrating on exploring multi-stage perspectives. For instance, AGCG [13] proposed a visual servoing approach based on a binocular localization model, adjusting the alignment between the unmanned aerial vehicle (UAV) and the target in the image to determine the target’s position. Utilizing multi-feature fusion recognition and projection imaging techniques, this method enhances the positioning accuracy of UAVs and unmanned ground vehicles (UGVs) in outdoor environments concerning moving targets. However, these methods have not yet proposed a specific aerial-ground collaboration research framework and have only undergone simulated testing, lacking support from actual datasets to validate the accuracy of the methods.

In recent years, with the development of drone technology, many high-quality datasets (such as VisDrone [14, 15, 16], HIT-UAV [17], DroneVehicle [18], etc.) have emerged, containing only images from the UAV perspective. The recently released MAVREC [19] dataset provides images from both ground and aerial perspectives, but lacks three-dimensional annotations and coordinate transformations between UAVs and UGVs, thus only usable as an object detection dataset, incapable of achieving target-level aerial-ground collaborative perception. In summary, current aerial-ground collaboration research mainly focuses on object detection from a single perspective, followed by its extension to other perspectives to supplement information. However, we aspire to integrate information from both ”aerial” and ”ground” domains to obtain more comprehensive collaborative perception results. This will aid in more accurately understanding and addressing targets in complex environments.

As collaborative perception tasks mature in the field of autonomous driving, significant progress has been made in collaborative work among homogeneous agents. For example, HM-ViT [20] demonstrates efficient collaborative capabilities by fusing point cloud and optical image data. Additionally, in the context of drone swarm collaboration, When2com [21] proposed a grouping-based cooperative method that successfully reduces the bandwidth requirements. These studies mainly focus on collaborative perception tasks within the domain and do not address challenges in cross-domain scenarios.

To address challenges in cross-domain collaborative perception, some high-quality datasets such as DAIR-V2X [22] and V2X-Sim [23] have emerged, along with proposed collaborative methods. However, most cross-domain collaborative methods [24, 25, 26, 27, 28] still rely on Lidar data. For instance, FFNet [29] solves the problem of asynchronous timing between vehicles and infrastructure using point cloud data, while TransIFF [30] utilizes instance-level features to enhance the robustness of feature fusion. However, due to the large storage space occupied by Lidar data, there is a desire to directly use optical images for cross-domain collaborative perception to reduce storage space usage.

In summary, existing research has made significant progress in collaborative perception tasks within the field and partially addressed challenges in cross domain scenarios. Afterwards, we need to further explore and develop methods that directly utilize optical images for cross domain collaborative perception.

Depth estimation tasks [31, 32, 33], as essential auxiliary tasks in 3D visual detection [34, 35, 36, 37], play a crucial role in the modeling process, where accurate depth values help avoid distortions. With the continuous maturity of depth estimation algorithms, some new methods have made remarkable progress. For example, News-CRF treats depth estimation tasks as pixel-level classification problems, making them applicable to CRF applications and achieving advanced depth estimation results.

However, most depth estimation in the collaborative perception domain still relies on Lidar data. For depth estimation from optical images, it primarily relies on the depth estimation scheme proposed in LSS [38]. In collaborative perception, CoCa3D [39] fully utilizes the depth collaboration probability of each intelligent agent to modulate depth values. Although these methods have improved the accuracy of depth estimation to some extent, there is still room for optimization, especially in fully utilizing collaborative information. We hope to fully utilize collaborative information by employing CRF to optimize depth estimation schemes for more accurate feature information. This may include interacting information among multiple intelligent agents and jointly optimizing depth estimation tasks with other relevant tasks to obtain more comprehensive and accurate depth estimation results.

In our current framework, we employ the BEVDET4D [40] network as the baseline model, which is an enhanced BEV feature extraction framework. The framework consists of two main components: the backbone (including the neck) and the head. The backbone network employs ResNet [41] to extract both low-level geometric features and high-level semantic information. Subsequently, it utilizes the BEV generation method proposed in LSS to translate these features and semantic information into the BEV feature space, thereby obtaining BEV features. The head uses the CenterPoint [42] detector head for 3D target detection. However, this method is traditionally applied to generating BEV from look-around images. By modifying the BEV generation process and introducing timestamp concepts, we establish a collaborative task framework named UVCPNet capable of time synchronization through timestamps. UVCPNet is an aerial-ground collaborative framework for enhanced BEV features that combines multi-domain information. The overall framework of the method is illustrated in Fig. 2.

First, we input a set of RGB images of vehicles and UAVs. Second, by combining the features extracted from these RGB images with the estimated depth information, we produce the corresponding BEV feature map for each end. It should be emphasized that the required depth information is obtained through the CDO module. The module optimizes the depth estimation network by using CRF combined with the context semantic information of two perspectives, so as to obtain accurate depth estimation results.

Third, we obtain cross-domain BEV features from multiple ends and achieve alignment and enhancement of these features by the CDCA module. This module combines feature information from the ”Aerial” and ”Ground” domains, facilitating effective cross-domain alignment and feature enhancement. By adjusting the weight of BEV features in each domain, we fuse the enhanced BEV feature map. Finally, we use 3D target detection method to process the fused BEV feature results, and then decode the BEV representation into 3D targets for target detection.

Currently, to generate a BEV from optical images, we need to perform a 2D to 3D transformation [43], represented by the following equation:

In this context, $d$ represents the depth value of the image, which is the distance of each pixel point from the camera’s focal point. $\mathbf{K}$ is the intrinsic matrix, $\mathbf{R}$ is the rotation matrix of the camera, and $\mathbf{T}$ is its translation matrix. Therefore, to obtain accurate 3D information, it is crucial to acquire precise depth estimation data.

In existing BEV-based methods aimed at addressing depth estimation inaccuracies, such as BEVdepth [44] and Solofusion [45], additional supervision signals in the form of depth information are necessitated. This requirement imposes additional computational burdens on the agent and increases the volume of information that the agent must process. Our method leverages the contextual semantic information inherent in RGB images as implicit supervision signals to optimize the depth estimation network. This approach eliminates the need for additional supervision signals, thereby reducing the burden on agents while simultaneously enhancing depth estimation accuracy. We hope to conduct depth estimation without adding any supervisory information.

The accuracy of depth estimation plays a pivotal role in determining the precision of spatial information for targets within the BEV space [38]. Nevertheless, the majority of cooperative algorithms still rely on basic convolutional neural networks (CNNs) for depth estimation. If we consider depth estimation as a segmentation task and represent each class as a specific depth range, CRF can be utilized to enhance the quality of depth estimation. Since each domain observes the same region, the depth information for the same target should theoretically be consistent after transformation.

However, in current supervised methods, the sampling of feature maps is relatively large, causing the feature maps to lose a significant amount of adversarial information, thereby compromising the effectiveness of CRF optimization for depth. To address this issue, CRF can be employed to refine the semantic information inherent in the optical image context of both the ”Aerial” and ”Ground” domains. Furthermore, leveraging semantic information priors within the depth estimation network can mitigate the challenge of limited depth supervision. This approach significantly improves depth consistency at the pixel level.

We aim to use semantic information from multiple domains for comprehensive optimization. Given that each domain observes the same target, the semantic information within each domain exhibits high similarity. On the feature map of RGB image feature extraction in each domain, we let $\{X_{1}^{m},...,X_{N}^{m}\}$ represent the $N$ pixels in the feature graph of the subsampled $m$ field, and $\{D_{1}^{m},...,D_{k}^{m}\}$ denote $k$ discrete depth values. The responsibility of the depth network is to assign each pixel to various depth values, expressed as $d^{m}=\{x_{1}^{m},...,x_{N}^{m}|xi^{m}\in\{D_{1}^{m},...,D_{k}^{m}\}\}$. For depth information $d$ within each feature map, our objective is to minimize its energy cost $E(d|s)$ [46]:

Among these, $\psi_{u}(x_{i})$ represents a unary potential, serving to gauge the cost associated with the initial network output:

$P(x_{i}|d)$ is the probability that the depth information of pixel $x_{i}$ is d. Building upon prior research [47, 48], we define the binary potential as follows:

Here, $\bar{S}_{i}$ and $\bar{S}_{j}$ represent the semantic information of each image block with the same subsampling step in different domains, denoted by $m$ and $n$ respectively. Additionally, $|x_{i}-x_{j}|$ signifies the compatibility between depth information, quantifying the distance between their centers in the real world. Utilizing CRF as an optimization layer acting upon the depth estimation network, we ultimately obtain the optimized depth information $\tilde{d}$.

In essence, by employing the CDO module, we achieve more precise depth estimates for pixels without the need for additional supervision signals. Consequently, this leads to an enhanced accuracy in generating the BEV feature map for each domain.

In our collaborative framework, the input consists of a set of RGB images containing vehicles and UAVs of varying heights. After these input optical image features are converted to BEV space, because there are significant differences between different domains in the field of view information and noticed information, it is necessary to align the information obtained across domains. To address this misalignment issue, we introduce a cross-domain cross-adaptation (CDCA) module. As shown in Fig.3, in our module, we perform collaborative cross-adaptation on the obtained BEV images to achieve a well-fused BEV feature map.

The module exploits feature correlation across domains to facilitate cross-domain adaptation, thereby enhancing feature information and minimizing positional errors. Upon acquiring the BEV feature map of each domain, it is necessary to apply consistent adaptation operations to the information from different domains in order to obtain aligned BEV feature maps. We extract multi-scale feature information from the BEV feature map of each domain, and use these multi-scale information to adapt to the feature information of other domains. After conducting correlation analysis between the information from other domains and the information from the current domain, we align and enhance the feature maps accordingly. After obtaining the enhanced feature information, we perform feature fusion to obtain the final BEV feature map.

CDCA initially requires extracting multi-scale network features. We employ a FPN [49] structure to sample the BEV feature map $f_{i}$ across four layers, resulting in the feature map $f_{i}^{m}$ after sampling. To ensure consistency in network feature size across each scale, we utilize a Multi-Scale (MS) module to scale the feature map to the same size. After receiving the sampled feature map information $f_{i}^{m}$ transmitted from other domains, we cascade the BEV feature map information at the same level to obtain the cascaded feature map information $f^{m}$ at this level:

i represents the domain of the BEV feature map, and m represents the level of the cascading feature map.

Then, we analyze the correlation of these cascaded feature maps $f^{m}$:

Within this process, $\beta_{i}^{m}$ represents the autocorrelation analysis results of the features $f_{i}$ within the domain and the sampled feature maps within the same domain, as well as the cross-correlation analysis results between this domain and other domains. $\omega_{i}^{m}$ represents the weight information calculated through correlation analysis for each cascaded feature map, denoted as $f^{m}$, $f_{i}^{s}um$ represents the enhanced and aligned BEV feature map generated by fusion. By performing this operation, we can ascertain the adaptability outcomes for each scale and derive adaptability weights $\omega_{i}^{m}$ for each scale. Subsequently, we compute the enhanced feature graph information $f_{i}^{sum}$ within each domain.

After transforming the information from multiple domains into the BEV space, aligning and enhancing the BEV feature maps, we proceed with the fusion of BEV features. However, due to variations in the observation capabilities of each domain towards the target, ground perspectives typically capture more feature information, while aerial perspectives tend to gather relatively less target information. This leads to inconsistencies in the amount of information contained within the BEV feature maps. To address this issue, we utilize BEV space attention to fuse the BEV feature maps from both aerial and ground domains. This means that the final fused feature map will pay more attention to the BEV feature maps with higher information content, enabling them to play a more crucial role in the final fusion result, thereby enhancing the effectiveness of feature fusion.

This module enhances the model’s attention to different domains by adjusting the weights of the BEV feature maps between the input domains. Firstly, the module connects the cascaded BEV feature maps of the two domains to obtain the concatenated feature map information $K_{\text{cat}}$, $V_{\text{cat}}$ of the aerial-ground domains. Subsequently, we utilize the input feature information as $Q_{\text{veh}}$ and $Q_{\text{uav}}$ on both sides. Finally, the weights of different domains are computed to guide the fusion process:

Since the output of cross-domain attention in the module is the weighted sum of $K_{\text{cat}}$, which contains all information from the two domains, the essence of the module is to bring the two domains closer to a common center point. Specifically, by leveraging the shared $K_{\text{cat}}$ in the cross-domain attention mechanism, this module encourages convergence between the two domains towards a central representation. Finally, a learnable parameter $\lambda$ is set to balance the BEV feature maps from the two domains after the attention mechanism:

The module adjusts the ratio of the two parts of the feature maps by learning the weights in different domains. Through this operation, we can obtain a BEV fusion feature map with more accurate information.

The loss of the experiment is mainly composed of two parts, including smooth L1 Loss for bounding box $L_{bbox}$ and gaussian focal loss for classification $L_{cls}$ and cross-entropy loss for direction $L_{dir}$. The final loss function is as follows:

where $\lambda_{bbox}$, $\lambda_{cls}$ and $\lambda_{dir}$ represent the weights of $L_{bbox}$, $L_{cls}$ and $L_{dir}$ respectively.

Currently, publicly available collaborative 3D target detection datasets are limited to low-altitude cooperative sensing tasks between vehicles and infrastructure. This gap highlights the absence of datasets for high-altitude collaborative perception involving both vehicles and UAVs. In this study, we utilize the Carla simulation software to construct the V2U-COO dataset, designed specifically for air-to-ground cooperative sensing tasks. This dataset serves as a basis for our investigation into 3D target detection within cooperative sensing tasks.

Given the limitations of existing publicly accessible collaborative 3D target detection datasets, which are confined to low-altitude vehicle and infrastructure cooperative sensing tasks, there is a notable gap in datasets for high-altitude collaborative perception involving both vehicles and UAVs. To address this, we utilized the Carla simulation software to develop the V2U-COO dataset, specifically tailored for air-to-ground cooperative sensing tasks. This dataset facilitates our investigation into 3D target detection within these cooperative sensing scenarios. Furthermore, to validate the effectiveness of our method with real-world data, we conducted experimental tests on the DAIR-V2X, an open vehicle-road collaboration dataset.

We evaluate the performance of the network using the evaluation metrics provided by Nuscenes [50]. These metrics include the mean average precision (mAP), mean translation error (mATE), mean scale error (mASE), mean orientation error (mAOE), mean velocity error (mAVE), mean attribute error (mAAE), and the NuScenes detection score (NDS), which considers both comprehensive accuracy and true positives (TP).

The average precision of object detection is a crucial metric for assessing detection accuracy. However, in NuScenes, the calculation of mAP integrates threshold matching of average precision (AP) without employing Intersection over Union (IoU). Instead, it uses the 2D center distance (d) on the ground plane to compute AP, thereby decoupling the impact of object size and orientation on AP calculation. Here, d is set to $D=\{0.5,1,2,4\}$ meters. The formula for mAP calculation is as follows:

For each TP index, we calculate as follows:

NDS is computed using the TP metric, where half is based on mAP, and the other half is based on the quality of detection performance in terms of position, size, orientation, attributes, and velocity (ATE, ASE, AOE, AVE, AAE). As mAVE, mAOE, and mATE can exceed 1, we constrain each metric between 0 and 1 in the following formula:

The framework proposed in this study is experimented with using PyTorch on an RTX-4090 GPU to evaluate its efficacy. To assess the effectiveness of the proposed collaborative framework, we ensure uniformity by employing the same training detection model architecture across all agent nodes. We opt for ResNet-50 as the pre-trained model and subsequently train it on respective datasets.

During the training phase, parameters remain static at each agent, ensuring that updates occur solely within the collaborative module. This guarantees that all improvements stem from the collaborative framework, facilitating a comprehensive evaluation of the network’s capabilities. In the training detection phase, input images are standardized to 704x256 dimensions, with training extending over 40 epochs. We employ a standard gradient optimizer with a weight decay of 0.0005. Additionally, the learning rate is initialized at 0.001.

In this section, we conduct experiments based on the proposed V2U-COO dataset. Firstly, to assess the effectiveness of the collaborative network, we compare the results obtained from using only vehicle observations or only UAV observations with those obtained from collaborative network observations under the same network architecture. The specific mAP improvement curve is illustrated in the figure, demonstrating that the collaborative platform outperforms both the vehicle-only and UAV-only approaches. Additionally, to evaluate the effectiveness of each module within UVCP, we conduct extensive experiments on the dataset. To ensure fairness, experiments are conducted in the same environment. The overall experimental results are summarized in the Table II. It can be observed that compared to the baseline model, UVCP achieves an average precision improvement of approximately 6.1%.

To rigorously assess the efficacy of our proposed approach, we conducted a series of comparative experiments against both established BEV-based methodologies and select collaborative techniques. Our evaluations included comparisons with BEV-based systems such as BEVDet4D, BEVerse, and BEVFormer, alongside the collaborative approach EMIFF. Detailed outcomes of these comparisons are delineated in Table VII. The results demonstrate a significant enhancement in average precision with our method, exhibiting improvements of approximately 9% over BEVDet4D, 11% over BEVFormer [51], and 8% over BEVerse [52]. Additionally, when compared to the collaborative method EMIFF [53], our approach showed an improvement of about 6%. These findings underscore the superior performance of our proposed method in terms of average precision, highlighting its robustness and effectiveness in the tested scenarios.

To assess the efficacy of our cooperative framework and evaluate its performance within real-world contexts, we conducted validation experiments on the DAIR-V2X dataset. This dataset, comprising a rich array of images from real-world scenarios, presents significant challenges due to various adverse conditions that inherently increase the complexity of detection tasks. The experimental outcomes, as presented in Table VIII, indicate that the application of our proposed method led to an approximate increase of 2.7% in average precision. These results not only confirm the effectiveness of our framework but also demonstrate its robustness and adaptability in realistic settings.

In this section, we present a comparative analysis of the visualization results obtained from individual observations by vehicles and UAVs, as well as those enhanced through the incorporation of our collaborative framework. The analysis reveals that observations from single agents often suffer from challenges such as missed or duplicate detections. However, by integrating the collaborative framework, the observational outcomes for each agent are significantly improved. This enhancement is evident across multiple scenarios, confirming the effectiveness of the collaborative approach in providing more accurate and reliable detection results.