Learning for Vehicle-to-Vehicle Cooperative Perception under Lossy Communication

3D object detection is one of the most critical ways to the success of autonomous driving perception. Based on recently available sensor modality [27], 3D detection method can be roughly divided into three categories: (1) Camera-based detection methods where approaches detect 3D objects using a single or multiple RGB images [28, 29, 30]. For example, CaDDN [28] utilizes the depth distributions combined with the image features to generate bird’s-eye-view representations for 3D object detection. ImVoxelNet [29] constructs a 3D volume in 3D space and samples multi-view features to obtain the voxel representation for 3D object detection. DETR3D [30] uses queries to index into extracted 2D multi-camera features to directly estimate 3D bounding boxes in 3D spaces. (2) LiDAR-based detection methods where these methods typically convert LiDAR points into voxels or pillars, leading in voxel-based [31, 32] or pillar-based object detection methods [33, 34, 35]. PointRCNN [36] proposes a two-stage strategy based on raw point clouds, which learns rough estimation first and then refines it with semantic attributes. Some methods [31, 32] propose to split the space into voxels and produce features per voxel. However, 3D voxels are usually expensive to process. To address this issue, PointPillars [37] propose to compress all the voxels along the z-axis into a single pillar, then predict 3D boxes in the bird’s-eye-view space. Moreover, some recent methods [38, 39] combine voxel-based and point-based approaches to detect 3D objects jointly. (3) Camera-LiDAR fusion detection method where it presents an approach fusing information from both image and LiDAR points, which is a trend in 3D object detection. How to align the image features with point clouds is challenging in multimodal fusion. To solve this challenge, some methods [40, 41, 42] use a two-step framework, where detecting the object in 2D images in the first stage, then using the obtained information to further process point clouds for 3D detection. While other works [43, 44] develop end-to-end fusion pipelines and leverage cross-attention mechanisms to perform feature alignment. Our work in this paper focuses on the cooperative point cloud based 3D object detection to achieve fast processing and real-time performance [33, 34]; pillar-based approach would be used in our following experiments.

The performance of a 3D perception method highly depends on the accuracy of 3D point clouds. However, LiDAR cameras suffer from refraction, occlusion, and long-range distance, so the single-vehicle system could become unreliable under some challenging situations [10]. In recent years, Vehicle-to-Vehicle (V2V) / vehicle-to-infrastructure (V2X) cooperating system has been proposed to overcome the disadvantages of the single-vehicle system by using multiple vehicles. The collaboration among different vehicles enables the 3D perception network to fuse information from different sources.

Some former methods use Early fusion to share raw data among different vehicles. For example, Cooper [45] fused the point clouds from different connected autonomous vehicles and made predictions based on the aligned data. AUTOCAST [46] exchanged sensor readings from different sensors to broaden the perceptive field for a single vehicle. Other methods use Late fusion to integrate the 3D detection results from each vehicle. Rawashdeh et al. [47] proposed a machine learning based method that shares the dimension and the location of the center point for each tracked object. Some other late fusion methods [48, 49] also adopt point clouds as sensor data from both vehicle and infrastructure. While early fusion requires large bandwidth and data transfer speed, late fusion may generate undesirable results due to the biased individual prediction. In order to find the balance between data load and accuracy, recent methods focus on Intermediate fusion by sharing intermediate representations. F-cooper [12] applied voxel features fusion and spatial feature fusion from two cars. V2VNet [13] employed a graph neural network to aggregate features extracted by LiDAR from each vehicle. V2X-ViT [15] proposed a vision Transformer architecture to fuse features from vehicles and infrastructures. Cui et al. [50] proposed a Point-based Transformer for point cloud processing, which can integrate collaborative perception with control decisions. Tu et al. [51] proposed an efficient and practical online attack network in a multi-agent deep learning system based on intermediate representations. Luo et al. [52] utilized attention modules to fuse the intermediate feature and enhance feature complementarity. Lei et al. [53] proposed a latency compensation module to realize intermediate feature-level synchronization. Hu et al. [54] proposed a spatial confidence-aware communication strategy to use less communication to improve performance by focusing on perceptually critical areas. OPV2V [10] utilized a self-attention module to fuse the received intermediate features. CoBEVT [11] proposed local-global sparse attention that captures complex spatial interactions across views and agents to improve the performance of cooperative perception. However, these fusion methods are all with the assumption of ideal communication, which would suffer dramatic performance drop with lossy communication in the real world. To address this issue, we design a special V2V Attention Module (V2VAM), including intra-vehicle attention of ego vehicle and uncertainty-aware inter-vehicle attention to enhance the V2V interaction.

V2V and V2X communication can improve the safety and reliability of autonomous vehicles by exchanging information with surrounding vehicles. However, communication among vehicles may bring new issues to this research field [55]. Due to the nature of the connectivity, lossy communication is inevitable in wireless channels. Some factors like channel errors, network congestion, and delay deadline violation can cause packet losses during the transmission of data in the wireless network [56]. Low latency and high reliability are two common challenges for V2V communication. For example, in the pre-crash sensing scene, the maximum latency is only 20 ms, and data delivery reliability must be greater than 99$\%$ [55, 57]. Several works have proposed specific resource allocation schemes to ensure latency and reliability of V2V communication systems by using Lagrange dual decomposition and binary search  [58], greedy link selection  [59], or federated learning  [60]. Some studies also aim to improve the V2V communication security from different aspects such as authentication, data integrity, and data protection [61].

Lossy Communication (LC) is also a critical issue in V2V communication. According to studies on single-hop broadcasting, the obstacle (vehicles, buildings, etc.) between transmitter and receiver will result in signal power fluctuations, thus causing packet loss [55, 62, 63, 21]. The shared data could also be interfered with by other signals or modified by attackers before arriving at its destination, thus leading to lossy data. In this work, we aim to eliminate the lossy communication by proposing an LC-aware repair network and improving the robustness of the V2V perception network.

Image denoising is one of the longstanding challenging tasks in computer vision. The primary sources of noise [65] are shot noise, where a Poisson process with variance equal to the signal level, and read noise, where an approximately Gaussian process is caused by diverse sensor readout effects. To denoise them, some deep learning-based methods [66, 67, 68] use denoising networks that generate a filter for every pixel in the desired output to constrain the output space and thereby prevent the impact of artifacts. Inspired by these architectures, to handle the common V2V communication challenges i.e., lossy communication, we design a customized LC-aware repair network for intermediate feature recovering from other CAVs.

The framework of the LC-aware repair network is shown in Fig. 3, which is an encoder-decoder architecture with skip connections. This network generates a specific per-tensor filter kernel to jointly align and recover the input damaged feature to produce a recovered version of the output feature. The input feature for LC-aware repair network is $S\in\mathbb{R}^{c\times h\times w}$, then a tensor-wise kernel $K$ is generated and applied to $S$ to produce the recovered output feature $\hat{S}\in\mathbb{R}^{c\times h\times w}$. the specific tensor-wise filter kernel could be simply formulated as

and the value at each tensor $t$ in our output feature $\hat{S}$ is

where $\circledast$ denotes the matrix dot product. $K\in\mathbb{R}^{(k\times k)\times h\times w}$ is a tensor-wise kernel, and each tensor in channel dimension $K^{t}\in\mathbb{R}^{k\times k}$ is a per-tensor kernel and can be applied to the $k\times k$ neighborhood region of each tensor $t$ in the input feature $S\in\mathbb{R}^{c\times h\times w}$by multiplication. The $Conv(\centerdot)$ denotes the tensor-wise network and is used to perceive the input feature and generate the suitable kernel for each tensor.

To acquire the repaired output feature $\hat{S}$, the tensor-wise filtering $\circledast$ of the input damaged feature could largely preserve the feature detailed without corruption. Therefore, a large kernel size $k$ is desired to leverage the rich neighborhood information of each tensor fully. In our experiment, the kernel size $k$ is set to $5$ due to memory limitations.

The LC-aware repair loss function $\mathcal{L}_{LC}(\hat{S},\hat{S}^{g})$ is the tensor-wise $L1$ distance between the ground truth original feature $\hat{S}^{g}$ before suffering lossy communication and the repaired feature $\hat{S}$. The repair loss can be defined as

Self-attention mechanism [69] has emerged as a recent advance to capture long-range interaction; The key idea of self-attention is to calculate the response at a position as a weighted sum of the features at all locations, with the interaction between features determined by the features themselves rather than their relative location, as in convolutions. In this paper, after receiving the recovered intermediate feature, we aim to leverage the intermediate deep learning features from multiple nearby CAVs to improve perception performance based on V2V communication. We design a customized intra-vehicle and inter-vehicle attention fusion method by considering the lossy communication situation to enhance interaction between ego CAV and other CAVs. Moreover, we adopt a criss-cross attention module in our proposed V2V attention method, which can be leveraged to capture contextual information from full-feature dependencies more efficiently and effectively.

After obtaining the intra-vehicle attention and inter-vehicle attention, all of them would be fed into the max pooling and average pooling layers separately to obtain abundant spatial information, then they are concatenated as the input for the 2D convolutional layer with ReLU activation function. Therefore, the final fusion feature output $\mathbf{A}^{out}$ in V2V attention module is

where $\boldsymbol{F}$ denotes a set of max pooling, average pooling, and convolution layers. For 3D object detection, we use the smooth L1 loss for bounding box regression and focal loss [71] for classification. The final loss is the combination of detection and LC-aware repair loss $\mathcal{L}_{LC}$ as follows,

where $\mu$ and $\lambda$ are the balance coefficients within range $[0,1]$.

Due to the difficulties of collecting the real-world CAV perception data for cooperative perception with lossy communication in realistic scenes, we use the digital-twin-based simulation dataset to validate the proposed method. The experiments are conducted on the public cooperative perception dataset OPV2V [10]. OPV2V is a large-scale open-source simulated dataset for V2V perception, which contains 73 divergent scenes with various numbers of connected vehicles, 11,464 frames, and 232,913 annotated 3D vehicle bounding boxes. These data are collected from 8 digital towns in CARLA [16], and a digital town of Culver City, Los Angeles with the same road topology. Following the default setting of OPV2V [10], we use $3,382$ frames and $1,920$ frames from OPV2V as the training set and validation set, respectively, and $2,170$ frames in CARLA Towns and $594$ frames in Culver City are used as testing set for all methods.

In the training stage, we adopt two schemes to observe the impact of different training data on V2V 3D object detection models. The Scheme I uses only ideal communication-based data for training, while the other Scheme II uses simulated lossy communication-based data as described above for training. The training parameter settings for both schemes are identical, and the only difference between them is the training data, which considers lossy communication in Scheme II All trained models are evaluated on V2V CARLA Towns and Culver City testing sets under both Ideal Communication and Lossy Communication scenarios. Specifically, all models use the PointPillar [37] as the backbone with the voxel resolution of $0.4$ m for both height and width. We adopt Adam optimizer [72] with an initial learning rate of $10^{-3}$ and steadily decay it every $10$ epochs using a factor of $0.1$. The coefficient of detection loss $\mathcal{L}_{det}$ is set to $1.0$, and that of LC-aware repair loss $\mathcal{L}_{LC}$ is set to $0.1$. We follow the same hyperparameters in V2X-ViT [15], and all models are trained on two RTX 3090 GPUs.

Table I shows the performance comparisons of all models that are trained with Scheme I, then tested on two communication types e.g., Ideal Communication and Lossy Communication, respectively. Under Ideal communication, all the cooperative perception methods significantly surpass NO Fusion baseline. In V2V CARLA Town testing set, our proposed V2VAM outperforms the other five advanced fusion methods to achieve $92.6\%$/$86.1\%$ for [email protected]/0.7, which is highlighted as bold text in Table I. In V2V Culver City testing set, CoBEVT [11] gets $87.7\%$/$74.8\%$ for [email protected]/0.7, while the V2VAM achieves the $88.5\%$/$78.5\%$ for [email protected]/0.7 as the best performance, which is higher than the second best fusion method CoBEVT [11] with an [email protected]/0.7 improvement of $1.6\%$/$3.7\%$. These results indicate cooperative perception methods can improve the perception performance than a single vehicle perception system under Ideal Communication, and our proposed fusion method V2VAM can enhance the interaction between ego vehicle and other vehicles efficiently, which achieves the best performance. However, under Lossy Communication testing scenario, all intermediate fusion methods have a drastic performance drop on two testing sets, and the accuracy of these methods is even less than NO Fusion. In V2V CARLA Town testing set, the cooperative perception performance of F-Cooper [12], V2VNet [13], OPV2V [10], and CoBEVT [11] decrease by $80.8\%$, $85.0\%$, $85.6\%$, and $82.5\%$ in [email protected], respectively. Obviously, all intermediate fusion methods without considering the lossy communication are not practical for deployment in the real world.

The result of 3D object detection on two OPV2V testing sets based on the training of Scheme II is presented in Table II. Under Lossy Communication, although all intermediate fusion methods have a better performance than Table I, which learned the lossy intermediate feature in the training stage. They still fail to handle lossy communication data resulting in the poor perception performance in  II. In V2V CARLA Town testing set, F-Cooper [12] got $49.2\%$, V2VNet [13] got $46.5\%$, CoBEVT [11] got $58.2\%$, and V2X-ViT [15] got $59.9\%$ in [email protected]. These four fusion methods are even worse than single-vehicle baseline NO Fusion, which indicates the highly negative impacts by lossy communication. While our proposed method can reach the best performance of $84.1\%$/$70.5\%$ for [email protected]/0.7 on V2V CARLA town testing set, and $84.6\%$/$66.3\%$ for [email protected]/0.7 on Culver City testing sets, respectively. The proposed method achieves the best performance under both Ideal Communication and Lossy Communication, which is highlighted in Table I. Obviously, our proposed LCRN module efficiently maintains the benefits of collaborations under lossy communication. The proposed method can also diminish the impact of lossy V2V communication to achieve excellent cooperative perception performance. Further, we visualize some 3D object detection results on V2V Culver City testing set under Lossy Communication, as shown in Fig. 5. Intuitively, these five comparison methods cannot handle loss communication appropriately, thus leading to some false negative proposals. While the proposed method improves the perception performance under lossy communication significantly.

As explained in [56, 73], several random issues such as the occurrence of obstacles, fast and changing vehicle speeds, distance between vehicles might result in lossy communication when sharing a set of communication data. To simulate the complex lossy communication in the real world, the sharing data is randomly selected by a uniformly distributed random probability $p\in[0,1]$ and then replaced by random noise within the range of original shared feature values. We design two ways of random selection to simulate different lossy communication types in the real-world V2V communication.

Lossy Communication (named as “Lossy”) on global feature: The shared feature after V2V metadata sharing is reshaped from 3D tensor to 2D matrix first (Fig. 6(b)). Then, as shown in Fig. 6(c₁-c₃), the reshaped feature is randomly selected by the global random probability $p$ and replaced by random noise within the range of original shared feature values.

Channelwise Lossy Communication (named as “$Ch$-Lossy”): Different with the “Lossy” type to simulate lossy communication on the reshaped global feature, “$Ch$-Lossy” type is to simulate lossy communication on different channels. As shown in as Fig. 6(d₁-d₃), given a shared feature $C\times H\times W$, $\lfloor p*C\rfloor$ channels are randomly selected by the channelwise random probability $p$ and replaced by random noise within the range of original shared feature values.

Finally, the simulated lossy feature is reshaped back to its original shape of $C\times H\times W$ and then received by ego vehicle. In our experiment, Scheme II utilizes the simulated lossy communication data by the “Lossy” type to train models, and then we use the models trained in Scheme II to test both “Lossy” and “$Ch$-Lossy” simulated data. Table IV shows the performance comparisons of several methods with the two lossy communication types. The proposed method achieves the best performance under both “Lossy” and “$Ch$-Lossy” communication types.

The effectiveness of the two proposed components, V2VAM and LCRN, is investigated here. Based on training Scheme II, all methods are evaluated under Lossy Communication on V2V CARLA Town and Culver City testing sets, respectively. AveFuse is used as the baseline fusion method, which just averages all intermediate features. As shown in Table III, the proposed V2VAM obtains $70.9\%$ in [email protected] and $58.3\%$ in [email protected] on V2V CARLA Town testing set, which is $7.7\%$ and $25.8\%$ higher than AveFusion in [email protected] and [email protected] respectively. Both Intra-vehicle attention and Inter-vehicle attention modules are quite effective for V2VAM if we remove one of them in V2VAM during the ablation study. By adding LCRN to the baseline method, AveFuse+LCRN achieves $69.8\%$ in [email protected] and $47.2\%$ in [email protected] on V2V CARLA Town testing set, with the improvement of $6.6\%$ in [email protected], and $14.7\%$ in [email protected]. Furthermore, our proposed method V2VAM+LCRN achieves the best performance on both V2V CARLA Town and Culver City testing set. Obviously, both V2VAM and LCRN components are beneficial for improving the final performance of 3D object detection in lossy communication scenarios.