On the Federated Learning Framework for Cooperative Perception

It is essential that autonomous driving systems have the ability to perceive the environment accurately. This perception directly affects the efficiency of decision-making and overall safety. Traditional single vehicle perception systems, although have recently been equipped by the BEV perception paradigm which extracts BEV space feature from input sequences in perspective view or 3D space, facilitate performance of perception tasks such as 3D bounding box detection and BEV segmentation [29, 28, 30], yet still often struggle with occlusions and distant objects since being constrained by limited sensing capabilities. CP, enabled by V2X communications, marks a significant improvement, allowing CAVs to enhance perception through extensive wireless information sharing, encompassing V2V, V2I, and V2N communications. OpenCDA, introduced by [32], provides a versatile framework for developing and testing cooperative driving automation (CDA) systems within a simulated environment. Complementarily, the work [33] presented a large-scale simulated dataset for V2V perception, enriched with a diverse collection of scenes and annotated data. This dataset proves critical for advancing CP systems, as supported by [15], which underscores the expanded sensing capabilities facilitated by V2X. Further developments include the CoBEVT framework introduced in [31], a sparse vision transformer tailored for autonomous driving that leverages multi-agent multi-camera sensors, enabling collaborative perception across multiple vehicles. The work [34] discusses the integral role of advanced algorithms in optimizing CP systems. The works [35] and [36] explore the potential of integrating infrastructure-based sensors to bolster CP. There have been some notable advancements in perception frameworks recently. One of these is the introduction of V2X-ViT [37]. V2X-ViT is a novel vision Transformer designed to effectively combine information from on-road agents. Another important contribution is the work by Xu et al. [38]. They have created the first extensive real-world multimodal dataset for V2V perception. This dataset significantly enhances data availability for CP research. Lastly, Xiang et al. have introduced HM-ViT in their paper [39]. HM-ViT is an innovative multi-agent hetero-modal cooperative perception framework. It can accurately predict 3D objects in dynamic V2V scenarios. Collectively, these advancements represent a major leap towards more comprehensive and precise environmental perception in autonomous driving, addressing critical challenges and enhancing driving safety.

Federated learning, initially proposed as FedAvg by [40], marks a significant shift in machine learning by enabling distributed model training across multiple systems without accessing users’ raw data, thus enhancing data privacy. A primary challenge in federated learning is the management of data heterogeneity and distribution shifts, which complicate efficient model optimization. The work [41] introduces a method to mitigate client heterogeneity by incorporating proximal terms, ensuring that local model updates align with the global model. Additionally, the work [42] utilizes knowledge distillation techniques to aggregate locally computed logits, allowing for the synthesis of cohesive global models without requiring uniform local model architectures. Further addressing this heterogeneity, the works [43, 41] propose algorithms that adjust the training objective to tackle these challenges effectively. The work [44] introduced SCAFFOLD, which employs control variables for variance reduction to correct ‘client-drift’ in local updates. Moreover, the work [45] presents FedSAM, an algorithm that integrates sharpness aware minimization (SAM) as a local optimizer with a momentum-based federated learning algorithm to enhance synergy between local and global models.

In CP involving CAVs and RSUs, the exchange of sensor data raises significant privacy concerns. This data, sourced from vehicles and smart city infrastructure, can reveal sensitive information about individuals, such as location and personal habits. To address this, the work [46] applied federated learning to object detection tasks in AVs, capitalizing on the method’s ability to train models without compromising data privacy. However, the broader application of federated learning in CP was initially underexplored. Recent studies such as [21], [22] and [47] discuss the application of federated learning in CP settings. The work [48] introduces an innovative federated learning framework, H2-Fed, designed to handle Hierarchical Heterogeneity, significantly improving conventional pre-trained deep learning models for CP. Furthermore, the work [49] proposes a new client selection pipeline using V2X communications, optimizing client selection based on predicted communication latency in real CP scenarios. These developments underscore the potential for federated learning for collaborative model training in privacy-sensitive environments. Despite these advances, implementing CP through federated learning presents challenges, primarily due to heterogeneity in data distribution, computational resources, and network connectivity among devices. This variability can significantly affect learning efficiency and effectiveness. To address these challenges, the work [50] introduces a fair and efficient algorithm to manage the imbalanced distribution of data and fluctuating channel conditions. Moreover, the work [27] proposed FedBEVT, a federated transformer learning approach that focuses on BEV perception to tackle common data heterogeneity issues in CP.

In this paper, we aim to establish a federated learning framework, $FL$, consisting of $M$ clients that each equipped with a unique label $m\in\{1,2,...,M\}$, for the BEV semantic segmentation task under V2X CP setting. The $m$-th client contains a set of data samples denoted by $\mathbf{N}_{m}=\{(\mathbf{N}_{m,i},\mathbf{Y}_{m,i})\}_{i\in\{1,2,...,|\mathbf{N}_{m}|\}}$, where $|\cdot|$ denotes the number of elements in the set, $\mathbf{N}_{m,i}$ is the set of images in the $i$-th data sample captured from a multi-view camera system, and $\mathbf{Y}_{m,i}$ is the set of corresponding BEV semantic masks. Within this framework, each client $m$ trains a local model $G_{m}$ independently using the calculated gradient, by minimizing a primary optimization objective loss function $L_{m,i}$ for each data sample in their dataset, and then transmits the model updates to a central server. Thus this loss function is crucial for assessing and minimizing the error throughout the distributed network of clients $m$. The server integrates these updates to improve the global model. This cycle of local training and central updating repeats across several communication rounds to refine the global model optimally.

To demonstrate the effectiveness of our proposed federated learning framework, we follow FedBEVT [27] to employ the same transformer-based BEV segmentation network as network model $G_{m}$ and $G_{g}$, on each client and server respectively. Such network model comprises an image feature encoder utilizing a CNN, a BEV transformer with a multi-layer attention structure (incorporating both sparse cross-view and self-attention mechanisms), and a BEV decoder. This configuration effectively encodes images from multiple cameras and transforms them into BEV features. These features are decoded to produce semantic segmentation outputs on the BEV perspective.

Our framework introduces a novel aggregation algorithm and a customized loss function designed to tackle two primary challenges for CP setting: (i) the heterogeneity of data and distribution shifts across different clients, and (ii) the varying contributions of different clients to the convergence of the global model. These enhancements are vital to ensure robust and precise model performance in autonomous driving environments.

To address the specific challenges of applying federated learning in CP, particularly client drift due to diverse sensor types and Non-IID data distributions, we introduce the federated dynamic weighted aggregation (FedDWA) algorithm for our federated learning framework $FL$. This algorithm updates the client model described by:

where $G_{m}$ represents (the weights of) the local model of the $m$-th client, $\eta_{m}$ is the local step-size, and $\partial_{i}(G_{m}^{-})$ denotes the gradient of the $i$-th data sample for the local model in the previous communication round.

Each client uses a local control variable $c_{m}$ to account for variances in model updates, aligning local objectives with the global model to mitigate client-specific drifts, and $c_{m}^{-}$ denotes the local control variable in the previous communication round. Both $c_{m}$ and $c_{g}$ are initialized as 0.

Inspired by [44], we maintain a global control variable $c_{g}$ on the server, aggregating information from all clients to guide the overall direction of model updates. Diverging from [44], which uses the average of local variables to update $c_{g}$, we utilize Kullback-Leibler divergence (KLD) [51] to dynamically weigh the contributions of different clients like:

where $T_{m}$ is the intermediate variable, as shown in:

and

The KLD is calculated between the predicted data distribution of the global model from the previous communication round, $P_{g}^{-}(\mathbf{N}_{m,i})$, and the distribution of $m$-th local client, $P_{m}(\mathbf{N}_{m,i})$. Where $P_{g}^{-}(\mathbf{N}_{m,i})$ is approximated by using the segmentation output of global model from previous communication round $G^{-}_{g}$, and $P_{m}(\mathbf{N}_{m,i})$ is approximated by using the segmentation output of each local model $G_{m}$. To transform the segmentation output into a distribution, a log softmax function is applied to convert the segmentation output into log probabilities, e.g.:

where $V_{j,k}$ denotes the $(j,k)$-th pixel values in the segmentation output matrix $G_{m}(\mathbf{N}_{m,i})$ and $\sigma$ represents the softmax function. The same calculation is applied for $P_{g}^{-}(\mathbf{N}_{m,i})$.

Each client uses gradients to update the local control variable $c_{m}$:

where

The global model is updated as outlined below:

where $\eta_{g}$, the learning rate, plays a crucial role in balancing the updates, and $G_{g}$ represents the global model. This equation captures the aggregation of differences between the global and local models, promoting the convergence of the global model. To provide an comprehensive understanding of FedDWA, the pseudo code is illustrated as Algorithm 1.

As discussed before, it is critical to define a suitable loss function $L_{m,i}$, to calculate the gradient of $i$-th data sample for each client’s local model, $\partial_{i}(G_{m})$. The cross-entropy loss function is widely used within federated learning frameworks to train individual clients. This function effectively gauges the performance of classification models that output probabilities between 0 and 1. Nevertheless, the inherent data heterogeneity across clients, particularly with non-IID data, complicates the convergence to a global optimum.

To address this issue, we introduce an enhanced local training loss function termed dynamic adjusting loss (DALoss):

where $\mathcal{S}\left(G_{m}(\mathbf{N}_{m,i}),\mathbf{Y}_{m,i}\right)$ denotes the standard cross-entropy loss for the $i$-th data sample of client model $G_{m}$ against the true labels $\mathbf{Y}_{m,i}$, and $Q_{m,i}$ is the dynamic control term defined by:

where $C\cdot\mathcal{D}_{\mathrm{KL}}\left(P_{g}^{-}(\mathbf{N}_{m,i}),P_{m}(\mathbf{N}_{m,i})\right)$ scales the quadratic penalty based on the KLD, measuring the discrepancy between the probabilistic distributions of the global and local models from the previous communication round. $C$ is a regularization constant, and $||\cdot||_{2}$ denotes the $L^{2}$ norm. This divergence quantifies the differences, ensuring that local updates contribute significantly toward global model convergence.

This modification incorporates a dynamic adjustable regularization mechanism to better align each client’s model updates with the global model. Such dynamic adjustment of the regularization parameter allows our approach to adapt to the evolving nature of the training process and the unique characteristics of each client.

By integrating KLD into the loss function, the learning process is encouraged to minimize deviations between data distribution in different clients, effectively aligning the local models closer to the global model. Specifically, the KLD is employed as a weighting factor in the L2-norm calculation between the global and local models, as shown in Eq. (11). A large KLD indicates a significant discrepancy in the distributions, resulting in an increased weight. This increment in weight amplifies the loss, consequently accelerating the convergence of the local model towards the global model. By minimizing this divergence between local updates and the global model, the learning process adaptively adjusts to the unique data distributions present in each client. Therefore, our approach ensures that local updates contribute positively towards a more generalized global model, mitigating the impact of data distribution shifts and data heterogeneity.

While the design of the loss function allows for larger updates when there are significant discrepancies between the local and global models, it is crucial to manage the potential instability might be introduced by these aggressive updates. To avoid such instability, we use a regularization constant $C$, which can penalize large changes in model weights and provide a counterbalance to aggressive updates. We also use adaptive learning rates which can adjust the step size based on the training dynamics that we reduce the learning rate after several aggregation steps thus tempering the update magnitude.

We do the experiment on the dataset provided by FedBeVT [27], which extends the original OpenV2V [33] dataset only contained car objects by adding two types of vehicles: truck and bus with different camera sensor installation positions. This enhancement broadens the diversity and volume of the scenarios represented, providing a robust foundation for testing our model.

Following the protocol established in [27], our study orchestrates a collaborative framework among four distinct industrial clients to assess the efficacy of our proposed object detection model under various operational conditions Specifically, the dataset comprises:

• •

  Client 1 Bus: 1,398 frames in the training set and 413 frames in the testing set.

• •

  Client 2 Truck: 1,459 frames in the training set and 356 frames in the testing set.

• •

  Client 3 Car A: 11,636 frames in the training set and 3,546 frames in the testing set.

• •

  Client 4 Car B: 7,142 frames in the training set and 10,191 frames in the testing set.

For a fair comparison between our federated learning strategy and state-of-the-art approach FedBEVT [27], our model architecture adheres to the configuration described in [27]. This involves processing camera images observed by clients. Similar to [27], we begin by sending these images through a 3-layer ResNet34 encoder, which is proficient at extracting and encoding image features across various spatial dimensions. This encoding process yields feature tensors at differentiated spatial resolutions: $64\times 64\times 128$, $32\times 32\times 256$, and $16\times 16\times 512$.

Subsequently, we employ a focused attention cross (FAX) transformer operation, which is attention-based and skillfully manages interactions between BEV embeddings and the encoded image features, conceptualized as query, key, and value within this framework. Finally, a 3-layer bilinear module is used to transform these enriched BEV features. In our model configuration, the hyper-parameter $C$ defined in Equation (11) is set to 0.1. For other hyper-parameters, we adopt the configurations detailed in [27]. We selected these settings due to their proven effectiveness in similar contexts, ensuring that our model benefits from established best practices while maintaining consistency with state-of-the-art for fair comparison.

In our comparative study, we evaluate the performance of our proposed algorithm, FedDWA, against several established methods, including FedAvg [40], FedRep [52], FedTP [53], and FedBEVT [27]. The evaluation focuses on the average intersection over union (IoU) achieved by models across three use cases (UCs) and the number of communication rounds required to attain optimal IoU values.

The pixel-wise IoU metric is a common evaluation metric for BEV semantic segmentation. It quantifies the overlap between the predicted segmentation and the ground truth, providing a measure of accuracy at the pixel level. The IoU is calculated as follows:

where TP is the number of true positive pixels, FP is the number of false positive pixels, and FN is the number of false negative pixels.

The experimental analysis presented in Table I provides a comprehensive comparison of our federated learning framework against these prominent approaches.

Performance metrics include training loss, IoU, and the number of communication rounds (Com.) required to achieve peak IoU, offering a detailed assessment of the efficacy and efficiency of our approach within a federated learning context.

The key findings for each client scenario are as follows:

• •

  Client 1 Bus: Our method shows a significant improvement, achieving a 10.72$\%$ IoU after 235 communication rounds, demonstrating effective data heterogeneity management and communication efficiency.

• •

  Client 2 Truck: Our approach excels, reaching a 15.91$\%$ IoU, marking a notable improvement over local training and indicating robust adaptability and superior management of client-specific data characteristics.

• •

  Client 3 Car A: Achieving the highest IoU of 21.30$\%$ among all methods, our approach excels in complex vehicular environments, underscoring its potential to optimize the federated learning framework across diverse datasets.

• •

  Client 4 Car B: This client achieves a 19.35$\%$ IoU in just 100 communication rounds, highlighting both the efficacy and efficiency of our approach and establishing it as a leading method in federated learning for BEV perception.

Fig. 1 illustrates the relationship between the number of communication rounds and IoU during the testing phase for the four clients, providing a detailed view of test progression. Notably, the truck client and car client B reached optimal IoU with fewer communication rounds, potentially due to data distributions more closely aligned with the predictions of their local models. However, after reaching peak IoU, there is a noticeable decline in performance, suggesting possible overfitting as the models become excessively tailored to their local datasets, to the detriment of generalization.

The ablation study summarized in Table II elucidates the incremental impact of our DALoss and the intrinsic benefits of our proposed FedDWA framework. This study methodically analyzes the improvements offered by DALoss. The addition of DALoss consistently improves performance across all clients. When DALoss is added to FedBEVT [27], there is a noticeable improvement in IoU percentages for Client 2 (from 7.33$\%$ to 11.97$\%$) and modest gains across other clients. This indicates that DALoss facilitates better generalization. Our proposed FedDWA framework, even without DALoss, shows better performance than FedBEVT. For example, FedDWA achieves an IoU of 20.83$\%$ for Client 3 and 18.28$\%$ for Client 4, which are significant improvements over FedBEVT’s 18.40$\%$ and 16.16, respectively. This enhancement underscores the effectiveness of our weighting approach in distributing model updates across different clients in a way that respects their unique data distributions. The combination of FedDWA and DALoss shows the most substantial gains in performance. For instance, Client 2’s IoU increases from 14.88$\%$ with only FedDWA to 15.91$\%$ with the addition of DALoss, and similar incremental benefits are observed for other clients. The highest performance is observed in Client 3, where the IoU reaches 21.3$\%$. Fig 2 illustrates the visualization of an example network model output across different clients. It compares the results among the ground truth, the current state-of-the-art approach FedBEVT [27], the FedDWA only, and both FedDWA and DALoss on bus client, truck client and car client. The comparison shows that the model incorporating both FedDWA and DALoss achieves better performance.