Communication-Efficient Personalized Federal Graph Learning via Low-Rank Decomposition

Federated Learning (FL) methods [34, 29] aim to learn a global model by aggregating the local models of all clients. However, achieving a global model that generalizes well to each client is challenging due to data heterogeneity. Personalized Federated Learning (PFL) shifts the focus from a traditional server-centric federation to a client-centric one, better suited for non-IID data. Existing PFL methods can be categorized into three types. The first type of method is based on regularization. Scaffold [26] introduces proximal regularizers to mitigate client-side drift and accelerate training. FedProx [29] incorporates a regularization term to balance local training with the global model, promoting faster convergence. These methods improve training and inference efficiency by controlling model parameters. However, selecting regularization parameters often requires tuning through cross-validation. Inappropriate parameter choices can over-penalise personalised parameters and lead to over-fitting of local data. The second type involves personalized aggregation methods to learn local models. FedAMP [35] generates aggregated models for individual clients using attention-inducing functions and personalized aggregation. APPLE [30] performs local aggregation within each training batch rather than just local initialization. However, these methods are typically designed for cross-silo FL settings, which require all clients to participate in each iteration. The third type separates the global and personalization layers, combining global knowledge with models personalized for each client. This paper focuses on this type of method, which is widely studied for its adaptability and learning effectiveness. For instance, FedRep [27] divides the backbone into a global model and client-specific headers, fine-tuning the headers locally. pFedMe [28] learns additional personalized models for clients using Moreau envelopes. FedSLR [31] employs a two-stage proximity algorithm for optimization. Although these methods can help the model better adapt to client data, they lead to increased communication overhead as parameters are transmitted separately.

Compared to FedRep, pFedMe, and FedSLR, our method downloads a low-rank and compressed global model from the server, significantly reducing communication complexity in upstream and downstream links. We also employ accelerated local training techniques, effectively reducing communication frequency. Additionally, while the methods above are tailored for images and unsuitable for complex non-Euclidean data like graphs, our method targets highly heterogeneous cross-domain datasets. It effectively reduces communication costs while maintaining high accuracy, making it more suitable for highly heterogeneous cross-domain graph tasks.

While numerous studies have investigated federated learning, federated graph learning (FGL) remains underexplored. Unlike Euclidean data, such as images, graph data is inherently more complex, introducing new challenges and opportunities for FGL [36, 37]. FGL enables clients to train robust GNN models in a distributed manner without sharing private data, thus expanding its application scenarios. A common strategy for training personalized federated graph learning models is client-side clustering. For instance, in IFCA [38], clients are dynamically assigned to multiple clusters based on the latest graph model gradient. However, a drawback of this method is that the clustering results are significantly influenced by the latest gradients from the clients, which are often unstable during local training. To address this issue, GCFL [24] considers a series of gradient specifications in the client cluster, and CTFL [39] updates the global model based on representative clients for each cluster. Maintaining a shared global model in each cluster increases the number of parameters and computational requirements as the number of clients grows, leading to poor scalability. Recent approaches have focused on reducing communication costs while addressing non-IID challenges. For instance, FedStar [40] mitigates the non-IID issue by employing feature decoupling. Meanwhile, FedGCN [41] reduces communication overhead by minimizing the frequency of communication with a centralized server and incorporates homomorphic encryption to further enhance privacy. Despite these advancements, these methods face challenges in highly heterogeneous cross-domain scenarios, as they can only fine-tune global model parameters to fit local data.

CEFGL significantly reduces the server’s computational cost compared to clustering-based multi-model FGL methods, as it only requires generating a unified global model for all clients. Additionally, compared to FGL methods that only adjust local models, the proposed method employs a hybrid model that combines low-rank global components with sparse personalized components. This hybrid model is more adaptable to the highly heterogeneous cross-dataset graph learning task.

In robust principal component analysis (RPCA) [42], the matrix $\bm{M}$ is decomposed into a low-rank matrix $\bm{W}$ and a sparse matrix $\bm{S}$, where $\bm{W}$ captures the principal components retaining the maximum information in $\bm{M}$, while $\bm{S}$ represents the sparse outliers. Motivated by RPCA, we decompose the local model into a shared low-rank component and a private sparse component, expressed as:

where $rank(\bm{W})$ is the rank of the matrix $\bm{W}$, $\|\bm{S}\|_{0}$ is the number of nonzero elements in $\bm{S}$, and $\nu$ is the parameter used to trade-off between these two terms. Since the rank function and the $L_{0}$-norm are non-convex, we can minimize their convex proxies as

where $\|\bm{W}\|_{*}$ is the trace paradigm number of $\bm{W}$, i.e., the sum of the singular values of $\bm{W}$. $\|\bm{S}\|_{1}$ is the $L_{1}$-norm, i.e., the sum of the absolute values of all elements in $\bm{S}$.

Local training In the local training phase, we first use a multilayer perceptron (MLP) to transform feature information into vectors of consistent dimensions. Subsequently, we employ a dual-channel encoder to represent the low-rank component of global knowledge and the sparse component of personalized knowledge. Inspired by Scaffnew [33], we introduce a correction term $\bm{h}_{i}^{t}$ to record the local gradient, accelerating local training and mitigating client drift. This term contains information about the client’s model updating direction, akin to the client-drift value of the global model relative to the local model. The optimization process for learning the low-rank component is expressed as follows:

where $\eta$ represents the learning rate. $\alpha$ is a hyper-parameter used to balance the weight between the global model and the prior local model. $\bm{\Theta}^{t}$ is the global model downloaded from the server in the $t$-th iteration, and $\bm{W}_{i}^{t}$ is the low-rank component on client $i$ at the $t$-th iteration. The function $f$ represents the neural network transformation, and the cross-entropy loss function is employed. Initially, $\bm{W}_{i}^{0}=\bm{\Theta}^{0}$.

After completing the local training, we update the correction term in each round as follows:

Fine-tuning In the fine-tuning phase, we freeze the low-rank component $\bm{W}_{i}^{t}$ learned in the local training phase and fine-tune the sparse component using the initialized global parameters to fuse personalized knowledge with global knowledge. The optimization process of sparse component learning can be represented as follows:

where $\operatorname{Sparse}(\cdot)$ represents the sparsification function. We explore two types of sparsification: threshold sparsification and Top-$k$ sparsification. Threshold sparsification involves setting model parameters with absolute values smaller than a given threshold $\mu$ to zero, thus achieving parameter sparsity. Top-$k$ sparsification retains the $k$ parameters with the largest absolute values in the parameter matrix, setting the rest to zero.

After completing the local training and fine-tuning, the client sends the trained low-rank component $\bm{W}_{i}^{t}$ and the updated correction term $\bm{h}_{i}^{t+1}$ to the server. The server aggregates the client models and performs the low-rank decomposition, which can be represented as follows:

where $K$ denotes the number of clients participating in training, $|\mathcal{D}_{i}|$ denotes the total number of samples on client $c_{i}$, $|\mathcal{D}|$ denotes the total number of samples on $K$ clients, and $\operatorname{Low-rank}(\cdot)$ denotes the low-rank regularization. In this paper, the low-rank method we use is truncated singular value decomposition (TSVD), i.e., only singular values larger than a specific threshold $\lambda$ and their corresponding vectors are retained.

In the algorithm described above, the model after server aggregation is low-rank, reducing communication costs to some extent. However, since the local model uploaded by the client remains dense, the uplink communication cost is not reduced. To address this, we adopt a strategy of updating the local parameters several times before aggregation and integrating quantization compression into the algorithm.

Quantization compression involves converting model parameters from raw floating-point values to representations using fewer bits. While this reduces the accuracy of parameter representation, it significantly reduces the storage space and memory footprint of the model, thereby enhancing its efficiency during inference. For any vector $\bm{x}\in\mathbb{R}^{d}$, with $\bm{x}\neq 0$ and a number of bits $r>0$, its binary quantization $Q_{r}$ is defined componentwise as follows:

where $\xi_{i}(\bm{x},2^{r})$ is independent random variable and $\|\cdot\|_{2}$ is $L_{2}$-norm. The probability distribution is given by

where round($\cdot$) means round to the nearest integer.

Our proposed algorithm reduces the need for frequent client-server communication, significantly lowering communication frequency. Importantly, our method is less susceptible to data leakage than other methods, as the uncertain local step size prevents the server from accurately recovering the client’s actual gradient. Additionally, clients use quantization compression to reduce parameter size from 32 bits to 4 bits before uploading them to the server. Similarly, the server compresses the model before sending it to the client. This compression is particularly beneficial for downloading the model from the server, which can be costly due to network bandwidth limitations. Overall, these strategies significantly reduce both uplink and downlink communication costs.

To comprehensively evaluate the performance of CEFGL, we conducted experiments using three commonly used graph neural network architectures: GIN, GCN, and GraphSage. TABLE 1 and TABLE 2 present the average test accuracy and standard deviation for graph classification across six non-IID cross-dataset and three single-dataset settings. Our results consistently demonstrate that CEFGL outperforms all baseline methods across the three GNN architectures. In the highly heterogeneous cross-dataset graph learning task, CEFGL achieves an accuracy that is 2.11% higher than FedPerGCN, 2.82% higher than SpreadGNN, and 6.92% higher than GCFL. This significant improvement highlights the ability of CEFGL to effectively learn and generalize across diverse datasets, leveraging its dual-channel encoder to capture global and personalized knowledge efficiently. In the single dataset task, some personalization solutions exhibit performance degradation in the IID setting, failing to match the performance of global solutions like FedAvg. However, CEFGL maintains the highest performance in the IID setting, which indicates that its sparse personalization information effectively complements global information without overwhelming it, thereby preserving model generalization. This balance between personalization and generalization is crucial for maintaining high performance across different data distributions.

The superiority of CEFGL is further highlighted by its performance consistency across various setups. In non-IID settings, where data heterogeneity poses significant challenges, the architecture of CEFGL ensures effective communication and learning, reducing client drift and enhancing convergence. In IID settings, the ability of CEFGL to retain high accuracy showcases its adaptability and strength in more uniform data distributions. These comprehensive results underscore the effectiveness of CEFGL in both heterogeneous non-IID and IID settings. By consistently outperforming existing methods, CEFGL proves its superiority in federated graph learning, offering a reliable solution for real-world applications where data distribution can vary significantly.

To evaluate the impact of the low-rank hyperparameter $\mu$ on the performance of the algorithm, we vary $\mu$ and record the low-rank rate and classification accuracy of the model $\bm{\Theta}^{t}$ for different values of $\mu$ using GIN as the backbone, as shown in TABLE 3. A $\mu$ value 0 indicates no low-rank regularization, resulting in dense models susceptible to non-IID data from other clients, leading to suboptimal performance in federated environments with heterogeneous data. As $\mu$ increases from 0, we observe an initial improvement in classification accuracy and significant reductions in communication cost and the number of parameters. This is because low-rank regularization helps filter out noise, capture common patterns in the data, and enhance the generalization of models across clients. For instance, the model balances compactness and representativeness at $\mu=0.001$, improving communication efficiency and accuracy. This optimal setting maximizes the extraction of global knowledge while maintaining a manageable number of parameters. However, if $\mu$ is set too large, for instance, 0.01, the model becomes overly constrained by the low-rank regularization. This excessive constraint can cause the model to lose important information for accurate classification, resulting in significant performance degradation. In conclusion, the low-rank hyperparameter $\mu$ plays a crucial role in balancing the trade-off between communication cost and classification accuracy. By choosing an appropriate $\mu$, our method enhances the extraction of global knowledge and improves overall model performance, demonstrating the effectiveness of using low-rank regularization in federated graph learning.

To assess the impact of the sparsity of the personalization component, we fix the low-rank hyperparameter $\mu$ at 0.001 and vary the sparsity hyperparameter $\lambda$ using GIN as the backbone. As shown in TABLE 4, a $\lambda$ value of 0 results in a dense personalization component, leading to unsatisfactory model performance. Dense models tend to overfit the local data, which is particularly problematic in federated learning scenarios with heterogeneous data distributions. As $\lambda$ increases, the number of parameters in the personalization model decreases, significantly improving accuracy. This improvement can be attributed to sparse regularization, which filters out redundant information, allowing the model to focus on the specific features of the data. We also evaluate sparsity using the Top-$k$ sparsity method, where $k=\beta\times N_{s}$ and $N_{s}$ denote the total number of parameters in the personalized model. As shown in Figure 2, the optimal performance can be achieved by using only 10% of the parameters. This suggests that appropriate sparse regularization enhances the performance of models. The model becomes more efficient and generalizable by effectively capturing local knowledge with fewer parameters.

The sparsity in the personalization component ensures that the model enhances the capture of local knowledge, thereby improving model performance. This is crucial in federated learning, where data from different clients can vary significantly. By controlling the sparsity of the personalization component, our method balances model flexibility with the need to maintain generalization across different data sources. In summary, these results highlight the crucial role of sparsity hyperparameters ($\lambda$ or $\beta$) in the efficiency and effectiveness of personalization components in federated learning. This adaptability makes our method an excellent choice for federated learning scenarios with IID and non-IID data distributions.

To verify the rationale and validity of the low-rank global component and the sparse personalized component, we compare several variants of CEFGL using GIN as the backbone, with results shown in TABLE 5. When the global component is zeroed out, severe performance is degraded. This is primarily due to the limited number of parameters and the absence of information exchange between clients. In this case, the local model relies solely on the sparse personalization component, which is insufficient for comprehensive learning, especially in a federated setting with diverse data distributions. Conversely, when the personalization components are zeroed out, the model becomes a standard federated graph learning algorithm. This method proves unsuitable for highly heterogeneous cross-dataset scenarios, as it lacks the adaptability provided by personalized components. Although the performance of this pure global component variant shows improvement over the pure personalization component, it still falls short of the hybrid model. The hybrid model, which integrates low-rank global and sparse personalized components, achieves superior performance. This configuration benefits from the complementary strengths of both components. The global component captures common patterns and facilitates information sharing across clients. In contrast, the personalized component adapts to local data variations, enhancing the overall flexibility and generalization capability of the model. These observations underscore that integrating low-rank global and sparse personalized components is crucial for achieving optimal performance, demonstrating the ability of the model to balance global knowledge extraction and local adaptation. This balanced method is particularly advantageous in federated learning with non-IID data.

We explore the effect of varying the expected number of local iterations on model performance. The expected number of localized iterations is defined as $1/p$, where $p$ denotes the communication probability. For instance, $p=0.1$ means that the client communicates with the server, on average, once every ten localized iterations. To understand the impact of this parameter, we investigate the model accuracy and loss for $p\in\{0.1,0.2,0.4,0.6,0.8,1.0\}$. The results, shown in Figure 3, reveal several key insights:

• 1.

  Model Performance and Convergence: Smaller values of $p$ (i.e., more localized training) do not significantly affect the overall performance or convergence of the model. The accuracy and loss metrics remain relatively stable across different values of $p$, indicating that the model is robust to changes in communication frequency.

• 2.

  Communication Cost: Lowering $p$ effectively reduces the communication frequency between the client and the server. Reduced communication frequency translates to lower communication costs, which is especially beneficial in federated learning scenarios where communication overhead can be a significant bottleneck.

• 3.

  Privacy Considerations: Variable localized training step sizes prevent the server from accurately recovering the actual gradient of the client, thereby avoiding privacy leakage to some extent.

While performance metrics remain stable, it is essential to consider the trade-off between communication frequency and the potential risk of model divergence. Excessive localization (very low $p$) may result in the clients’ models drifting too far apart before synchronization, potentially affecting long-term convergence. However, this risk appears minimal within the tested range ($0.1\leq p\leq 1.0$). The experiments demonstrate that lowering the communication probability $p$ can reduce communication costs and enhance privacy without significantly affecting model performance or convergence. This finding highlights the flexibility and robustness of our method, making it well-suited for federated learning environments where communication efficiency is critical. By adjusting the communication probabilities, we can balance maintaining model accuracy and reducing communication overhead.

We investigate the application of quantization compression $Q_{r}(\cdot)$ in CEFGL, varying the number of bits as $r\in\{4,8,16,32\}$. Using the quantization method proposed by Alistarh et al. [46], we present the results after 200 communication rounds in Figure 4. Using 4-bit quantization, which corresponds to a communication cost of only 12.5%, does not significantly degrade model performance. Accuracy and convergence speeds remain comparable to those achieved with higher bits quantization levels. It demonstrates that CEFGL effectively retains essential training information even with lower granularity parameters. Additionally, low-bit quantization reduces the amount of data transferred between the client and the server, significantly lowering communication costs.

The proposed method effectively reduces communication overhead through low-bit quantization without sacrificing model performance or convergence speed. This highlights the efficiency of our method, making it well-suited for large-scale federated learning deployments where communication efficiency is critical and enabling implementation in resource-constrained environments such as mobile and edge devices.

Due to fluctuating network connections, clients in mobile environments may unexpectedly exit in one iteration and rejoin in another. To evaluate the performance of various FGL methods under these conditions, we conduct experiments as shown in Figure 5. Unlike many existing methods with a fixed client drop rate, we sample the drop rate $\rho$ from a Beta distribution in each iteration. Larger values of $\rho$ indicate greater instability. The experimental results reveal several significant findings. The accuracy of most FGL methods decreases under unstable conditions. This is expected, as inconsistency in client participation undermines the ability of the model to aggregate information efficiently. However, CEFGL maintains the highest accuracy even under extremely unstable conditions, i.e., when $\rho$ is sampled from $\beta(10,1)$. The ability of CEFGL to maintain its dominance and consistent performance in unstable environments can be attributed to two main factors:

• 1.

  Unforced aggregation: CEFGL does not require aggregation at every iteration, allowing the model to continue learning even when some clients are unavailable.

• 2.

  Sparse Personalisation Component: The sparse personalization component in CEFGL effectively complements global knowledge. By focusing on the critical local information, the model adapts to changing environments without relying solely on continuous client engagement.

CEFGL is particularly suitable for FGL environments with unstable clients. It is designed to cope with the challenges posed by network fluctuations and to leverage sparse personalization to enhance the overall robustness and adaptability of the model. This makes CEFGL ideal for deployment in mobile and other dynamic environments where maintaining high accuracy and consistent performance is critical.