The Impact of Cut Layer Selection in Split Federated Learning

Federated Learning (FL) was introduced by McMahan et al. (McMahan et al. 2017) as a distributed learning paradigm where multiple clients collaborate to train a model while keeping their data local. In FL, each client trains a complete copy of the model on their local data, and a model synchronization server aggregates these models to create a global model. The most widely-used FL algorithm, FedAvg, performs weighted averaging of client models based on their local dataset sizes.

Data heterogeneity represents one of the fundamental challenges in FL. When clients have heterogeneous data distributions, FedAvg can suffer significant performance degradation due to the notorious client drift issue (Hsu, Qi, and Brown 2019). Several approaches have been proposed to address this challenge: SCAFFOLD (Karimireddy et al. 2020) uses control variates to correct for client drift in non-IID settings, while FedPer (Arivazhagan et al. 2019) maintains personalized layers on clients while sharing base layers globally. Other approaches like FedDF (Lin et al. 2020) employ ensemble distillation for better heterogeneous model aggregation, and FedProx (Li et al. 2020) adds a proximal term to limit local updates from diverging from the global model. Recent works have introduced novel approaches to tackle heterogeneity: FedUV (Son et al. 2024) introduces regularization terms to emulate IID settings locally by controlling classifier variance and representation uniformity. MOON (Li, He, and Song 2021) leverages model-level contrastive learning to correct local training. FedAlign (Sun et al. 2024) addresses data heterogeneity through data-free knowledge distillation, using a generator to estimate global feature distributions and align local models accordingly.¹¹1While these techniques were primarily designed for FL, they can be easily implemented in SFL.

Split Learning (SL), introduced by Gupta and Raskar (Gupta and Raskar 2018), takes a different approach to distributed learning. Instead of training complete models on clients, SL splits a neural network at a cut layer, with initial layers on clients and remaining layers on the server. In SL, clients process data through their local layers up to the cut layer, then send activations to the server. The server completes the forward pass, computes gradients, and sends these gradients back to clients for local model updates.

Several works have explored SL’s potential for handling heterogeneous data distributions. SplitAVG (Zhang et al. 2022) combines network splitting with feature map concatenation to train an unbiased estimator of the target data distribution. Li and Lyu (Li and Lyu 2024) provided theoretical convergence guarantees for split learning, suggesting potential advantages over FedAvg on heterogeneous data. COMSPLIT (Ninkovic et al. 2024) introduced a communication-aware SL framework for time series data, incorporating early-exit strategies to handle devices with heterogeneous computational capabilities. FedCST (Wang et al. 2024) proposed a hybrid approach combining device clustering with SL to address both data and device heterogeneity. AdaSplit (Chopra et al. 2021) focused on improving SL’s performance across heterogeneous clients while reducing bandwidth consumption through adaptive mechanisms. In the domain of graph neural networks, SplitGNN (Xu et al. 2023b) addressed heterogeneity in graph data through a split architecture with heterogeneous attention mechanisms. RoofSplit (Huang et al. 2023) proposed an edge computing framework that optimally splits CNN models across heterogeneous edge nodes based on Roofline theory (Williams, Waterman, and Patterson 2009).

Split Federated Learning (SFL) was originally proposed by Thapa et al. (Thapa et al. 2022) as a hybrid approach combining FL and SL. Since its introduction, several works have explored various aspects of SFL, from privacy and security to system optimization and scalability.

Privacy preservation has been a key focus in SFL research. Li et al. (Li et al. 2022) proposed ResSFL, a framework designed to be resistant to model inversion attacks during training through attacker-aware training of the feature extractor. Khan et al. (Khan et al. 2022) conducted the first empirical analysis of SFL’s robustness against model poisoning attacks, demonstrating that SFL’s lower-dimensional model updates provide inherent robustness compared to traditional FL. Zhang et al. (Zhang et al. 2023) demonstrated that deeper cut layers in SFL improve privacy by reducing reconstruction accuracy at the cost of increased client computation and communication overhead. Zheng et al. (Zheng, Chen, and Lai 2024) introduced PPSFL, incorporating private group normalization layers to protect privacy while addressing data heterogeneity.

System optimization and efficiency have been another major research direction. Mu et al. (Mu and Shen 2023) developed CSE-FSL to reduce communication overhead and server storage requirements through an auxiliary network and selective epoch communication. Gao et al. (Gao et al. 2024) proposed PipeSFL, a fine-grained parallelization framework that addresses client heterogeneity through priority scheduling and hybrid training modes. Xu et al. (Xu et al. 2023a) tackled the challenge of heterogeneous devices by jointly optimizing cut layer selection and bandwidth allocation.

Several works have focused on enhancing SFL’s performance and applicability. Yang et al. (Yang et al. 2022) introduced RoS-FL specifically for U-shaped medical image networks, incorporating a dynamic weight correction strategy to stabilize training and prevent model drift. Li et al. (Li et al. 2023) demonstrated SFL’s effectiveness in healthcare settings, achieving comparable performance to centralized and federated approaches while improving privacy and computational efficiency. Shin et al. (Shin et al. 2023) proposed FedSplitX to handle system heterogeneity in foundation models by implementing multiple partition points and auxiliary networks. Liao et al. (Liao et al. 2024) developed MergeSFL, which combines feature merging and batch size regulation to improve accuracy and training efficiency. Zhu et al. (Zhu et al. 2024) proposed ESFL, which jointly optimizes user-side workload and server-side resource allocation for heterogeneous environments.

Recent work has begun to examine the implications of cut layer selection, though primarily from system-level perspectives. Lee et al. (Lee et al. 2024) provided a comprehensive analysis of how cut layer placement affects both energy consumption and privacy preservation in SFL systems, demonstrating that cut layer selection creates important trade-offs between computational efficiency and data protection. Shiranthika et al. (Shiranthika et al. 2023) investigated the robustness of different cut layer positions against network packet loss, finding statistical evidence that deeper split points can provide advantages in certain applications. While these works provide valuable insights into system-level tradeoffs, they do not address the question of how cut layer selection affects model performance. Note that recent work has examined SFL convergence properties (Han et al. 2024b), but it does not analyze the specific effects of cut layer selection. Our work addresses this gap by providing the first comprehensive analysis of how cut layer selection affects model performance across various datasets, architectures, and data distributions.

We conduct experiments on CIFAR-10 (Krizhevsky, Hinton et al. 2009) and HAM10000 (Tschandl 2018) with ResNet-18, and CIFAR-100 (Krizhevsky, Hinton et al. 2009) and Tiny ImageNet (Le and Yang 2015) with ResNet-50. The architectures use cut layer $L_{c}\in\{1,2,3,4\}$, where $L_{c}=i$ denotes the network is cut following the $i$-th residual block. These cut points correspond to the boundaries between macro-residual blocks in ResNet architectures (He et al. 2016). We set the number of clients $K=100$, except for Tiny ImageNet, where $K=10$. The models, numbers of clients, and rounds of training used for each dataset are enumerated in Table 1. The local training epochs for each client were set as $E=5$. Hyperparameters for all algorithms are enumerated in Table 2.

We consider both IID and non-IID scenarios. In our non-IID scenario, we use a label-based Dirichlet distribution (Hsu, Qi, and Brown 2019; Li, He, and Song 2021) with $\mu=0.1$ to simulate highly imbalanced datasets across clients. Here, $\mu>0$ is a tunable parameter, where a smaller value of $\mu$ refers to a more imbalanced partition and $\mu=\infty$ corresponds to IID data partition. We exclude the HAM10000 dataset from this process because the dataset itself is highly label skewed. The results for IID and non-IID settings are presented in Tables 3 and 4, respectively. Bold values indicate the best performance for each dataset, while underlined values show the second-best results.

Our experiments demonstrate that SFL, particularly SFL-V2, often outperforms FL. In IID settings, SFL-V2 ($L_{c}=1$) consistently outperforms FedAvg across all datasets. For CIFAR10, SFL-V2 achieves $92.30\%$ accuracy compared to FedAvg’s $85.25\%$, a significant $7.05\%$ improvement. The performance gap widens in non-IID scenarios. On CIFAR100, SFL-V2 ($L_{c}=1$) reaches $52.38\%$, while FedAvg achieves $42.60\%$, a substantial $9.78\%$ difference. This advantage is maintained with increasing dataset complexity: For non-IID TinyImageNet, SFL-V2 ($L_{c}=1$) achieves $30.14\%$ versus FedAvg’s $28.33\%$.

Our results suggest that SFL-V2’s superior performance stems from its unique architecture combining local and centralized computation. With an early cut layer, most of the model computation occurs at the training server, where the server-side model processes features from all clients without requiring weight aggregation. Unlike FedAvg, where models must reconcile entirely separate local training trajectories through weight averaging, SFL-V2’s server-side model directly updates on client features through back propagation. This more direct form of learning appears to be more effective than weight aggregation, particularly for heterogeneous data distributions. The sequential processing of client features by a single shared server-side model, combined with client-side model aggregation, enables SFL-V2 to learn global patterns more efficiently than traditional FL approaches. This helps explain why SFL-V2 can surpass FedAvg, particularly when the cut layer is placed early in the network.

While our experiments demonstrate SFL-V2’s strong performance in non-IID settings, several important theoretical questions remain unexplored. First, we lack a theoretical framework explaining why SFL-V2’s architecture provides advantages at certain cut layer selections. Understanding this could help optimize cut layer placement and guide architectural choices for different types of data heterogeneity. Additionally, while we’ve shown empirical improvements, we haven’t established theoretical guarantees for the advantage of SFL-V2 over FedAvg. A comprehensive theoretical analysis would need to account for the complex interplay between data heterogeneity, model architecture, and system constraints like client computation and communication resources.

Although SFL maintains some level of privacy by keeping part of the model on client devices, the improved performance with lower cut layers in SFL-V2 raises important privacy considerations. As more layers are shifted to the server, there’s an increased risk of potential privacy leakage through model inversions or inference attacks. Further research is needed to quantify these risks and develop mitigation strategies, possibly through the integration of differential privacy techniques or secure multi-party computation.