Split Learning without Local Weight Sharing
To Enhance Client-side Data Privacy

A deep model, denoted as $h_{\theta}:\mathcal{X}\mapsto\mathcal{Y}$, is a hypothesis that maps an input $x\in\mathcal{X}$ to an output $y\in\mathcal{Y}$. Model training involves finding the parameters (weights) $\theta$ that accurately capture the relationship between $\mathcal{X}$ and $\mathcal{Y}$. In order to ensure user data privacy during model training, SL [2] divides the layers of the deep model into multiple parts. In a simple vanilla setting, the model is split into two components: $h_{\theta}=f_{u}\cdot g_{w}$. The localized part $f_{u}$ contains the initial layers, while the remaining part $g_{w}$ is deployed on the server, which is typically the most computationally intensive component. During training, the client performs forward propagation on its local data batch and sends the resulting output (referred to as intermediate data or smashed data) along with the corresponding ground-truth labels $\left(f_{u}(x^{batch}),y^{batch}\right)$ to the server. The server continues forward propagation on the received smashed data to compute the loss between $y^{batch}$ and $g_{w}(f_{u}(x^{batch}))$. The gradients of the loss function are then backpropagated at the server until the split layer, at which point the deep model is cut/split. The split layer’s gradients are sent back to the client, where the remaining backpropagation is performed locally, all the way to the first layer. Based on the computed gradients, both the client and the server update their respective weights, $u$ and $w$. This process, known as simple vanilla SL, serves as the core mechanism for many other variants, including SL with multiple clients and our proposed approach.

SL can be extended to train a deep model on $N\geq 2$ clients. The deep model is also split into two parts, $f_{\theta}=f_{u}\cdot g_{w}$, where $f_{u}$ is distributed to all clients ($f_{u_{i}}$ to client $C_{i}$), and $g_{w}$ is deployed on the central server. The training procedure involves utilizing data from multiple clients in a round-robin fashion. In this setting, when the training process of client $C_{i-1}$ is completed, client $C_{i}$ receives the weights $u_{i-1}$ of $C_{i-1}$ to initialize its own weights $u_{i}$. Then, client $C_{i}$ continues training on its own data, collaborating with the server following the vanilla setting. Once the training is finished, client $C_{i}$ shares its trained weights, $u_{i}$, with the next client $C_{i+1}$ [2]. This process continues until the last client $C_{N}$ completes training, and the weights $u_{N}$ trained by client $C_{N}$ are the model weights that are passed back to all clients for inference.

The model training process in SL is typically performed sequentially among the clients, which can introduce significant latency. To address this issue, several studies have focused on improving the training speed. In [8], the authors set up the mini-batch of each client proportional to its local data size, allowing for parallel processing of the training model. All clients are initialized with the exact weights, and after each iteration, the gradients are averaged before being updated on the clients. This synchronization strategy ensures that all clients have the same model weights during training.

SplitFed learning (SFL) [9] is an innovative approach that combines the strengths of FL and SL. In SFL, clients perform forward propagation in parallel on their respective data and send the smashed data to a central server. Upon receiving the gradients from the server, the clients execute the backpropagation step and then send the updated weights to a Fed server. The Fed server aggregates the weight updates using an averaging function ($Avg(\cdot)$) and disseminates a single update to all clients. Similar to [8], in SFL, after each global epoch, clients synchronize their models with identical weights, which also renders them susceptible to model inversion attacks in a white-box setting such as an adversarial client which possesses the same model weights as the victims.

Critical privacy vulnerabilities of SL are based on the fact that a neural network is naturally predisposed to be functionally inverted [17]. That is, the smashed data exposed by clients may be exploited to recover the raw input data. Therefore, privacy protection techniques in SL typically aim to minimize data leakage from the smashed data. For example, the noise defense approach [18, 19] applies additive Laplacian noise to the smashed data before sending it to the server. By introducing noise, the target model is no longer a one-to-one function, making it harder for an attacker to learn the mapping from smashed to raw data. Another method involves adding latent noise through binarization [20] to reduce the correlation between smashed and input data. However, these mechanisms require efforts to mitigate the impact of noise perturbation on model accuracy [21].

The work [22] aims to reduce raw data leakage by adding an additional distance correlation-based loss term to the loss function. This distance correlation loss minimizes the correlation between the raw and smashed data, ensuring that the smashed data contains minimal information for reconstructing the raw data while still being valuable for achieving model utility. In [20], the authors extend this idea by suggesting that the additional loss term can be any leakage metric, not limited to distance correlation. However, the application of an extra loss term may still result in privacy leakage because the smashed data exposes too much information to be adequately protected by a single leakage metric in the loss function [17]. To overcome this limitation, the authors in [23] propose client-based privacy protection, which employs two different loss functions computed on the client and server sides. In line with this approach, [24] designs a framework consisting of two steps: a pre-training step that establishes a feature extractor with strong model-inversion resistance, and a resistance transfer step that initializes the client-side models using the feature extractor. This framework requires sufficient computational resources for pre-training on a source task and may be vulnerable during the early training epochs. Similarly, [25] proposes a learnable pruning filter for selectively removing channels in the latent representation space at the split layer (smashed data) to prevent various state-of-the-art reconstruction attacks. However, the pruned network in [25] may not be suitable for deployment on low-end clients due to the requirement of a large amount of computing power. To preserve both data privacy and label information, [26] employs sample diversity to mix the smashed data of client-side models and create obfuscated labels before transmitting them from clients to the server. The mixed smashed data maintains a low distance correlation with the raw data, thereby preventing private data from being individually reconstructed. However, it should be noted that this mixing technique does not effectively reduce data leakage as intended when performing inference on a single data sample.

In multi-head SL (MHSL) [27, 28], the authors explore the feasibility of SFL without requiring client-side synchronization. The objective is to reduce the extra communication and computation overhead at the client side using synchronization. The study further considers the case which the server gains information of raw data through the clients’ smashed data, but the evaluation does not determine which approach, MHSL or SFL, results in less information leakage. Moreover, the analysis solely focuses on leakage from visualizing smashed data, which exhibits limited quality as demonstrated in [20]. Federated Split Learning (FSL) in [23] is another study that considers not sharing clients’ neural network weights to speed up the training time. The authors propose setting up the same number of clients and edge servers for parallelizing the training with the aim of reducing learning latency. In contrast, our approach prioritizes privacy and we design the non-local-weight-sharing mechanism to address the privacy attacks under an extended threat model. Additionally, we evaluate our scheme across multiple scenarios and data distributions.

In general, data is often distributed among clients in an imbalanced manner. For example, some sensors may be more active than others, resulting more data from certain sources. Similarly, in domains like healthcare, larger institutions tend to have more patient data available [11]. In a classification task, under balanced data distribution, each client holds samples from all classes in similar quantities. However, in an imbalanced data distribution, each client still possesses samples from all classes, but the total number of samples for each class is imbalanced. It is important to note that the ratio of samples between classes at each client remains similar to the overall dataset ratio. In the study from [11], the authors investigate three different data distributions for user data: balanced, imbalanced, and non-IID (non-independent identically distributed). Their findings reveal that SL performs well (compared to FL) under both balanced and imbalanced data while being highly sensitive to non-IID data. Therefore, this study mainly focuses on investigating and evaluating SL specifically under balanced and imbalanced data settings.

In traditional SL, only the server is assumed to be honest-but-curious [29], meaning it follows the training procedure but may have a curiosity about the raw data from clients. This threat model is assumed in most of the literature on privacy protection techniques in SL. In our study, we extend this assumption to include honest-but-curious clients as well. To the best of our knowledge, our work is the first to consider privacy attacks from both honest-but-curious clients and server in SL.

We define the leakage of user data as the disparity between the reconstructed data and the original private raw data of a client. The quality of the reconstruction can be evaluated using various metrics such as mean squared error (MSE), structural similarity index measure (SSIM), peak signal-to-noise ratio (PNSR), Kullback–Leibler divergence, and so on [13, 30, 20, 31]. In our work, we utilize SSIM as the main metric for measuring data leakage.

To lessen the adverse effects of model inversion attacks, we propose a non-local-weight-sharing method at the client side for SL. The proposed P-SL algorithm is presented in Alg. 1, followed by computation analysis.

Fig. 2 illustrates the architecture of P-SL, highlighting its differences from SL and SFL. The proposed P-SL is based on multi-client SL, where multiple clients connect to a central server, and there is no communication among themselves (such as sharing snapshots [2] - local weights) or the use of a Fed server (for local model aggregation [9]). Alg. 1 presents the collaborative training procedure between clients and the server in the proposed P-SL. In the initial phase, clients and the server receive their corresponding parts, $C_{i}\leftarrow f_{u}$ and $S\leftarrow g_{w}$, from a split model, $h_{\theta}=f_{u}\cdot g_{w}$. Then, they initialize their model weights, $u_{i}$ and $w$. During a global epoch, following a round-robin manner, each client $C_{i}$ starts its training with the server, following the simple vanilla SL procedure, which is demonstrated by the inner while loop (lines $2-10$). Note that the box (lines $5-8$) represents the operations executed at the server, and the transmission of data between clients and the server (e.g., smashed data, labels, gradients, etc.) is done via a network connection. Once the training of $N$ clients is completed, we have $N$ different local models combined with the server model to form $N$ distinct deep models (i.e. $h_{\theta_{i}}=f_{u_{i}}\cdot g_{w}$ where $1\leq i\leq N$). After training, each client performs inference on its live data using its local private model in combination with the shared server model. In contrast to SL and SFL, P-SL maintains the client-server collaboration but prohibits weight exchanges among clients, thereby reducing local computation, which will be examined in the following.

The convergence guarantee of pure SL on homogeneous data is straightforward and can be reduced to the case of SGD [32]. However, the non-local-weight-sharing of P-SL introduces additional challenges due to the presence of multiple different local models that can be individually combined with the server model, denoted as $h_{\theta_{i}}=f_{u_{i}}\cdot g_{w}$ for training. Let us consider $g_{w}$ as the primary deep model being trained on the smashed data of all clients. If the training (smashed) data remains stable (with minimal changes), the convergence of P-SL can be simplified to the SGD case. Therefore, the convergence of P-SL relies on the convergence of each client. After a few training rounds, when a client’s local model has converged, indicating small local weight updates, the corresponding smashed data from that client also stabilizes. Stable training data facilitates the convergence of the server-side model’s training. Additionally, the learning of the server-side model is influenced by the size of the training data, specifically the size of the smashed data. A larger smashed data size leads to a more extensive training dataset, resulting in a more complex server-side model (e.g., more layers) required to learn and memorize. This insight aligns with the experiment findings in [27, 28], where the authors evaluate model accuracy with different cutting points (which determine the division of the deep model for deployment on clients and the server, respectively). Performance tends to degrade when the client model is thicker (possessing more local layers) because it requires more time to converge, and the corresponding smashed data size also increases. However, for low-end devices, it is preferable to have thin client models, which will facilitate the learning performance of P-SL, as previously discussed.

To analyze total computation and communication costs of P-SL and provide a comparison to SL and SFL, we assume a balanced data distribution for simplicity. Let us consider the following variables: $N$ as the number of clients, $|\mathcal{X}|$ as the total number of dataset items, $S$ as the size of the split layer (the last layer of $f_{u}$), $C^{P}$ as the computation cost for processing one forward and backward propagation on $f_{u}$ with one data item, $C^{U}$ as the cost for updating a client’s local weights from the received weights from the previous client or the Fed server, and $|U|$ as the size of the local model $f_{u}$. Table I demonstrates that the formulated computation and communication costs at the client side in P-SL are lower than SL and SFL, respectively, due to the absence of local weight sharing. The reduction in costs depends on the size of the local model and is independent on data distribution. The factor of $2$ in the communication costs represents the uploading of smashed data and the downloading of corresponding gradients ($2\frac{|\mathcal{X}|}{N}S$), or the uploading and downloading of local models ($2|U|$) at the client side.

In the proposed P-SL, each client conducts collaborative training with the server separately. This results in high latency and large idle time at the client side, as only one client is active during training. Therefore, we can process clients’ training simultaneously if multiple server instances are available. In Fig. 3, paralleling P-SL follows the steps below in each round:

1. 1.

   Setup phase: During this phase, all clients receive the same model $f_{u}$, and the server starts with model $g_{w}$. The server sets up a pool of $m$ instances (in this example, there are two instances).

2. 2.

   Client computation: Clients connect to the server and are associated with available instances. Then, they perform forward propagation on their local models using their local data in parallel and independently. After this, they send their smashed data to the server.

3. 3.

   Server computation: The corresponding server instances perform forward-backward operations on the received smashed data from the clients and send back the computed gradients.

4. 4.

   Client-server collaboration: The collaborative training between a client and the corresponding server instance is indicated by label ①. Upon completing the training, resulting in a pair client-server model, $f_{u_{i}}\cdot g_{w_{j}}$, the server instance becomes available and waits in the pool for the next client to connect (label ③).

5. 5.

   Server model aggregation: When a server instance becomes available after training, a snapshot of the server model weights, $w_{j}$, is recorded (label ②). After a certain period of time or a predetermined number of snapshots is recorded, the server aggregates (using the $Avg(.)$ function) all snapshots to form a new version of the server model weights, $w^{*}$. Then all server instances, $g_{w_{j}}$, update their weights to the new aggregated weights, $w^{*}$, for the next round of training.

It should be noted that the aggregation of the server models, $g_{w_{j}}$, is performed asynchronously, and the degree of parallelization depends on the number of server instances. The parallelization of P-SL differs from the client-side parallelization in SFL, as it does not require the Fed server for local model aggregation.

Using the selected datasets and local data distributions described in the previous section, we implement P-SL with $N=6$ clients and a central server. After training, we measure the learning performance of each client when performing inference on a test set collaboratively with the server ($h_{\theta_{i}}=f_{u_{i}}\cdot g_{w}$). We compare the results with multiple SL, referred to as $m$SL, where we set up $N$ different SL processes between $N$ client-server pairs ($h_{\theta_{i}}=f_{u_{i}}\cdot g_{w_{i}}$). The key difference between P-SL and $m$SL is the collaboration at the server side. In $m$SL, there are $m$ distinct server instances, each associated with a different client, and they do not collaboratively aggregate server models. Conversely, in P-SL, we utilize only one server instance, allowing the server model to learn from the smashed data of the entire dataset. We also include the results of SL and SFL for benchmarking reference in Table V.

The training accuracy of each client with the Fashion dataset is presented in Table VI, while the results with the CIFAR10 dataset are visualized in Fig. 5. With $m$SL, the accuracy of each client depends on the number of data samples held by that client, as expected. Therefore, under balanced data, these clients have similar accuracy (around $90\%$ with Fashion), while their accuracy ranges from lower ($C_{1}$ with $78.6\%$) to higher ($C_{6}$ with $92.1\%$) values with imbalanced data (see Table VI). We can visually observe similar results in Fig. 5, which shows the results using CIFAR10, a more complex and difficult dataset that leads to a higher accuracy difference between clients with fewer and more data samples.

In the proposed P-SL, despite the separate training of local models, the learning performance is better than $m$SL attributing to the shared server model, which aggregates knowledge from all clients. P-SL achieves a $3\%$ higher accuracy with the Fashion dataset and a $12\%$ higher accuracy with the CIFAR10 dataset compared to $m$SL under a balanced data distribution. Under an imbalanced distribution, the results are even more impressive as we observe significant accuracy improvements for clients with fewer data, such as $C_{1}$, $C_{2}$, etc. (see Fig. 5b). This demonstrates the benefit of P-SL in collaborative learning, even without weight sharing among clients. We also compare our results with SL and SFL, which achieve state-of-the-art collaborative learning performance. By sharing local models among clients, knowledge is aggregated at both the client and server sides, resulting in higher accuracy for SL and SFL compared to P-SL, which only aggregates knowledge at the server. In summary, our experiments demonstrate that P-SL, without local weight sharing, still benefits collaborative learning between multiple clients and a central server. Under imbalanced data distribution, clients with less data can learn more by training with clients with more data.

In our experiments, we follow a fixed training order for the clients, starting with $C_{1}$, followed by $C_{2}$, and so on up to $C_{6}$. This fixed training order may have an impact on the accuracy performance since the learning process can be influenced by the presence of more or less data samples during training, especially under an imbalanced data distribution. However, a detailed investigation of the training order and its effects is deferred to the next section for further analysis and discussion.

We conduct experiments to evaluate the privacy preservation of P-SL. During the training process, we employ model inversion attacks to reconstruct the raw data of all clients using the smashed data that clients send to the server. Specifically, these experiments are conducted with six clients under a balanced data distribution. Whereas all clients are engaged in training, we train a decoder using the local weights and data of one client, which in our case is client $C_{1}$ (representing a malicious client who is curious about the data of other clients). The trained decoder is used to reconstruct raw data from the smashed data that any client sent to the server. Based on this experiment setup, we evaluate the privacy preservation by measuring the amount of data leakage from clients’ raw data. Data leakage is quantified using the SSIM [36], which is a perceptual metric that assesses image quality degradation. SSIM provides a measure of similarity between the raw and reconstructed images and has been commonly used in previous works to evaluate data leakage, such as in [13, 31, 37]. Unlike other metrics such as MSE or PSNR, SSIM offers a more intuitive and interpretable metric, where values range between $0$ and $1$. A value of $0$ indicates the least similarity, while a value of $1$ represents the highest similarity, indicating the most leakage.

Fig. 6 demonstrates the reconstructed images from other clients assuming that $C_{1}$ is malicious. The decoder is trained using $C_{1}$’s local model and its raw data, resulting in clear reconstructions from $C_{1}$’s smashed data. However, the quality of reconstruction significantly drops when applying the decoder to the smashed data of other clients. The reconstructed images from $C_{2}$ and $C_{3}$ in Fig. 6 are vague and contain a high level of noise compared to the raw images. The numerical results presented in Table VII reveal that the reconstruction quality (SSIM value) of all clients (except $C_{1}$) in P-SL is only around $0.5$. On the other hand, with SL and SFL, as partially visualized in Fig. 1, $C_{1}$ can almost entirely reconstruct the raw data of all other clients, with SSIM values exceeding $0.95$. This is similar to $C_{1}$ self-reconstructing its own data.

The last row of Table VII presents the leakage of P-SL measured by MSE between the raw and reconstructed data. The results show that the errors in reconstructing data for clients $C_{2}$ to $C_{6}$ are more than $100\times$ higher than the reconstruction error for $C_{1}$. While a value of $0$ in the MSE metric indicates full leakage (no reconstruction error), there is no upper bound for non-leak or partial leak scenarios. Additionally, the correlation of leakage among clients does not exhibit a clear and meaningful trend when using MSE, unlike when using SSIM. Therefore, we primarily present the leakage measure using the SSIM metric. In [31], the authors made a similar observation and suggested using SSIM to measure leakage instead of MSE and its related PSNR.

We also conduct experiments with imbalanced data and obtain similar results, as presented in Table VIII. Based on the experiment results, we can conclude that P-SL outperforms SL and SFL in preserving data privacy at the client side. However, it is important to note that the SSIM values between reconstructed and raw images in P-SL are still significantly different from $0$, indicating the presence of leakage. This leakage can be attributed to the query-free attack described in [13], where the attacker does not require knowledge of the target model or the ability to query it. The only assumption for this type of attack is that the attacker and the victim share the same data distribution. In our experiments, the data is uniformly distributed among all clients, regardless of the number of samples they have. Therefore, the local model of the attacker acts as the shadow model for the query-free attack, leading to partial data leakage. Furthermore, an interesting observation from Table VIII is that attackers with more data (e.g. $C_{6}$) can reconstruct higher quality images compared to the ones with fewer data (e.g., $C_{1}$).

Fig. 7 demonstrates the reconstructed results from other clients assuming that $C_{1}$ is malicious, regarding experiments on the ECG dataset. Similar to Fig. 6, the decoder trained on $C_{1}$’s local model and data can effectively reconstruct raw input from $C_{1}$’s smashed data, with the similarity quantified as $0.15$, $0.99$, and $3e-4$ in terms of DTW, DC, and MSE metrics, respectively. However, the reconstruction quality significantly drops when applied to the smashed data of other clients (i.e. $C_{2}$ and $C_{3}$). The mean similarity is $0.69$, $0.86$, and $7e-3$, as measured by DTW, DC, and MSE, respectively. Note that, lower DTW, higher DC, and lower MSE values indicate higher similarity between raw data and the reconstructed results. The experiment results with 1D time-series data further demonstrate the effectiveness of our P-SL in mitigating privacy attacks at the client side of SL.

The experiments are conducted on the Fashion dataset with varying noise levels, pruning ratios, and regularization strength of DP, DISCO, and ResSFL, respectively. The results are shown in Fig. 8, where we can observe the trade-offs of DP and DISCO (lines) in sacrificing accuracy for improved privacy. On the other hand, our P-SL does not provide a method for controlling the trade-off; however, its result (dot) stays above the lines, indicating that P-SL preserves more privacy (larger dissimilarity in reconstruction) with less accuracy sacrificed. According to the experiments, P-SL achieves privacy similar to applying DP with $\epsilon=1$, while maintaining an accuracy comparable to utilizing DP with $\epsilon=3$. Regarding ResSFL, a simulated attacker is required in training the local model to resist model inversion attacks. However, simulating a weak attacker does not help defend against a strong real attacker, as we can achieve high-quality reconstructed results with a more complex decoder than the one used in training. Furthermore, requiring additional model training (adversarial networks in DISCO and ResSFL) would lead to extra costs for resource-constrained clients in SL. Comparing to other related works, even though lacking flexibility in managing accuracy sacrificed, our P-SL achieves better accuracy-privacy trade-offs with less processing overhead at the client side. Following, we continue to present further evaluation of P-SL such as in dynamic environments.