Federated Split Learning with Only Positive Labels for resource-constrained IoT environment

The paradigm of federated learning (FL), presented in Figure 2 (a), emphasizes the federated averaging (FedAvg) algorithm for local model aggregation [7]. Throughout the training phase, the server initially sets up the global model $W_{t}$, distributing it to all the participating clients. Upon receipt of the model $W_{t}$, every client $k$ proceeds to train the global model utilizing its local data, where $\mathcal{S}^{k}$ represents the number of training samples maintained by client $k$, and $\mathcal{S}$ signify the cumulative count of training samples across all clients. Subsequently, each client communicates the locally updated model $W_{k,t}$ back to the server. The server then undertakes the task of aggregating these models to refresh the global model to $W_{t+1}$. This iterative process, often called a ’round’, persists until the model converges.

Distinct from FL, where every client trains the entire neural network, split learning (SL)  [8] segments a neural network-based model into a minimum of two model portions. These model portions are then trained separately by the distributed parties, such as clients and servers. An elementary representation of SL is depicted in Figure 2 (b), with a cut layer acting as the division layer that splits the entire network into two model portions. The initial model portion $W^{\textup{C}}_{k,t}$ is trained and managed by the client, whereas the second model portion $W^{\textup{S}}_{k,t}$ is under the server’s control. During the training phase, both forward and backpropagation processes occur within the network. As shown in Figure 2 (b), the client initiates forward propagation on the input data and transmits the cut layer activations, referred to as the smashed data $\left(A_{k,t}\right)$, to the server. The server then conducts forward propagation on this smashed data, calculating the subsequent loss. The backpropagation begins from this point, yielding the gradients of the smashed data $\left(\nabla\ell\left(A_{k,t};W^{\textup{S}}_{k,t}\right)\right)$. Once this gradient is computed, it is transmitted back to the client, initiating the client’s backpropagation process. In the context of SL training/testing, the server does not have access to the clients’ model portions and data, bolstering privacy. Along with privacy advantages, each client is only required to train a model portion that usually comprises a few layers, with the majority of layers housed in the server. Consequently, this reduces the computational load on the client. The learning performance (for instance, model accuracy and convergence) of SL has not yet been thoroughly explored when the data is non-Independent and Identically Distributed (non-IID) or unevenly distributed, which has been taken into consideration in this study.

SL significantly diminishes the computational needs on the client side by operating only on a smaller model portion. Nonetheless, it necessitates sequential iterations over each client, thereby extending training time in instances where multiple clients are involved. In FL, clients typically parallelize their interactions with the server, thereby facilitating faster completion of training compared to SL. However, this method imposes a high computational overhead as each client must train the entire model. Recently, a hybrid approach combining the benefits of both SL and FL has been introduced, known as SFL [4]. Within SFL, all clients calculate independently and concurrently. These clients transmit/receive their smashed data to/from the server simultaneously. The synchronization of the client-side model portion, that is, the creation of the global client-side network, is achieved through the aggregation (for instance, weighted averaging) of all local networks on the client side on a separate server, termed the ”fed server”. Two distinct methodologies for server-side model portion synchronization were presented in [4], leading to the creation of two specific versions of SFL:

• •

  SFLV1: The first methodology involves the independent and concurrent training of the smashed data of each client, resulting in an equal number of server-side model portions and clients. Subsequently, all the model portions are amalgamated (e.g., through weighted averaging) to form the global server-side network, a variant known as SFLV1.

• •

  SFLV2: The second methodology entails sequential server-side model portion training on each client’s smashed data (note that clients can still send their smashed data concurrently). This approach maintains a single copy of the model portion on the server side, which becomes the global server-side network once the main server has processed all the smashed data, referred to as SFLV2.

In our setup, SFLv2 splits the full model into two portions; client-side model $W^{\textup{C}}$ and server-side model $W^{\textup{S}}$. At the individual client level, a client-side model is denoted as $W^{\textup{C}}_{k}$, where $k\in[N],[N]:=\{1,\ldots,N\}$, $k$ represents the client, and $N$ indicates the maximum number of clients.

We assume that each client has data from only one class, i.e., positive labels. Let $\mathcal{Z}$ be the specific set of instances, and $\mathcal{X}$ be a subset of $\mathcal{Z}$ employed in a specific batch. Given that there are $V$ classes indexed by a set $[V]$, the number of distinct sets $[V]$ equals $N$, with both $|[V]|$ and $N\in\mathbb{N}$. We assume that the $N$ clients collaboratively train for $t$ epochs, where $t\in\{1,\ldots,T\}$, and communicate with the server in a random order, once per epoch. Assume that $g_{\boldsymbol{\theta}^{\textup{C}}}$ represents the function of client-side model parameters. With this notation, the smashed data transmitted from the $k$-th client using its private data $\boldsymbol{X}_{k,t}$ can be formulated as:

Let $\mathcal{F}\subseteq\left\{f_{s}:\boldsymbol{A}_{k,t}\rightarrow\mathbb{R}^{V}\right\}$ be a set of scorer functions in the server-side model portion $W^{\textup{S}}$, which when given the smashed data $\boldsymbol{A}_{k,t}$, assigns a score to each of the $V$ classes. In particular, for $v\in[V],f_{s}(\boldsymbol{A}_{k,t})$ gives the probability of the $v$-th class for ${\boldsymbol{A}_{k,t}}$ received from the client-side model portion, as measured by the scorer $f_{s}\in\mathcal{F}$. For simplicity, scorers are of the form

where $g_{\boldsymbol{\theta}^{\textup{S}}}:\boldsymbol{A}_{k,t}\rightarrow\mathbb{R}^{v}$ maps the instance $\boldsymbol{A}_{k,t}$ to a $v$-dimensional vector to produce the scores (or logits) for $V$ classes as $g_{\boldsymbol{\theta}^{\textup{S}}}(\boldsymbol{A}_{k,t})$.

In DCML with only positive labels, the $k$-th client has $n_{k}$ instances of all only label $k$. We postulate that each client possesses a subset of the comprehensive dataset, denoted as $\mathcal{S}=\cup_{k\in[{N}]}\mathcal{S}^{k}$. Here, the subset $\mathcal{S}^{k}$ represents the collection of $V=\sum_{k\in[N]}v_{k}$, where $v_{k}$ corresponds to the instances and label pairs jointly accessible to each client. This relationship can be explicitly defined as $k:\mathcal{S}^{k}=$ $\left\{\left(\boldsymbol{x}_{1},k\right),\ldots,\left(\boldsymbol{x}_{n},k\right)\right\}\subset\mathcal{X}\times[V]$.

Our objective is to minimize the loss function $\ell:\mathbb{R}^{V}\times[V]\rightarrow\mathbb{R}$ to derive a score function $\mathcal{F}$ that accurately classifies instances. The loss function $\ell$ assesses the server-side scorer $f_{s}$ quality, considering $y^{j}_{k}$ as the corresponding label to the smashed data $\boldsymbol{A}^{j}_{k,t}$ corresponding to a data sample $\boldsymbol{x_{j}}$. It operates on the input-output pairs $(\boldsymbol{A}^{j}_{k,t},y^{j}_{k})$ to minimize the empirical risk estimation based on the activations received from the $k$-th client, as follows:

In SFLv2, each client’s $W^{\textup{C}}_{k}$ trains with the server-side model at least once but in random order as follows:

1. 1.

   Initially, all clients receive the client-side model portion with parameters $\boldsymbol{\theta}^{\textup{C}}$ and $W^{\textup{C}}$ to all clients. At the same time, the server-side receives the server-side model portion with $\boldsymbol{\theta}^{\textup{S}}$ and $W^{\textup{S}}$.

2. 2.

   For each client $k\in[N]$, the $k$-th client transmits the smashed data $\boldsymbol{A}_{k,t}$ to the server-side model $W^{\textup{S}}$. Utilizing the empirical risk estimation, the client-side and server-side model components are updated accordingly:

   For client-side:

   $\displaystyle\boldsymbol{\theta}_{k,t}^{\textup{C}}=\boldsymbol{\theta}_{t}^{\textup{C}}-\eta\cdot\nabla_{\boldsymbol{\theta}^{\textup{C}}}\hat{\mathcal{R}}\left(g_{\boldsymbol{\theta}^{\textup{C}}};\mathcal{S}^{k}\right).$

   (4)

   $\displaystyle W_{k,t}^{\textup{C}}=W_{t}^{\textup{C}}-\eta\cdot\nabla_{W^{\textup{C}}}\hat{\mathcal{R}}\left(g_{\boldsymbol{\theta}^{\textup{C}}};\mathcal{S}^{k}\right).$

   (5)

   For server-side:

   $\displaystyle\boldsymbol{\theta}_{k,t}^{\textup{S}}=\boldsymbol{\theta}_{t}^{\textup{S}}-\eta\cdot\nabla_{\boldsymbol{\theta}^{\textup{S}}}\hat{\mathcal{R}}\left(f_{s};\boldsymbol{A}_{k,t}\right).$

   (6)

   $\displaystyle W_{k,t}^{\textup{S}}=W_{t}^{\textup{S}}-\eta\cdot\nabla_{W^{\textup{S}}}\hat{\mathcal{R}}\left(f_{\textup{s}};\boldsymbol{A}_{k,t}\right).$

   (7)

3. 3.

   Once the clients update their model parameters as mentioned in the equation 4 and 5, they send updated model parameters to the federated server. After receiving all updated model parameters $\left\{\boldsymbol{\theta}_{k,t}^{\textup{C}},W_{k,t}^{\textup{C}}\right\}_{k\in[N]}$, the federated server updates the parameters of the global client-side model using federated averaging as:

   $$\boldsymbol{\theta}_{t+1}=\sum_{k\in[N]}\boldsymbol{\theta}_{k,t}^{\textup{C}};\quad W_{t+1}=\sum_{k\in[N]}W_{k,t}^{\textup{C}}.$$

   (8)

For the multi-class classification taking criterion as the cross-entropy loss over the server-side model portion is expressed as:

where $z_{ij}$ is the server-side model’s output (logits) for the smashed data $\boldsymbol{A}_{k,t}[i]$ and class $j$, $S$ denotes the number of samples in the batch, $V$ denotes the number of classes.

SFLv2 demonstrates significant constraints in managing positive labels, as evidenced by the results presented in Table I. To tackle this challenge, we introduce an enhanced version of the SFLv2, which we call SFPL. The algorithm of SFPL is depicted in Algorithm 1 and 2. It comprises four main functions: Client, ClientFedServer, GlobalCollectorFunction, and ServerSideModelFunction. A brief description of each component’s functionality, illustrated in Figure 1, is presented in the following:

• •

  Client: Clients initiate with a weight matrix $W_{k,t}^{\textup{C}}$, perform forward propagation on local data $X_{k}$, producing smashed data $\boldsymbol{A}_{k,t}$. These, along with true labels $\boldsymbol{Y}_{k}$, are sent to the GlobalCollectorFunction, returning gradients $d\boldsymbol{A}_{k,t}$. Clients use this for back-propagation, calculating gradients $\triangledown\ell_{k}({W_{k,t}^{\textup{C}}})$, updating weights via ${W_{k,t}^{\textup{C}}}\leftarrow{W_{k,t}^{\textup{C}}}-\eta\triangledown\ell_{k}({W_{k,t}^{\textup{C}}})$, and await ClientFedServer(${W_{k,t}^{\textup{C}}}$) completion.

• •

  GlobalCollectorFunction: The global collector function collects activations and true labels from clients, shuffles and sends them to the server-side model function, receives gradient of $\mathbf{A}_{k,t}$, de-shuffles, and sends back to clients.

• •

  ServerSideModelFunction: The server-side model function receives shuffled activations and labels, initializes global model weights $W^{\textup{S}}_{t}$, computes predicted labels $\hat{\boldsymbol{Y}_{k}}$, evaluates loss and determines gradient $d\mathbf{A}_{k,t}$ to send back to GlobalCollector.

• •

  ClientFedServer: The function collects client model weights, computes the global model’s average weights excluding the batch normalization layer to mitigate its impact, and updates each client’s model accordingly.

Catastrophic interference is a phenomenon observed in machine learning where a model loses previously acquired knowledge upon being fine-tuned on a new task. In SFLv2, catastrophic interference arises on the server side when client-side models train on single-class data. The server-side model encounters clients’ smashed data in random, sequentially ordered batches during each epoch, treating each batch as a learning task. SFLv2 aims to minimize cumulative loss across tasks. In a scenario with $N$ clients, each task $q\in\{1,2,\dots,Q_{n}\}$, where $Q_{n}$ represents the task of learning labels for the $n$-th client, can be considered sequential tasks for the server-side model. With $L_{q}(\theta_{q})$ representing the loss on task $q$, the optimal total loss can be expressed as:

As the server-side model trains sequentially with batches from each client in an epoch, it exhibits higher accuracy for the class associated with the last visited client and lower accuracy for the class of the first visited client. The graphical representation depicted in Figure 3 presents a pattern of accuracy dynamics for labels 1 and 8 during the training process of the CIFAR-100 dataset [9] using the SFLv2 model. Following a certain number of epochs, a steep decrease in accuracy for label 8 is noticeable post the 25th epoch, coinciding with a significant increase in accuracy for label 1. It is noteworthy that in circumstances where a marked boost in accuracy for a specific label is detected, the final iteration of the batch SFLv2 model is primarily linked with the client associated with that particular label, in this instance, label 1. This pattern undergoes a shift at the 148th epoch, where a sharp ascent in accuracy for label 8 is observed, accompanied by a decline in accuracy for label 1. This shift is again correlated with the last label being trained upon, which in this case, is label 8.

Weight divergence in the context of non-IID refers to the discrepancy in the model weights of SFLv2 when compared to those of standard stochastic gradient descent (SGD). Weight divergence in SFLv2 learning refers to the discrepancy between aggregated client-side model parameters under two different data distribution scenarios: IID data and non-IID data on the client side. In a study [10], the authors demonstrated that aggregating FL models in a non-IID setting results in model weight divergence when compared to FL models in an IID setting. Weight divergence statistics were utilized to measure this divergence.

For SFLv2, weight divergence statistics can be defined with certain assumptions. Suppose the client-side global model, after an epoch $e$ with an IID distribution, is $\boldsymbol{w}^{\textup{ SGD}}$, and with a non-IID distribution after the same number of epochs, is $\boldsymbol{w}^{\textup{FedAvg}}$. The weight divergence statistics can be defined as:

The theoretical and empirical evaluations showed weight divergence for federated learning (FL) with both IID and non-IID data [10]. This remains applicable to SFLv2, where client-side model aggregation occurs under IID and non-IID data scenarios on the client side.

In SFLv2, the batch normalization layer parameters, including the mean and variance, are typically aggregated as part of the client-side model portion. However, studies in [11] assert that in a non-IID FL framework, model parameters are contingent upon each client’s local dataset and influenced by its data distribution. Designating the $k$-th client’s activation mean and variance as $\mu_{k}$ and $\sigma_{k}^{2}$ respectively, the batch normalization layer normalizes these using the following equation:

where $x_{i}$ is the activation of the $i$-th neuron, $\epsilon$ is a small constant added to avoid division by zero, and $\hat{x}_{i}$ is the normalized activation. As each client’s local data distribution may vary, their mean and variance estimates can also differ. Averaging the means and variances of all clients may not accurately capture the distribution of activations for each individual client. In the study by Li et al. [11], it has been demonstrated both theoretically and empirically that in federated learning (FL), the global model converges more quickly when batch normalization (BN) layers are not aggregated. This finding is extended to SFLv2, where client-side model weights are aggregated at the end of each epoch.

In this section, we assess the performance of SFPL, SFLv2, and FedAws. It is important to acknowledge that the FedAws framework may not be optimally suited for resource-constrained IoT devices with computational limitations capped at $475.136$ K Flops. Nevertheless, we incorporate FedAws in our comparative analysis, as it shares a comparable learning setting that exclusively relies on positive labels, thereby providing valuable insights and context for the evaluation of SFPL and SFLv2. For experiments with SFPL and SFLv2 frameworks, the DL model under consideration was partitioned at the first layer, with the initial layer assigned to the client side and the rest of the model allocated to the server side. Finally, we assess the experiments in terms of precision@1 and F1-score on the CIFAR-10 and CIFAR-100 test datasets. Additionally, recall and accuracy metrics are included in our evaluation table to offer complementary perspectives that contribute to a comprehensive assessment of the model’s performance.

Additionally, we examine the impact of using the current mean and variance (CMSD) strategy, where the batch normalization layer for the client-side model portion is not aggregated during the aggregation step. Instead, the mean and variance of the batch under test are utilized during testing. In contrast, the running means and standard deviation (RMSD) strategy aggregates the batch normalization layer for the client-side model portion during the aggregation step, using the learned running mean and variance for the batch normalization layer during testing. It is important to note that due to the unavailability of the FedAws code and the inability to reproduce results, we have taken the FedAws results from its ICLR research paper [6].

The empirical findings presented in Table V demonstrate the efficacy of SFPL in mitigating the constraints inherent in the SFLv2 learning framework while training DL models solely utilizing positive labels. In the CIFAR-100 dataset, SFLv2 reported relatively low accuracies, approximately 1.4% for the R56 architecture and 2.05% for the R32 architecture, indicating a failure to learn effectively. Conversely, the SFPL framework significantly enhanced performance, achieving 72.16% and 66.77% accuracy for the R56 and R32 architectures respectively. A similar trend was observed in the CIFAR-10 dataset, with the R32 and R8 architectures’ accuracies stagnating at 10% under SFLv2, while SFPL significantly boosted them to 92.33% and 85.15%, respectively.

Table V also presents additional evaluation metrics, such as Precision@1, recall, and F1 score, to corroborate that the DL models effectively learned all classes when employing the SFPL framework. Furthermore, the results indicate that the CMSD setup yielded superior outcomes compared to the RMSD setup during the testing phase, specifically when a single-class batch was utilized for model evaluation. Conversely, SFPL’s performance declined when an IID batch was employed for testing.

Lastly, SFPL surpassed FedAws’ Precision@1 scores for the CIFAR-100 R-56 and R-32 architectures by 3.71% and 1.65%, respectively. Nevertheless, the performance remained comparable for the CIFAR-10 R-32 and R-8 architectures.

In this section, we present an extended study to analyze the performance of the SFPL framework in various training and testing scenarios. Specifically, we investigate three sub-settings of the SFPL framework, as discussed in section VII. For each sub-setting, we evaluate the performance of the SFPL framework while introducing both CMSD and RMSD setup on the client side. This study aims to gain insights into the effectiveness and limitations of the SFPL framework across these different scenarios.

1. 1.

   SFPL: IID dataset at training and inference phase

   Based on the results presented in Table VI, it can be inferred that the RMSD setup outperforms the CMSD setup for all combinations of model architectures and datasets considered. These findings suggest that, within the context of the SFPL framework, performing BN layer aggregation during the client-side model aggregation process contributes to improved performance during the inference phase.

2. 2.

   SFPL: non-IID dataset at training and IID test dataset at inference phase

   The results in Table VII reveal that the RMSD setup consistently surpasses the CMSD setup across all examined model architectures and dataset combinations. This indicates that within the SFPL framework, employing BN layer aggregation during client-side model aggregation for non-IID training and IID inference scenarios leads to enhanced performance during inference.

3. 3.

   SFPL: non-IID dataset at training and inference phase

   The results in Table VIII demonstrate that the CMSD setup consistently surpasses the RMSD setup across all model architectures and dataset combinations, a considerable performance difference. This can be attributed to the utilization of non-IID testing datasets, enhancing the accuracy of the BN layer’s statistics for the client-side model in the CMSD setup through the test batch’s current mean and variance. Thus, in the SFPL framework for non-IID training and testing, aggregating the batch normalization layer during the training aggregation phase could potentially detriment the overall performance.