Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy

In response to the above-mentioned constraints on the uplink payload budget and limitations on the local computational power, we develop a novel scheme to perform model-based transfer FL and refer to this scheme as feature-based federated transfer learning (FbFTL). In FbFTL, rather than uploading the gradients, the input and output of the subset of DNN to be trained are uploaded with the goal to reduce the uplink payload. This is one of the key differences from prior FL and FTL strategies. In this paper, after describing the proposed FbFTL scheme, we analyze its overall uplink payload and provide comparisons with prior schemes to quantify the gains. Specifically, we test this approach by transferring the VGG-16 convolutional neural network (CNN) model [41] trained with ImageNet dataset [42] to CIFAR-10 dataset [43], and show that FL, two types of FTL, and FbFTL require uploading 3216 Tb, 949.5 Tb, 599 Tb, and 6.6 Gb of data, respectively, until performance convergence. With this, we demonstrate that the proposed FbFTL outperforms FL and FTL substantially by reducing the upload requirements by at least 5 orders of magnitude. We note that the ITU standard of 5G uplink user experienced data rate is only 50 Mb/s [44], and even the 6G uplink user experienced data rate at Gbit/s level may not sufficiently support such huge uploading requirements of regular FL. Additionally, to our best knowledge, existing works on improving FL efficiency (such as FedAvg [4], sparsification [45, 46] and quantization [47, 48] with or without error feedback [49, 50], federated distillation [51, 52, 53], pruning [54, 55] or partially trainable network [56, 57], and over-the-air computation [58, 59, 60]) still consider the transmission of gradient updates and can achieve a relatively limited reduction in payload and experience degradation in performance. For instance, the payload reduction is of only two orders of magnitude of the original payload on the same CIFAR-10 dataset [61, 62]. Therefore, FbFTL is still a more effective approach in reducing the uplink payload even after the efficiency of FL has been improved via the aforementioned methods. We further show that FbFTL has substantially lower downlink payload, and requires significantly less computational power from the edge devices, and therefore it is much more friendly in terms of facilitating the training tasks on clients with limited power budget.

To validate our approach, we further analyze the robustness of FbFTL against FL and FTL. We show significant reduction on packet loss rate (PLR) with FbFTL with the same block loss rate (BLR), and demonstrate the robustness of FbFTL with insufficient training data. While there have been numerous studies on gradient quantization and sparsification to reduce the uplink payload [63, 64], we illustrate that FbFTL also achieves significant compression rate by quantization while the validation accuracy is barely affected. Furthermore, we analyze the privacy leakage to a potential adversary. We define the label privacy leakage and feature privacy leakage for both feature-based and gradient-based frameworks. To our best knowledge, this is the first work that divides the privacy into different categories, defines each type, and proposes mitigation approaches. Via experimental results, we show that FbFTL has better performance (e.g., classification accuracy) with the same level of privacy protection when each client obtains a small set of samples.

In summary, our main contribution can be listed as follows:

• •

  We propose the FbFTL scheme that uploads extracted features instead of gradients to reduce uplink payload by five orders of magnitude.

• •

  We analyze the robustness of FbFTL, and demonstrate that it maintains the payload advantage when strategies such as sparsification, quantization and error feedback are deployed.

• •

  We study the privacy guarantees in terms of entropy based formulations that quantify the uncertainty and also by utilizing differential privacy mechanisms.

• •

  We show that a small batch size is preferred to protect label privacy of shuffled batches, and FbFTL has better performance (such as in terms of accuracy) given the differential privacy constraint for input privacy in a small batch.

The remainder of the paper is organized as follows. In Section II, we discuss the preliminary concepts regarding FL and FTL, describe the system design via FedAvg, and demonstrate their uplink payload. In Section III, we introduce the proposed FbFTL algorithm, introduce the learning structure and system design, and compare its payload with FL and FTL. In this section, we also evaluate the performance of FbFTL via simulations, and provide comparisons with FL and FTL. In Section IV, we illustrate the robustness of FbFTL in the presence of packet loss, data insufficiency, and quantization. In Section V, we define the label privacy leakage and the feature privacy leakage. For label privacy, we discuss the leakage with statistical information and the leakage with query, and investigate mitigation approaches to avoid these privacy leakages. For feature privacy leakage, we illustrate the mitigation approach via differential privacy. Finally, we conclude the paper in Section VI. The list of all notations in the paper is provided in Table I as a quick reference.

We address a common FL task in which a PS coordinates a set $\mathcal{U}$ of $U$ clients to cooperatively train a DNN model. Each client $u$ possesses a local dataset $\mathcal{K}_{u}$ with $K_{u}$ samples for training. In this dataset, the $k$th sample $\textbf{s}_{u,k}\in\mathcal{K}_{u}$ is comprised of an input vector $\textbf{x}_{u,k}\in\mathbb{R}^{N_{0}}$ and an output vector $\textbf{y}_{u,k}\in\mathbb{R}^{N}$. The DNN, represented by the function $f_{\boldsymbol{\theta}}(\textbf{x})$, maps the input x to an output $\hat{\textbf{y}}$ (as an estimate of y) with trainable parameter vector $\boldsymbol{\theta}$. The goal of the FL training process is to update the parameter vector $\boldsymbol{\theta}$ using the training samples from each client in order to minimize the expected loss $\mathbb{E}(L(f_{\boldsymbol{\theta}}|\textbf{s}_{test}))$ on the unseen sample $\textbf{s}_{test}$ with the same distribution as the training samples. For most DNNs with classification tasks, the output label y is an axis-aligned unit vector with one element equal to 1 indicating the class of this sample, and all others equal to 0. In this case, the loss function is typically the categorical cross-entropy:

In order to minimize the loss and keep the training samples local, the authors in [4] proposed an iterative distributed optimization scheme called FedAvg with the following steps:

1. 1.

   The PS initiates trainable parameters $\boldsymbol{\theta}$, and broadcasts the model structure $f$ with non-trainable parameters to each client.

2. 2.

   The PS chooses a subset of $UC$ clients (with $C$ denoting the fraction of clients in this iteration) and broadcasts the trainable parameters $\boldsymbol{\theta}$.

3. 3.

   Each client performs stochastic gradient descent (SGD) to obtain the parameter update corresponding to each training sample: $\nabla_{\boldsymbol{\theta}}{L(f_{\boldsymbol{\theta}}|\textbf{s}_{u,k})}$.

4. 4.

   Each client sends the sum of the updates over all local samples $\textbf{g}_{u}=\sum_{k=1}^{K_{u}}{\nabla_{\boldsymbol{\theta}}{L(f_{\boldsymbol{\theta}}|\textbf{s}_{u,k})}}$ and $K_{u}$ to the PS.

5. 5.

   The PS updates the parameter $\boldsymbol{\theta}$ with $\boldsymbol{\theta}-\sum_{u=1}^{U}{\frac{\alpha}{K_{u}}\textbf{g}_{u}}$ where $\alpha$ is the learning rate that controls the training speed.

6. 6.

   Return to step 2) until convergence.

Let us assume that the number of iterations in FL with FedAvg is $I^{FL}$ (in the absence of any processing and transmission failures). This implies that the total number of times the clients upload $\textbf{g}_{u}$ is $I^{FL}UC$ within the training process. Assume further that the DNN includes $M$ layers with trainable parameters (such as convolutional layer and fully connected layer, i.e., dense layer) and $M^{\prime}$ layers without trainable parameters (such as pooling layer and residue connection). The $m$th trainable layer has $T_{m}$ trainable parameters. Thus, for each trainable parameter, the client uploads one float number of $d$ bits for the corresponding element in the update $\textbf{g}_{u}$ during each iteration. Therefore, the overall uplink payload to train a DNN via FL with FedAvg is

and we will set this as a benchmark to compare with three other training methods in the remainder of this paper.

TL is an effective learning technique that utilizes knowledge from a different domain to enhance the performance in the target domain. FTL incorporates the privacy-preserving distributed learning paradigm into conventional TL in order to address the challenges of spectrum limitations in wireless applications and training data scarcity. In this paper, we consider FTL to be performed in the same fashion as in the previous subsection, where one PS orchestrates $U$ clients. Based on the difference between the domains, FTL can be divided into three different categories: instance-based FTL, feature-based FTL, and model-based FTL [6, 65]. The former two types of FTL assume similarity in the distribution of the input and output, and hence we in this paper focus on the model-based FTL which only assumes similarity in the functionality to extract a high-dimensional description from the input data.

As shown in Fig. 1, we consider FTL with a DNN model pre-trained with an open-source dataset that corresponds to a similar but not the same task as the source model. The goal here is to reduce the number of iterations during training. Many DNNs can be partitioned into two sections. The first section generates the high-dimensional features from the sample input data with general information. The second section carries out operations for a particular task, such as classification, and is less likely to be transferable to a new task. To move the knowledge of the source model into a new task before training, the feature extraction section of the source model is directly transferred to the new target model, while the task-specific part is randomly initiated. FTL can be performed by retraining all the parameters with new data. However, there typically exist a large number of parameters to train, and in order to reduce the training time and data requirements, a common approach is to perform FTL by fixing the feature extraction part and simply updating the task-specific part. In particular, we select one fully connected layer $m_{c}$ (which is Layer 4 in Fig. 1), close to the output without a paralleling path (such as residue connection), and randomly initiate all trainable parameters of layers $m_{c},m_{c}+1,\ldots,M$ (which are Layer 4 and Layer 5 in Fig. 1). During the distributed training process, all copied parameters are fixed, and we only update the parameters of layers $m_{c},\ldots,M$.

Similar to the previous sub-section on FL, we determine the least requirements on successful uplink payload for FTL until convergence. For FTL that updates all parameters, assuming that FTL with FedAvg is iterated $I^{FTL}_{f}$ times, the overall uplink payload required for training is

On the other hand, assuming FTL with FedAvg that only updates the task-specific part with $I^{FTL}_{c}$ iterations and assuming that each sum update $\textbf{g}_{u}$ consists of $\sum_{m=m_{c}}^{M}{T_{m}}$ trainable parameters, the overall uplink payload for training is

Typically, training from scratch requires more training samples, and thus we have $I^{FTL}_{c}UC\approx I^{FTL}_{f}UC<I^{FL}UC$. For most of the models, the majority of the DNN structure is dedicated to the feature extraction, and obviously $\sum_{m=m_{c}}^{M}{T_{m}}<\sum_{m=1}^{M}{T_{m}}$. Therefore,

While we in this paper focus on FL schemes that generalize to the majority of use cases, there are also recent works that improve the performance under certain assumptions and can be used in FL. We list these well-known methods, along with FL, FTL, and the proposed FbFTL, in Table II and provide a comparison. Below, we briefly describe the key assumptions of these methods, and highlight the differences of the proposed FbFTL scheme.

Specifically, elastic weight consolidation (EWC) [66] focuses on a large number of different tasks simultaneously and the transfer between them by remembering the importance of each weight. The performance is typically measured on all experienced tasks. EWC is inspired by neuroscience and Bayesian inference, and it aims to quantify the importance of each weight to the tasks the model has previously learned. The fundamental idea is that the weights critical to prior tasks should be altered less when new data is encountered. In this paper, we consider a different problem, where we transfer to a single target task from a single source model on another task whose performance is no longer considered. In this problem setting, there is no clear difference between EWC and plain SGD in [66] in terms of both training performance or training time, but EWC does introduce extra communication payload and computation cost in FL. On the contrary, FbFTL achieves five orders of magnitude payload reduction.

Split federated learning (SFL) [67] partitions the neural network into two segments: the initial part near the input undergoes training in a federated learning manner using a “Fed Server”, and the subsequent part near the output is trained using split learning with a “Main Server,” which receives the smashed data. Compared to FbFTL, the federated learning segment utilizing the Fed Server operates in the same way as FL and requires significant uplink and downlink payload. Hence, this part alone is typically much more communication-expensive compared to FbFTL. The advantage of SFL lies in its similar performance to FL and not needing a pre-trained model as it updates all parameters. We will present the experimental performance comparison in Table III in Section III-D.

Personalized federated learning (PFL) [68] considers clients with large number of heterogeneous local samples, enabling generalization to the heterogeneous local distribution. However, we in this paper consider a large number of clients with a small set of local samples from each. As explained in Section V-A and Section V-C, such a setting better protects label privacy. Therefore, personalized federated learning focuses on a different problem from ours. We cannot apply this approach to FbFTL because personalized FbFTL violates the shuffle-batch assumption, unless we train the global model by FbFTL and then retrain locally. Although PFL has better local performance than FL with large heterogeneous batches, it requires several independent local batches and significantly more local computation. Furthermore, it does not allow quantization and sparsification since the weights instead of gradients are uploaded. Therefore, it is comparatively even less communication-efficient when quantization and sparsification are deployed in other methods, as shown in Section IV-C and Table V.

As depicted in Fig. 2, in FbFTL we consider the model-based TL on a source DNN model $f^{\prime}$ pre-trained on a different task. We choose one fully connected layer $m_{c}$ without paralleling path as the cut layer, and divide the new target model $f\leftarrow f^{\prime}$ into two parts. Those layers before layer $m_{c}$ (which is layer 4 in Fig. 2, i.e., $m_{c}=4$ ) are regarded as the feature extraction sub-model $f^{1}_{\boldsymbol{\theta}^{1}}$, and the parameters $\boldsymbol{\theta}^{1}$ for the new target model $f$ are copied from the pre-trained source model $f^{\prime}$ and fixed without any further update. The other layers are regarded as the task-specific sub-model $f^{2}_{\boldsymbol{\theta}^{2}}$, and all trainable parameters $\boldsymbol{\theta}^{2}$ in the new target model $f$ are randomly initiated for training with the $u$th client’s $k$th training sample $\textbf{s}_{u,k}=\{\textbf{x}_{u,k},\textbf{y}_{u,k}\}$ for all $u$ and $k$.

The forward pass of the full model $f_{\boldsymbol{\theta}}(\textbf{x}_{u,k})=f^{2}_{\boldsymbol{\theta}^{2}}(f^{1}_{\boldsymbol{\theta}^{1}}(\textbf{x}_{u,k}))$ maps the input $\textbf{x}_{u,k}$ to the estimated output $\hat{\textbf{y}}_{u,k}$, and we note that the output of the feature extraction sub-model $\textbf{z}_{u,k}=f^{1}_{\boldsymbol{\theta}^{1}}(\textbf{x}_{u,k})$ is also the input of the task-specific sub-model $f^{2}_{\boldsymbol{\theta}^{2}}$. Note that we require only the task-specific sub-model to be trained. In this setting, each client generates the features $\textbf{z}_{u,k}$ from input $\textbf{x}_{u,k}$ and sends these features to the PS only once. Subsequently, in FbFTL, the PS performs the gradient back-propagation iteratively without sending any feedback to the clients. Contrary to this, in FL and FTL, the parameter update, which is based on the gradients, highly relies on the parameters in the current training iteration, and the same data sample may generate different parameter updates in different iterations within the training process. Therefore, in FL and FTL, we either require many more training samples, or need to upload updates multiple times for the same sample. However in FbFTL, each client only needs to upload the intermediate output $\textbf{z}_{u,k}$ and output $\textbf{y}_{u,k}$ once, instead of iteratively uploading the gradients. The PS may deem these as the input and the output of the training sample for the task-specific sub-model $f^{2}_{\boldsymbol{\theta}^{2}}$. Such pairs of samples are not correlated with the model parameters $\boldsymbol{\theta}^{2}$, and therefore they can be used in different training iterations without downloading or uploading anything again. We provide the steps of the FbFTL algorithm in Algorithm 1 below.

As emphasized above, a key distinction of FbFTL is that the training samples of task-specific sub-model are uploaded rather than the direct gradient updates. With this, FbFTL provides additional important benefits that can lead to further improvements in the training performance and efficient management in practice. One of the most important benefits is the significant reduction of packet loss rate given the same batch loss rate since FbFTL has much less data in each batch, as we will illustrate in detail in Section IV-A. One other benefit is the waiving of the requirement of synchronization between clients, since there is only one iteration in FbFTL. The other benefits pertain to hyper-parameter fine-tuning, dataset balancing, and enabling flexible training batch size selection, as detailed below.

Specifically, one of the most important additional benefit is that FbFTL enables iterative fine-tuning of the SGD optimizer hyper-parameters such as the learning rate. To obtain the optimized DNN, one needs to find the optimized hyper-parameters that provide the best convergence performance. In FbFTL, the model can be trained from scratch several times at the PS to identify the best hyper-parameter setting without additional communication with the clients. On the other hand, in FL and FTL, the entire online training process has to be run several times, resulting in a much higher cost compared to the ideal uplink payload $P$.

Another benefit of FbFTL is the dataset balancing. If the overall dataset is imbalanced and hence the samples with certain types of output appears much more frequently than those of other outputs, it is hard for FL and FTL to distinguish this via gradient updates, and such imbalanced data distribution could significantly degrade FL performance [69, 70]. However, FbFTL with direct output information enables techniques, such as re-sampling specific classes or merging near-identical classes, to improve dataset imbalance.

One more benefit of FbFTL is to lift the constraint of $UC$ and $K_{u}$ on the training batch size. To avoid over-fitting to the training dataset, one needs to validate the performance on a separate validation set of data to identify the optimal number of training iterations to stop training and conclude the final model. It is straightforward for FbFTL to divide the obtained dataset into training and validation subsets. However, the gradient-based FL frameworks call for extra effort for the communication system to meticulously perform the training process and validation process with the desired order and number of samples. Additionally for the training process, due to the broadcast nature of the downlink in wireless FL and FTL, all selected clients in the same communication iteration receive the same parameters. Hence, each SGD mini-batch has a larger size than $\sum_{u=1}^{UC}K_{u}$, and an overwhelmingly large SGD mini-batch size may delay the training process and require more training iterations. The mini-batches in these gradient-based FL frameworks are also on the order of given clients’ samples. Therefore, when the clients’ sample distribution is biased, we cannot shuffle the samples between iterations and have to accept the loss in the final performance. However, FbFTL may choose any rational size of SGD mini-batch without the constraints of the communication system, and can reshuffle the data in each training iteration.

Note that FbFTL has the major benefit of requiring one-time communication between the clients and the PS. In addition to this, FbFTL has much less data in a single upload batch compared to that of each sample in the upload batch of FL and FTL. Let us assume that in the fully connected layer $m$ with bias, the number of input nodes and the number of output nodes are denoted by $N^{-}_{m}$ and $N^{+}_{m}$, respectively Then, the number of trainable parameters in this layer is given by $T_{m}=N^{+}_{m}(N^{-}_{m}+1)$. Compared to the dimension of $N^{-}_{m_{c}}$, the amount of information that we need to represent $\textbf{y}_{u,k}\in\{1,\ldots,N\}$, i.e., $\log_{2}{N}$ bits, is negligible. Note that in FbFTL, gradients are computed at the PS. Therefore, FedAvg cannot be applied and each sample is required to be uploaded separately. Therefore for FbFTL, the uplink payload for each sample is $dN^{-}_{m_{c}}$, and the overall successful uplink payload required for training is

Compared to the uplink payload in (4) of FTL with FedAvg (in which only the task-specific sub-model is updated), the ratio of each upload for single sample between FTL and FbFTL is

Therefore,

The number of extracted features for many state-of-the-art models is larger than $10^{3}$, and TL usually requires a relatively deeper task-specific sub-model, and therefore $N^{+}_{m_{c}}$ can be large. For the cross-device federated learning, the clients do not obtain huge local datasets, $\sum_{u=1}^{U}{K_{u}}<I^{FTL}_{c}UC$. Therefore, we have $P^{FbFTL}\ll P^{FTL}_{c}$. We note that FbFTL and FTL that updates the task-specific sub-model have the same performance at every iteration because the only difference between the two methods is the communication model but not the numerical process to generate the gradient updates. Combining this observation with the conclusion in (5), we have

and therefore we expect extremely smaller uplink payload in FbFTL compared to FTL and FL.

We also note that FbFTL has the smallest downlink broadcast payload and the least local computation compared to FL and FTL. For FL, FTL that updates all parameters, FTL that updates the task-specific sub-model, and FbFTL, the overall downlink broadcast payloads, respectively, are

We note that FbFTL consumes less computation time and power for training in total, and also transfers a proportion of computation from the client devices to the PS. In FL and FTL, each client must complete one full forward pass and one back-propagation of the parameters to be trained for each training sample at each iteration. However, in FbFTL, each sample is only processed once by the client, and each client only needs to complete the forward pass of feature extraction sub-model, while all other computations are completed at the PS. Such shift of computational load to the PS is particularly beneficial if the end users and devices are severely limited in their computational capabilities and power resources (for instance, in IoT networks) and the PS is equipped with advanced processors and has access to more resources.

In this subsection, we use time complexity to estimate the computation time and energy consumption. For each fully connected layer $m_{1}$ with $N^{-}_{m_{1}}$ input nodes and $N^{+}_{m_{1}}$ output nodes, there are $T_{m_{1}}=N^{+}_{m_{1}}(N^{-}_{m_{1}}+1)$ trainable parameters, and each sample requires $O(N^{+}_{m_{1}}(N^{-}_{m_{1}}+1))=O(T_{m_{1}})$ time complexity for matrix multiplication during forward pass, and the same time complexity for matrix multiplication during back-propagation. Each 2-dimensional convolutional layer $m_{2}$ with stride 1 in each direction and same padding has the input shape $\{a^{-},b^{-},c^{-}\}$ where $N^{-}_{m_{2}}=a^{-}b^{-}c^{-}$, kernel shape $\{c^{+},a^{\prime},b^{\prime},c^{-}\}$ where $T_{m_{2}}=c^{+}(a^{\prime}b^{\prime}c^{-}+1)$, and output shape $\{a^{-},b^{-},c^{+}\}$ where $N^{+}_{m_{2}}=a^{-}b^{-}c^{+}$. Thus, each sample requires $O(c^{+}(a^{\prime}b^{\prime}c^{-}+1)a^{-}b^{-})=O(a^{-}b^{-}T_{m_{2}})$ time complexity for matrix multiplication during forward pass, and the same during back-propagation. Comparatively, the time complexity of activation functions, max-pooling layers and dropout layers is negligible. Therefore, we denote the time complexity of each trainable layer $m$ for a single training sample as $O(X_{m})$, and the overall time complexity required for all clients as $O(X)$. Specifically, the overall time complexities at clients for the methods of FL, FTL that updates all parameters, FTL that updates the task-specific sub-model, and FbFTL, respectively, are

where $\propto$ stands for “proportional to”. Typically, we have $X^{FbFTL}<X^{FTL}_{c}\ll X^{FTL}_{f}<X^{FL}$.

In this section, we consider the application of FL, FTL and FbFTL to the VGG-16 CNN model [41] and transfer the knowledge learned from ImageNet dataset [42] to CIFAR-10 dataset [43].

ImageNet is a vast online database containing over 14 million images, each with hand-annotated labels describing the classification types or intended outputs for training. The pre-trained source model we use for TL was created on the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC2012, [71]), and the images come from 1000 different categories. As an illustration, we provide 10 samples with their labels in Fig. 3.

CIFAR-10 is a database of $60000$ images, each with one of $N=10$ distinct labels. Out of these images, $50000$ images are used for training and $10000$ images for testing. Fig. 4 showcases the first ten samples with their labels. Note that these samples have simple labels and appear more blurred compared to those in ImageNet due to their lower resolution/dimension.

In Fig. 5, we depict the structure of VGG-16 used for training on CIFAR-10. For TL, we consider the first half of the layers marked blue as the feature extraction sub-model $f^{1}_{\boldsymbol{\theta}^{1}}$ and directly transfer this sub-model from that trained on ImageNet. We deem the latter part marked yellow as the task-specific sub-model $f^{2}_{\boldsymbol{\theta}^{2}}$ and randomly initiate this part. For FbFTL, the dimension of the intermediate output is $N^{-}_{m_{c}}=4096$.

To train the model on CIFAR-10, we utilize Nvidia GeForce GPU with CUDA to run the algorithms with PyTorch [72]. We assume that there are $U=6250$ clients in total, each iteration takes a fraction $C=1.28\times 10^{-3}$ of all clients, and each batch contains $K_{u}=8$ samples. In the two most commonly used deep learning tools TensorFlow (including Keras) [73] and PyTorch, the default data type of each number has 32 bits and therefore $d=32$ bits. The learning rate is $10^{-2}$, the momentum of the optimizer is $0.9$, and the L2 penalty is $5\times 10^{-4}$.

In Table III, we compare the performances of FL, SFL, FTL updating the full model (FTL_f), FTL updating the task-specific sub-model (FTL_c), and FbFTL. For increased fairness in the comparison of payloads, we further demonstrate the performances FL^low and FTL${}_{f}^{low}$, which are the FL and FTL_f algorithms that terminate training process when the validation accuracy reaches those of FTL_c and FbFTL (i.e., $\approx 86\%$). As we have analyzed, the other algorithms require $IUC$ successfully uploaded batches, while FbFTL only requires $\sum_{u=1}^{U}{K_{u}}$ batches. Also, FL, SFL, FTL_f and FTL_c require uploading $d\sum_{m=1}^{M}{T_{m}}$ bits, $d(K_{u}N^{-}_{m_{c}}+\sum_{m=m_{c}}^{M}{T_{m}})$ bits, $d\sum_{m=1}^{M}{T_{m}}$ bits, and $d\sum_{m=m_{c}}^{M}{T_{m}}$ bits, respectively, for each batch, while FbFTL only requires uploading $dN^{-}_{m_{c}}$ bits for each batch. In the third row of the table, we observe that the FbFTL algorithm significantly reduces the uplink payload per batch by four orders of magnitude (i.e. a factor of $10^{-4}$) compared to the other algorithms for each client. Additionally, in the fourth row, FbFTL leads to a reduction of five orders of magnitude (i.e. a factor of $10^{-5}$) in the total uplink payload during training when compared to the other algorithms. These results demonstrate that FbFTL is apparently the most efficient scheme. FbFTL also results in a substantial decrease in the overall downlink payload. If larger models are trained for more complex tasks or if the size of the training dataset is more limited, the difference in payload could be even greater. In (14) through (17), we have described the overall computation time complexity required for training at clients as $O(X)$. In the second-to-last row of the table, we quantify and provide this time complexity for each method by counting the number of multiplications among floating numbers. We readily observe that FbFTL requires the least computation time and power consumption at the clients, and has two orders of magnitude reduction compared to FL, SFL and FTL_f, since it only runs the forward pass over feature extraction sub-model for each sample one time. We further note that for FbFTL, the computation complexity at the PS is also low with $X=3.00\times 10^{14}$. Furthermore, we note that different number of clients $CU$ in each iteration does not affect the performance of FbFTL, but a large number of clients $CU$ reduces the performance of FL and FTL, since it defines the minimum “training batch size” $CUK_{u}$. The performance of FL and FTL will decrease with an overwhelmingly large training batch size, which is another benefit of FbFTL. We in the experiment pick a small $CU=8$ so that the benchmark schemes (i.e., FL, FTL_f, and FTL_c) have the best performance (at which point FTL_c and FbFTL have the same performance). Moreover, we also observe that FTL_c and FbFTL only update a small portion of the parameters and exhibit a slight decrease in validation accuracy, which is the trade-off for the reduced payload. However, we will show in Section IV and Section V that under communication efficiency or privacy constraints, FbFTL may also prevail in terms of validation accuracy in certain situations.

Furthermore, we note that in transfer learning, there is an additional trade-off between privacy protection, performance and payload. When partitioning a model into feature extraction sub-model and task-specific sub-model, choosing the cut layer closer to the output better preserves data privacy, while picking a cut layer closer to the input improves the training performance. In order to demonstrate such a trade-off, we next consider a language model as our application scenario. In FL, FTL, and FbFTL on natural language processing tasks, we consider tasks where the model and the intermediate features are not proprietary or private, and thus we assume that the clients are willing to share them while keeping the local data private As an example, we show the results on a conversation summary task SAMSum [74] with 32128 distinct token types out of 14732 training dialogues and 819 testing dialogues. For instance, dialogue ID 13728867 in SAMSum is “Olivia: Who are you voting for in this election? Oliver: Liberals as always. Olivia: Me too!! Oliver: Great”, and ground truth summary is “Olivia and Olivier are voting for liberals in this election.” For this task, we deploy a language model FLAN-T5-small [75], which is a transformer with 110 million parameters, including 8 encoders and 8 decoders. This model is pre-trained and the dataset is relatively small, so we do not have FL in this case. We assume that there are $U=7366$ clients, each has $K_{u}=2$ dialogues, and each iteration takes a fraction $C=5.43\times 10^{-4}$ of all clients. Our experiment is based on HuggingFace [76] with learning rate $2\times 10^{-4}$.

In Table IV, we compare the performances of FTL and FbFTL in terms of ROUGE-1 score, which measures the match between the generated text and reference text. A larger ROUGE-1 indicates better performance. We note that FTL_f trains all components including encoders, decoders, embedding and the final linear layer. On the other hand, FTL_c and FbFTL freeze embedding, and may or may not freeze several encoders close to the input prompts. The number of trained encoders is given in the second row of Table IV. For instance, the performance results in the third and fourth columns are for FTL_c and FbFTL that have trained all encoders (and hence have not frozen any of them), while the other columns provide the performances with 4 or 2 encoders trained (indicating that 4 or 6 encoders close to the input are frozen). We do not need to freeze decoders since label privacy leakage converges to zero with shuffled batches, as will be shown in Section V. In Table IV, we observe that FbFTL reduces the uplink and downlink payload by similar orders of magnitude as in the VGG-16 experiment. Furthermore, we notice that training less layers or encoders leads to a slight reduction in ROUGE-1 score but it does not guarantee lower payload. While FTL may require more iterations to arrive convergence, FbFTL may need to broadcast a larger feature extraction sub-model.

As we have demonstrated in the previous section, gradient-based FedAvg FL frameworks including FL and FTL upload the gradient update $\textbf{g}_{u}=\sum_{k=1}^{K_{u}}{\nabla_{\boldsymbol{\theta}}{L(f_{\boldsymbol{\theta}}|\textbf{s}_{u,k})}}$ iteratively, where each batch contains the gradient summation from all samples of the client. In contrast, our proposed FbFTL uploads the data for each sample only once, and each batch contains extracted features $\textbf{z}_{u,k}$ and output $\textbf{y}_{u,k}$ from one sample.

In (7), we have shown that each batch for gradient-based FL is much larger than each batch for FbFTL (by about $10^{4}$ larger in our experiments), and we assume that the packets to transmit both types of batches consist of multiple transmission blocks of the same size. For both types of packets, we consider measuring the robustness of all frameworks against packet loss caused by block losses due to network congestion or link outage (e.g., as a result of deep fading in wireless networks). For each learning framework whose packet consists of $n_{b}$ transmission blocks, we consider the same block loss rate (BLR). The PLR without retransmission is

Obviously, PLR highly depends on the value of $n_{b}$. FbFTL has significantly lower packet size and consequently we expect much lower PLR for the given same BLR. We show the significant performance difference among different learning frameworks in our experiments at the end of this section.

If, on the other hand, we allow at most $n_{r}$ retransmissions for all packets and assume the channel state of each transmission to be independent and identically distributed (i.i.d.), we can lower the PLR and achieve similar packet-level reliability. However, this is realized at the cost of higher total uplink payload. Specifically for each learning framework requiring uplink payload $P$, the expected uplink payload with retransmissions is

We note that $\sum_{n=1}^{n_{r}}(\text{BLR})^{n}$ is the expected number of retransmissions, which is the same for all different learning frameworks. If the transmission has a fixed bandwidth, $\sum_{n=1}^{n_{r}}(\text{BLR})^{n}$ also represents the delay factor of the total transmission. We further note that if $\text{BLR}=0$, then $P(\text{BLR})=P$. Otherwise, $P(\text{BLR})=P+P\sum_{n=1}^{n_{r}}(\text{BLR})^{n}>P$. Notice that the increase in the payload $P\sum_{n=1}^{n_{r}}(\text{BLR})^{n}$ grows with $P$ and $n_{r}$. Hence, learning frameworks with higher payload experience a higher increase in payload when retransmissions are introduced.

At the end of Section III-A, we have discussed the benefit that FbFTL does not require additional online uploads from the clients to test different sets of hyper-parameters, while gradient-based frameworks need to run the entire uploading process multiple times. In practice, in addition to hyper-parameters, another intangible aspect prior to training is whether there exists a sufficient number of participating clients and training samples. If the planned set of samples is insufficient to train the neural network, gradient-based frameworks require rerunning the overall process with more clients, which leads to high consumption and potential waste of both computational and transmission resources. However, FbFTL only requires the new clients to compute and upload their batches, and hence there is no waste of transmission resources.

To further reduce the uplink payload of gradient-based frameworks, there have been extensive works on gradient compression, including gradient sparsification [77, 78] and gradient quantization [63, 64], especially signSGD, the extreme case in which each element is reduced to be binary valued without scaling [79]. Several recent studies utilize error feedback (or quantization and sparsification with memory) that reduces the error in compression at client’s local device to improve the updates in future iterations [50, 49].

For gradient-based frameworks utilizing gradient update $\textbf{g}_{u}$ with $X_{g}$ elements, we denote the sparsification function as $\mathcal{S}_{r}(\cdot):\mathbb{R}^{X_{g}}\rightarrow\mathbb{R}^{X_{g}}$, where $r\in(0,1]$. This function keeps the value of $rX_{g}$ elements in the vector with the highest absolute values, and set the value of all other elements to 0. We denote the quantization function as $\mathcal{Q}_{q}(\cdot):\mathbb{R}^{X_{g}}\rightarrow\mathbb{R}^{X_{g}}$ where $q\in\mathbb{Z}^{+}$, and this function quantizes each element in the vector to $q$ bits within the max/min range. Furthermore, we denote the error memory vector for client $u$ at iteration $t$ as $\textbf{m}_{u,t}\sim\mathbb{R}^{X_{g}}$. Therefore, the process of quantization and sparsification with error feedback at iteration $t$ is described as follows:

where $\textbf{g}^{\prime}_{u}$ is uploaded to the PS, and the local error memory vector is updated to $\textbf{m}_{u,t+1}$. For initialization, $\textbf{m}_{u,1}=\textbf{0}$.

Similarly, for FbFTL utilizing extracted feature $\textbf{z}_{u,k}$ with $X_{z}$ elements, we may also apply sparsification and quantization. However, error feedback is not applicable, because there is only one upload iteration in FbFTL, and the extracted feature is not additive unlike the gradient. Therefore, the process of quantization and sparsification for FbFTL is described as follows:

where $\textbf{z}^{\prime}_{u,k}$ is uploaded to the PS.

In both cases, the compression rate is close to $2^{d/q}/r$ where $d$ is the number of bits for the representation of the original data type. We note that there is also a potential drawback in the practical deployment of sparsification and error feedback. On the one hand, while all other operations including training, communication, inference, quantization and error feedback have time complexity no more than $O(X^{\prime})$ where $X^{\prime}$ is the total number of parameters in the neural network, we notice that depending on the sparsification ratio $r$, sparsification requires up to $O(X_{g}\log{X_{g}})$ time complexity for gradient-based frameworks and $O(X_{z}\log{X_{z}})$ time complexity for FbFTL. Typically, we have $O(X_{z})\ll O(X_{g})=O(X^{\prime})$, and FbFTL can achieve more significant time complexity reduction in local sparsification computation compared to gradient-based frameworks. On the other hand, by compensating for the compression error in future iterations, the error feedback significantly improves the training performance when the compression rate is high, i.e., $r$ and $q$ are low. However, the error feedback requires $O(X^{g})$ additional memory throughout the entire training process and not just during the local training. Also, we demonstrate below in the experimental results that the gradient-based frameworks do not prevail even with error feedback.

In this section, we numerically demonstrate the aforementioned robustness metrics in our experiments with the VGG-16 CNN model on transfer learning from ImageNet dataset to CIFAR-10 dataset.

In Fig. 7, we show the PLR (without retransmissions) of different learning frameworks when the block size equals the batch size of FbFTL (i.e., the BLR equals the PLR of FbFTL). As we have shown in the row of uplink payload per batch of Table III, the batch sizes of other learning frameworks are about $10^{4}$ times larger than that of FbFTL. Due to the huge difference in the batch sizes, the PLR curves of the other learning frameworks almost reach $1$ when the PLR of FbFTL is less than $0.001$ (or equivalently when BLR$<0.001$). The difference is still substantially high if a few rounds of retransmissions are allowed in gradient-based frameworks. Fig. 7 provides the magnified plot of Fig. 7 for BLR$<0.001$, and we note that the curve of FL and the curve of FTL that updates full model overlap since they have the same number of parameters to update and therefore have the same batch size. In this figure, we again observe that the PLR curves of learning mechanisms other than FbFTL quickly approach $1$ within the considered range of BLR $<0.001$, while the PLR of FbFTL (being equal to BLR) stays below $0.001$.

Since FbFTL has significantly lower PLR, we subsequently analyze its performance with different amount of data. In Fig. 8, we see that FbFTL has relatively high validation accuracy even with only 10% of the samples. Indeed, when we have only $0.1\%$ of the samples (i.e., the case with 50 samples), the accuracy is $82.5\%$ even with PLR$=0.5$. Furthermore, we observe that as PLR increases, the accuracy curves remain relatively flat, and experience sharp drops only when PLR approaches 1. Therefore, FbFTL is considerably robust against data insufficiency and PLR. We note that such robustness requires significantly more training iterations $I^{\prime}$ such that $IU$ and $I^{\prime}U^{\prime}$ have the same order of magnitude, where $I$ is the original number of training iterations with all $U$ participating clients, and $I^{\prime}$ is the number of training iterations with limited data from $U^{\prime}$ clients. In the case of smaller number of participating clients $U^{\prime}\ll U$ and PLR $=0$, FbFTL requires the same level of computational power consumption and more total downlink payload compared to the original case with $U$ clients. However, there is a huge reduction in total uplink payload and total local computational power consumption, because FbFTL only requires uploading once for each participating client. In comparison, gradient-based frameworks require uploading in every training iteration, and therefore there is no such benefit in terms of reduced uplink payload and local computations, while they also suffer from higher downlink payload. Similarly, we have the performance curves for FLAN-T5-small in Fig. 9.

In Table V, we show the validation accuracy of each framework with data from all $U$ clients and PLR $=0$, but for different values of quantization size $q$ bits and sparsification ratio $r$. In the experiment, we pick the same set of values for $q$ and $r$ as in [77, 78]. Note that the highest reduction in uplink payload is achieved when we set $r=0.001$ and $q=2$. However, even in this case, the uplink payload of gradient-based frameworks (i.e., FTL, FTL_f, FTL_c) is still greater than that of FbFTL with $r=1$ and $q=32$. Moreover, with such drastic reduction, gradient-based schemes achieve lower accuracies compared to that of FbFTL with $r=1$ and $q=32$. Therefore, even without sparsification and more restrictive quantization (e.g., $q=8$ or $q=2$), FbFTL outperforms gradient-based frameworks in terms of both accuracy and payload reduction. We can further reduce the uplink payload of FbFTL by choosing $r<1$ and $q<32$. We note that FbFTL in the extreme case of $r=0.001$ only keeps 4 elements in the extracted features to distinguish among 10 classes. In this setting, the information is severely limited and is insufficient to make an accurate prediction, resulting in accuracy levels of $65\%$ for FbFTL. If, on the other hand, we pick a certain threshold on the validation accuracy, for instance 82%, FbFTL requires the sparsification ratio to be no lower than $r=0.01$ (which is $10$ times more than that of the best gradient-based framework), but still maintains a reduction of more than four orders of magnitude (i.e., reduction of $10^{-4}$) in total uplink payload compared to the gradient-based algorithms while achieving the same validation accuracy. Such good performance of FbFTL without error feedback is due to the robustness of extracted features against noise. If we consider the error from compression as random noise, the extracted features from a well-trained source model is typically robust to noise while gradient update does not necessarily have the same level of robustness.

First, we analyze the label privacy leakage. While FbFTL directly uploads the output in each batch for each sample, we note that FL and FTL with FedAvg that update the final layer also leak the output, and hence label privacy leakage also occurs in these cases. According to [80], the count of each output type in a batch can be numerically solved given the average gradient. Adversaries with certain prior knowledge on the training data are able to gain further knowledge and reconstruct the input from the average gradient, such as deep leakage [81] or gradient inversion [82, 83]. Although the following label privacy analysis applies to both feature-based FbFTL and gradient-based FedAvg frameworks, we in label privacy analysis use the word “batch” to indicate all uploaded information from one client with multiple samples. Compared to the high-dimensional gradient $\textbf{g}_{u}$ or feature $\textbf{z}_{u,k}$, the output y also potentially reveals clients’ private information but up to a certain degree. In this setting, we analyze the conditions under which the label privacy leakage (with statistical information) vanishes. We also address the role of shuffling the output information from all clients as a way of hiding the client’s address from uploaded content when adversary has access to outputs.

To validate these approaches, we analyze the private output information leakage of a specific batch to a potential adversary without client addresses. According to [84], the privacy loss of a query can be evaluated as the difference in the privacy before and after the query. In our setting, we determine the amount of label privacy via the uncertainty from the adversary’s perspective, and quantify it by utilizing the entropy formulation.

As shown in Fig. 10, we consider a common learning task in which the output $\textbf{y}\in[0,1]^{N}$ is an axis-aligned unit vector with one element having a value of 1 indicating the label associated with the given input, and all others equal to 0. We assume that each sample $\{\textbf{x}_{u,k},\textbf{z}_{u,k},\textbf{y}_{u,k}\}$ is independent and identically distributed (i.i.d.). Each client $u\in\mathcal{U}$ transmits one batch with $K_{u}=K$ samples, including the outputs/labels $\{\textbf{y}_{u,k}\}_{k=1}^{K}$. Therefore for this client, there are $N^{K}$ different possible ordered batches of outputs in total (where $N$ is the number of possible different labels that can be associated with the input).

Assuming that the order in the batch does not contain information, we use $\mathcal{B}$ to denote the set of possible batches without order from client $u$, and the privacy information of the real batch $b$ from client $u$ is the sum vector $\sum^{K}_{k=1}\textbf{y}_{u,k}=[n^{y}_{1,b},n^{y}_{2,b},\ldots,n^{y}_{N,b}]$, where $\sum^{N}_{a=1}n^{y}_{a,b}=K$. Subsequently, the number of possible batches without order is the combination with the replacement of $N$ items taken $K$ times and is given by

We denote the index of client $u$’s batch as a random variable $B$, and the index of the uploaded batch as $B=b\in\mathcal{B}$. We denote the type of the label $\textbf{y}_{u,k}$’s distribution as $P_{y}$. Typically to achieve better performance, in most of the machine learning applications we desire a uniform distribution over different labels, and we denote the uniform distribution over $N$ labels as $P_{y,\text{uni}}=[\frac{1}{N},\frac{1}{N},\ldots,\frac{1}{N}]$.

To evaluate the adversary’s knowledge on one given batch $B=b$ of a target victim client, we consider the presumed distribution $\textbf{P}_{B}$ of the batch $B$ from the adversary’s perspective to quantify the adversary’s uncertainty through the entropy $H(B)$ of the given batch.