PS-FedGAN: An Efficient Federated Learning Framework Based on Partially Shared Generative Adversarial Networks For Data Privacy

In this work, we aim to develop a novel GAN-based FL framework for a global/common task in a distributed learning setup. For convenience, we will use image classification as an illustrative example henceforth. Consider a reliable central server with limited access to client training data to achieve a desirable accuracy on the global task. Here, we assume non-IID data distributions and vulnerable communication channels among users, which are common in practical applications. For example, in smart healthcare, one learning task could be to train neural networks for the detection of a specific disease. One clinic may have a unique set of brain images from patients whereas other neighboring hospitals may also have access to several types of similar diseases with a plethora of examples. A global model based on all distributed data to detect these diseases could benefit all hospitals and provide better service for patients. In such a collaborative system, local data privacy is a critical concern. Also, adding noises to the dataset to hide sensitive information may lead to unwanted information distortion and/or artifacts. In this work, we address the problem of how to preserve local privacy while preserving the original data statistics.

Motivated by existing GAN-based FL, we develop a novel GAN publishing mechanism for FL with privacy preservation and communication efficiency. In alignment with other GAN-related FL works, we assume a system with a centralized server in the cloud and multiple distributed clients/users in communication with the server. Each client has access to enough resources to train a local GAN model. Note that, although we apply conditional GANs (cGAN)  Mirza and Osindero (2014) to alleviate the need for label detection via pseudo-labeling, the principle of our proposed scheme generally applies to all types of generative models.

To assess the effectiveness of our framework in preserving privacy, we consider the presence of adversarial attackers. In order to model potential attacks from these adversaries, we make the assumption that an attacker has the capability to eavesdrop on the communication channels connecting local users and the server, as depicted in Figure 1. The attacker’s objective is to estimate the data distribution of the local user through reconstruction attacks.

As illustrated in Figure 1, in conventional GAN-based FL methods, such an attacker would have unrestricted access to the entire GAN model. However, our proposed PS-FedGAN is specifically designed to address the security vulnerabilities associated with fully shared GAN models.

We now delve into the structures of the proposed PS-FedGAN. In order to address the privacy concerns arising from full GAN sharing, we introduce a partial sharing approach within our PS-FedGAN framework. Specifically, we propose training two generators: generator $G_{u}$ at the local user’s end and generator $G_{s}$ at the server side for each user. In order to bridge the training of two generators, i.e., one at a local user and the other at the server connected by a communication link, we share a common discriminator $D_{u}$ which is trained only at the local user.

To visually depict the update process of PS-FedGAN, we present the user-server communication flow leading up to a single-step update of the global model ($C_{l}$) in Figure 2. In the case of a single user, the training process begins with the local user training a local GAN model consisting of generator $G_{u}$ and discriminator $D_{u}$. After completing a single batch training at the local user’s end, the trained discriminator $D_{u}$ is shared with the server through the PS-FedGAN publishing mechanism $\mathcal{M}_{p}$. The details of this publishing scheme will be further elaborated in Section 2.2.1. In our study, we consider an untrustworthy communication channel where attackers may potentially gain access to the PS-FedGAN publishing mechanism $\mathcal{M}_{p}$. On the server side, we initialize a separate generator $G_{s}$ for each user, following the guidelines of PS-FedGAN Algorithm 1. Subsequently, the server updates the corresponding $G_{s}$ based on the information received through $\mathcal{M}_{p}$.

Before updating the global model $C_{l}$, we wait for a specific number of updates from the user side, denoted as $N$. In this context, $N$ can represent an epoch for the local user. Consequently, we carry out $N$ updates on both the local user’s GAN and the corresponding $G_{s}$ in parallel. The generated data from all the updated $G_{s}$ models, corresponding to different local users, is combined with the server-available data to update the global model $C_{l}$. This cycle of updates and combination of generated data is repeated until $C_{l}$ converges to the desired point.

When deploying the FL models, we assume each local user and the server have agreed on a secret seed and the same architecture for $G_{u}$ and $G_{s}$. The secret seed serves to initiate the weights of $G_{u}$ and its corresponding $G_{s}$. Note that each user is assigned its own dedicated generator in the cloud server. Also, we impose no constraint on $D_{u}$. PS-FedGAN follows a batch-wise training process where we update $G_{s}$, $G_{u}$, and $D_{u}$ at each step. Once $G_{u}$ and $G_{s}$ are initiated, local GAN training commences with $D_{u}$. In the following subsections, we provide a detailed explanation of the PS-FedGAN publishing mechanism $\mathcal{M}_{p}$ followed by an elaboration of the training process for both the local users and the cloud server in each communication round.

To assess the performance of PS-FedGAN against potential attackers, we focus on reconstruction attacks specifically. These attackers are denoted as $\mathcal{A}_{R}$. For any $\mathcal{M}(\theta)$, an attacker $\mathcal{A}_{R}$ attempts to reconstruct the training samples and assume $\mathcal{I}$ represents a reconstructed sample (images in this manuscript), i.e.,

In our experimental setups, we consider two types of attackers: $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$. The bottleneck faced by any attacker against our model lies in the hidden generator model that is prone to information leakage. Therefore, the process of reconstructing synthetic data helps preserve privacy in terms of the initial weights and network architecture, as further elaborated in Section 3. Predicting the exact architecture has become challenging owing to advancements in deep learning techniques.

For an attacker to obtain information, it is crucial to eavesdrop from the very beginning and ensure they do not miss any round of communication. Such requirements prove difficult for potential privacy attackers. Additionally, the attacker must acquire knowledge of the initial weights of the generator, to be discussed in Section 3. However, in practice, it is typically challenging, and often impossible, for an attacker to accurately estimate the generator’s structure due to various practical constraints such as power, latency, and hardware capabilities. The complexity and variability of generator architectures play a significant role in preserving privacy. The diverse range of architectures available for generators, coupled with their complexity, makes it highly challenging for an attacker to accurately infer the precise model structure. This inherent difficulty further enhances the preservation of privacy in the system. By utilizing complex and variable generator architectures, PS-FedGAN adds an additional layer of protection against potential privacy breaches.

Therefore, to define new attacks, we first introduce a factor $r\in[0,1]$. We assume that attackers $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ have the same architecture of $G_{u}$ and the same weights and bias terms of $G_{u}$ in every deep learning network layer except the first. Let $w_{1}$ and $b_{1}$ be the layer 1 weight and bias term of $G_{u}$. Layer 1 weight $w_{a1}$ and bias term $b_{a11}$ of $\mathcal{A}_{1}$ are set to $w_{a11}=rw_{1}$ and $b_{a1}=b_{1}$, respectively. Similarly for $\mathcal{A}_{2}$, we set layer 1’s weight unperturbed as $w_{a21}=w_{1}$, and set layer 1’s bias to $b_{a21}=rb_{1}$. Clearly, these two attackers’ knowledge of $G_{u}$ parameters are only slightly different from the true parameters with a multiplicative factor $r$ on Layer 1’s weights and biases, respectively.

We first show the convergence of two generators and a discriminator trained according to Algorithm 1. Suppose that the GAN trained at a local user consists of a generator $G_{u}$ capturing a probability distribution of $p_{gu}$, and a discriminator $D_{u}$. Let the local user’s data distribution be $p_{data}(x)$. The local user (client) trains $G_{u}$ with $z\sim p_{z}$. We have the following properties on model convergence.

Proposition 1. Two generators $G_{u}$ and $G_{s}$ trained on Algorithm 1 with a shared discriminator $D_{u}$ converges to the same optimal discriminator $D_{u}^{*}$ as in Goodfellow et al. (2014) and it uniquely corresponds to the given $G_{u}$, i.e.,

In Proposition 1, we see that the $D_{u}$ in PS-FedGAN converges to the same discriminator if we train $G_{u}$ and $D_{u}$ without $G_{s}$. That is, training $G_{s}$ in the cloud does not hamper the convergence or performance of the local GAN training. On the other hand, we have a unique $D_{u}$ given $G_{s}$. This property solidifies the convergence of $G_{s}$ to $G_{u}$ which is characterized in the following Propositions regarding the generator models.

Proposition 2. Two generators $G_{u}$ and $G_{s}$ trained according to Algorithm 1 with a shared $D_{u}$ would converge to a unique $G^{*}=G_{u}^{*}=G_{s}^{*}$ which captures $p_{data}$.

This proposition establishes that two distributed generators trained using PS-FedGAN converge to the same generative model. Moreover, this model is the same as the optimized model that one can obtain via classical GAN training. Another vital observation is that both $G_{u}$ and $G_{s}$ capture the user-side data distribution.

Proposition 3. Any other generator $G_{\mathcal{A}}$ failing to capture the weights and architectures of $G_{s}$ or $G_{u}$, either in the initial state or in any single communication round, would fail to characterize the data distribution $p_{data}$.

This proposition provides us with insights into the capacity required by an attacker. To attack the proposed model, an attacker would need to predict a generator $G$ using the information obtained from $\mathcal{M}_{p}$. However, to successfully carry out this attack, the attacker would need to possess precise knowledge of the generator’s architecture, initial weights of either $G_{s}$ or $G_{u}$, and would need to monitor and capture every round of communication. In the next section on experiments, numerical results shall substantiate the need for these requirements and further demonstrate the difficulty an attacker would face when attempting to breach the privacy of the PS-FedGAN model.

We now discuss the DP properties of the proposed methods.

Let us denote the discriminator by $D=f(data)$ and the generator by $G=g(D,z)$, where $z$ represents noise. From the post-processing property in Dwork and Roth (2014), $g(f(D))$ is DP if $f(data)$ is DP. Thus, the generator $G$ is DP given $D$ is DP. Furthermore, if the training process is based on original data, FL-GAN follows DP Xin et al. (2022).

In practical scenarios of PS-FedGAN, accessed models of $D$ with quantized channel noise is DP  Amiri et al. (2021) on the original weights of $D$, i.e., $W_{D}$. Since an attacker can only gain access to discriminator $D$ during communication round in the proposed PS-FedGAN framework, any generators reconstructed by the attackers from the hacked $D$ is also DP on $W_{D}$.

Proposition 4. Any generator $G$ reconstructed by the attacker shall be DP in a communication channel with quantization noise or channel induced error.

This proposition shows that the attackers’ estimated GAN model is DP on the original model weights $W_{D}$ sent from a client to the server. Considering the mutual information of shared models and original data, we have $I(data,W_{D})\leq I(data,data)$. Therefore, if we preserve privacy in $W_{D}$, we also preserve some privacy in the original data.

Proposition 5. $\mathcal{M}_{p}$ preserves privacy compared to full data sharing.

Our study considers three distinct scenarios of heterogenous user cases, which are based on the work presented in  Li et al. (2022b). Each case involves a total of 10 users and the following details:

• Split-1:

  In this case, training data is divided into 10 shards, each containing samples from a single class. Each user is randomly assigned one distinct shard.

• Split-2:

  This case generates 20 training data shards, each consisting of samples from a single class. Two shards are randomly assigned to each user without overlap.

• Split-3:

  This case generates 30 shards, each containing samples from a single class. Three shards are randomly assigned to each user without overlap.

As a utility measure, we select the classification accuracy of the global model in a supervised setup. We compare our results against several existing FL alternatives: FedAvg (FA)  McMahan et al. (2016), FedProx (FP)  Li et al. (2020a), SCAFFOLD (SD)  Karimireddy et al. (2020), Naivemix (NM)  Yoon et al. (2021), FedMix (FM) Yoon et al. (2021), and SDA-FL (SL) Li et al. (2022b) as baselines. We also compare our partially-shared PS-FedGAN (PS) with the fully-shared GAN (FG) as a performance benchmark. The results of Split-1/2 are shown in Table 1. The performance of Split-3 can be found in the Appendix.

Table 1 shows the superior performance of the proposed PS-FedGAN over most existing methods except for the FG. More specially, we see a significant improvement in PS-FedGAN in Split-1 for the CIFAR10 dataset. Compared with full-GAN (FG) sharing, our partially-shared GAN achieves similar performance but at significantly reduced communication cost and privacy loss as further shown in Table 4. Note that in the above test, the SDA-FL method requires an infinite privacy budget to achieve the desired utility. This indicates that SDA-FL faces significant challenges in balancing privacy and utility. Additionally, it is worth mentioning that other alternatives provide more real data for the classifier from Split-1 to Split-3, whereas in our proposed method, we maintain a constant amount of real data on the server (1%). Similar results can be seen for Split-3 in the Appendix.

We now evaluate the privacy of PS-FedGAN with respect to reconstruction attacks, which is often viewed as one of the most dangerous attacks. Here, we utilize the MNIST dataset. To evaluate the efficacy against reconstruction attacks, we use several proxies to measure the privacy leakage, including normalized mean square error (NMSE), structural similarity index (SSIM)  Dangwal et al. (2021b), and classification accuracy. We consider two different non-IID user setups:

• •

  Setup-1 includes three users. User-1 has access to classes {0,1}, user-2 has access to classes {2,3,4}, and user-3 has access to remaining classes {5,6,7,8,9}. Also attacker model $\mathcal{A}_{1}$ described in Section 2.3 is used.

• •

  In Setup-2, we consider 10 users with data splitting in Split-1, where attacker model $\mathcal{A}_{2}$ is applied here in reconstruction attacks.

Classification Accuracy: To evaluate the attacker’s classification accuracy on reconstructed images, we first consider Setup-1. In this setup, we assume that the attacker has access to all the elements released by the releasing mechanism $\mathcal{M}p$. Furthermore, we assume that the attacker can accurately guess the exact generator architecture. Note that at the beginning of training, the generator weights are different for $\mathcal{A}_{1}$ ($w{a11}=rw_{1}$), from the cloud generator as mentioned earlier. The generators are trained simultaneously at the user, the server, and the attacker.

We then pick an attacking classifier trained on the original MNIST training set and infer data generated by the attacker’s generator. Here, the attacker assumes that the user works with MNIST data. Table 2 illustrates the attacker’s performance in different $r$. From Table 2, we see that the attacking gains better accuracy with increasing $r$. In Table 2, we evaluate the classification accuracy at the cloud using the same attacking classifier, which reflects the potential of an ideal attacker and the utility at the cloud. Table 2 shows that an attacker with some disparity in the initial weights can only perform like a random guess. Note that in this scenario, user-1 has 2 classes, user-2 has 3 classes, and user-3 has 5 classes. These results suggest that an attacker must obtain very accurate information on architecture and initial model weights to achieve higher and nontrivial inference accuracy. Since such requirements are improbable and impractical, our results establish the robustness of our proposed method against classification attacks.

Next, we consider $\mathcal{A}_{2}$’s classification accuracy on reconstructed images. For this attacker, the difference between its generator and the cloud generator lies the first layer bias term ($b_{a21}=rb_{1}$). We evaluate $\mathcal{A}_{2}$’s performances on Split-1. Figure 3 illustrates the attacker’s performance for different $r$. Figure 3 shows that the attacking gains better classification with increasing $r$. Similar to the first type of attackers in $\mathcal{A}_{1}$, these results suggest that an attacker from $\mathcal{A}_{2}$ also needs to have very accurate architecture knowledge and the initial weights. This test case further establishes the robustness of PS-FedGAN robust against classification attacks.

Reconstruction Quality and Similarity As presented in the paper Dangwal et al. (2021a), reconstruction quality and similarity can be used as a measure of privacy leakage. For this, we consider 2 metrics, i.e., SSIM and NMSE, in Setup-1. Table 3 compares the corresponding generations of the attacker and the cloud based on the similarity between each generated image. From Table 3, we can see that the NMSE values achieved by the attackers are high while SSIM (maximum 1) is very low. These results indicate that the attacker-generated images cannot capture the true data distribution. We shall further show in Appendix that, if the attacker deviates from the actual generator weights, its convergence would be to a trivial point. As a result, no underlying user data properties are captured, as illustrated by the visual examples in Figure 4. More results and discussions are provided in the Appendix.

Table 4 illustrates the number of parameters that need to be shared between each user and the cloud server per communication round in the full GAN vs PS-FedGAN. From the table, we can see a significant saving in PS-FedGAN compared to classical full GAN sharing. The advantage applies similarly to various different GAN architectures.

Federated Learning (FL) offers a simple and efficient method for privacy-protected distributed learning. In classic FL, local data collections are often non-IID, unbalanced, and widely distributed, typically with limited communication bandwidth McMahan et al. [2016]. To reduce communication overhead and maintain privacy, a Federated Average (FedAvg) algorithm is proposed. FedAvg aggregates gradient updates from different users by combining several local updates in a single round of communication Konečný et al. [2016].

The convergence of FedAvg for non-IID data is discussed in Li et al. [2020b]. However, as shown in Zhao et al. [2018], the accuracy loss of FedAvg can be up to $\sim 55\%$ in a non-IID setup for some datasets. How to effectively handle non-IID data distributions among different users remains an open FL question. First, FedProx Li et al. [2020a] is proposed as a generalized FedAvg for heterogeneous networks. SCAFFOLD is another extension of FedAvg that incorporates predictive variance reduction techniques Karimireddy et al. [2020]. In Luo et al. [2021], the authors propose a classifier calibration with virtual representation, leveraging a Gaussian Mixer Model to sample virtual features. As an alternative solution, the authors of  Yoshida et al. [2020] proposed raw data sharing in a cooperative learning mechanism. In  Li et al. [2020b], the authors proposed the use of some initial training data e.g., 5%, to handle non-IID data with traditional FL algorithms. Some other classic methods for non-IID data include GAN sharing  Li et al. [2022a, b], Cao et al. [2023], synthetic data sharing  Hao et al. [2021], Jeong et al. [2018] and global sub-dataset sharing Zhao et al. [2018].

Training GANs locally before sharing trained models with a centralized server is a popular approach for handling both IID and non-IID data distributions. In Zhang et al. [2019], the authors applied a conditional GAN (cGAN) and shared local classifier and generator with the central server which trains a global classifier and a generator to guide local users. Applying similar concept, the authors of Wu et al. [2021] proposed to incorporate model splitting to keep part of the cGAN (discriminator) and part of the hidden classifier by sharing the generator and a global classifier. In addition, as introduced in Li et al. [2022b], Cao et al. [2023], one can share the entire local GAN with the server and create a synthetic dataset to train a global GAN. Similarly, full GAN sharing has been discussed in  Li et al. [2022a]. Only shared generators with the maximum mean discrepancy are aggregated. To address privacy concerns, DP Dwork [2006] has been applied in GAN-based FL Xu et al. [2019], Li et al. [2022b], while a privacy budget is introduced to balance privacy and utility.

Proposition 1. Two generators $G_{u}$ and $G_{s}$ trained on Algorithm 1 with a shared discriminator $D_{u}$ converges to the same optimal discriminator $D_{u}^{*}$ as in Goodfellow et al. [2014] and it uniquely corresponds to the given $G_{u}$, i.e.,

Proof: In the proposed PS-FedGAN, Algorithm 1 trains $D_{u}$ using $G_{u}$ while $G_{s}$ has no influence on $D_{u}$. Therefore, as illustrated in Goodfellow et al. [2014], the discriminator training is valid with the value function $V(G_{u},D_{u})$ denoted by

Using the Radon-Nikodym theorem, we have the following conclusions:

Then, the value function can be recalculated as

Let $g(x)=a\log(x)+b\log(1-x)$. The derivatives for stationary points can be calculated as

and

which give the unique maximizer $x=\frac{a}{a+b}$ with $\frac{\partial^{2}g(x)}{\partial x^{2}}<0$ for $a,b\in(0,1)$.

Hence, the optimal $D_{u}^{*}$ can be calculated by

Here, the unique maximizer implies the uniqueness of $D_{u}^{*}$ for a given $G_{u}$.

Proposition 2. Two generators $G_{u}$ and $G_{s}$ trained according to Algorithm 1 with a shared $D_{u}$ would converge to a unique $G^{*}=G_{u}^{*}=G_{s}^{*}$ which captures $p_{data}$.

Proof: Following Proposition 1, Algorithm 1 converges to $D_{u}^{*}$. Therefore, in the training of the generator, we have the following optimization formulations, i.e.,

and

According to Algorithm 1, after each epoch of training in $G_{u}$ and $G_{s}$, we have $G_{u}^{{}^{\prime}}=G_{s}^{{}^{\prime}}$. This is because the input to both networks and initial weights are the same, leading to the same loss with the same gradients to be updated in the backpropagation. Therefore, the training of $G_{s}$ can be viewed as the traditional GAN training with $G_{s}$ and $D_{u}$. Since $z$ is shared in the communication, we have

Moreover, if $G_{s}$ captures a distribution $p_{gs}$, from Eq. (12), we have

which could be further calculated via

The uniqueness of $D_{u}^{*}$ in Eq. (6) and Eq. (14) leads to $p_{gu}=p_{gs}$. Thus, $G_{s}$ captures the same distribution as $G_{u}$. Hence, we can train two generators to capture the same distribution by only sharing a discriminator, without sharing original data explicitly. If $G_{u}$ achieves $G_{u}^{*}$, we have $G_{u}^{*}=G_{s}^{*}$. Then the optimization problems in Eq. (10) and Eq. (11) reduce to

In order to prove that $G_{s}^{*}$ captures $p_{data}$, we refer to Goodfellow et al. [2014]. For $G^{*}$, from the viewpoint of game theory, $D_{u}$ fails to distinguish between true and fake data. Then, we have

which leads to $p_{gu}=p_{data}$. Now, we have $p_{gs}=p_{data}$. Thus, the server generator also captures the data distribution of the corresponding local user. Hence, $G^{*}$ captures $p_{data}$. The uniqueness of $G^{*}$ shall follow the same conclusion from Goodfellow et al. [2014]. To prove this, we first assume that $p_{gu}=p_{data}$. Then the value in Eq. (14) can be calculated as

This provides us with the global minimum. On the other hand, for any $G$ and $D_{u}^{*}$, Let $M(G)=\max_{D}V(G,D)$. Then, we have,

After some manipulations, $M(G)$ can be calculated as

where JSD is the Jenson-Shannon Divergence which is non-negative. This yields to $-\log(4)$ which is the global minimum. Finally, it gives us $p_{data}=p_{gu}$, and the uniqueness of $G^{*}$ is proved.

Proposition 3. Any other generator $G_{\mathcal{A}}$ failing to capture the weights and architectures of $G_{s}$ or $G_{u}$, either in the initial state or in any single communication round, would fail to characterize the data distribution $p_{data}$.

Proof: From Proposition 1 and Proposition 2, we have $p_{gu}=p_{data}$ and $p_{gs}=p_{data}$ which are made possible only by a unique $G^{*}$ and $D_{u}^{*}$. As shown before, to obtain $G^{*}=G_{u}^{*}=G_{s}^{*}$, we need to have $G_{u}^{{}^{\prime}}=G_{s}^{{}^{\prime}}$ at each step, implying that $G$ needs to capture all communication rounds. Therefore, these conditions restrict $G$ to capture $p_{data}$.

In this section, we provide more visual results for the robustness of the proposed methods against the attacks of $\mathcal{A}_{1}$. Figure 7 illustrates the cloud-generated images compared to $\mathcal{A}_{1}$ generated images for the SVHN dataset for three users at some intermediate step (epoch 20) while Figure 8 show the regenerated samples in the FMNIST dataset. Shown as the visualized results, $\mathcal{A}_{1}$ fails to capture the underlying data statistics as well as to regenerate the meaningful synthetic data samples.