Federated clustering with GAN-based data synthesis

In traditional centralized learning, all data is typically stored on a central server. For this learning protocol, one problem is hardware constraints that it places a high demand on the storage and computing capacity of the central server when the dataset is large. And another problem is privacy risks and data leakage Zhu et al. (2021). To handle the first problem, distributed data parallelism Dean et al. (2012) uses multiple devices to train model replicas in parallel using different data subsets. Nevertheless, this method still necessitates access to the entire dataset for division into subsets, thus retaining inherent privacy risks.

Federated learning McMahan et al. (2017), a recent research focal point, offers a promising solution by enabling the collaborative training of a global model through the fusion of local models trained on client devices, all while safeguarding user privacy. Despite its potential to alleviate both hardware and privacy concerns, federated learning presents several challenges, which are outlined as follows Li et al. (2020):

• •

  Expensive Communication. In practice, federated networks may encompass millions of client devices, rendering network communication substantially more costly than local computation Huang et al. (2013); Van Berkel (2009). Crafting communication-efficient models involves minimizing both communication rounds and message volumes.

• •

  Systems Heterogeneity. Client devices often possess diverse communication, computation, and storage capabilities due to variations in wireless network connectivity, energy resources, and hardware limitations. Consequently, only a fraction of devices may effectively participate in training a large model Bonawitz et al. (2019), and device disconnections during training must be anticipated. Thus, a robust federated model should accommodate low device participation and device failures.

• •

  Data Heterogeneity. In federated learning, local data distributions across client devices can differ substantially, influenced by how each client utilizes their local device. This phenomenon, known as non-IID data or data heterogeneity, can impede convergence and degrade model performance Zhu et al. (2021). A natural solution to this challenge involves moving beyond the traditional single-center framework, where only one global model is trained. Instead, constructing a multi-center framework, which simultaneously trains multiple global models based on client clustering Ghosh et al. (2020); Zec et al. (2022) or data clustering Dennis et al. (2021); Stallmann and Wilbik (2022); Chung et al. (2022), may offer a more viable approach.

• •

  Privacy Concerns. Although the classic federated averaging method McMahan et al. (2017) helps mitigate shallow data leakage from raw data, Zhu et al. Zhu et al. (2019) demonstrated its potential to lead to deep data leakage through shared gradients and parameters. This deep data leakage allows the reconstruction of original data from these gradients and parameters. To enhance privacy in federated learning, various works have employed differential privacy Dwork et al. (2006); Dwork (2011); Dwork et al. (2014), a popular privacy protection technique Geyer et al. (2017); Zhao et al. (2020); Wei et al. (2020). However, while improving privacy, differential privacy can significantly degrade model performance.

Numerous centralized algorithms have been adapted for federated settings in recent years, encompassing reinforcement learning Wang et al. (2019, 2020); Zhang et al. (2021), classification algorithms Brisimi et al. (2018); Yu et al. (2020); Chen and Chao (2021), clustering algorithms Dennis et al. (2021); Stallmann and Wilbik (2022); Chung et al. (2022), and more. Nevertheless, these extensions are often discouraged by the challenges of federated learning in terms of model efficiency and model performance. This work specifically addresses the challenges associated with federated clustering.

As shown in Fig. 1, the local similarity is insufficient to group local data correctly, while a global perspective excels in this regard. However, the global real dataset cannot be obtained since the local data on client devices are forbidden to share. Hence, the key here is how to measure the global similarity without sharing private data.

To tackle this challenge, k-FED Dennis et al. (2021) and federated fuzzy c-means (FFCM) Stallmann and Wilbik (2022) have extended two classic centralized clustering algorithms, K-means (KM) MacQueen (1967) and fuzzy c-means (FCM) Bezdek et al. (1984), to the federated setting, respectively. Specifically, each client device runs a classic centralized clustering algorithm on the local data to generate several cluster centroids and uploads them to the central server. Then, the central server can construct $k$ global cluster centroids by running KM on the uploaded local cluster centroids. The classic centralized clustering algorithm used in k-FED is KM and that used in FFCM is FCM. However, the resulting global cluster centroids often exhibit fragility and sensitivity to varying non-IID levels, leading to suboptimal and non-robust model performance.

Given a federated dataset, the non-IID level quantifies the heterogeneity degree among local data distributions, and it remains independent of the global distribution. Hence, model performance can remain unaffected by different non-IID levels if a good approximation of the global real data can be constructed. Additionally, such an approximation may facilitate the more effective capture of global similarity characteristics without sharing private data. Motivated by this, we introduce a simple but effective federated clustering framework based on GAN-based data synthesis, named synthetic data aided federated clustering (SDA-FC). It requires only one communication round, can run asynchronously, and can handle device failures. It comprises two main steps: global synthetic data construction and cluster assignment. In the former, the central server endeavors to construct a global synthetic dataset using multiple local GANs Goodfellow et al. (2014), trained on local data. For the latter, the central server first applies KM or FCM to the global synthetic dataset, resulting in $k$ global cluster centroids. Then, the final clustering result can be obtained based on the proximity of local data points to these centroids.

The vanilla GAN Goodfellow et al. (2014) architecture comprises two networks: the generator and the discriminator. These networks engage in a two-player game wherein they continuously improve each other. The generator aims to produce synthetic samples that closely resemble real data, thereby deceiving the discriminator, whose role is to differentiate between real and generated samples. The game concludes when the discriminator can no longer distinguish between real and generated samples, indicating that the generator has learned the underlying real data distribution and reached the theoretical global optimum.

The objective function of the vanilla GAN is defined as:

where $G$ is the generator that inputs a noise $z$ and outputs a generated sample, $\mathcal{N}$ is Gaussian distribution, $D$ is the discriminator that inputs a sample and outputs a scalar to tell the generated samples from the real ones, and $p_{r}$ is the distribution of real data. While GAN have demonstrated remarkable success in various applications, adversarial training is known for its instability and susceptibility to mode collapses Metz et al. (2017), resulting in both high-quality and low-diversity generated samples. Mode collapses imply that the model captures only a portion of the real data’s characteristics. To address this, Mukherjee et al. Mukherjee et al. (2019) introduced an additional categorical variable into the generator’s input, promoting a clearer cluster structure in the latent space and enhancing sample diversity.

To mitigate mode collapses in our SDA-FC framework, we also use a mixture of discrete and continuous variables as the input of the generator by following Mukherjee et al. (2019). The new objective function is defined as:

where $\mathcal{U}$ is a uniform random distribution with the lowest value 1 and the highest value $k$, and $e_{u}$ is a one-hot vector with the $u$-th element being 1.

Given a real world dataset $X$, which is distributed among $m$ clients, i.e. $X=\bigcup_{i=1}^{m}X^{(i)}$. The goal of federated clustering is to cluster data based on a global similarity measure while keeping all data local Stallmann and Wilbik (2022). However, direct measurement of global similarity is infeasible without sharing private data. This is why federated clustering is a challenging task to handle.

To handle this, we propose the synthetic data aided federated clustering (SDA-FC) framework, which is designed to require only a single round of communication between clients and the central server. It consists of two main steps: global synthetic data construction and cluster assignment, which are detailed below.

Creating a universal non-IID benchmark dataset in the realm of federated learning remains a challenging endeavor due to the inherent complexity of federated learning itself Li et al. (2022); Hu et al. (2022). Here, following ref. Chung et al. (2022), we simulate different federated scenarios by partitioning the real-world dataset into $k$ smaller subsets (each one corresponds to a client) and scaling the non-IID level $p$ in clients, where $k$ is the number of true clusters. For a client with $s$ datapoints, the first $p\cdot s$ datapoints are sampled from a single cluster, and the remaining $(1-p)\cdot s$ ones are randomly sampled from any cluster. Extremely, $p=0$ means the local data across the clients is IID, and $p=1$ means the local data across the clients is completely non-IID.

As shown in Table LABEL:datasets, we select two grey image datasets MNIST and Fashion-MNIST, two color image datasets CIFAR-10 Krizhevsky et al. (2009) and STL-10 ¹¹1Note that, to reduce the computational cost of baseline methods and to use the same network structure for CIFAR-10, we performed a preprocessing step to resize the images in STL-10 to 32 $\times$ 32. Coates et al. (2011), and a time series dataset Pendigits Keller et al. (2012) for comprehensive analysis. In SDA-FC, all local GANs are trained with the Adam Optimizer Kingma and Ba (2014). More detailed hyperparameter settings can be found in the Appendix A. Our code will be made publicly available to facilitate reproducibility.

To evaluate the effectiveness of SDA-FC, we employ two state-of-the-art federated clustering methods as baselines: k-FED Dennis et al. (2021) and federated fuzzy c-means (FFCM) Stallmann and Wilbik (2022). Additionally, to contextualize the performance of federated clustering relative to centralized clustering, we report numerical results for K-means (KM) and fuzzy c-means (FCM) in centralized scenarios, denoted as KM_central and FCM_central, respectively. In all experiments, the default fuzzy degree in FCM-based methods is set to 1.1.

The clustering performance based on NMI Strehl and Ghosh (2002) and kappa Liu et al. (2019) is shown in Table 3 and Table 4. One can observe that: 1) For the KM-based methods, both metrics consistently indicate the superiority of our proposed method over k-FED in terms of both effectiveness and robustness. For the FCM-based methods, the two metrics give different ranks. Although NMI is the most commonly used metric, recent researchs highlight its limitations, rendering Kappa a more reliable alternative Liu et al. (2019); Yan et al. (2022). Here, we also contribute some new cases to show that NMI fails to accurately reflect the performance of clustering results. As shown in Fig. 2, SDA-FC provides clustering results more aligned with the ground truth, yet NMI ranks it lower (0.6806 for FFCM and 0.6654 for SDA-FC), which is perplexing. In contrast, Kappa produces a more reasonable ranking (0.6523 for FFCM and 0.6611 for SDA-FC). Moreover, on the CIFAR-10 dataset, we establish an oracle baseline with ground-truth centroids (computed using ground-truth labels) in a centralized scenario, which assigns clusters based on cosine distance from local data to the centroids. It is intuitive that KM-based or FCM-based methods cannot surpass the oracle baseline. However, NMI ranks them inversely (0.1009 for the oracle baseline, 0.1051 for FFCM with a fuzzy degree of 2 when p = 0.75, and 0.1059 for FFCM with a fuzzy degree of 2 when p = 1), while Kappa provides more reasonable values (0.1992 for the oracle baseline, 0.1634 for FFCM with a fuzzy degree of 2 when p = 0.75, and 0.1854 for FFCM with a fuzzy degree of 2 when p = 1). Hence, for FCM-based methods, our proposed approach excels in terms of both effectiveness and robustness. 2) There exists a discernable gap between federated clustering and centralized clustering, but the SDA-FC framework mitigates this gap.

While we have demonstrated the effectiveness of the SDA-FC framework in conjunction with KM or FCM, we expect that SDA-FC can serve as a versatile framework for addressing more intricate challenges in federated clustering, including its integration with deep clustering methods. The crux lies in ensuring that the global synthetic dataset generated by SDA-FC is a good approximation of the real one.

For each image dataset, we generate a global synthetic dataset of the same size as the real dataset. We showcase part of the generated images randomly in Fig. 4 and visualize the global data distribution using t-SNE Van der Maaten and Hinton (2008) in Fig. 5. One can see that: 1) Despite instances where objects in generated images are unrecognizable, SDA-FC effectively captures the fundamental data characteristics. 2) The global synthetic datasets and the global real ones exhibit substantial overlap, affirming that the former serves as a good approximation of the latter. 3) The cluster structure is more evident in grayscale image datasets compared to color image datasets. This elucidates the superior clustering performance observed in grayscale datasets, as outlined in Table 4.

In practice, certain client devices may experience disconnections from the server due to factors such as wireless network fluctuations or energy constraints. Consequently, specific data characteristics from the disconnected devices may be lost, potentially affecting clustering performance. This means that the sensitivity analysis of clustering performance to device failures is worth investigating.

To handle such problem, following Li et al. (2020), we simply ignore the failed devices and continue training with the remaining ones. The rate of disconnection among all devices is denoted as the disconnection rate. Our experiments simulate various disconnected scenarios on MNIST by scaling the disconnection rate and run four federated clustering methods on them. As shown in Fig. 3, one can see that: 1) The proposed methods exhibit superior effectiveness and robustness in all considered scenarios. 2) The sensitivity of clustering performance to device failures is positively correlated with the non-IID level $p$. Larger $p$ values result in reduced mutual substitutability of data characteristics across clients (one extreme case is that the data across the clients is completely non-IID when $p=1$), exacerbating the adverse impact of device failures on clustering performance.

In summary, our findings reveal that: 1) The proposed methods outperform baselines in terms of both effectiveness and robustness. 2) Kappa emerges as a more reliable metric compared to NMI. 3) The global synthetic dataset generated by local GANs serves as a commendable approximation of the global real dataset. 4) Our proposed method demonstrates heightened resilience to device failures. 5) The sensitivity of clustering performance to device failures intensifies with higher heterogeneity levels, as characterized by $p$.