Fed-TGAN: Federated Learning Framework for Synthesizing Tabular Data

Our privacy-preserving model initialization comprises two steps as shown in Fig. 3a and 3b.

Step 1. Each of the $P$ clients extracts the statistical properties of the local data and sends them to the federator. The information sent is different based on the column type. For any categorical column $j$, each client $i$ computes and sends in the category frequency distribution $X_{ij}$. This information is used in three ways. First, the federator uses all distinct categories to build the label encoder $LE_{j}$ for column $j$. A label encoder is a table which maps all possible distinct values of a categorical column into their corresponding rank in one-hot encoding. Second, the frequency information is used to build an aggregated global frequency distribution $X_{j}$ for column $j$. Third, the sum of the frequency values is used to compute the number of table rows: i) available locally $N_{{i}}$ at each client $i$; ii) available globally $N$ across all clients. The global label frequency distribution $X_{j}$, $N_{{i}}$ and $N$ are needed to estimate the similarity of clients’ local data for computing the clients’ weights for model aggregation. If no categorical columns are present in the tabular data, the client sends out $N_{{i}}$ instead.

For any continuous column $j$, each client $i$ fits and sends in the parameters of a VGM model $VGM_{{i}j}$. To estimate the global distribution of column $j$ the federator uses $VGM_{{1}j}$, $VGM_{{2}j}$, $\dots$, $VGM_{{P}j}$ to create the data sets $D_{{1}j}$, $D_{{2}j}$, $\dots$, $D_{{P}j}$ with $N_{{1}}$, $N_{{2}}$, $\dots$,$N_{{P}}$ data points. The federator then uses these data sets to fit a new global VGM model $VGM_{j}$ for column $j$²²2It might be possible to fit the global model directly from the parameters of the local models by, e.g., adapting (DBLP:conf/icpr/BruneauGP08, ). This is left for future work.. $VGM_{j}$ is used as the final encoder for column $j$.

Step 2. The federator distributes all the column encoders $LE_{j}$ and $VGM_{j}$ to each client. Clients use this information to encode the local data and initialize the local models. Models initialized by using the same encoders will have the same input/output layers. This solves the first challenge outlined in Sec. 2. Note that the number and structure of the internal layers used for the generator and discriminator networks are predefined and independent of the data. In our evaluation against the MD structure this information is also used by the server to initialize the hosted generator network. Note that in this process the federator never directly accesses the local data of the clients, only their statistical distribution, thus this addresses the second challenge from Sec. 2.

After model initialization, federator uses the collected global data statistics to pre-compute the weights for each client. These weights are used in training during the model aggregation (shown in Fig. 3c) to smooth convergence in the presence of skewed data across the clients. The weights calculation process is presented in Fig. 4.

Step 0 is to build a $P$ $\times$$Q$ divergence matrix $\bm{S}$ where $P$ is the number of clients and $Q$ is the number of columns. Each matrix element $S_{ij}$ is the divergence of client $i$ for column $j$ when compared to the global statistics of column $j$. The metric used depends on the type of column.

• •

  Categorical columns use the Jensen-Shannon Divergence (JSD) (jsd, ). The JSD between two probability vectors $p$ and $q$ is defined mathematically as $\sqrt{\frac{D(p||m)+D(q||m)}{2}}$ where $m$ is the point-wise mean of $p$ and $q$, and $D$ is the Kullback-Leibler divergence (Joyce2011, ). The JSD distance metric is symmetric and bounded between 0 and 1 enabling a hassle-free interpretation of results. For each categorical column $j$ and client $i$ we compute $S_{ij}$ as JSD between $X_{ij}$ and $X_{j}$, i.e., $S_{ij}=\text{JSD}($$X_{ij}$, $X_{j}$ $)$.

• •

  Continuous columns use the Wasserstein Distance (WD) (wgan_test, ). The first Wasserstein distance between two distributions $u$ and $v$ is defined as $WD(u,v)=inf_{\pi\in\Gamma(u,v)}\int_{\mathbb{R}\times\mathbb{R}}|x-y|d\pi(x,y)$ where $\Gamma(u,v)$ is the set of probability distributions on $\mathbb{R}\times\mathbb{R}$ whose marginals are $u$ and $v$ on the first and second factors, respectively. It can be interpreted as the minimum cost to transform one distribution into another where the cost is given by amount of distribution to shift times the distance it must be shifted. For each continuous column $j$, we use the data sets $D_{{i}j}$ created previously for each client $i$ to compute $S_{ij}$ as the WD between $VGM_{{i}j}$ and $VGM_{j}$.

Step 1 normalizes the matrix $\bm{S}$ across the $P$ clients for each table column $j$. This is done by dividing each matrix element by the sum of the elements in the corresponding matrix column. This step maintains the relative divergence between different clients with respect to the global column data distribution while allowing to give the same importance to all columns (all columns sum up to 1).

Step 2 aggregates the divergence across the different table columns $j$. This is done via a sum along the rows of the matrix. For each client $i$ the resulting score $SS_{i}$ can already represent the divergence between client and global data distribution, but it does not yet take into account possible difference in the amount of local data available at each client.

Step 3 fuses the divergence in data values and data quantity at each client. Step 3 first normalizes the divergence metric between 0 and 1 across the clients. Then it uses the complement to represent similarity instead of divergence and combines it with the ratio of local data available with respect to the global data, i.e., $\frac{N_{i}}{N_{all}}$. The resulting $SD_{i}$ take into account differences in both number of values and distribution of values of the local vs. global data. It takes into account all different dimensions given by the different columns addressing the third challenge from Sec. 2.

Step 4 computes the final weights $W_{i}$. The $W_{i}$ for each client $i$ are obtained by passing the $SD_{i}$ to a softmax function. $W_{i}$ is the weight that the federator will use when it aggregates the model from client $i$.

Fed-TGAN is implemented using the Pytorch RPC framework.This choice makes it easy to control the flow of the training steps from the federator. Clients just need to join the group, then wait to be initialized and assigned work. To parallelize the training across all clients, RPC provides a function rpc_async() which allows the federator to make nonblocking RPC calls to run functions at a client. To implement synchronization points, RPC provides a blocking function wait() for the return from previously called function rpc_async(). The return of rpc_async() is future type object. Once the wait() is called on this object, the process is blocked until the return values are received from the client. The federator starts the training on all clients via rpc_async(). Then it waits for the new model from each client via the wait(). Once all models are received, they are aggregated into a single model using the clients weight and redistributed to the clients via rpc_async(). Once all clients confirm the reception of the updated model (via wait()) the federator starts the next round of training. We plan to provide the source code via Github after publication of the paper.

One weakness of current RPC framework from Pytorch v1.8.1 is that it does not support the transmission of tensors directly on GPU through RPC call. This means that each time when we collect or update the model weights we need to pay an extra time cost to detach the weights from GPU to CPU or reload the weights from CPU to GPU.

Datasets. We test our algorithm on four commonly used machine learning datasets. Adult, Covertype and Intrusion – are from the UCI machine learning repository (UCIdataset, ), and Credit is from Kaggle (kagglecredit, ). Due to our computational limitation, we randomly sample 40k data from each of above datasets. The details of each dataset are shown in Tab. 1.

Baselines. We compare Fed-TGAN against 3 baselines: (i) multi-discriminator structure, (ii) vanilla federated learning structure and (iii) centralized approach abbreviated as MD-TGAN, vanilla FL-TGAN, and Centralized, respectively. The aim is to learn a CTGAN model from distributed clients using the three frameworks on the basis of CTGAN’s default settings for encoding features (ctgan, ). Specifically, we use 10 as the limiting number for estimated modes for the VGM encoders for each continuous column and use one-hot-encoding for categorical columns. We re-implement all baselines using the Pytorch v1.8.1 RPC framework.

MD-TGAN clients swap discriminator models with each other at the end of each training epoch (mdgan, ). For a fair comparison with MD-TGAN, we force also Fed-TGAN and vanilla FL-TGAN to share the model weights with the federator at the end of each training epoch. Due to this, the notion per round commonly referred in FL studies equals to per epoch in this paper, if not otherwise stated. Vanilla FL-TGAN is identical to Fed-TGAN, except it uses identical weights for all clients equal to $\frac{1}{P}$ where $P$ is the number of clients. Due to the different learning speed per epoch of the four frameworks, for a fair comparison we fix the number of epochs so that the training time is similar. In particular we use to 500, 500, 500, and 150 epochs for Fed-TGAN, Vanilla FL-TGAN, centralised, and MD-TGAN, respectively. We repeat each experiment 3 times and report the average.

Testbed. Experiments are run under Ubuntu 20.04 on two machines. Each machine is equipped with 32 GB memory, GeForce RTX 2080 Ti GPU and 10-core Intel i9 CPU. Each CPU core has two threads, hence each machine contains 20 logical CPU cores in total. The machine are interconnected via 1G Ethernet links (measured speed: 943Mb/s). One machine hosts the federator, the other all the clients. When not otherwise stated both federator and clients use the GPU for training. For experiments in Sec. 5.4, when CPU is used to host clients for Fed-TGAN and MD-TGAN, CPU affinity (by taskset command in Linux) is used to bind each client to one logical CPU core to reduce interference between different processes.

To evaluate the performance of the generator, a 40k synthetic data is sampled from the trained generator at the end of each epoch. We use two metrics to quantitatively measure the statistical similarity between the real and synthetic data:

Average Jensen-Shannon divergence (Avg-JSD). Used for categorical columns (see Sec. 4 for its definition). First, we compute the JSD between the synthetic and real data for each categorical column. Second, we average the obtained JSDs to obtain a compact comprehensible score, abbreviated as Avg-JSD. The closer to 0 Avg-JSD is, the more realistic the synthetic data is.

Average Wasserstein distance (Avg-WD). Used for continuous columns³³3We use WD over JSD for continuous columns since JSD is not well-defined when the synthetic values lie outside of the original value range from the real dataset, i.e., the KL divergence is not defined when comparing the similarity of probability distributions with non-overlapping support. (see Sec. 4 for its definition). Unlike JSD, WD is unbounded and can vary greatly depending on the scale of the data. To make the WD scores comparable across columns, before computing the WD we fit and apply a min-max normalizer to each continuous column in the real data and apply the same normalizer to the corresponding columns in the synthetic data. We average all column WD scores to obtain the final score abbreviated as Avg-WD. The closer to 0 Avg-WD is, the more realistic the synthetic data is.

We first evaluate how realistic the generated synthetic data is. Sec. 5.3.1 designs an experiment where all the clients contain the whole dataset, this is to test the performance of each framework under the ideal case. Then in Sec. 5.3.2, we implement a scenario where data on clients are IID, but quantities of data are highly imbalanced across all the clients. The objective of this experiment is to show the effect of our model aggregation weighting method. In the end in Sec. 5.3.3, an ablation analysis is designed where one of the clients has much higher amount of data, but the data quality is low. We want to show the efficacy of our table-similarity aware weighting method in the calculation of model aggregation weights.

Above experiments all focus on the quality of generation. In this section, we study the training efficiency of MD-TGAN and Fed-TGAN. The first experiment scenario is the same as we discussed in Sec. 5.3.1. The entire FL system consists of 5 clients, and each of them possesses the full set original dataset. Fig. 8a shows the time distribution for MD-TGAN and Fed-TGAN during one training epoch on the Intrusion dataset. The Calculation on C is estimated at server or federator. It means when server or federator calls the training on clients, the time is calculated from the start of first client to the finishing of all clients. For MD-TGAN, clients all need to wait for synthesized data from generator. Its Calculation on C extracts this part of time, and adds it to Communication. Communication counts the time for exchanging model weights, swapping discriminator between clients, or sharing training data between server (or federator) and clients.

We can see that for Fed-TGAN, the calculation time on federator is negligible since it is only averaging model weights. Fed-TGAN has a slightly higher calculation time on clients because it trains both generator and discriminator networks on clients. Moreover, the communication time of MD-TGAN is much higher. Because for updating generator or discriminator in MD-TGAN, the generator needs to send the generated data from generator to each discriminator. Since MD-TGAN only has one server, the above tasks can not be distributed. Fig. 8a shows that Fed-TGAN saves more than 200% of the time taken by MD-TGAN per epoch. The communication time of Fed-TGAN is only 30% of what MD-TGAN uses.

In the second experiment, instead of aggregating models at the end of each epoch for every client, we vary the number of local training epochs before aggregating models for Fed-TGAN. Fig. 8b shows the total training time with variation on local epochs per round for Fed-TGAN, where we fix all the clients to train for 500 epochs in total. Therefore with more local epochs per round, we are left with less rounds for the whole training process. For local epochs 1, 10, 25 and 50, the corresponding round numbers are 500, 50, 20 and 10. The massive time decrease between local epoch 1 and others is simply because of the reduction of model aggregations. The differences among other numbers of local epochs are not that significant. Fig. 9 shows the generation results under different local epochs. We see for categorical columns, the Avg-JSD converges for all to a small value. For continuous columns, the Avg-WD for Fed-TGAN with 10 local epochs per round converges fastest and provides the best result until 1150s. This result indicates that it is possible to speed up training of Fed-TGAN by utilizing more local epochs while still preserving the statistical similarity between real and synthetic datasets. However, increasing the local epochs to a large value can potentially lead to over-fitting on the local data of clients ultimately deteriorating performance. Thus, the local epoch number introduces a trade-off between efficiency and performance.

Next, we evaluate time consumption of one epoch for Fed-TGAN and MD-TGAN with varying factors. First, we fix the number and type of data (10K IID data from Intrusion dataset) on each client, and vary the number of clients from 5 to 20. For computing resource limitations, all the experiments with varying number of clients are implemented using CPUs on client side with the server (or federator) side using the GPU. To limit interference between different processes, CPU affinity is used to bind each client to one logical CPU core. Fig. 10a clearly shows that Fed-TGAN scales better than MD-TGAN with number of clients. In MD-TGAN the central server becomes increasingly the bottleneck when adding clients due to large amount of data exchanged with each client. Second, we fix the number of clients to 5 and vary the number of IID sampled data from the Intrusion dataset. We vary the number of rows from 10k to 40k. The experiments are implemented on both CPU and GPU for clients. The result in Fig. 10b shows that with increasing the number of data rows on each client, Fed-TGAN and MD-TGAN both experience an increase in the training time per epoch. The difference between the two algorithms is small when training happens on CPU. But when using GPU, we have an increasing difference with increasing amount of data on clients. The reason is because when sharing data between the client and the server, tensors that are on GPU must first be detached from GPU to CPU for being sent through the Pytorch RPC framework. Since Fed-TGAN trains all tabular GAN models locally on each client, the training process is highly accelerated by GPUs. As clients in the federated setting only need to detach the model from the GPU to CPU at the end of training to share them with the when exchanging messages. And so, since the server shares message more times in MD-TGAN than the federator in Fed-TGAN, the GPUs do not accelerate the training process of MD-TGAN as much as for Fed-TGAN.

For calculating the weights for merging local client models during the initialization process, we only use the individual column data distributions to compare local data distributions with the global distributions. But for tabular data, inter-dependency between columns is also an important factor.

The reason that we do not use it is due to privacy. Since the server (or federator) cannot collect real data from clients, inter-dependencies between columns cannot be inferred only according to distributions of each column.

But analysis of column inter-dependency for client’s data is not useless. Recall the experiment in Sec. 5.3.3. One malicious client contains 40k data, which is only one data repeated 40k times. Other 4 clients each contains 10k IID data sampled from original data. For a dataset of one row repeated 40k times, the correlation between every two columns is 0, since they are just two constant columns. Therefore, the analysis of self-reported column inter-dependency may not improve the similarity calculation under the current privacy-preserving rule, but it may still help to identify some types of malicious clients.

Further note on Privacy-Preserving Technologies. FL indeed emerges as a viable solution that enables a collaborative distributed learning without disclosing original data. Orthogonal to FL, there is an array of privacy-preserving techniques that can be jointly applied to further strengthen privacy guarantee of GANs and FL, namely differential privacy (DP) (dwork2006our, ; DPGAN, ; pategan, ) and homomorphic encryption (HE) (gentry2010computing, ; crossSilos, ). Exploring advanced privacy enhancing technologies is beyond the scope of this paper and will be addressed in our future work.