Differentially Private Synthetic Medical Data Generation using Convolutional GANs

One of the most promising privacy-preserving approaches is Synthetic Data Generation (SDG). Synthetically generated data can be shared publicly without privacy concerns and provides many collaborative research opportunities, including for tasks such as building prediction models and finding patterns. As SDG inherently involves a generative process, Generative Adversarial Networks (GANs) [4] attracted much attention in this research area, due to their recent success in other domains.

GANs are not reversible, i.e., one may not use a deterministic function to go from the generated samples to the real samples, as the generated samples are created using an implicit distribution of the real data. However, a naive usage of GANs for SDG does not guarantee the system being privacy-preserving by just relying on the fact that GANs are not reversible, as GANs are already proven to be vulnerable [5].

This problem becomes much more severe when such privacy violations can have serious consequences, such as when dealing with patient sensitive data in the medical domain. The primary ambiguity here is to understand how a system can claim to preserve the original data’s privacy. More precisely, how private is a system? How much information is leaked during the training process? Thus, there is a need to measure the privacy of a system – to be able to judge if a system is privacy-preserving or not.

Differential Privacy (DP) [6] provides a mechanism to ensure and quantify the privacy of a system using a solid mathematical formulation. Differential privacy recently became the de facto standard for statistical exploration of databases that contain sensitive private data. The power of differential privacy is in its accurate mathematical representation, that ensures privacy, without restricting the model for accurate statistical reasoning. Furthermore, utilizing differential privacy, one can measure the privacy level of a system.

Differential privacy, as a strong notion of privacy, becomes crucial in machine learning and deep learning as the system by itself often employs sensitive information to augment the model performance for prediction. Although many different attack types can jeopardize a machine learning model’s privacy, it is the safest to assume the presence of a powerful adversary with complete knowledge of the pipeline, training, and model [7]. Hence protecting the system against such an adversary, or at the very least, measuring the upper bound of the privacy leakage in this scenario, gives us a full view of the system’s privacy. A fully differentially private system guarantees that the algorithm’s training is not dependent on an individual’s sensitive data.

One of the most adopted approaches to ensure differential privacy, under a modest privacy budget, and model accuracy, is Differentially Private Stochastic Gradient Descent (DP-SGD), proposed in [8], which is the basis for many different research works [9, 10]. At its core, DP-SGD (1) bounds the sensitivity of the algorithm on individuals by clipping the gradients, (2) adds Gaussian noise, and (3) performs the gradient descent optimization step. As of now, SGD under a DP regime is one of the most promising privacy-preserving approaches.

Due to the rapid rise in the utilization of differential privacy, practical issues related to tracking the privacy budget became a subject of many discussions. Despite its great advantages, the proposed notion of differential privacy has some disadvantages due to what is called the composition theorem [6, 11]. The composition theorem indicates that, for some composed mechanisms, the privacy cost simply adds up, leading to the discussion of privacy budget restriction. This is problematic, especially in deep learning, as the model training is an iterative process, and each iteration adds to the privacy loss. To tackle the shortcomings of the differential privacy definition, Rényi Differential Privacy (RDP) has been proposed as a natural relaxation of differential privacy [12]. As opposed to differential privacy, RDP is a more robust notion of privacy that may lead to a more accurate and numerically stable computation of privacy loss.

There are four primary challenges in the synthetic data generation research in the medical domain: (1) Preserving the privacy. The majority of the existing works do not train the model in a privacy-preserving manner. At best, they only try to address privacy with some statistical or machine learning-based measurements. (2) Handling discrete data. When discrete data is involved, many of the methods using GANs face some difficulty since GAN models are inherently designed for generating continuous values. In particular, they especially fail when there is a mixture of continuous and discrete data. (3) Evaluation of synthetic data quality. One of the challenging issues regarding the evaluation of GANs and the quality of synthetic data in a real-world domain is: How can we measure the quality of a synthetically generated data? However, this becomes particularly more critical in the medical domain since the use of low-quality synthetic data for building prediction models can have dire consequences and may even jeopardize human lives. (4) Temporal information and correlated features. The temporal and local correlation between the features is often ignored. Incorporating such information is important in the medical domain since the disease incidents, or individual patients’ records, often have meaningful temporal/correlation patterns. The quality of the generated data can be significantly improved by incorporating such dependencies in the data.

To address the problems described before, here we propose a privacy-preserving framework that employs Rényi Differential Privacy and Convolutional Generative Adversarial Networks (RDP-CGAN). Our work makes the following contributions:

• •

  We track and compute privacy loss with stable numerical computations and have tighter bounds on the privacy loss function, which leads to better performance under the sample privacy budget. We achieve this by employing Rényi differential privacy in our model.

• •

  Our model can effectively handle discrete data and capture the information from a mixture of discrete-continuous data by creating a compact, unified representation through Convolutional Autoencoders (CEs) for unsupervised feature learning.

• •

  In addition to measuring the quality of synthetic data with statistical measurements, we also employ a labeled synthetic data generation mechanism for assessing the quality of the synthetic data in a supervised manner.

• •

  To incorporate temporal and correlation dependencies in the data, we utilize one-dimensional convolutional neural networks.

We demonstrate that our model generates higher quality synthetic data under the same privacy budget. Likewise, it can provide a higher level of privacy under the same level of synthetic data quality. We also empirically show that, regardless of privacy considerations, our proposed architecture generates higher quality synthetic data quality as it captures temporal and correlated information.

The rest of this part is organized as follows: Section II investigate some prior research efforts related to synthetic medical data generation and differentiates our work from other existing works. Section III describes preliminary concepts required to comprehend our proposed work. Section IV provides the structural and implementation details of the proposed system. Section III demonstrates the comparison results with state-of-the-arts by privacy analysis and the quality of generated synthetic data. Finally, Section VI concludes our findings in this paper.

Autoencoder are a kind of neural network architectures that consists of two encode and decode functions. $\boldsymbol{Enc(.)}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{d}$ and $\boldsymbol{Dec(.)}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{n}$ aim to transpose the input $\boldsymbol{x}\in\mathbb{R}^{n}$ to the latent space $\mathcal{I}\in\mathbb{R}^{d}$ and then reconstruct $\boldsymbol{\hat{x}}\in\mathbb{R}^{n}$. In an ideal scenario, a perfect reconstruction of the original input data can be achieved, i.e., $x=\hat{x}$. The autoencoders usually employ Binary Cross Entropy (BCE) and Mean Square Error (MSE) losses for binary and continuous inputs, respectively.

We utilize autoencoders in our model for capturing low-dimensional representations for both discrete and continuous variables.

Differential Privacy establishes a guarantee of individuals’ privacy via measuring the privacy loss (associated with any information release extracted from a database) by a mathematical definition [11, 6]. The most widely adopted definition is the $(\epsilon,\delta)$-differential privacy.

Two datasets $D$, ${D}^{\prime}$ that only differ by one record (e.g., a patients’ record) are called neighbor datasets. The notion of neighboring datasets emphasizes the sensitivity of any individual private data. The parameters $(\epsilon,\delta)$ denote the privacy budget, i.e., being differentially private does not mean we have absolute privacy. It only indicates our confidence level of privacy, given the $(\epsilon,\delta)$ parameters. The smaller the $(\epsilon,\delta)$ parameters are, the more confident we become about our algorithm’s privacy as $(\epsilon,\delta)$ indicates the privacy loss by definition.

The $(\epsilon,\delta)$-DP with $\delta=0$ is called $\epsilon$-DP which is the initially proposed definition [11]. It provides a more substantial promise of privacy as even a tiny amount of $\delta$ might be a dangerous privacy violation due to the distribution shift [11]. One of the most important reasons in the usage of $(\epsilon,\delta)$-DP is the applicability of advanced composition theorems [6, 29].

The strong composition (see Definition 1) enables tighter upper bound for compositions of $(\epsilon,\delta)$-DP steps compared to the basic composition theorem [6]. One particular problem with the advanced composition theorem is that iterating this process leads to the rapid growth of parameters as each employment of the advanced composition theorem (each step of the iterative process) lead to a various selection of possible $(\epsilon(\delta),\delta)$ values. To address some of the shortcomings of the $(\epsilon,\delta)$-DP definition, the Rényi Differential Privacy (RDP) has been proposed in [12] based on Rényi divergence (Eq. 2).

The Rényi divergence, as a generalization of the Kullback–Leibler divergence, is precisely equal to the Kullback–Leibler divergence for the order $\alpha=1$. The special case of $\alpha=\infty$ equals to:

which is the log of the maximum ratio of the probabilities over $P^{\prime}(x)$. The order of $\alpha=\infty$ establishes the connection between the Rényi divergence and the $\epsilon$-DP. A randomized mechanism $\mathcal{A}$ is $\epsilon$-differentially private if for two neighbor datasets $D$ and ${D}^{\prime}$, we have:

Now, based on these definitions, let us define Rényi differential privacy (RDP) [12] as a new notion of differential privacy.

There are two essential properties of the RDP definition (Definition 3) that are required here.

The above two propositions form the basis for preserving privacy in our approach. Proposition 1 determines the calculation of the privacy cost in our framework as a composition of two structures (autoencoder and convolutional GAN) and Proposition 2 is applicable when one wants to calculate the extent to which our system is differentially private based on the traditional $(\epsilon,\delta)$ notion (see Definition 1).

We build a one-dimensional convolutional autoencoder (1D-CAE) based architecture. The main goal of such a 1D-CAE is to: (1) capture the correlation between neighboring features, (2) represent a new compact feature space, (3) transform the possible discrete records to a new continuous space, and (4) simultaneously model discrete and continuous phenomena, which is a challenging problem [22].

We enforce privacy by adding noise, and by clipping gradients of both the encoder and decoder (see Algorithm 1, Autoencoder Pretraining Step), as in some previous research works [10, 33]. One might claim adding noise to the decoder part should suffice as the discriminator only has access to the decoder part of the CAE. We assert that such an action might jeopardize the privacy of the model since we cannot rely on the autoencoder training to be trusted and secure as it has access to the real data. The details of autoencoder pretraining, demonstrated in Algorithm 1 are as follows:

• •

  We pre-train the autoencoder for $n_{ae}$ steps (which will be determined based on the privacy budget $\epsilon$), and hence it varies by changing the desired level of $\epsilon$.

• •

  We divide a mini-batch into several micro-batches.

• •

  For each micro-batch of size 1, we calculate the loss (line 7), calculate the gradients (line 8), and clip the gradients to bind the model’s sensitivity to the individuals (line 9). The operation $\left\|\boldsymbol{g_{i}}()\right\|_{2}$ indicates the $\ell_{2}$ norm over the micro-batch gradients, and $C$ is an arbitrary upper bound for the gradients’ norm.

• •

  The Gaussian noise ($\mathcal{N}(0,\sigma_{ae}^{2}C^{2}\mathbb{I})$)will then be independently added to the gradients of each micro-batch and will be aggregated (line 10).

• •

  The last step is where the optimizer performs the parameter update (line 11).

The pseudocode for the Convolutional GAN is given in Algorithm 2, under the procedure GAN Training Step. As can be observed in Algorithm 2, we only enforce differential privacy on the discriminator as only it has access to the real data. To avoid the issue of mode collapse [34], we train the CGAN model, using Wasserstein GAN [35] since it is an efficient approximation of Earth Mover (EM) distance and is shown to be robust to the mode collapse problem during model training [35].

The Earth-Mover (EM) distance or Wasserstein-1 distance represents the minimum price in transforming the generated data distribution $\boldsymbol{P_{g}}$ to the real data distribution $\boldsymbol{P_{r}}$:

$\vartheta(x,y)$ refers to how much “mass” should be moved from $x$ to $y$ to transform $\mathbb{P}_{x}$ to $\mathbb{P}_{g}$.

However, as the infimum in (3) is intractable, based on the the Kantorovich-Rubinstein duality [36], the WGAN propose the Eq. 4 optimization:

For simplicity of the definition, infimum and supremum indicate the greatest lower bound and the least upper bound, respectively.

Using the distance metric and $K=1$, Eq. (5) is equivalent to:

It is clear that for computing the Wasserstein distance, we should find a 1-Lipschitz function (see Definition 4 and Eq. 6.. The approach is to build a neural model to learn it. The procedure is to construct a discriminator D without the Sigmoid function, and output a scalar instead of the probability of confidence.

Regarding the privacy considerations, the generator has no access to real data directly (although it has access to the gradients from the discriminator); only the discriminator will have access to it. We propose only to train the discriminator under differential privacy for this aim. More intuitively, this approach considers the post-processing theorem in differential privacy as follows.

Following the argument above, we consider $\boldsymbol{D}$ and $\boldsymbol{G}$ as discriminator and generator functions, respectively. Since the generator is an arbitrarily randomized mapping on top of the discriminator, enforcing the differential privacy on the discriminator suffices to guarantee that the overall system ensures differential privacy, and we do not need to train a private generator as well. The generator training procedure is given in Algorithm 2, lines 17-20. As can be observed, we do not have a private generator and the loss function is the regular generator loss function of the WGAN method [35].

For the GAN discriminator, we used five convolutional layers similar to the autoencoder (encoder), except for the last layer that we have another dense layer, of output size 1, for decision making. For all layers, we used PReLU activation [37], except for the last layer that does not use any activation. For the generator, we used transposed convolutions (also called fractionally-strided convolutions) similar to the ones used in [38]. However, we use 1-D transposed convolutions. The GAN generator has an input noise of size 100, and its output size is set to 128.

In the autoencoder and for the encoder, we used an architecture similar to the GAN discriminator, except we discard the last layer of the discriminator. For the decoder, as done in the generator, we used transposed convolutional layers in the reverse order of the ones used for the encoder. For the encoder, we used PReLU activation layers except for the last layer of the encoder where Tanh was used to match the generator output. For the decoder, we also used PReLU activation except for the last layer where a Sigmoid activation function was used to bound the range of output data to the $[0,1]$ to ideally reconstruct the discrete range of $\{0,1\}$ aligned with the input data. The decoder input size (the encoder output size) is equal to the GAN’s generator output dimension.

Since our model inputs sizes may change for different datasets, we modify the input pipeline of our architecture by varying the dimensions of convolution kernels, stride for the GAN discriminator, and the autoencoder, to match the new dimensionality. The GAN generator does not require any change since, for all experiments, we used the same noise dimension as mentioned above.

As tracking the RDP privacy accountant [8] is computationally more precise than the regular DP, we based our privacy loss calculation on RDP. Then, the RDP computations can be transformed to DP based on the proposition 2. The Gaussian noise additive procedure is also called Sampled Gaussian Mechanism (SGM) [39]. For tracking privacy loss, we use the following Theorem.

It can be proven that $A_{\alpha}\leq B_{\alpha}$ and the RDP privacy computation can solely be focused on upper bounding $A_{\alpha}$ which can be calculated with a closed-form bound and numerical calculations [39]. We use the same numerical calculations here. However, that bounds $\epsilon$ for each step. The overall bound of $\epsilon$ for the entire training process can be calculated by $\epsilon_{total}=num\_steps\times\epsilon$ for any given $\alpha$ (see proposition 1). To determine the tighter upper bound, we try multiple $\alpha$ values and use the minimum $(\epsilon)$ and its associated $\alpha$ (obtained by RDP privacy accountant) to compute the $(\epsilon,\delta)$-DP employing proposition 2.

System privacy budget: One important question is: how do we calculate the $(\alpha,\epsilon)$-RDP for the combination of an autoencoder and GAN training? Consider proposition 1, in which $\mathcal{X}$, $\mathcal{U}_{1}$, and $\mathcal{U}_{2}$ are the input and output spaces of the autoencoder and the output space of the discriminator. The mechanisms $\mathcal{A}$ and $\mathcal{B}$ are the autoencoder and discriminator, respectively. Consider $\mathcal{U}_{1}$ is what the discriminator observes after decoding of the fake samples, and $\mathcal{X}$ is the space of real inputs. Hence, the proposition 1 directly results in having the whole system with $(\alpha,\epsilon_{ae}+\epsilon_{gan})$-RDP by fixing the $\alpha$. But we cannot guarantee that we can have a fixed $\alpha$, and the RDP has a budget curve parameterized by it. The procedure of fixing $\alpha$ is as follows.

• •

  Assume we have two systems $S_{1}$ and $S_{2}$ such that one is the autoencoder, and the other is the GAN. Hence we have two systems which $(\alpha_{1},\epsilon_{1})$-RDP and $(\alpha_{2},\epsilon_{2})$-RDP.

• •

  Without loss of generality, we assume $\epsilon_{1}\leq\epsilon_{2}$. We pick $\alpha_{total}=\alpha_{2}$. Now we have system $S_{2}$ which $(\alpha_{total},\epsilon_{2})$-RDP.

• •

  For $S_{1}$ we pick $\alpha_{total}=\alpha_{1}$ and calculate $\epsilon^{\prime}$ that $\epsilon\leq\epsilon^{\prime}$.

• •

  Now, the total system $(S_{1},S_{2})$ satisfies $(\alpha_{total},\epsilon_{2}+\epsilon^{\prime})$-RDP.

Here, we split the dataset to train $\mathcal{D}_{tr}$ and test sets $\mathcal{D}_{te}$. We utilize $\mathcal{D}_{tr}$ to train the models, and then generate symthesized samples $\mathcal{D}_{syn}$ using the trained model. We set $|\mathcal{D}_{syn}|=|\mathcal{D}_{tr}|$. Recall from Section IV that although for different datasets we alter the architectures associated with the input size, the encoder output space (decoder input space) and the generator output space will always have the same dimensionality. Furthermore, the encoder input space, the decoder output space, and the discriminator input space will also have the same dimensionality. It is worth noting that all of the reported $\epsilon$ values are associated with the $(\epsilon,\delta)$-DP definition with $\delta=10^{-5}$ unless otherwise stated.

Both the convolutional autoencoder and convolutional GAN were trained with the Adam optimizer [40] with the learning rate of $0.005$, with a mini-batch size of 64. For both generator and discriminator we used Batch Normalization (BN) [41] to improve the training. We used one GeForce RTX 2080 NVIDIA GPU for our experiments.

We compare our framework with various different methods. Depends on the experiments, some of those methods may or may not be used for comparison depending on their characteristics. For further details, please see Table I.

To evaluate this performance, we used the following datasets for our experiments:

1. 1.

   MIMIC-III [42]: This dataset consists of the medical records from around 46K patients. The MIMIC-III dataset [42] includes the medical records of 46K patients.

2. 2.

   Kaggle Cervical Cancer [43]: This dataset covers patients records and aims to classify the cervical cancer. There are multiple attributes in the dataset of discrete, and continuous types and one attribute associated with the class label (e.g., Biopsy as the target for classification).

3. 3.

   UCI Epileptic Seizure Recognition [44]: In the UCI Epileptic Seizure Recognition dataset [44], the task is to classify the seizure activity. The features are the values of the Electroencephalogram (EEG) records at various time points.

4. 4.

   PTB Diagnostic ECG [45]: The electrocardiograms (ECGs) in this dataset [45] are used to classify the heart activity as normal or abnormal. This dataset contains 14552 samples, out of which 4046 are classified as normal, and 10506 are abnormal activities.

5. 5.

   Kaggle Cardiovascular Disease [46]: This Kaggle datasetis used to determine if a patient has cardiovascular disease or not from a variety of features such as age, systolic blood pressure and diastolic blood pressure. This dataset has a mixture of discrete and continuous feature types as well.

6. 6.

   MIT-BIH Arrhythmia [47]: The MIT-BIH Arrhythmia database [48] includes ambulatory ECG recordings, created from 47 subjects. records obtained from a mixed group of inpatients (requires overnight hospitalization) and outpatients (usually do not require overnight hospitalization) at Boston’s Beth Israel Hospital. Unlike other datasets in this set of experiments, for the MIT-BIH Arrhythmia database, we are dealing with a multicategory classification task. The ECG signals correspond to normal (negative) and abnormal cases (positive). There are a total of five classes in this dataset. Class zero is associated with normal cases. The rest of the classes are associated with different arrhythmias and myocardial infarction which we call abnormal.

We compare our model with different benchmark methods (see Table I). Depends on the nature of experiments, the models are used as benchmarks. For example, if the experiments are associated with different privacy settings, the models that do not preserve the privacy, will not be used for comparison.

1. 1.

   MedGAN [13]: MedGAN consists of an architecture that is designed to generate discrete records. It has an autoencoder and a vanilla GAN. MedGAN does not preserve privacy. MedGAN can be used for unsupervised synthetic data generation. We used MedGAN open-source implementation¹¹1https://github.com/mp2893/medgan.

2. 2.

   TableGAN [14]: TableGAN consists of three components: A generator, a discriminator, and an auxiliary classifier that aims to augment the semantic integrity of synthesized data. Similar to our method, TableGAN uses Convolutional Neural Networks. TableGAN requires labeled training data and will be used in experiments with a supervised setting. We used TableGAN open-source implementation²²2https://github.com/mahmoodm2/tableGAN.

3. 3.

   DPGAN [9]: DPGAN enforces differential privacy by clipping the weights rather than the gradients. It uses WGAN to stabilize the training and use multi-layer perceptions for its architecture. DPGAN does not require labeled data and hence will be used in both supervised and unsupervised settings. We used DPGAN open-source implementation ³³3https://github.com/illidanlab/dpgan.

4. 4.

   PATE-GAN [16]: PATE-GAN proposes a differentially private model formed based on the modification of the PATE mechanism. We utilized our own implementation of the PATE-GAN as its code was not available publicly.

We chose electronic health records after converting them to a high-dimensional binary discrete dataset to demonstrate the privacy-preserving potential of the proposed model. Here, we have $\mathcal{D}_{tr}\in\{0,1\}^{R\times|\mathcal{T}|}$, $\mathcal{D}_{te}\in\{0,1\}^{T\times|\mathcal{T}|}$, and $\mathcal{D}_{syn}\in\{0,1\}^{S\times|\mathcal{T}|}$, where $|\mathcal{T}|$ represents the feature size. The goal is to capture the information from a real private dataset, and use that to synthesize a dataset with similar distribution in a privacy-sensitive manner. In this section, we can compare our method with models that do not require labeled data (for unsupervised synthetic data generation) and are also privacy-reserving method (see Table I). In the experiments of this section, to pretrain the autoencoder, we used Eq. 1 as the loss function.

Dataset Construction: We used MIMIC-III dataset [42] as an example for high-dimensional and discrete dataset. From MIMIC-III, We extracted and used 1071 unique ICD-9 codes. The dataset has different discrete variables (e.g., diagnosis codes, procedure, etc.). Assuming there are $\boldsymbol{|V|}$ discrete variables, we can represent the patient’s record using a binary vector $\boldsymbol{v}\in{\{0,1\}}^{|V|}$. The $i^{th}$ variable is set to one if it is available in the patient’s record. In our experiments, $\boldsymbol{|V|}=1071$.

Evaluation Metrics: To assess the quality of synthetically generated data, we used two evaluation metrics:

1. 1.

   Maximum Mean Discrepancy (MMD): This statistical measure indicates the extent to which the model captures the statistical distribution of the real data. In a recent study [49], MMD demonstrated most of the desired features of an evaluation metric, especially for GANs. This is used in an unsupervised setting since there are no labeled data for statistical measurements. To report MMD, we compared two samples of real and synthetic data, each of size 10000.

2. 2.

   Dimension-wise prediction: This setting aims to illustrate the inter-connection between features, i.e., if we can predict missing features using the features available in the dataset. We select top-10 and top-50 most frequent features (ICD-9 codes in the patients’ history) from training, test, and synthetic sets. In each run, one testing dimension ($k$) from $\mathcal{D}_{syn}$ and $\mathcal{D}_{tr}$ will be selected as $\mathcal{D}_{syn,k}\in\{0,1\}^{N\times 1}$ and $\mathcal{D}_{tr,k}\in\{0,1\}^{N\times 1}$. The remaining dimensions ($\mathcal{D}_{syn,\backslash k}\in\{0,1\}^{N\times 1}$ and $\mathcal{D}_{tr,\backslash k}\in\{0,1\}^{N\times 1}$) will be utilized to train a classifier that aims to predict $\mathcal{S}_{te,k}\in\{0,1\}^{N\times 1}$ from the test set. We employed Random Forests [50], XGBoost [51], and Decision Tree [52]. We report the averaged performance over all predictive models (trained on synthetic data) and for all features using the F1-score (Figure 3).  We compare models by considering different privacy budgets by fixing $\delta=10^{-5}$ and varying $\epsilon$.

As can be observed in Figure 2 and Figure 3, without enforcing privacy ($\epsilon$), our model performs better compared to DPGAN. This is due to the fact that by using 1-D convolutional architecture, our model captures the correlated information among adjacent ICD-9 codes in the MIMIC-III dataset. Top-features indicate the most frequent features in the database.

To further investigate the effect of CGANs and CAEs in capturing correlated features, we conduct some experiments without enforcing any privacy and compare our method with existing methods. As can be observed in Table II our method outperforms other methods. Note that when the number of top features are increased, our method shows even better performance compared to other methods since capturing correlation for higher number of features becomes more feasible with convolutional layers.

In this part, we consider a supervised setting and generate labeled synthetic data. For the experiments of this part, we need labeled data. The supervised setting here includes various classification tasks on different datasets. In the experiments of this section, to pretrain the autoencoder, we used Eq. 2 as the loss function.

Data Processing: For all the datasets except for MIMIC, we follow the same data processing steps that are described in Section V-B. For the MIMIC-III dataset, we used one of the top three codes across hospital admissions [42], which is 414.01 (associated with coronary atherosclerosis) and extracted patients diagnosed with that specific medical code. The classification task will be mortality prediction. We used 12,127 unique admissions and variables such as demographic information (Marital status, ethnicity, and insurance status), admission Information (e.g., days of admission), treatment information (cardiac defibrillator implant with/without cardiac catheterization), diagnostic information (respiratory disorder, cancer, etc.), and lab results (kidney function tests, creatine kinase, etc.) forming a total of 31 variables. Each admission is considered as one data sample. Since multiple admissions might be associated with one patient we will have multiple samples per patient. This comes from the fact that, in a 1-year interval, a patient may survive, but within the next few years and with new medical conditions, the same patient may not survive in the 1-year observation windows.

Evaluation: Let us consider the following two settings: (A) Train and test the model on the real data as the baseline (results shown in Table III). (B) Train on the generated data and test on the real data (this setting is used to quantify the quality of synthetic data). Note that the class distribution of the generated synthetic data must be identical to the real data. Setting (B) can demonstrate how well the model has performed the task of synthetic data generation. The closer the performance of setting (B) is to setting (A), the better the model is in terms of the quality of the generated synthetic data. We conducted different sets of experiments. We conduct ten runs (E=10), for each experiment, and report the averaged AUROC (Area Under the ROC curve) and AUPRC (Area Under the Precision-Recall Curve) for the models’ evaluations. The AUPRC metric is being utilized here since it operates better than AUROC for an imbalanced classification setting.