Permutation-Invariant Tabular Data Synthesis

Generative adversarial networks (GAN) are a recently developed algorithm [16] for synthetic data generation. A GAN consists of two components: a generator ($G$) that learns to produce realistic synthetic data, and a discriminator ($D$) that tries to distinguish real data from synthetic (fake) data. Both $G$ and $D$ are neural networks, e.g., Fully-Connected Networks (FCNs) or Convolutional Neural Networks (CNNs). In the training process, $G$ and $D$ play an adversarial game described as follows:

where $x$ is the real sample, $z$ is the random input signal given to $G$, $G(z)$ is the synthetic sample, and $D(\cdot)$ is the probability of a sample being real from the perspective of $D$. The goal of $G$ is to minimize the chance that its generated samples are identified as synthetic, whereas $D$ maximizes the chance of correctly distinguishing real and synthetic samples.

An autoencoder (AE) is an unsupervised learning algorithm that learns a mapping from high-dimensional inputs to low-dimensional representations [14, 17], namely latent vectors. It consists of two models, an encoder ($Enc$) and a decoder ($Dec$). $Enc$ takes a high-dimensional input and compresses it to a latent vector, and $Dec$ uses the latent vector to reconstruct the original input. $Enc$ and $Dec$ are trained as a whole and penalized for creating output that deviates from the input. The loss function is defined as follows:

where $\theta$ and $\phi$ are the parameters of $Dec$ and $Enc$, $x_{i}$ is a high-dimensional input, $Enc_{\phi}(x_{i})$ is the latent vector, $Dec_{\theta}(Enc_{\phi}(x_{i}))$ is the reconstructed input, and $N$ is the total number of samples.

We focus on deep-learning approaches for tabular data synthesis and skip the discussion of classical methods such as Copulas [18, 19] and Bayesian Networks [20]. Table I summarizes the recently developed deep learning methods for tabular data synthesis in terms of models, network architecture, and datasets. MedGAN [21] is designed for aggregated electronic health records (EHRs), which only have count and binary features. Since EHRs are high-dimensional and sparse [22], medGAN uses a pre-trained autoencoder to learn compact representations of the input data and thereby simplifies the GAN’s task. MedGAN is improved by [22], where the standard GAN loss is replaced by Wasserstein loss with gradient penalty, and the new model is named medWGAN. However, different from AE-GAN, medGAN and medWGAN are limited in generalizing to real-world scenarios because they only consider count and binary features.

A few recent tabular data synthesizers are suitable for general data types, including table-GAN [8], CTGAN [11], TVAE [11], and CTAB-GAN [12]. CTGAN, TVAE, and CTAB-GAN use Variational Gaussian Mixture (VGM) to encode numerical features and one-hot encoding for categorical features. Moreover, CTAB-GAN defines the mixed datatype and proposes a new encoding method. In addition, CTGAN, TVAE, and CTAB-GAN adopt the training-by-sampling technique to handle highly-imbalanced distributions. Despite their effectiveness in tabular data synthesis, these models overlook and do not abide by the key property of column permutation invariance.

In computer vision similar concepts have been brought up and investigated, including permutation invariance [23, 24], translation invariance [25, 26, 27], and translation equivalence [28]. Permutation invariance means that the output of a neural network stays the same despite permutations of its input. For example, the classification of an image should not change after adjusting the object location in the image.

Motivated by that, we define column permutation invariance in tabular data synthesis as follows. The performance of a tabular data synthesizer should not be affected by permutations on the input column order. To the best of our knowledge, column permutation invariance has not been researched by the prior art on tabular data synthesis.

Initially designed for images, CNNs use a set of convolution kernels to slide over the input feature space, abstract high-dimensional features, and then aggregate them into knowledge about the input. Due to the limited kernel size, CNNs only learn local relations between neighboring features and fall short to capture global dependencies.

The focus of CNNs on local relations hinders high quality tabular data synthesis. In contrast to image data, tabular data do not have necessarily strong local relations. Highly-correlated features can be very far apart, and their dependencies can be complex and irregular [29]. These characteristics make modeling tabular data extra challenging for CNNs [30, 31], despite their remarkable performance in many machine learning tasks [32].

Since CNNs capture mainly local relations, CNN-based tabular data synthesizers are sensitive to column permutations. We use table-GAN to verify this assumption. We test it with five datasets arranged using three column orders, namely the original order, order by type, and order by correlation. Order by type means putting all continuous columns on the left of the table, and all categorical columns on the right. Order by correlation means placing highly-correlated columns on the left and weakly-correlated columns on the right. For each order, we train the model separately and calculate the Wasserstein-1 distance (WD) between real and synthetic data. Every experiment is repeated 5 times. Table II summarizes our results. The best, i.e. lowest, distance values are highlighted in bold. The last column shows the maximum WD change in percent across all three column permutations. The results show that table-GAN is most sensitive on the Intrusion dataset with a maximum difference in WD of 14.90%.

Efficient representation of categorical features is one of the main challenges in tabular data synthesis. In the state-of-the-art, table-GAN uses label encoding to transform categorical features into numerical ones and normalizes them with min-max normalization. This method often leads to sub-optimal performance due to the artificial order in categorical features [33]. In contrast, CTGAN, CTAB-GAN, and TVAE use one-hot encoding to represent categorical features. Despite its simplicity and effectiveness, one-hot encoding introduces many zeros to the input data and thus increases sparsity.

Representing numerical features in tabular data is relatively straightforward. The most common method is mapping them into [-1, 1] with min-max normalization. However, the authors of CTGAN, TVAE, and CTAB-GAN adopt mode-specific normalization, which uses Variational Gaussian Mixture to represent multi-modal numerical features. Although this method improves the quality of the synthetic data, we find it leads to sparse input because one-hot encoding is used to represent multiple modes.

Our experiments show that sparse tabular data causes sensitivity to column permutations. We compare CTAB-GAN, a model using one-hot encoding and mode-specific normalization, with table-GAN, a model using label encoding and min-max normalization. Note that label encoding and min-max normalization do not change the dimensionality of the input data, whereas one-hot encoding and mode-specific normalization increase sparsity. Table III shows our experiment results of CTAB-GAN. Compared to table-GAN (see Table II), CTAB-GAN synthesizes more realistic data (shown by a lower WD which is about $1/4$ of table-GAN’s WD), but with higher sensitivity to column permutations on all datasets. The average maximum change in WD across the five datasets is 38.67%, whereas in table-GAN it is 6.82%.

To further highlight the sparsity of the encoded data we visualize the input of table-GAN and CTAB-GAN in Figure 2. We note that the sparsity is determined by the sum of all levels of all variables, i.e., the number of modes per continuous variable and the discrete levels per discrete variable. We reshape each row of a table into a square matrix to make it compatible with CNNs. In table-GAN, one row of the Adult dataset is represented by a $4\times 4$ matrix, but in CTAB-GAN a $24\times 24$ matrix is needed due to one-hot encoding and mode-specific normalization which increases the number of zeros, i.e. purple matrix cells in the figure accounting for roughly 97% area. This increase in sparsity makes CTAB-GAN more sensitive to column permutations than table-GAN since the average distance between related columns (pixels) is increased.

Theoretically, fully-connected networks (FCNs) should be robust to column permutations because all features are connected. However, we find FCNs are not fully permutation invariant. Tests with CTGAN and TVAE, two FCN-based tabular data synthesizers, on five datasets show an average WD of $1.87$ with an average maximum change of 18.62% for CTGAN and $1.80$ with 14.89% for TVAE (details skipped due to space constraints). Overall, the state-of-the-art tabular data synthesizers either provide the high quality synthetic data or are resilient to column order permutations, struggling to make a good trade-off.

Following [11], we use mode-specific normalization for numerical features and one-hot encoding for categorical features. Mode-specific normalization preserves the multi-modal distribution of numerical features and improves the performance of tabular data synthesizers [11, 12]. One-hot encoding is a simple yet effective way to convert categorical values to numerical ones without losing too much information. Certainly, one-hot encoding and mode-specific normalization causes sparse input, but the autoencoder in AE-GAN maps the sparse encoded input into compact latent vectors solving this issue.

Since we identify sparsity as one of the main reasons for tabular data synthesizers’ sensitivity to column permutations, one natural solution is to use an autoencoder to extract the features of tabular data and compress them into compact latent vectors. Such an advantage can apply to all kinds of tabular data synthesizers.

We use two three-layer fully connected networks as $Enc$ and $Dec$. Based on our study of mainstream open-source implementations of autoencoders [34, 35, 36], fully-connected networks with 2-4 layers are common choices for autoencoders.

Another important design choice is the length of the latent vector, i.e., the output size of $Enc$ and the input size of $Dec$. This determines the autoencoder’s capacity to represent high-dimensional data. We choose this parameter based on the size of the input dataset. For datasets with a large number of columns, we increase the length of the latent vector to ensure that complex relations between columns can be well represented, therefore helping the generator to synthesize realistic data.

The core of AE-GAN is a GAN, which has two competing networks, namely the generator, $G$, and the discriminator, $D$. Figure 4 shows their architectures. Both $G$ and $D$ have three fully-connected layers which are more resilient to permutations of elements in the (more compact) latent vector. In the discriminator, the fully connected layers are followed by leaky ReLU activation and the final output is the validity of the input. In the generator, the fully-connected layers are followed by batch normalization and leaky ReLU activation, and the final output is the synthetic latent vector. The design is based on the architecture of WGAN-GP [37].

To enhance the training of $G$ and thus the quality of the generated tabular data, we introduce an auxiliary classifier $C$ to the GAN. This design is inspired by [8], where an auxiliary classifier is added to maintain the semantic consistency of synthetic data. Note that the input to $C$ is the reconstructed data from $Dec$, rather than the latent data from $Enc$ and $G$ because we want $C$ to learn the semantic relations between columns directly. Specifically, given a categorical target column with several categories, $C$ learns to classify which category a sample belongs to according to the columns other than the target column. This is how $C$ differs from the $D$: $D$ determines the “realness” of a sample based on all columns in the latent space, whereas $C$ learns the relationship between the target column and all other columns in the reconstructed space. By combining $C$ with the GAN, we simultaneously leverage the flexibility of unsupervised training with the control provided by supervised training, thereby improving the quality of the synthetic data.

The training of AE-GAN requires four loss functions: autoencoder loss $\mathbb{L}_{AE}$, generator loss $\mathbb{L}_{G}$, discriminator loss $\mathbb{L}_{D}$, and classifier loss $\mathbb{L}_{C}$.

We test two training strategies: disjoint training and joint training. For disjoint training we first train the AE until convergence and then train the GAN with the classifier while utilizing the compression power of the AE. For joint training, inspired by TimeGAN [38] and the hypothesis of possible training synergies, we first pre-train the autoencoder for a certain number of epochs and then co-train it with the GAN and the classifier. Ablation tests, details in Section VI-D, show that disjoint training achieves lower training losses. Hence we use disjoint training for all results if not specified.

Our evaluation of tabular data synthesizers focuses on the statistical difference and machine learning utility difference between real and synthetic data. Two metrics quantify the statistical difference.

Wasserstein-1 Distance (WD) measures the difference between two continuous/discrete 1-dimensional distributions. We use this metric to compare the per features difference between real and synthetic data.

Difference in Correlation Matrix (Dif. Corr.) measures how well the cross-column correlations⁶⁶6Note that we use “correlation” as a general term. For two numerical features, it refers to the Pearson correlation coefficient; for two categorical features, it is their Cramer’s V; and for a categorical features and a numerical features, it means their correlation ratio. are captured by a tabular data synthesizer. We calculate the difference between the correlation matrices of the real and synthetic table as follows:

where $Corr^{R}_{i,j}$ and $Corr^{F}_{i,j}$ are the correlation coefficients between features $i$ and $j$ in the real and synthetic correlation matrices.

We measure machine learning utility as the performance differences of machine learning models trained on the real and synthetic data. Specifically, we first train four machine learning models with real and synthetic data separately. Then we obtain their average prediction accuracy and compute the difference. The difference is small if the synthetic data has high machine learning utility.

AE-GAN is evaluated on four aspects: column permutation invariance, statistical similarity and machine learning utility of the synthetic data, and training time. We aim to verify if AE-GAN is robust to column permutations and achieves good synthesis quality compared with the state-of-the-art. Additionally, we evaluate the scalability of AE-GAN by analyzing its training time. Table V summarises the results averaged on five datasets.

Column Permutation Invariance. Similar to our empirical analysis, we arange training data in three different orders, i.e., original order, order by type, and order by correlation, and test the performance of AE-GAN. The second column of Table V shows the sensitivity to column permutations of the baseline models and AE-GAN, averaged on five datasets. We find AE-GAN ranks second in permutation invariance among the five models. Table-GAN is the most permutation-invariant model because it does not have the sparsity issue caused by one-hot encoding and mode-specific normalization. In contrast, CTAB-GAN, TVAE, CTGAN and AE-GAN adopt one-hot encoding for categorical features and mode-specific normalization for numerical features and thus have sparse input, leading to higher sensitivity. However, since AE-GAN has an autoencoder to compress the input, it is more robust to column permutations than CTAB-GAN, TVAE, and CTGAN.

Synthesis Quality Comparison. We evaluate the quality of the synthesized data with two metrics: statistical difference and ML utility difference between real and synthetic data. Table V shows that CTAB-GAN is the best model in synthesis quality because its synthetic data have the lowest statistical difference and ML utility difference compared with real data. Table-GAN is the worst among all models. In the rest three models, AE-GAN is better than CTGAN and TVAE on Dif. Corr, but worse on WD. However, the ML utility of AE-GAN is better than CTGAN and TVAE. The results show that the autoencoder of AE-GAN helps preserve the correlation between different features, but for each feature, the statistical difference between real and synthetic data may increase due to the information loss caused by data compression. Future work may look into optimizing the trade-off between the performance boost for GAN and the compression loss caused by compact representations.

Training Time Analysis. A model with a short training time can scale up to large datasets. It also requires fewer hardware resources than slow models given the same input. We compare the scalability of AE-GAN with the baseline models by analyzing their total training time. Table V shows that AE-GAN is faster than CTGAN and CTAB-GAN, but slower than table-GAN and TVAE. Table-GAN has the shortest training time because of its simple data representation, which leads to small input size. TVAE is faster than AE-GAN because AE-GAN has an auxilliary classifier. Nonetheless, AE-GAN is significantly faster than CTAB-GAN, speeding up as much as 6 times.

Summary. AE-GAN achieves the best tradeoff between permutation invariance, synthesis quality, and training time compared with state-of-the-art tabular data synthesizers. It is more permutation-invariant than CTAB-GAN, CTGAN, and TVAE, and it has better synthesis quality than table-GAN, CTGAN, and TVAE in terms of ML utility. Although table-GAN is less sensitive to column permutations and takes less time to train than AE-GAN, its synthesis quality is much worse. In a similar vein, although CTAB-GAN is better than AE-GAN in synthesis quality, it is much slower and more sensitive to column permutations. Compared with CTGAN and TVAE, AE-GAN is more permutation-invariant, has better ML utility, and takes a similar time to train.

We conduct an ablation study to understand the influence of the design choices we made with AE-GAN. We change the data representations, model architecture, and training algorithm to test their effect. Table VI summarizes the results of the ablation study.

Without Mode-Specific Normalization (MSN). We use mode-specific normalization in AE-GAN to normalize numerical features. Although it preserves the multi-model distribution of numerical features, it increases the sparsity in training data. We replace it with min-max normalization to understand its effect. Table VI shows that after removing mode-specific normalization, the WD becomes worse on all datasets, meaning that mode-specific normalization improves the synthesis quality. However, it also makes AE-GAN more sensitive to column permutations. After removing it, the sensitivity to column permutations decreases on the Loan, Credit, and Covtype datasets. In conclusion, mode-specific normalization increases sensitivity to column permutations, but it improves synthesis quality.

Without One-hot and Mode-Specific Normalization. To further reduce sparsity in the input data, we remove one-hot encoding and mode-specific normalization together. Similar to table-GAN, we pre-process categorical and numerical features using min-max normalization. We found that the WD is worse than only removing mode-specific normalization, which proves that one-hot encoding can enhance synthesis quality. Moreover, the sensitivity to column permutations decreases on all datasets except the Adult dataset after removing one-hot encoding and mode-specific normalization, especially on datasets with a high proportion of categorical features such as the Covtype and Intrusion datasets. The results verify again that reducing sparsity can enhance permutation invariance.

Without Auxiliary Classifier. We use an auxiliary classifier to improve the synthesis quality of AE-GAN. After removing the auxiliary classifier, the Wasserstein distance between real and synthetic data worsens on all datasets except the Loan dataset. Overall, the average WD on five datasets increases from $2.669$ to $2.773$ after removing auxiliary classifier, showing that the classifier improves synthesis quality.

Co-training AE and GAN. The AE and GAN in AE-GAN are trained separately. To study whether co-training AE and GAN can improve the synthesis quality, we first pre-train the AE for 300 epochs and then train it together with GAN. Surprisingly, the results show that co-training makes the synthesis quality worse. We find that the training loss of AE is already low after pre-training. However, during co-training, the feedback from GAN increases AE’s loss and makes it unstable.

We evaluate the proposed feature sorting algorithm on table-GAN and CTAB-GAN, two CNN-based tabular data synthesizers, because this algorithm is designed to alleviate the limitations of CNN as explained in Section V.

Table-GAN. Table VII shows the effect of the feature sorting algorithm on table-GAN. A negative change in Dif. Corr. or WD means the difference between synthetic and real data becomes smaller. That is, the feature sorting algorithm helps tabular data synthesizers generate more realistic data. The results show that the feature sorting algorithm works best on the Credit dataset, where Dif. Corr. and WD are decreased by 12% and 4%. It also improves the results on the Intrusion dataset, where WD is reduced by 16%, whereas Dif. Corr slightly increases by 3%. However, it does not influences much the Loan and Covtype datasets, where the Dif. Corr. and WD change less than 5%. Moreover, the results on the Adult dataset become worse, with Dif. Corr and WD increase by 24% and 3%.

The algorithm performs best on the Credit dataset because of the simple correlations between its features. Using $\displaystyle\pm 0.2$ as the threshold for high correlation, only Time and Amount are strongly-correlated with other features. All other features have a close-to-0 correlation. Besides, Time and Amount are only correlated with 3 and 5 features, respectively. With such a small number of correlated features, capturing their relation in the convolution process is easy once we group them together.

The algorithm also alleviates the CNN boundary effect on the highly-correlated features of the Credit dataset. In the original order, Time and Amount are the leftmost and rightmost columns in the table, and many of their correlated features are far apart. However, after feature sorting, these features are in the middle of the table therefore reducing the boundary effect.

In contrast to the Credit dataset, the other four datasets have a larger number of correlated features. For example, in the Adult dataset most features are correlated with at least one other feature, and seven features are correlated with more than three features. Due to the limited kernel size, it is challenging for CNNs to capture all the cross-column relations even after putting the highly-correlated features together. Besides, our algorithm is based on pairwise correlation, but putting one pair of highly-correlated features together could possibly separate another pair of highly-correlated features, which explains why sometimes Dif. Corr. and WD become worse after applying the feature sorting algorithm. In this case, domain knowledge is required to effectively group the correlated features and arrange them in a good order.

CTAB-GAN. To understand whether our feature sorting algorithm works when sparsity is involved, we test it on CTAB-GAN, and the results are summarized in Table VIII. Surprisingly, the algorithm can reduce Dif. Corr. and WD by more than 10 % on all datasets except the Credit dataset. On the Loan dataset, the Dif. Corr. and WD are decreased by 57% and 29% after feature sorting, meaning that the algorithm can effectively improve the statistical similarity between synthetic and real data.

Compared with table-GAN, CTAB-GAN has more performance gain after feature sorting. This is due to the sparsity issue caused by the encoding methods of CTAB-GAN, i.e., mode-specific normalization for numerical features and one-hot encoding for categorical features. Since the input data are sparse after encoding, putting the highly-correlated columns together can drastically reduce the distance between correlated columns, and therefore improves CTAB-GAN’s ability to capture the relation between highly-correlated columns.

To summarize, our feature sorting algorithm can improve the performance of CNN-based table synthesizers, especially when the input tabular data are sparse. For dense tabular data, it also works if the relation between correlated features is relatively simple.