Boosting Data Analytics with Synthetic Volume Expansion

The Syn framework empowers data analytics by applying statistical methods on a synthetic sample $\tilde{\bm{Z}}^{(m)}=(\tilde{\bm{Z}}_{i})_{i=1}^{m}$. This sample is generated by a generative model trained on raw data $\bm{Z}^{(n)}=(\bm{Z}_{i})_{i=1}^{n}$ through fine-tuning a pre-trained model, leveraging insights from various similar studies. These models include GPT (Brown et al., 2020; Bubeck et al., 2023; OpenAI, 2023), diffusion models (Ho, Jain and Abbeel, 2020; Song, Meng and Ermon, 2020), normalizing flows (Dinh, Krueger and Bengio, 2014; Dinh, Sohl-Dickstein and Bengio, 2016; Kingma and Dhariwal, 2018), and GANs (Frid-Adar et al., 2018; Liu et al., 2020). For an in-depth understanding of the generation processes for diffusion models and flows, readers refer to (Liu, Shen and Shen, 2023). In this framework, the cumulative distribution function (CDF) $\tilde{F}$ of $\tilde{\bm{Z}}^{(m)}$ estimates the CDF $F$ of $\bm{Z}^{(n)}$. To produce high-fidelity synthetic data, fine-tuning a pre-trained generative model is recommended, which involves transferring knowledge from previous studies. If pre-trained models are unsuitable, constructing a generative model from scratch is a viable, though less preferred, alternative. The quality of the generated data hinges on the choice of generative model and the effectiveness of knowledge transfer from similar studies. For an illustration, we detail the synthetic data generation process using a diffusion model in Figure 1. Furthermore, to demonstrate the importance of knowledge transfer, we fine-tune a pre-trained tabular diffusion model (Shwartz-Ziv and Armon, 2022) on the Adult-Male dataset to apply this knowledge to the Adult-Female dataset in Section 3.3, where male and female distributions exhibit distinct differences. For a detailed explanation of the impact of knowledge transfer, readers refer to Section 2.5.

It is imperative to underscore the pivotal role of knowledge encapsulated in pre-trained models for improving generation accuracy through fine-tuning. Nevertheless, directly accessing the pre-training data for pre-trained models is often infeasible, impeded by privacy concerns, extensive storage needs, and data inconsistencies (Sweeney, 2002; Quionero-Candela et al., 2009; Labrinidis and Jagadish, 2012; Carroll, 2006), as seen in models like GPT-4. Furthermore, the distributions of pre-training datasets, especially from different sources, might not always mirror that of raw data, as illustrated by the Adult-Male and Adult-Female examples in Section 3.3. Considering these challenges and the nuances of real-world scenarios, we omit pre-training data from our raw data composition throughout this article.

To yield $\tilde{F}$ directly, one can utilize some revertible generative models such as normalizing flows and Roundtrip GAN (Liu et al., 2020), acting as a nonparametric estimate of $F$. For other generative models, such as diffusion models and GPT, one can typically obtain $\tilde{F}$ from synthetic data employing Monte Carlo methods, which we elaborate on in Section 2.4. In the numerical examples presented in this paper, we utilize a tabular diffusion model (TDM, Kotelnikov et al. (2023)) and GPT for synthetic data generation.

Subsequently, we explore the precision advantages offered by the Syn framework.

In estimation and prediction, leveraging a statistical method on a synthetic sample gives rise to an estimate, denoted as $\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)})$, of a parameter vector $\bm{\theta}$. The effectiveness of this method gauges through a specific error metric $\text{R}(\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)}))=\operatorname{E}L(\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)}),\bm{\theta})$, which would theoretically improve the statistical accuracy of $\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)})$ with an infinite amount of synthetic data, provided $\tilde{F}$ perfectly replicates $F$. Here, $\operatorname{E}$ symbolizes the expectation under $F$, while $L(\cdot,\cdot)$ represents a loss function quantifying the discrepancies between $\widehat{\bm{\theta}}$ and $\bm{\theta}$. However, numerical insights from Section 3.2 reveal the existence of a reflection point, denoted as $m_{0}=\arg\min_{m\geq 1}\text{R}(\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)}))$, which delineates a relationship between the synthetic sample size $m$ and the accuracy augmentation for this method. This point $m_{0}$ is governed by the generation error measured by metrics such as the total variation distance between $\tilde{F}$ and $F$, defined as $\text{TV}(\tilde{F},F)=\sup_{B}|P_{F}(B)-P_{\tilde{F}}(B)|$, where $P_{F}$ and $P_{\tilde{F}}$ are probabilities measures induced by $F$ and $\tilde{F}$ and $B$ is any event.

To estimate $m_{0}$, we optimize its empirical risk measure across $m$ on an independent cross-validation sample from the original resources, which yields an optimizer $\hat{m}$ as an estimate of $m_{0}$. For instance, in scenarios where the risk measure is the generalization error in binary classification, its empirical risk is the test error on a test dataset obtained from the original resources, approximating a classifier’s generalizability.

To investigate the theoretical aspects of the generational effect concerning the accuracy of $\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)})$, we clarify the notation: Let $\text{Gr}^{(m)}=\text{R}(\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)}))-\text{R}(\widehat{\bm{\theta}}(\bm{Z}^{(m)}))$ represent the discrepancy between the synthetic and raw errors for a sample size $m$, where $\text{R}(\widehat{\bm{\theta}}(\bm{Z}^{(m)}))=\operatorname{E}L(\widehat{\bm{\theta}}(\bm{Z}^{(m)}))$ denotes the risk incurred when employing a raw sample $\bm{Z}^{(m)}$ of size $m$.

The premise regarding the growth of $Gr^{(m)}$ in $m$ in Theorem 2.1 suggests that the divergence between the synthetic risk and the risk escalates more rapidly when $m\leq m^{*}$, beyond which it remains higher than $Gr^{(m^{*})}$. This condition implies a substantial generation error. According to Theorem 1, a considerable generation error results in the synthetic risk $\text{R}(\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)}))$ reaching its minimum at a specific $m_{0}$, beyond which the synthetic error begins to worsen. This outcome indicates that an increase in synthetic sample size does not necessarily improve estimation or prediction accuracy in the presence of significant generation error, a phenomenon further illustrated in Section 3.2. In contrast, when the generation error is minimal, the synthetic risk remains controlled, and the optimal $m_{0}$ tends to be substantially large or potentially infinite, as suggested by Theorem 2.2.

Next, we establish a bound on the synthetic risk to offer insight into the conditions under which accuracy improvements may arise. Assume that $\bm{Z}^{(n)}$ is independently and identically distributed according to $F$ while $\tilde{\bm{Z}}^{(m)}$ is independently and identically distributed according to a conditional distribution $\tilde{F}\equiv F_{\bm{Z}\mid\bm{Z}^{(n)}}$ given $\bm{Z}^{(n)}$.

Theorem 2.2 posits that training a method trained on synthetic data can lead to an accuracy gain, provided that the generation error that governs the synthetic risk $\text{R}(\widehat{\bm{\theta}}(\tilde{\bm{Z}}^{(m)}))$ is small. Hence, high-fidelity data can mitigate the synthetic risk in a method, thereby enabling a large optimal synthetic size $m_{0}$ for further amplifying the precision.

The existing literature provides insights into the magnitude of generation error as represented by $\text{TV}(\tilde{F},F)$. For instance, Theorem 5.1 in Oko, Akiyama and Suzuki (2023) specifies bounds for a diffusion model, given the data-generating distribution is a member of a Besov space. It’s worth noting that the boundary defined in Oko, Akiyama and Suzuki (2023) pertains to $\operatorname{E}\text{TV}(\tilde{F},F)$, with $\operatorname{E}$ symbolizing the expectation relative to $F$. However, one may extend this to the in-probability convergence rate for the random quantity $\text{TV}(\tilde{F},F)$ by leveraging Markov’s inequality, which provides the convergence rates for $\text{TV}(\tilde{F},F)$ based on the raw sample size $n$ and/or the pre-training sample size.

We now introduce Syn-Test, a novel inference tool using high-fidelity synthetic data to boost any test’s power by expanding the sample size of raw data. Syn-Test yields two distinct advantages. First, it employs synthetic data to gauge the null distribution of any test statistic by Monte Carlo methods as in the bootstrap approach (Efron, 1992), circumventing analytical derivations. This methodology proves particularly powerful for unstructured data inferences, including texts and images (Liu, Shen and Shen, 2023). Second, Syn-Test identifies the optimal synthetic data size, optimizing a test’s power while maintaining a suitable control of Type-I errors. For illustration, we refer to Section 3.3.

Given a raw sample, Syn-Test employs two nearly equal-sized subsamples $\mathcal{S}_{1}$ and $\mathcal{S}_{2}$, partitioned from a training sample for fine-tuning a pre-trained generative model. It also uses a separate validation sample $\mathcal{S}_{3}$ for validating model training. One generative model generates synthetic data using $\mathcal{S}_{1}$ for null distribution estimation, while the other uses $\mathcal{S}_{2}$ for computing the test statistic. Figure 7 illustrates the splitting scheme and Syn-Test process. Syn-Test also empirically determines the optimal synthetic data size $m_{0}$ to control the Type-I error. By swapping the roles of $\mathcal{S}_{1}$ and $\mathcal{S}_{2}$, Syn-Test can de-randomize the partition, transitioning the original inference sample size $n$ to a synthetic inference sample size of $m$. This swapping mechanism proves especially advantageous in scenarios with low generation error. Crucially, abundant synthetic data can enhance the size of the raw data, even when sample splitting results in a reduced inference sample size (Wasserman and Roeder, 2009; Wasserman, Ramdas and Balakrishnan, 2020).

Syn-Test encompasses four steps, using a significance level $\alpha$, a tolerance error $\varepsilon$, and a Monte Carlo size $D$. Syn-Test goes as follows:

Step 1: Controlling Type-I Error. Generate $D$ distinct synthetic samples $(T(\tilde{\bm{Z}}_{1}^{(m,d)}))_{d=1}^{D}$ of size $m$ by refining a pre-trained generative model with $\mathcal{S}_{1}$ under $H_{0}$, using $\mathcal{S}_{3}$ as a validation set for model tuning to avoid overfitting. Compute the empirical distribution of the test statistic $T$ using $(T(\tilde{\bm{Z}}_{1}^{(m,d)}))_{d=1}^{D}$. Define a rejection region $C_{m}$ at a significance level where $\alpha>0$.

Step 2: Optimizing Synthetic Size through Tuning. Execute Step 1, but use $\mathcal{S}_{2}$ instead of $\mathcal{S}_{1}$ to produce $\tilde{\bm{Z}}^{(m,d)}_{2}$. Utilize the empirical distribution from $(T(\tilde{\bm{Z}}_{2}^{(m,d)}))_{d=1}^{D}$ to find the empirical Type-I error, denoted $\tilde{P}(C_{m})$, for the $C_{m}$ created in Step 1. To effectively control the Type-I error, we propose two distinct strategies for identifying $\hat{m}$: an aggressive and a conservative approach. The aggressive approach selects the largest $m$ that maintains the estimated Type-I error within the desired limit. In contrast, the conservative one chooses the smallest $m$ about failing to control the estimated Type-I error. Mathematically,

1. 1.

   Aggressive: $\hat{m}=\max\{m:\tilde{P}(C_{m})\leq\alpha+\varepsilon\}$.

2. 2.

   Conservative: $\hat{m}=\min\{m:\tilde{P}(C_{m})\leq\alpha+\varepsilon\ \text{ and }\tilde{P}(C_{m+1})>\alpha+\varepsilon\}$.

In practice, we recommend adopting a conservative approach, as it more effectively manages Type-I errors, although it may be less powerful compared to a more aggressive strategy.

Step 3: Calculating the P-value. With the determined $\hat{m}$, produce synthetic data $\tilde{\bm{Z}}_{2}^{(\hat{m})}$ by fine-tuning the pre-trained generative model using $\mathcal{S}_{2}$ with $\mathcal{S}_{3}$ as a validation set. Calculate the test statistic for $T(\tilde{\bm{Z}}_{2}^{(\hat{m})})$ and determine the P-value, $P^{1}$, leveraging the null CDF estimated from $\mathcal{S}_{1}$ in Step 1.

Step 4: Combining the P-values. Repeat Step 3 by interchanging the roles of $\mathcal{S}_{1}$ and $\mathcal{S}_{2}$ to compute the P-value $P^{2}$. Combine P-values via Hommel’s weighted average (Hommel, 1983):

where $C=\sum_{q=1}^{2}\frac{1}{q}=3/2$ and $P^{(q)}$ is the $q$-th order statistic of $\{P^{1},P^{2}\}$.

Hommel’s method excels in controlling the Type-I error relative to many of its peers, ensuring that $\mathbb{P}\big{(}\bar{P}\leq\alpha\big{)}\leq\alpha$ under $H_{0}$. While effective, there are also alternative strategies such as the Cauchy combination (Liu and Xie, 2020). To expedite the search of an estimated $m_{0}$ $\hat{m}$, we may consider techniques such as Bisection (Burden and Faires, 2001) or Fibonacci Search (Kiefer, 1953).

Syn-Test facilitates valid inference without necessitating many model assumptions, specific data distributions, or an infinitely large inference sample. Its validity and power primarily hinge on the small generation error, a condition usually met by generative models trained on sufficiently large datasets. Additionally, Syn-Test permits a fine-tuned generator to estimate the bias of a test statistic through a Monte Carlo approach. This estimation can serve various purposes in downstream analysis, including debiasing. We defer this methodology in the future study.

The Syn Framework enables synthetic data generation, mirroring raw data distributions. It is adept at tackling statistical challenges in both unsupervised and supervised realms. It can also derive $\tilde{F}$ directly from some generative models such as normalizing flows (Dinh, Krueger and Bengio, 2014; Dinh, Sohl-Dickstein and Bengio, 2016; Kingma and Dhariwal, 2018). Subsequently, we introduce Syn-Slm, a streamlined approach for supervised learning tasks. Unlike the methods discussed in Section 2.2, Syn-Slm directly models the conditional distribution of the outcome given predictors without relying on an additional predictive model for synthetic data. This method simplifies the process by directly modeling the conditional distribution of the outcome.

Supervised learning aims to predict an outcome based on a set of predictors, denoted as $\bm{X}$. Consider a scenario with a one-dimensional outcome variable $Y$ and define $\bm{Z}=(Y,\bm{X})$. In this context, the goal is to estimate a statistical functional, represented as $\phi(F_{Y|\bm{X}})$, where $F_{Y|\bm{X}}$ is the conditional CDF of $Y$ given $\bm{X}$. Our Syn-Slm method introduces a plug-in estimate $\phi(\tilde{F}_{Y|\bm{X}})$, where $\tilde{F}_{Y|\bm{X}}$ represents the conditional CDF of a synthetic $Y$ given $\bm{X}$. This approach, focusing on estimating $\tilde{F}_{Y|\bm{X}}$, is notably distinct from conventional methods, which often estimate $\tilde{F}_{\bm{X}|Y}$, such as generating images from a specific class. However, as hypothesized by Zhang et al. (2023b), this type of generative modeling could outperform direct predictive modeling, as demonstrated in their work using imputation techniques. To illustrate, if $\phi(F_{Y|\bm{X}})=\operatorname{E}(Y|\bm{X})$ signifies the conditional expectation of $Y$, then $\phi(\tilde{F}_{Y|\bm{X}})=\operatorname{E}(\tilde{Y}|\bm{X})$, where $\tilde{Y}$ is derived from $\tilde{F}_{Y|\bm{X}}$, resulting in a Syn-Slm estimate. An example is provided in Section 3.1, and a simulated example is presented in Appendix B.

In contrast, the Syn prediction method described in Section 2.2 employs a two-step process: it initially generates joint synthetic data $\tilde{\bm{Z}}=(\tilde{Y},\tilde{\bm{X}})$, followed by training a predictive model using $\tilde{\bm{Z}}$. For instance, Syn-Boost in 3.2 utilizes TDM (Kotelnikov et al., 2023) as the generator and CatBoost (Dorogush, Ershov and Gulin, 2018) for prediction. While straightforward in implementation, methods akin to Syn-Boost may incur prediction errors from predictive modeling and generation errors from generating $\bm{Z}$. In contrast, Syn-Slm directly models the conditional generation of $Y|\bm{X}$ and estimates $\tilde{F}_{Y|\bm{X}}$ via a Monte Carlo approach, offering a fully non-parametric solution. Additionally, Syn-Slm enables the estimation of various statistical functionals $\phi(F_{Y|\bm{X}})$, such as $E(Y|\bm{X})$, $\text{Var}(Y|\bm{X})$, and quantiles of $Y|\bm{X}$. In contrast, methods like Syn-Boost require training different predictive models under varying assumptions to estimate these quantities. A detailed comparison between Syn-Boost and Syn-Slm is presented in Table 1.

Knowledge transfer elevates generation accuracy by infusing task-specific generative models with pre-trained knowledge from relevant studies. From the perspective of dimension reduction, we dissect knowledge transfer in two scenarios. In the first situation, consider a generative model $g_{\bm{\theta}}$ parametrized by $\bm{\theta}$. Originally trained on an extensive dataset for a generation task, the model undergoes subsequent fine-tuning on a smaller but similar dataset to account for distribution shift, resulting in model $g_{\bm{\theta}^{\prime}}$, where the architecture remains consistent across both models, with the essence of knowledge transmission occurring via the transition from $\bm{\theta}$ to $\bm{\theta}^{\prime}$ amid the fine-tuning. In the other situation, a robust pre-trained model undergoes training across multiple tasks, characterized as $(f_{1},\ldots,f_{t})\circ h$. Here, $f_{i}$ defines the output function tied to the $i$-th task, $h$ is the shared representation function, and $\circ$ denotes functional composition. Given a learned representation $h$, one only fine-tune $f_{0}$ during its optimization phase for $f_{0}$ (Tripuraneni, Jordan and Jin, 2020). As the generative model refines its precursor, it absorbs the precisely calibrated representation $h$. This knowledge transfer thus can augment the generation precision through fine-tuning with a heightened accuracy of the learned $f_{0}$, facilitating its dimension reduction. It is pivotal to acknowledge that within this configuration, $f_{0}\circ h$ and $f_{i}\circ h$; $i=1,\ldots,t$, only share the same architecture in $h$. An alternate strategy entails concurrent fine-tuning of $f_{0}$ and $h$ to derive a representation explicitly for $f_{0}$.

The Syn framework capitalizes on knowledge transfer to bolster its overall efficacy, streamlining the synthetic data generation process. For a visual representation of how a pre-trained generative model is fine-tuned on a specific dataset to facilitate knowledge transfer, refer to Figure 1. In Section 3.3, we illustrate knowledge transfer in generative models using a pre-trained model based on adult male data (Kohavi et al., 1996), subsequently fine-tuned with adult female data for downstream analysis. As demonstrated in Figures 3, 4, and Table 3, the fine-tuned model adeptly captures the data distribution, even with a limited size of raw samples.

This subsection presents sentiment classification applied to the benchmark dataset, IMDB (Maas et al., 2011). This task involves assigning emotions expressed in the text into positive or negative sentiments based on the opinions reflected in each text. The dataset comprises $50,000$ polarized movie reviews, categorized as “positive” or “negative” sentiments. These labels correspond to movie scores below four or above seven out of ten, where no movie has more than 30 reviews to prevent significant class imbalance. Here, we use $49,000$ of these reviews as our training data, reserving $1,000$ reviews for testing. We compare Syn’s generative approach against its conventional counterpart in a downstream prediction task, utilizing three models, GPT-3.5, DistilBERT, and LSTM models (Chen et al., 2023).

GPT-3.5 functions primarily as a text completion model, predicting the succeeding token as the sentiment label. Although essentially a completion model, GPT-3.5 is a conditional generative model that aligns with Syn-Slm, which we adapt for predictive tasks. In contrast, DistilBERT generates a fixed-size embedding of a review, which is then passed to an appended classification head to deduce sentiment likelihood. Unlike GPT-3.5’s token generation approach, DistilBERT’s technique aligns more closely with traditional predictive modeling with transfer learning for supervised tasks. Additionally, we train a traditional LSTM model from scratch, eschewing prior knowledge. Like DistilBERT, the LSTM processes an embedding and feeds it into a classification head, rendering it a predictive model.

Table 2 compares GPT-3.5, DistilBERT, and LSTM in six performance metrics. The extensive collection of pre-trained models likely contributes to GPT-3.5’s outstanding performance. On the other hand, LSTM’s underperformance stems from its inability to transfer knowledge. Knowledge transfer plays a crucial role in model performance. For details on training configurations, refer to the Supplementary Materials.

This subsection investigates the generational phenomenon and challenges associated with enhancing the precision of gradient-boosting for regression and classification (Breiman, 1997; Friedman, 2002). It also focuses on the implications for the quality of synthetic data generation. Within the Syn framework, we designate the boosting method tailored for synthetic data as Syn-Boost. Despite the surge of diverse predictive models, the capabilities of the Syn framework in predictive modeling remain largely untapped. To highlight this potential, we draw contrasts between Syn-Boost and its traditional supervised counterparts: specifically, the boosting algorithm — CatBoost (Dorogush, Ershov and Gulin, 2018) and FNN — a fully connected neural network that leverages insights from a pre-trained model, both of which are traditional. Syn-Boost presents a strategy to harness knowledge transfer in boosting, effectively addressing the transfer learning challenge inherent to the boosting method.

This subsection concerns testing the relevance of features for predicting the outcome of the response variable $Y$ by a machine learner using a candidate feature vector $\bm{X}$.

Define the subvector $\bm{X}_{S}$ by $\bm{X}_{S}=(X_{j}:j\in S)$, where $S$ is a subset of the features. Our objective is significance testing of $\bm{X}_{S}$ in its functional relevance to $Y$. To assess the influence of $\bm{X}_{S}$, we use the differenced risk $R(f^{*})-R(f_{S^{c}}^{*})$. Here, $f_{S^{c}}=f(\bm{X}_{S^{c}})$, and $f^{*}$ and $f^{*}_{S^{c}}$ are the optimal prediction functions in the population, defined as $f^{*}=\operatorname*{\arg\min}_{f}R(f)$ and $f^{*}_{S^{c}}=\operatorname*{\arg\min}_{f_{S^{c}}}R(f_{S^{c}})$. The risks, $R(f)$ and $R(f_{S^{c}})$, are given by $R(f)=\operatorname{E}\big{(}L(f(\bm{X}),Y)\big{)}$ and $R(f_{S^{c}})=\operatorname{E}\big{(}L(f(\bm{X}_{S^{c}}),Y)\big{)}$, where $\operatorname{E}$ represents the expectation. Now, we introduce the null and its alternative hypotheses $H_{0}$ and $H_{a}$:

Rejecting $H_{0}$ at a significance level implies feature relevance of $\bm{X}_{S}$ for predicting $Y$. It is worth mentioning that we target the population-level functions $f^{*}$ and $f^{*}_{S^{c}}$ in (5).

In (5), Dai, Shen and Pan (2024) developed an asymptotic test tailored to black-box learning models. Building upon this foundation, we illustrate how Syn-Test can bolster the power of this traditional test on raw samples by enlarging the synthetic data size while circumventing the necessity to derive the asymptotic distribution of a test statistic.

For Syn-Test, we adhere to Steps 1-4, as delineated in Section 2.3, to examine the relevance of feature set $\bm{X}_{S}$ to outcome $Y$, employing CatBoost (Dorogush, Ershov and Gulin, 2018) as the learning algorithm. Here, we engage a diffusion model, TDM (Kotelnikov et al., 2023), to generate synthetic data. Initially, we adapt the original test statistic from (Dai, Shen and Pan, 2024) to suit synthetic data as follows:

where $R_{m}(\cdot)$ denotes the empirical risk, evaluated on an inference sample $\tilde{\bm{Z}}_{1}^{(m,d)}$ in Step 1 and $\tilde{\bm{Z}}_{1}^{(m)}$ with $m=\hat{m}$ in Step 3 of Syn-Test, $\tilde{f}$ and $\tilde{f}_{S^{c}}$ denote the estimated predictive function function with and without $S$, and $SE$ denotes the standard error. Note that a large value of $T$ indicates the relevance of tested features to the response.

In (6), we reject $H_{0}$ if $T$ manifests as large. To compute the test statistic values in Steps 1 and 3, we generate an additional synthetic sample of size $2N$ and split it evenly into two subsamples. Using the first subsample, we train a CatBoost model $\operatorname{E}(Y|\bm{X})=f(\bm{X})$ to forecast $Y$ employing all features $\bm{X}$, resulting in $\tilde{f}$. In parallel, we train another CatBoost model $\operatorname{E}(Y|\bm{X}_{S^{c}})=f(\bm{X}_{S^{c}})$ using features $\bm{X}_{S^{c}}$, yielding $\tilde{f}_{S^{c}}$. By employing the synthetic sample to compute full and null predictive models, yielding $\tilde{f}$ and $\tilde{f}_{S^{c}}$, we can mitigate the intrinsic bias and asymptotics highlighted in (Dai, Shen and Pan, 2024) stemming from a limited inference size. This behavior is demonstrated in Figure 8 (middle row).

To refine a generative model under the null hypothesis, researchers often employ permutation by replacing redundant predictor vectors $\bm{X}_{S}$ with irrelevant values (Dai, Shen and Pan, 2024). However, this approach may not preserve the correlation structures between $\bm{X}_{S}$ and $\bm{X}_{S^{c}}$. Addressing this issue, we introduce a novel method that maintains these correlation structures while ensuring the feature irrelevance of $\bm{X}_{S}$ on $Y$ given the rest of the features. Our procedure involves two steps:

1. 1.

   We first train a predictive model to estimate $\operatorname{E}(\bm{X}_{S}|\bm{X}_{S^{c}})=g(\bm{X}_{S^{c}})$.

2. 2.

   We generate synthetic data tuples $(Y,\bm{X}_{S},\bm{X}_{S^{c}})$ using this model. Then, we modify these tuples by replacing $\bm{X}_{S}$ with the predicted values $g(\bm{X}_{S^{c}})$, resulting in new tuples $(Y,g(\bm{X}_{S^{c}}),\bm{X}_{S^{c}})$. This process ensures compliance with a specific subclass under the risk invariance of $H_{0}$ by creating conditionally independent tuples, as described by Dai, Shen and Pan (2024).

Finally, we designate an MC size of $D=1,000$ for estimating both the null distribution and the Type-I error in Step 1 of Syn-Test. The parameters are set as $\alpha=0.05$, $\varepsilon=0.01$, and the optimal $\hat{m}$ will be tuned based on the ratios $m/n\in\{1,2,\ldots,20\}$.