Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data

Gradient-boosted decision trees (GBDT) algorithms (e.g., XGBoost [5]) currently stand as the predominant models on most tabular prediction tasks due to their commendable data-efficiency and robustness [16, 8]. In view of the remarkable strides observed in deep learning across various fields [29, 28, 10], there has been a burgeoning interest in extending this success to the tabular data prediction tasks [3, 16, 4, 30, 11, 7, 20, 31]. Many of these approaches adopt a Transformer-like structure [31, 7, 26, 27] and demonstrate superior performance compared to alternative deep learning structures [8]. Yet, current deep tabular models still encounter several challenges: (i) Deep tabular models, serving as rotationally invariant approaches with a larger parameter space, demand substantial amounts of training data [21]. As a result, deep learning models typically yield suboptimal results on small-scale tabular datasets compared to data-efficient GBDTs [8]. (ii) Due to the absence of a “common” correlation on tabular data [33], most current deep tabular models are still trained separately on individual tabular datasets and have not effectively leveraged the benefits of transferring knowledge from other tabular datasets, a strategy successfully employed in other domains such as images and text data.

Unlike spatial and sequential correlations of features in images and texts consistently maintain coherence across various datasets [33], tabular data presents a unique challenge as it lacks common correlations [13]. This poses distinctive hurdles for cross-table pretraining on tabular datasets. Despite the numerous pretraining methods proposed for tabular data [3, 33, 26, 30, 35], the majority of these approaches concentrate on intra-table pretraining, which avoids confronting to handle the heterogeneity between datasets but also forfeits the opportunity to leverage reusable knowledge from other datasets. Recent studies have taken a step forward by achieving transfer learning on tables with shared label and feature spaces [30, 6, 18], facilitating to reutilize partially overlapped feature processors. Levin et al. [17] proposed a pseudo-feature approach to align upstream and downstream feature sets for table pretraining within the same domain. Despite this, challenges persist when extending these approaches to arbitrary cross-table pretraining. Recently, XTab [35] pioneered cross-dataset pretraining using composable components, leading to only marginal performance gains.

We utilize labeled datasets from the high-quality large tabular data benchmark OpenTabs V1 [32]. We systematically exclude datasets containing features with free text. Additionally, exclude duplicated datasets from the database, ensuring no overlap between the pretraining and downstream datasets. We opt to omit multi-class datasets in this work following [8], as a multi-classification task can be decomposed into multiple binary ones, and the multi-class datasets are relatively infrequent in tabular dataset collections. In total, we have 294 binary classification datasets and 366 regression datasets for pretraining, and evaluate the pretrained XTFormer on 190 downstream tasks. The 190 downstream tasks are organized upon 38 datasets (20 binary classification datasets and 18 regression datasets), and each datasets are used under 5 settings. In pre-processing, we divide each dataset into training and test sets using an 80:20 random split. Additionally, we performed random sampling to extract 20% of the training samples for validation purposes. The 5 settings are conducted by: (1) all the training data and validation data are used (T-full), and limited data scenarios: (2) 200 training samples with 50 validation data (T-200), (3) 100 training data with 25 validation data (T-100), (4) 50 training data with 13 validation data (T-50), and (5) 20 training data with 5 validation data (T-20). Detailed descriptions of target datasets are given in Appendix B.

We employ the data preprocessing strategy following the FT-Transformer [7] setting. The quantile transformation [22] is applied onto raw features, and standardization is applied to regression targets. We consolidate all datasets into a unified dataloader, which allows to randomly sample a dataset per pretraining step. Each dataset undergoes pretraining with the shared model body, accompanied by its dataset-specific embedding layer, output layer, and learnable context vector $\mathbf{v}$. We utilize a stack of 4 transformer blocks to construct the shared part of XTFormer, with the feature embedding size of 192 and 8 attention heads. The number of basis linear layer in CaLinear is set to 4 ($M=4$). In both the pretraining and fine-tuning phases, we utilize the AdamW optimizer [19] with a learning rate set to 1e-4. Consistent with [7], a weight decay of 1e-5 is applied to all components, excluding the embedding layer, layer normalization, and the bias terms. All experiments are executed on NVIDIA A100 GPUs 5 times with different random seeds, employing a batch size of 1024. When testing algorithms under limited dataset scenarios, the variation in random seeds also introduce changes to the sampled training and validation data. For pretraining, we execute an average of 350 epochs for each dataset. This involves incorporating a warm-up phase for the dataset-specific components during the initial 20% of steps, followed by a linear decay to zero. The default epoch amounts of task calibration for full dataset scenario and limited training data scenarios are 240 and 40, respectively. We set the default number of refinement epochs to 5. The best checkpoints on the validation set is used. For our XTFormer, we only use hyperparameters without tuning.

We compare our XTFormer with various representative supervised or pretrained tabular prediction models, including (i) the dominant GBDTs algorithms: XGBoost [5] and CatBoost [23], (2) cutting-edge deep tabular models including FT-Transformer (FT-T) [7], SAINT [26], AutoInt [27], as well as (3) the open-source cross-table pretraining model XTab [35]. All the algorithms trained through supervised learning undergo hyperparameter fine-tuning for better performances. The hyper-parameter tuning settings follow their original configurations and the settings outlined in [7]. Detailed information about hyperparameter fine-tuning can be found in Appendix A. For XTab [35], we use the source code and follow its specific settings that reports the best scores with the given pretrained checkpoint while keeping other hyperparameters constant.

With the diversity of tabular datasets, no single model outperforms every competitor across all datasets [7, 31, 8]. We thus compare model performances by: (i) performance ranking across all approaches, based on the accuracy for classification and the mean squared error for regression; following the practice in previous works [35, 11]; and (ii) a win-tie-loss analysis for a direct comparison between two approaches.

We compare XTFormer’s performance against existing algorithms 190 tasks (on 38 datasets across five different settings). The performance rankings are presented in Table 1 and the raw results are given in Appendix C. It is obvious that XTFormer consistently outperforms all other methods in every setting, especially when the training and validation data are highly limited. Except for our approach, generally, deep learning approaches cannot beat hyperparameter-tuned XGboost and Catboost in most settings, corroborating the conclusions drawn in previous researches [7, 8]. In order to further inspect whether our approach has altered the landscape of the tabular prediction domain, we conducted an one-on-one comparison for XTFormer, XGBoost, and Catboost against previous deep learning models, as shown in Fig. 4. It is clear that XGBoost and Catboost outperform or achieve comparable performance to existing deep learning approaches across a majority of datasets and settings, while these deep learning approaches demonstrate superiority in fewer setting. In contrast, neither XGBoost nor Catboost surpasses XTFormer, suggesting a transformative impact on the landscape of deep learning and GBDTs competition. One can observe that our model outperforms baseline approaches in the vast majority of tasks (typically over 70%). For example, it outperforms classic GBDTs, XGBoost and Catboost, on 137 (72%) tasks, FT-Transformer on 144 (76%) tasks, and the previous tabular pre-training model XTab on 162 (85%) tasks.

The number of basis functions in CaLinear directly impacts the dimensionality of the established meta-function space. Intuitively, a higher count of basis functions leads to a larger function space. In this study, we investigate the impact of varying the number of basis functions on model performance. We conduct experiments using 1, 2, 4, and 6 basis functions in each CaLinear, re-pretraining the XTFormer from scratch for each setting. Evaluation is performed on both the T-full and a data-insufficient setting (T-100), and performance rankings are computed separately. As shown in Table 2, the model exhibits improved performance with an increasing number of basis functions. Models with 4 or 6 basis functions markedly outperform those with 1 and 2. However, the difference between 4 and 6 basis functions is not significant. Actually, using 4 basis functions in XTFormer has already yielded a substantial meta-function space with $n^{2L}=4^{8}=65,536$ dimensions, which is sufficient to represent the majority of potential functions.

In this study, we devise a $\texttt{M}_{cal}$ to process a learnable context vector $\mathbf{v}$ to generate coefficients, aiming to identify dataset-suited functions in the meta-function space. An intuitive alternative is to directly assign learnable coefficients for each CaLinear and optimizing them to obtain coefficients. To compare these two approaches, we substitute the $\texttt{M}_{cal}$ in the framework with learnable coefficients (also employing softmax like $\texttt{M}_{cal}$) and present the comparative results in Table 3. Remarkably, $\texttt{M}_{cal}$ consistently outperforms the approach using vanilla learnable coefficients across all settings, outperforming on approximately $\nicefrac{{2}}{{3}}$ of the datasets. We attribute the observed superiority of using $\texttt{M}_{cal}$ to two key factors: (i) Coefficient Correlation Modeling: $\texttt{M}_{cal}$ can model the correlations among these coefficients, which is more effective for target model searching in the function space; (ii) Parameter Efficiency: $\texttt{M}_{cal}$ allows us to optimize only $N$ learnable parameters ($\mathbf{v}\in\mathbb{R}^{N}$, $N$ is the number of feature). In contrast, the alternative method requires to learn $kN$ coefficients directly, where $k$ represents the number of CaLinear ($k=8$ in this paper). Given the potentially limited training data in tabular datasets, a reduced parameter count generally results in better performance.

(i) pretraining epoch. We compare the results of the pretrained XTFormer at different pretraining epochs with a baseline XTFormer version without pretraining. Referring to Fig. 5, it is evident that pretraining yields a noteworthy performance enhancement over the first 50 pretraining epochs (per dataset). Following this, a gradual ascent phase occurs with the pretraining epoch number increases. Importantly, we note no significant performance decline, indicating the absence of both overfitting and collapse during pretraining, even as we constructed a meta-function space of up to 65,536 dimensions. (ii) task calibration epoch. We evaluate the impacts of fine-tuning step settings in the task calibration phases and refinement phases, on original (T-full) and data-insufficient (T-20, T-100) scenarios. See Table 4, it is evident that on full datasets, as the number of task calibration steps increases, the model demonstrates an increasingly positive trend in performance. However, in the scenario of extreme data inefficiency, the trend is entirely opposite: referring to Table 5 and Table 6, an increase in the number of task calibration steps correlates with a decline in performance. This might be because the overly extensive fine-tuning process tends to result in overfitting when the training and validation samples are extremely limited. (iii) refinement epoch. Upon reviewing Table 4, 5, and 6, it is apparent that the refinement phase offers additional performance gains (compared to #.R=0), especially in the T-full setting. Besides, it is also apparent that our few-epoch refinement setting (5 epochs by default) does not hurt the performances even in highly data-insufficient scenarios (T-20 and T-100), implying the robustness of our approach.

Recall that our calibration step aims to find a good transformation from embedding layer to the output layer (i.e. a good feature-target function). In Fig. 6(E), we collect coefficients for various features from both upstream (learned in pretraining) and downstream (before refinement) tables, and use t-SNE to visualize their distribution. Each point represents one feature (i.e. one feature-target relation) from one tabular dataset. In Fig. 6(A-D), we visualize the feature-target distribution of a cluster of coefficients, by plotting the mean activation after the embedding layer (denoted as $\overline{z}_{in}$) against that of the embedding before the dataset-specific output layer (denoted as $\overline{z}_{out}$)²²2Note that we are actually interested in the relation of the feature and target vectors (input and output of the shared body of XTFormer), but we plot only the mean of these vectors ($\overline{z}_{out}$ vs $\overline{z}_{in}$) because visualizing the relation of vectors is hard. . It is apparent that the pre-trained meta-function space exhibits a degree of smoothness - features whose coefficients are close in the t-SNE space also exhibit similar feature-target distributions per (A-D). More importantly, upstream and downstream points close in the t-SNE plot also shows similar feature-target distributions. This provides insights into why XTFormer works: For any new feature from a new dataset (a downstream red dot), $\texttt{M}_{cal}$ identifies good coordinates ($\mathbf{c}$) for it in the meta-function space such that its neighbors (nearby upstream black dots) are similar in terms of feature-target relation, as visibly confirmed in Fig. 6(A-D). Since $\mathbf{c}$ pinpoints the final feature-target function per Eq. 3, the resulting calibrated XTFormer is also suitable for processing features similar to this new feature. In short, our approach performs an automatic clustering for feature-target functions, aiding in the discovery and utilization of similar mapping patterns in cross-table transfer. Another example is given in Appendix D.