GPT in Data Science:
A Practical Exploration of Model Selection

Large Language Models (LLMs) and Generative Pre-trained Transformers (GPT) are integral parts of AI’s Natural Language Processing (NLP) realm. While LLM is a broad category encompassing models that predict word sequences and can be used for various tasks such as text generation and translation, GPT, developed by OpenAI [9], is a specific LLM type. GPT, renowned for generating text akin to human writing, undergoes extensive pre-training before fine-tuning for specialized tasks. In essence, GPT is a subclass of LLMs, but not all LLMs are GPT models. Other prominent LLM examples include BERT, RoBERTa, and XLNet.

Variability modeling, widely explored for developing automated systems, refers to the process of identifying and representing the variabilities—or the ability of a product, item, or feature to change, evolve, or be customized—in order to harness automation opportunities. This concept is crucial in fields like software engineering, databases, and data warehouses, where understanding the common and unique aspects of a system is essential. Feature modeling, a key aspect of variability modeling, involves capturing the common and variable attributes of a system using abstract entities known as features.

In this context, feature models stand out as intuitive and effective tools for representing the features of a variant-rich software system. They not only facilitate an overall understanding of the system but also support various stages of development, such as scoping, planning, development, variant derivation, configuration, and maintenance, thus contributing to the system’s long-term success[10]. Feature models formally represent features, their relationships, and constraints. This approach enhances understandability, traceability, explainability, and maintenance of the system.

In our proposed reverse engineering approach, we use an automated method based on massive neural networks for decision-making. By extracting variability models from these automated systems, we aim to leverage the benefits of variability models—such as traceability, understandability, explainability, and maintenance—in a novel context.

Furthermore, feature diagrams provide a structured, visual method for addressing the complexities in machine learning applications. These diagrams not only highlight the intricate relationship between modeling assumptions and algorithm choices but also underscore their impact on performance and other critical evaluation criteria, including fairness.

Large Language Models (LLMs) are increasingly integral to data science, showcasing their potential in diverse areas such as data preprocessing, analytics, and even drug discovery [3]. The study by Chopra et al. [2] highlights the use of LLMs in data science, particularly for tasks like data preprocessing and analytics. However, challenges arise in the interaction between data scientists and LLM-powered chatbots, especially in areas like contextual data retrieval, prompt formulation for complex tasks, and adapting generated code. These insights lead to the proposal of design improvements, including data brushing and inquisitive feedback loops, to enhance AI-assisted data science tools.

Tools like DataChat AI [4], Anaconda Assistant, and Databricks Assistant, LLMs enable data scientists to engage in conversational interactions about their data. This interaction includes asking follow-up questions and receiving context-specific responses, greatly enhancing the user experience and efficiency in data management and analysis. Finally, Troy et al. [5] focus on enabling generative AI to produce SQL statements. They propose a tool that generates syntactically valid language sentences and integrate AI algorithms for semantic guidance, demonstrating the capability of LLMs in generating structured queries for specific purposes, like detecting cyber-attacks.

These studies collectively underscore the expanding role of LLMs in data science, from enhancing analytics and data management to contributing significantly to fields like drug discovery. However, the challenges, including effective communication with AI assistants and adapting AI-generated solutions to specific contexts, remain crucial areas for further development.

In the top-down approach, we directly posed a question to GPT to elicit its criteria for selecting a data science model based on given dataset characteristics. The question was formulated as: “Given a dataset, which modeling assumptions/factors do you consider to selecting a data science model?” This query aimed to uncover the explicit decision-making criteria used by GPT. However, it’s important to note that the response presented below may not necessarily reflect the actual decision-making process used by GPT internally; it could represent an idealized or generalized answer suitable for the given question.

To complement the top-down analysis, we employed a bottom-up approach, which involved practical applications of GPT’s decision-making process. We selected three different toy datasets, each with a clearly defined target variable. For each dataset, we asked GPT to recommend the most suitable machine learning model(s) to address the specific problem presented by the dataset. The prompt was structured as follows: “Given the dataset {x}, with the target variable in column {column y}, and with the objective of {problem description z}, identify the most suitable machine learning model(s) to solve this issue. Explain your choice(s) and the underlying modeling assumptions and factors that guided your decision. Outline your decision-making process in detail. If multiple models are viable, rank them in order of preference, and describe the criteria for transitioning from one model to another in the evaluation process.”

Figure 6 presents the feature diagram showcasing the factors and assumptions used in GPT model selection, integrating insights from both top-down and bottom-up approaches. It categorizes these into three main groups under Modeling Assumptions: dataset assumptions, functional requirements, and non-functional requirements, following the framework proposed by Tavares et al. [8]. At the root of this diagram is the Modeling Technique Selection feature, which encompasses a wide range of machine learning techniques. Previous figures depicted only the ML techniques recommended for specific applications, whereas this comprehensive diagram includes all models for reference, as detailed in Tavares et al. [8].

Although ethical concerns were raised in the top-down query and we represent it in our Modeling Assumptions feature diagram, GPT did not consider these when proposing model selections for any of the three datasets.

Below is a concise set of constraints derived from the figure and GPT outputs, encapsulating dataset attributes, problem types, evaluation metrics, and computational capacity. The guidelines also consider the iterative process of model improvement, advocating for an adaptive approach—progressing from basic to advanced models in response to performance results and overfitting indicators. The following exemplifies the structuring of these constraints:

1. 1.

   Dataset Constraints:

   • •

     Size $\geq$ MinimumSizeRequirement $\land$ Features $\leq$ MaximumFeaturesAllowed

   • •

     DatasetType = {Numerical, Categorical, Binary, Text, Image, Time-series}

   • •

     DatasetQualityConstraints = $\neg$(MissingData $\vee$ Outliers $\vee$ Noise $\vee$ Unbalanced)

2. 2.

   Algorithm Type Constraints:

   • •

     ProblemType = $\{$Classification $\vee$ Regression $\vee$ Clustering $\vee$ Dimensionality Reduction$\}$

   • •

     If Classification $\Rightarrow$ AlgorithmOptions = $\{$DecisionTree, NaiveBayes, NeuralNetwork, SVM, K-Nearest Neighbors, Logistic Regression, EnsembleMethods, DeepLearningModels$\}$

   • •

     If Regression $\Rightarrow$ AlgorithmOptions = $\{$LinearRegression, PolynomialRegression, RidgeRegression, LassoRegression, ElasticNet, EnsembleMethods, DeepLearningModels$\}$

   • •

     If Clustering $\Rightarrow$ AlgorithmOptions = $\{$K-Means, Hierarchical Clustering, DBSCAN, Gaussian Mixture Models, Mean Shift$\}$

   • •

     If Dimensionality Reduction $\Rightarrow$ AlgorithmOptions = $\{$PCA, t-SNE, LDA, QDA, Autoencoders$\}$

3. 3.

   Evaluation Criteria Constraints:

   • •

     If ProblemType = Classification $\Rightarrow$ PerformanceMetrics = {Accuracy, Precision, Recall, F1Score, AUC-ROC}

   • •

     If ProblemType = Regression $\Rightarrow$ PerformanceMetrics = {RMSE, MAE, R²}

   • •

     If ProblemType = Clustering $\Rightarrow$ PerformanceMetrics = {SilhouetteScore, DaviesBouldinIndex, CalinskiHarabaszIndex}

   • •

     If ProblemType = DimensionalityReduction $\Rightarrow$ PerformanceMetrics = {ReconstructionError, ExplainedVarianceRatio}

4. 4.

   Non-Functional Requirements Constraints:

   • •

     EvaluationCriteria $\Rightarrow$ (Interpretability $\vee$ Robustness $\vee$ Fairness $\vee$ Privacy $\vee$ Transparency $\vee$ Generalizability $\vee$ EthicalConsiderations)

   • •

     ModelComplexity $\Rightarrow$ (Simple $\vee$ Complex)

5. 5.

   Computational Resource Constraints:

   • •

     If LimitedResources $\Rightarrow$ Prefer {ModelsWithLessComplexity $\vee$ ReducedFeatureSets}

   • •

     If SufficientResources $\Rightarrow$ Capability to Implement {EnsembleMethods $\vee$ DeepLearningModels}

6. 6.

   Transition and Iterative Improvement Constraints:

   • •

     InitialModelSelection $\Rightarrow$ SimplerModels

   • •

     PoorPerformance $\Rightarrow$ TransitionToMoreComplexModels

   • •

     If OverfittingDetected $\Rightarrow$ ApplyRegularization $\vee$ ParameterTuning $\vee$ ModelSelectionReevaluation

In summary, this section presents the factors and variabilities that GPT takes into account when selecting a model for a dataset. This diagram and constraints can be further adapted to examine model selection for additional datasets, employing GPT’s heuristics to interpret decisions made by an automated GPT-based system, or to conduct tests and verification processes investigating the relationships between these factors.

Following [14], we divided our dataset into an 80% training set (239 patients) and a 20% test set (60 patients), using a stratified split to accommodate the dataset’s imbalance.

The Sklearn heuristic recommended LinearSVC, KNeighborsClassifier, SVC, and EnsembleClassifiers. In contrast, the GPT heuristic suggested Logistic Regression, Random Forest, Gradient Boosting Machines, and Support Vector Machine. Model selection was based on performance metrics like accuracy, precision, recall, and AUC-ROC, as suggested by GPT.

We executed the following models for the dataset:

Results including the top two models from GPT (Logistic Regression and Random Forest), the top two from Scikit-learn (LinearSVC and KNeighborsClassifier), and the baseline model (Random Forest) as used in [14] are:

Logistic Regression, LinearSVC, and RandomForestClassifier showed identical accuracy and precision. However, Logistic Regression (suggested by GPT) demonstrated superior ROC-AUC performance. ROC-AUC is a critical metric for binary classifiers, especially in imbalanced datasets, as it quantifies the model’s ability to distinguish between classes across all thresholds.

In the Diabetes Prediction subsection, we compared the effectiveness of models suggested by both Scikit-learn heuristics and GPT outputs. The Scikit-based heuristic recommended the SGDClassifier, and kernelApproximation, with alternatives like Linear SVC, Kneighbors Classifier, SVC, and Ensemble Classifiers for datasets under 100K samples. The GPT heuristic, on the other hand, suggested Logistic Regression, Decision Trees, Random Forests, GBM, and Neural Networks, emphasizing the transition between models based on performance metrics such as accuracy and AUC-ROC.

Our execution and evaluation of these models on the dataset revealed notable results. The Logistic Regression model, recommended by GPT, achieved an accuracy of 0.9604, precision of 0.8592, recall of 0.6388, and a roc_auc of 0.9625. In contrast, the SGDClassifier, as suggested by Scikit, showed similar accuracy (0.9599) and precision (0.8598), but slightly lower recall (0.6312) and identical roc_auc (0.9625).

Notably, the RandomForestClassifier and GradientBoostingClassifier, also recommended by GPT, outperformed both in terms of accuracy, with scores of 0.9721 and 0.9723 respectively, and showed superior recall and precision. This comparison indicates that while the Scikit-based heuristic offers competitive models, the GPT-suggested models, particularly RandomForestClassifier and GradientBoostingClassifier, display a slight edge in overall performance for this specific dataset.

In the Car’s Price Prediction subsection, we evaluated models suggested by both the Sklearn-based heuristic and GPT outputs. The Sklearn heuristic recommended RidgeRegression, SVR(kernel=linear), SVR(kernel=rbf), and EnsembleRegressors, whereas GPT suggested Random Forest Regressor, Gradient Boosting Regressors (like XGBoost, LightGBM), Linear Regression, Ridge/Lasso Regression, and Support Vector Regressor (SVR). Model performance was assessed based on RMSE, MAE, and R² score.

Upon execution, the Ridge Regression model, recommended by Sklearn, achieved an RMSE of 339373.6684, MAE of 122614.1828, and an R2_score of 0.6226. The SVR with RBF kernel, another Sklearn suggestion, however, had a significantly higher RMSE (568437.5465) and lower R2_score (-0.0588).

Comparatively, the Random Forest Regressor, a GPT suggestion, showed an RMSE of 360838.8792 and an R2_score of 0.5733, indicating slightly lower performance than Ridge Regression but notably better than SVR with RBF kernel. The GradientBoostingRegressor, also recommended by GPT, had an RMSE of 354280.1772 and an R2_score of 0.5887, performing comparably to the Random Forest Regressor.

Linear Regression and Lasso, both GPT suggestions, showed competitive performance, with RMSEs of 340865.1113 and 358057.2296, respectively, and similar R2_scores.

This comparison highlights that while Sklearn-based heuristics offer robust models like Ridge Regression, GPT’s recommendations, especially GradientBoostingRegressor and Random Forest Regressor, are equally competitive in terms of RMSE and R2_score. The overall performance of models suggested by GPT demonstrates their efficacy in handling this specific dataset for car price prediction.

This section presents an overview of the results from evaluating the efficacy of model selection heuristics provided by GPT and Scikit-learn engineers across different datasets. Our findings reveal interesting implications for the application of Large Language Models (LLMs) like GPT-4 in data science. In the classification datasets, the models suggested by GPT outperformed those recommended by Scikit-learn heuristics. Conversely, in the regression dataset, the model suggested by Scikit-learn engineers showed superior performance compared to GPT’s suggestions.

However, it’s important to note that several factors can influence these outcomes. The way datasets are split, parameter selection for algorithms, and the specific characteristics of each dataset play significant roles in the performance of the suggested models. These factors need to be carefully considered when interpreting the results and their applicability to real-world scenarios.

Overall, the results are positive and encouraging. We were successful in mapping the decision-making process of GPT for the selection of each algorithm. The suggestions provided by GPT achieved satisfactory results in all datasets, demonstrating its potential as a valuable tool in model selection.

A comprehensive investigation of various LLMs will provide insights into their unique decision-making processes. Establishing clear evaluation standards will enable us to compare and contrast these models. The integration of different generative models with data connectors, as exemplified in LangChain, offers a fertile ground for exploring how different LLMs interact and perform in a variety of contexts.

Prompt engineering and its influence on model heuristics is a significant aspect to explore. By adopting the approach proposed by [15], where LLMs are used to simulate multiple humans and replicate human subject studies, we can investigate how simulating data scientists with varied professional experiences impact GPT’s model selection. This approach aligns with the concept of Turing Experiments for language models, which aim to evaluate the extent to which language models can simulate different aspects of human behavior.

Our investigation revealed that while GPT’s strategic framework (top-down approach) acknowledges ethical considerations, these elements are not as emphasized in the practical application of the models. This gap suggests an area for future work, where we can explore potential biases and ethical implications of AI-assisted model selection, and how prompt engineering could be leveraged to explicitly include these ethical considerations. By incorporating these factors, we can ensure that the AI’s decision-making process not only focuses on performance metrics but also aligns with ethical guidelines and best practices.

One of the critical challenges with LLMs is addressing ‘hallucinations’ [16] – instances where the models generate contextually irrelevant or factually incorrect responses. A deeper analysis of the internal decision-making processes of these models is necessary to understand and mitigate these risks, which are crucial for the trust and safety in AI applications.

Applying our findings to larger datasets and more complex applications will significantly enhance our understanding of LLM decision-making. This will involve not only scaling up the size of the datasets but also tackling more challenging scenarios that reflect real-world complexity. Such applications will provide valuable insights into the robustness and adaptability of LLMs in diverse and demanding environments.