Exploring Variability in Fine-Tuned Models for Text Classification with DistilBERT

The work on Rethinking Learning Rate Tuning in the Era of Large Language Models [1] focuses on the critical role of learning rate in fine-tuning. The study proposes LRBench++, a tool for evaluating learning rate policies across LLMs and traditional DNNs, highlighting the need for tailored learning rate policies for LLMs. In contrast, our study expands beyond a singular focus on learning rate to evaluate the combined effects of learning rate, batch size, and epochs. Additionally, our approach incorporates both absolute and relative analyses of accuracy, F1-score, and loss to provide a holistic view of hyperparameter impacts on DistilBERT’s performance in text classification.

A recent study, Evaluating the Effectiveness of Fine-Tuning Large Language Models for Domain-Specific Tasks [2], fine-tuned LLaMA 2 using two distinct methods on a migration-related news dataset, illustrating the impact of self-supervised versus direct Q&A fine-tuning strategies. While this study concentrates on strategy effectiveness in aligning the model with domain-specific content, our work evaluates a broader range of hyperparameters and their trade-offs across core metrics. Unlike their focus on dataset alignment, we derive generalized fine-tuning strategies through polynomial regression analyses, applicable across different datasets and tasks.

Fine-Tuning Large Language Models for Task-Specific Data [3]examines the use of Parameter-Efficient Fine-Tuning (PEFT) and Quantized Low-Rank Adaptation (QLoRA) techniques to fine-tune LLaMA 2 and Falcon 7B models for an e-commerce dataset. This study demonstrated the contextual accuracy benefits of customized fine-tuning. In contrast, our study targets the foundational impacts of basic hyperparameters (learning rate, batch size, and epochs) on multiple performance metrics, producing insights for balanced tuning strategies rather than domain-specific optimizations.

The study Fine-Tune it Like I’m Five: Supporting Medical Domain Experts in Training NER Models Using Cloud, LLM, and Auto Fine-Tuning [4]introduces CRM4NER, a cloud-based system for managing, training, and fine-tuning Named Entity Recognition (NER) models. It incorporates automatic hyperparameter tuning and context-aware fine-tuning recommendations generated via a Large Language Model (LLM), aiming to support domain experts who may lack expertise in Machine Learning (ML). While CRM4NER emphasizes a user-friendly ML management system, our study directly assesses the fine-tuning effects of hyperparameters on model metrics, guiding users on tuning DistilBERT for optimal performance without requiring complex management systems.

Fine Tuning LLMs for Low Resource Languages [5] addresses fine-tuning strategies for optimizing LLMs in low-resource language settings, exploring language-specific techniques such as LoRA, QLoRA, instruction tuning, and Representation Fine-Tuning (ReFT). This study emphasizes language-specific tuning needs, contrasting with our approach, which examines hyperparameter impacts on DistilBERT’s text classification performance regardless of language resources. We focus on understanding the metric-specific influences of core hyperparameters, providing insights that are applicable across diverse linguistic contexts.

Achieving Peak Performance for Large Language Models: A Systematic Review [6] provides a comprehensive systematic literature review (SLR) of optimization and acceleration methods for large language models (LLMs), focusing on balancing high performance with practical constraints. While this review categorizes optimization strategies into LLM training, inference, and system serving, our work offers a more focused analysis of hyperparameters that directly affect DistilBERT’s classification performance. Our study contributes a metric-specific understanding of hyperparameter trade-offs, complementing high-level efficiency optimizations discussed in the review.

The study Fine-Tuned Understanding: Enhancing Social Bot Detection With Transformer-Based Classification [7] uses transformer-based models like BERT and GPT-3 to detect social media bots, achieving high F1-scores through fine-tuning. Unlike this bot-detection focus, our research evaluates the interplay of hyperparameters on general performance metrics. Our work seeks to balance multiple metrics rather than targeting domain-specific detection accuracy, expanding the application of our insights beyond niche uses.

LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding [8] uses a layout-aware fine-tuning approach for document understanding, incorporating layout-aware pre-training and supervised fine-tuning to enhance the comprehension of document structures. Our study differs by exploring fundamental hyperparameter adjustments rather than layout-specific tuning strategies, providing adaptable insights for general text classification applications across various NLP tasks.

Several studies have investigated fine-tuning DistilBERT for specific applications:

• •

  Cyberbullying Detection in Social Networks: A Comparison Between Machine Learning and Transfer Learning Approaches [9] used DistilBERT embeddings to detect abusive content with high F1-scores.

• •

  Hate Speech and Target Community Detection in Nastaliq Urdu Using Transfer Learning Techniques [10] applied DistilBERT to identify hate speech in low-resource languages, demonstrating the model’s adaptability in multilingual contexts.

• •

  Depression Classification From Tweets Using Small Deep Transfer Learning Language Models [11] employed DistilBERT for social media-based depression classification, comparing it with smaller models.

• •

  AGI-P: A Gender Identification Framework for Authorship Analysis Using Customized Fine-Tuning of Multilingual Language Model [12] developed AGI-P, fine-tuning DistilBERT for gender identification in a multilingual setting.

• •

  Performance Analysis of Federated Learning Algorithms for Multilingual Protest News Detection Using Pre-Trained DistilBERT and BERT [13] fine-tuned DistilBERT within a Federated Learning framework for protest news detection.

These studies leverage DistilBERT for specific text classification applications but focus primarily on single-task or dataset-specific fine-tuning effects. Our study, however, explores hyperparameter-driven variability across multiple metrics (accuracy, F1-score, and loss) in DistilBERT’s performance. By examining both absolute and incremental changes from a baseline model, our research uniquely contributes a comprehensive evaluation framework that informs adaptable fine-tuning strategies for balanced performance.

The data collection process involved identifying and extracting information about hyperparameters and performance metrics from models available on the Hugging Face platform. Our goal was to select NLP models specialized in text classification and gather relevant data from associated files.

• •

  Model Filtering: Selected the Natural Language Processing (NLP) category from Hugging Face’s available options.

• •

  Task Selection: Filtered for Text Classification, yielding 30 available models.

• •

  Base Model Selection: Chose DistilBERT (distilbert-base-uncased-finetuned-sst-2-english) due to its popularity and high number of downloads.

• •

  Fine-Tuned Models Extraction:

  • –

    Retrieved detailed information from 55 fine-tuned models associated with DistilBERT.

  • –

    Data Sources: Extracted from README.md and config.json files for each model.

The model selection and task design aimed to explore the efficiency and performance of fine-tuned models in text classification.

• •

  Base Model Selection: Chose DistilBERT (distilbert-base-uncased-finetuned-sst-2-english) for its efficiency and popularity.

• •

  Objective: Evaluated the impact of different hyperparameter configurations and compared results with other fine-tuned models and NLP tasks.

• •

  Task Design:

  • –

    Evaluated all 55 fine-tuned models to ensure consistent analysis across metrics.

We began our analysis with a comprehensive set of nine hyperparameters and nine evaluation metrics, following the initial configurations recommended in the model documentation. To streamline our focus, we refined this selection based on the frequency of observations across the 55 models analyzed. This approach enabled us to prioritize hyperparameters and metrics that consistently appeared and provided statistically significant insights into model performance.

Due to the limited statistical significance observed across a broader range of metrics and hyperparameters, our study focused on metrics and hyperparameters with a sufficient number of observations to allow for statistically significant fine-tuning strategies. This selection aimed to ensure a robust analysis of the most impactful elements on model performance.

• •

  Evaluation Metrics: We concentrated on three key metrics that provided enough observations to support meaningful statistical exploration:

  • –

    Accuracy - Selected as a primary measure for the overall correctness of model predictions, offering insight into general model performance.

  • –

    F1-Score - Focused on due to its balanced consideration of precision and recall, particularly important for assessing performance in cases with class imbalance.

  • –

    Loss - Included as it quantifies model prediction error, essential for evaluating the model’s error minimization during training and validation.

• •

  Hyperparameters: Three core hyperparameters were chosen, as they provided enough data points to examine statistically significant trends in fine-tuning:

  • –

    Learning Rate - An influential parameter for training speed and stability, providing sufficient variability in observations for meaningful analysis.

  • –

    Batch Size - Affects memory usage and convergence, offering adequate data to explore its relationship with model generalization and performance.

  • –

    Number of Epochs - Key in controlling the number of training iterations over the dataset, providing essential insights into learning and convergence patterns.

By focusing exclusively on these metrics and hyperparameters, we ensured a statistically valid basis for identifying consistent patterns and potential fine-tuning strategies. This approach provides a clearer understanding of core dependencies and allows for targeted adjustments in fine-tuning aimed at optimizing model performance in a statistically sound manner.

To assess the impact of selected hyperparameters on evaluation metrics, we employed polynomial regression analyses across two versions—Absolute and Relative. These analyses aimed to identify patterns, correlations, and potential optimizations through detailed regression models, heatmaps, scatter plots, and regression plots.

• •

  Polynomial Regression Analyses: We conducted polynomial regressions to examine the influence of hyperparameters (Learning Rate, Batch Size, and Number of Epochs) on key evaluation metrics (Accuracy, F1-Score, and Loss) in two distinct versions:

  • –

    Absolute Regression: This version utilized the absolute values of both the metrics and hyperparameters. It aimed to reveal the direct effects of hyperparameter values on model performance metrics (Accuracy, F1-Score, Loss) as observed post-training. Absolute regression thus reflects the foundational impact of hyperparameters on baseline metric performance.

  • –

    Relative Regression: In the relative regression, we used the differences in metric values between each fine-tuned model and the baseline model, as well as the differences in the hyperparameters applied in fine-tuned versus baseline models. This version provides insights into the incremental improvements and optimization effects of fine-tuning adjustments, specifically measuring how hyperparameter variations contribute to performance gains beyond the baseline.

• •

  Heatmaps and Scatter Plots: We generated heatmaps to visualize correlations between hyperparameters and metrics, identifying patterns that could inform fine-tuning adjustments. Scatter plots illustrated the spread and potential trade-offs between hyperparameters and metric outcomes, facilitating a more granular understanding of parameter interactions.

• •

  Regression Plots: In conjunction with the polynomial regression analyses, we produced regression plots to visualize fitted regression lines and residuals. These plots provided additional insights into model fit and variability, helping to identify areas where hyperparameter adjustments yield statistically significant improvements.

The objective of this subsection is to systematically evaluate the consistency of optimization strategies across the key metrics of accuracy, F1, and loss. By analyzing both absolute and relative regression results, we aim to identify patterns that highlight common and divergent impacts of hyperparameters across these metrics. This approach allows us to observe how adjustments to batch size, learning rate, and epochs influence each metric and to determine whether improvements in one metric, such as accuracy, align or conflict with performance outcomes in F1 and loss. Ultimately, this evaluation provides insights into cross-metric coherence, informing the design of balanced fine-tuning strategies that prioritize comprehensive model performance.

This subsection synthesizes the insights captured for each metric based on both absolute and relative regression analyses to formulate actionable fine-tuning strategies. By mapping hyperparameter impacts on accuracy, F1, and loss, we can derive specific, data-driven recommendations that address the unique optimization needs of each metric while also accounting for their cross-effects. The objective is to enable a strategic approach to fine-tuning that emphasizes incremental gains in each metric while maintaining overall model stability. These strategies highlight which parameters to prioritize initially, such as batch size and epochs, and where to make final refinements, particularly with learning rate adjustments, to achieve balanced improvements in accuracy, F1, and loss.

The implementation of our analyses leveraged several computational packages and libraries to streamline data processing, model training, statistical analysis, and visualization. Below is a summary of the key packages used in our study:

• •

  Python Core Libraries:

  • –

    NumPy - Provided efficient handling of numerical data and supported array-based computations essential for data manipulation and model input formatting.

  • –

    Pandas - Facilitated data manipulation, cleaning, and preparation, allowing us to manage datasets, perform feature engineering, and store results for analysis.

• •

  Statistical Analysis and Metrics:

  • –

    SciPy - Enabled statistical testing and analysis, including regression calculations and significance testing on hyperparameter impacts.

  • –

    Scikit-Learn - Supported computation of evaluation metrics (such as Accuracy, F1-score, Precision, and Recall) and provided tools for model evaluation, data splitting, and scaling.

• •

  Visualization and Plotting:

  • –

    Matplotlib and Seaborn - Used to generate detailed visualizations, including heatmaps, scatter plots, and regression plots. These visualizations facilitated a clearer understanding of hyperparameter effects and metric distributions across different configurations.

This section presents the results of analyzing the relationships between the dependent variables—accuracy, F1-score, and loss—and the independent variables (learning rate, batch size, and number of epochs). We report both absolute and relative regression analyses, along with heatmaps and scatter plots that help visualize the correlations and spreads across different hyperparameter configurations.