Adaptable and Reliable Text Classification using Large Language Models

Text classification has evolved through various machine learning (ML) methods, each with its strengths and limitations. Early rule-based approaches, such as decision trees like C4.5 [12], were simple but prone to overfitting and lacked flexibility. Probability-based models, such as Multinomial Naive Bayes (MNB) [13] and Hidden Markov Models [14], improved generalization, particularly in tasks like spam detection and speech recognition. Geometry-based methods, including support vector machine (SVM) [15], handled high-dimensional data but struggled with large datasets. Finally, statistical methods like K-nearest neighbors (KNN) [16] and Logistic Regression (LG) [17] provided effective solutions but required extensive preprocessing and often faltered with nonlinear data.

Deep learning (DL) has become a key technology in text classification, capable of handling complex language features. Convolutional neural networks (CNNs) text classification models [18] capture local textual features through convolutional layers. LSTM [19] and GRU [20], as optimized versions of RNNs, are particularly effective in addressing long-distance dependencies in text. [21] proposes an optical character recognition and classification method for cigarette laser code recognition, using a convolutional recurrent neural network to extract image features and utilizing BiLSTM for text classification. Transformer models, like BERT [22], achieve remarkable results in various NLP tasks by utilizing self-attention mechanisms. Specifically, the BERT model demonstrates powerful capabilities in text classification tasks. [23] proposes the text recognition framework Nbias for detecting and eliminating biases, including data, corpus construction, model development, and evaluation layers. The dataset is collected from various fields, and a transformer-based token classification model is applied. [24] proposes a semi-supervised generative adversarial learning method that improves the model’s classification performance with limited annotated data through generative adversarial networks. [25] introduces a hybrid model that combines BERT, LSTM, and Decision Templates (DT) for IMDB and Drug Review classification. However, these methods typically require substantial data for training, often necessitating extensive datasets to achieve optimal performance. This reliance on large training sets can pose challenges, especially when collecting or labeling data is difficult or impractical.

LLMs represent a significant advancement in the field of text classification, building on the DL foundations that have revolutionized NLP. These models, which include notable examples such as GPT [26], T5 [27], RWKV [28], Mamba [29], Gemini [30], PaLM [6], Llama [7], and Claude [31], leverage massive amounts of data and extensive training regimes to understand and generate human-like text. Their ability to capture nuanced language patterns and context makes them highly effective for text classification tasks across various domains.

Recent studies have begun to explore the practical applications of LLMs in specialized fields. For instance, [32] study the application of LLMs in sociological text classification, demonstrating their potential in social science research. [33] examine the performance and cost trade-offs when employing LLMs for text classification, focusing on financial intent detection datasets. Another study by [34] investigates the effect of fine-tuning LLMs on text classification tasks within legal document review, highlighting how domain-specific adjustments can enhance model performance. Furthermore, the research identified as [35] refines LLM performance on multi-class imbalanced text classification tasks through oversampling techniques, addressing one of the common challenges in ML.

Despite their strengths, there remains a gap in making LLMs accessible to users without deep technical expertise. Our system addresses this by leveraging pre-trained LLMs as out-of-the-box classifiers that require minimal adaptation. This system democratizes access to advanced NLP tools, offering scalable solutions for diverse applications without the steep learning curve typically associated with LLM deployment.

Our proposed system integrates LLMs to refine the traditional text classification system, as illustrated in Figure 3 based on our previous work [36]. The framework of our system presents a comprehensive strategy that capitalizes on the strengths of LLMs while mitigating their traditional limitations.

Initially, our system aggregates data from many sources, either public or private. Unlike traditional ML/NN methods, which often require extensive retraining or fine-tuning when confronted with new data types, our LLM-based system can effectively adapt to these varied inputs without additional training. This versatility is one of the key strengths of our approach.

Subsequently, the system harnesses domain-specific data through zero-shot prompting or few-shot learning techniques or by fine-tuning a pre-trained LLM. This adaptive phase meticulously tailors the LLM’s capabilities to suit the target domain’s particular linguistic features and contextual subtleties, thereby bolstering accuracy and relevance for classification tasks.

Furthermore, the involvement of domain knowledge is crucial but optional. They configure the system by establishing customized prompts that direct the LLM toward generating pertinent and contextually aware responses. This human-in-the-loop methodology guarantees that the system adheres to specific domain requirements and can adeptly manage intricate query scenarios.

Additionally, an LLM API serves as an intermediary between the model and user interface, enabling seamless real-time interactions. Through this user-friendly interface, users without expertise can effortlessly query the system for advice, classification results, sentiment assessments, predictions, or recommendations based on their input.

Lastly, our system incorporates an evaluation subsystem dedicated to continuously monitoring LLM performance. It scrutinizes accuracy and error rates while observing model behavior over time. Such vigilance facilitates perpetual enhancements and updates via model versioning and caching strategies.

By amalgamating these components, our system simplifies and elevates text classification processes in terms of adaptability and precision. It significantly diminishes reliance on domain knowledge for complex preprocessing or algorithmic design tasks—thereby democratizing access to cutting-edge NLP technologies across various sectors such as e-commerce and social media analytics.

We utilized several key metrics to assess the performance of LLMs as text classifiers. These metrics provide insights into the accuracy, precision, recall, and stability of the LLMs in handling classification tasks.

The first dataset consists of tweets related to the COVID-19 pandemic, curated by [37]. As shown in Table I, it comprises a total of 41,156 training instances and 3,798 test instances, categorized into negative, neutral, and positive sentiments.

The second dataset includes economic texts compiled by [38], designed for sentiment analysis within the financial domain. The dataset contains 3,876 training samples and 969 test samples distributed across negative, neutral, and positive classes, as detailed in Table II. This dataset includes 5 levels of sentiment, which were merged into 3 levels in this study.

For multi-class classification tasks beyond binary or ternary sentiment analysis, we utilize an e-commerce texts dataset provided by [39]. The training set includes 40,338 instances, while the test set contains 10,085 instances, as outlined in Table III.

Lastly, we incorporate an SMS Spam Collection dataset assembled by [40] to evaluate spam detection performance. This binary classification task involves distinguishing between normal messages and spam with a total of 4,457 training messages and 1,115 test messages presented in Table IV.

Our experiment setup is designed to evaluate the performance of various models across different categories, ensuring a comprehensive analysis of the proposed methods. The models are categorized as follows:

• •

  Traditional ML Algorithms: This category includes MNB, LG, RF, DT, and KNN.

• •

  NN Architectures: We utilize advanced deep neural network models such as RNN, LSTM, and GRU.

• •

  Zero-shot Learning (ZSL) Models: We explore zero-shot learning capabilities using transformer-based models, including BART (facebook/bart-large-mnli) and DeBERTa (microsoft/deberta-large-mnli).

• •

  LLMs: State-of-the-art LLMs including closed source models: GPT-3.5(gpt-3.5-turbo-0125), GPT-4 (gpt-4-1106-preview), Gemini-pro, and open source models: Llama3-8B(Llama3-8B-Instruct), Qwen-Chat(7B and 14B), and Vicuna-v1.5(7B and 13B) were assessed.

To maintain consistency in evaluation, the input processing was standardized for all traditional ML algorithms and NN architectures. Each model receives the same processed text derived from a uniform raw text processing pipeline applied to training and testing datasets. This standardization ensures that any observed variations in performance can be attributed more directly to the intrinsic capabilities of each model rather than disparities in input processing.

Conversely, unprocessed raw text from the testing dataset was used to fully leverage their natural language understanding abilities for zero-shot learning models and LLMs. It is important to note that this testing dataset remains consistent across all model types to provide a fair comparison.

In addition to these measures, we implemented a sampling strategy for dataset selection that respects the original label distribution within both training and test sets:

• •

  For datasets with more than 10,000 instances in their training set, only 10,000 instances were selected while preserving the original label distribution proportionally through stratified sampling.

• •

  Similarly, for test sets with more than 800 instances, only 800 instances were chosen based on their original label distribution.

This approach ensures that smaller datasets are fully represented while larger ones are sampled appropriately without introducing bias or altering their inherent class distributions.

Furthermore, when configuring prompts for LLMs within the experiments, uniformity is ensured by keeping prompts identical across different LLMs for the same dataset. When dealing with different datasets, a consistent core structure is maintained within prompts—only adjusting labels and dataset names as necessary—to minimize variability due to prompt differences.

Table V, VI, VII and VIII present the experimental results for all the models. Notably, when employing few-shot strategies or fine-tuning, they are indicated by “(S)” and “(F),” respectively.

Table V presents results for the COVID-19-related text dataset. All models have relatively low performance except the fine-tuned LLM model of Qwen-7B, which performed the best in all metrics with $0.8388$ in accuracy and $0.8433$ F1 score and clearly provided all the answers.

Traditional algorithms show poor accuracy and F1 scores. In contrast, NN-based models demonstrate superior performance, with GRU leading in both ACC and F1 metrics. Among LLMs, before fine-tuning, GPT-3.5 exhibits the highest ACC and F1 scores, outperforming other LLMs, including GPT-4, while the performance is below NN methods. However, once the fine-tuning method was employed, the Qwen-7B(F) outperformed all the other models, including GRU’s best model. The best accuracy increased from 0.6913, performed by GRU, to 0.8388, and the F1 score from 0.63332, performed by RNN, to 0.8433.

Table VI presents results for the e-commerce product text classification dataset. The GRU model shows the best performance among all models except for fine-tuned LLMs with an accuracy of $0.9387$ and an F1 score of $0.9383$, making it the leading model in these categories before considering fine-tuning. This illustrates the capability of GRU to handle sequence and context effectively, which is crucial for product text classification. Like Table V, traditional algorithms exhibit much lower accuracy and F1 scores than NN-based models. Among LLMs, GPT-based models also show impressive results before fine-tuning, with GPT-3.5 achieving slightly higher metrics than GPT-4. Applying fine-tuning techniques to LLMs such as Qwen-7B can result in superior accuracy of $0.9713$ and F1 scores of $0.9713$, making these models particularly effective for specialized tasks such as e-commerce product text classification.

Table VII presents results for the economic texts sentiment classification dataset. The models show a broad performance spectrum, with the best results observed in fine-tuned LLMs. Traditional models continue to exhibit relatively low accuracy and F1 scores. RF performs somewhat better within this group but remains significantly lower than advanced models with a $0.6375$ accuracy and a $0.4048$ F1 score. NN-based models perform adequately, with GRU notably achieving an accuracy of $0.6837$ and an F1 score of $0.5494$. However, their performance is outstripped by more sophisticated models. LLM models, like GPT-4 and Gemini-pro, show significant improvements over traditional models, with GPT-4 reaching an accuracy of $0.7638$ and an F1 score of $0.7659$, indicating robust capabilities in processing complex economic texts. The fine-tuned models Llama-3-8B(F) and Qwen-7B(F) exhibit exceptional performance, with Qwen-7B(F) standing out for its remarkable accuracy and F1 score improvements. It is the only model that surpasses 80% accuracy and F1 score.

Table VIII details the SMS spam collection classification results, showcasing a notable disparity in model effectiveness, with fine-tuned LLMs and NN-based models outperforming others by a wide margin. Once again, traditional models underperform compared to NN and some LLM models, with RF leading the traditional pack but not nearly matching the performance of advanced models. NN-based models show exceptionally high performance, with RNN achieving the best results with an accuracy of $0.9725$ and an F1 score of $0.9366$. For LLMs, while some models demonstrate their high abilities in detecting spam SMS with accuracy high to more than 90%, like GPT-4 and Qwen-14B, some models failed in this task with accuracy lower than 0.5, like GPT-3.5, Llama-3-8B, and Vicuna families which are far worse than traditional ML methods or NN models. Notably, fine-tuning dramatically enhances the performance of models like Llama-3-8B(F) and Qwen-7B(F), which achieved the highest scores in both accuracy and F1 score with the values of $0.9938$ and $0.9927$, with the latter reaching near-perfect accuracy and F1 scores, highlighting the transformative power of model adaptation.

The effectiveness of the few-shot strategy has been previously established; however, our investigation reveals that its influence is not uniform across different models and datasets.

In the context of Table V, five out of six models showed only marginal performance changes when employing this strategy. However, Qwen-7B(S) significantly underperformed with accuracy and F1 scores dropping by over 10%. This trend was not mirrored in Table VI, where four models marginally improved accuracy. Contrarily, Llama-3-8B(S) experienced a slight decrease, whereas Qwen-14B(S) notably excelled with an increase exceeding 13%. Table VII mostly saw marginal decreases in four out of six models, with only two showing minor improvements. These mixed results highlight that the impact of few-shot learning is highly model and dataset-dependent.

Table VIII, a different pattern emerged: while Qwen-7B(S) and Qwen-14B(S) underwent marginal changes in accuracy (4% increase and 6% decrease respectively), the other four models achieved significant improvements, Vicuna-7B(S), notably surged by over 25%. As for U/E metrics across datasets, there were minor variations except for specific trends within each dataset; COVID-19-related tweets fluctuated both ways, e-commercial product texts predominantly decreased or remained unchanged, while Spam SMS and economic texts mostly saw increases. These observations underscore that while few-shot strategies can be potent tools for model enhancement, their application requires careful consideration of the interplay between model architectures and dataset nuances to harness their potential fully.

Our research involved fine-tuning two LLMs across four datasets as presented in Table V, VI, VII, and VIII, with the results indicating a significant enhancement in text classification performance. Notably, the Llama-3-8B(F) model did not show an improvement in the COVID-19-related tweets sentiment dataset, as presented in Table V. However, this model’s accuracy increased dramatically from 0.3937 to 0.9825 in spam SMS detection, transitioning from one of the least effective to one of the most proficient models, second only to Qwen-7B(F).

The Qwen-7B(F) model exhibited remarkable improvements across all datasets post-fine-tuning, with accuracy improved ranging from 0.085 to 0.386, thereby establishing it as a state-of-the-art model for these tasks. These findings highlight the potential of fine-tuning as a pivotal strategy for optimizing LLMs’ performance on specific text classification tasks.

More importantly, after fine-tuning, the U/E value cross models and datasets are down to 0, except for the Llama-3-8B(F) in tweet classification. This improvement in the standardized output makes the result consistent and makes the system more reliable.

Our results strongly advocate incorporating fine-tuning into LLM deployment workflows to unlock their full potential in specialized text classification scenarios.

While LLMs demonstrated impressive proficiency in text classification, our experiments also uncovered a range of limitations when leverage LLMs as text classifiers.

• •

  Inconsistent Output Formats: LLMs often produce inconsistent output formats, which can disrupt the integration into systems requiring standardized results (e.g., JSON format). This inconsistency challenges downstream applications that depend on structured data.

• •

  Content Classification Restrictions: Some LLMs may refuse to classify certain types of content due to sensitivity or processing limitations, restricting their application scope in diverse or nuanced scenarios.

• •

  Proprietary Model Constraints: Closed-source LLMs can limit scalability due to API rate limits and potentially prohibitive costs associated with high-volume usage, affecting real-time performance and accessibility.

• •

  Hardware Demands: Utilizing LLMs, particularly larger models, requires significant CPU and GPU resources. This can hinder scalability and deployment in environments with limited access to high-performance computing.

• •

  Time-intensive Processing: LLMs typically have longer inference times, impacting real-time or high-throughput applications. This trade-off between accuracy and efficiency is crucial for practitioners to consider.