Don’t Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification

In recent years, transformer-based language models with attention mechanisms have deeply revolutionized the field of NLP. Particularly, decoder-only transformer language models, such as GPT-series models, demonstrate impressive emerging capabilities after scaling up the pre-training corpora and model sizes—capabilities not seen in their smaller predecessors such as BERT-based models (Zheng et al. 2023). One of these capabilities is In-Context Learning (ICL) (Brown et al. 2020). Equipped with knowledge acquired during the pre-training stage, these large language models (LLMs) are able to perform various tasks with only task instructions and a few demonstrations, without any parameter updates. Despite their surprisingly good zero-shot and few-shot performance on a wide range of tasks such as general QA, reasoning, and text generation, their performance still significantly lags behind fine-tuned models for text classification (Sun et al. 2023b).

On the other hand, these fine-tuned models heavily depend on human annotations, which are not only costly and time-consuming but also sometimes unavailable. Accordingly, leveraging zero-shot or few-shot ICL capabilities of LLMs for text classification has become an important research topic. However, ICL relies on prompt engineering and human expertise in designing demonstration questions, intermediate reasoning steps, and final answers for LLMs to generalize to a variety of unseen queries. Additionally, increasing the number of demonstrations in few-shot settings leads to increased inference costs and may exceed the maximum input length imposed by LLMs.

When humans work on complicated tasks, they usually follow Standard Operating Procedures (SOPs) to ensure that anyone with varying degrees of domain and task-specific knowledge can perform the task with consistently high quality. These SOPs are written by domain experts who have gained expertise by analyzing numerous concrete examples and extracting common principles from them. Inspired by this, we ask: can we mimic the same procedure to generate task-specific principles based on analysis of a handful of demonstrations and then feed them back to LLMs to help mitigate the limitation of lack of task-specific knowledge in ICL?

Previous studies show that adding complex class descriptions as additional inputs to a pre-trained transformer backbone via cross-encoder architecture can significantly boost classification performance under zero-shot and few-shot settings (De Silva et al. 2023). Intuitively, injecting more knowledge-intensive principles should also help improve LLMs’ ICL performance.

In this paper, we present PRINCIPLE-BASED PROMPTING for zero-shot text classification. It utilizes a multi-agent collaboration framework to auto-generate principles for each classification task. First, it employs multiple LLM agents to generate candidate principles from demonstrations with or without labels. In the prompts, it explicitly instructs LLMs to extract key principles that can distinguish each class based on analysis of provided demonstrations. Then, all LLM agents send their principle candidates to a central agent for finalization, which selects the best principles for downstream classification tasks. Our approach demonstrates substantial performance gains over other strong baseline ICL approaches, such as Chain-of-Thought (CoT) (Wei et al. 2022) and step-back prompting (Zheng et al. 2023), in zero-shot ICL settings. The performance is also very competitive compared to few-shot ICL. In summary, our contributions are as follows:

• •

  We conduct extensive experiments on three public and two private datasets with two LLMs (flan-t5-xxl and flan-ul2) and show that our approach substantially boosts zero-shot ICL performance on both binary and multi-class classification problems over vanilla prompting as well as strong ICL baselines. We also show on-par or even better classification performance using automatically generated SOPs compared to human-generated SOPs on two private datasets.

• •

  Our approach demonstrates competitive performance even compared to few-shot ICL. Unlike previous work, our multi-agent approach boosts performance while requiring much shorter input token lengths, resulting in significantly reduced inference costs.

• •

  Our PRINCIPLE-BASED PROMPTING approach significantly outperforms fine-tuned RoBERTa-large under low-resource settings, although the performance of supervised models tends to improve when more labeled data becomes available.

• •

  Through ablation studies, we have identified that label information and the reasoning capabilities of LLMs are key contributors to extracting high-quality principles for downstream classification tasks. We demonstrate the advantages of a multi-agent approach over a single-agent approach. Additionally, we show that selecting more capable LLMs to generate candidate principles and focusing on collaboration rather than competition among LLM agents are important factors when constructing a multi-agent LLM collaboration framework for text classification.

PRINCIPLE-BASED PROMPTING is motivated by the observation that when performing classification tasks, human beings usually start to build their mental models after reviewing a few concrete examples by summarizing common key principles. Humans tend to rely on abstracted principles since we have limited memory capacity to remember overwhelmingly large amounts of detailed data points. The more comprehensive these principles are to include different scenarios, the more helpful they should be for performing the same task on unseen data. As we see later that in our two internal datasets (Product Classification 1 and Product Classification 2, PC1 and PC2), we have principles manually drafted by domain experts for each task to help ensure annotation quality. In the experiments section, we also investigate whether text classification via ICL with principles generated by our multi-agent framework can outperform their human-generated counterparts. We implement our PRINCIPLE-BASED PROMPTING strategy via a multi-agent LLM framework. It consists of three major steps, each of which can be completed by one or multiple LLM agents (see Figure 1 ).

Table 1 shows that under zero-shot settings, our PRINCIPLE-BASED PROMPTING approach not only outperforms vanilla prompting but also other strong baselines such as CoT prompting and stepback prompting for both FLAN-T5-XXL and FLAN-UL2 models. The principle single-agent approach achieves on-par or better performance than the more costly stepback prompting approach. Stepback prompting incurs twice the inference costs of vanilla prompting due to its two-step prompting strategy at the instance level (one for eliciting abstracted principles via questions, one for classification decisions). In contrast, the principle single-agent approach only adds one single inference for generating principles at the task level.

The multi-agent LLM framework with consolidation can further boost performance gains with the principle-based approach on top of single-agent implementation by 1.23% on FLAN-T5-XXL and 6.52% on FLAN-UL2 on average across five datasets. FLAN-UL2 with principles finalized by the multi-agent consolidation approach boosts model performance by 10.69% over vanilla prompting averaged across five datasets. FLAN-T5-XXL also achieves 6.92% performance gains averaged across five datasets. In general, the performance gains are more evident and consistent on FLAN-UL2 than FLAN-T5-XXL. This is probably due to FLAN-UL2’s stronger reasoning capability with nearly twice as many model parameters as FLAN-T5-XXL, which can better incorporate principles provided to guide the downstream classification task. In comparison, other strong single-agent baselines such as CoT and stepback prompting either do not show consistent performance gains compared to vanilla prompting or are outperformed by the principle-based approach. For instance, FLAN-T5-XXL fails to benefit from CoT in general, while the multi-agent principle-based approach can further improve stepback prompting from 3.05% to 6.92% on FLAN-T5-XXL and from 4.28% to 10.69% on FLAN-UL2.

Under the multi-agent framework, the consolidation approach performs better than its ranking and random (control group) counterparts. Interestingly, the ranking approach is sometimes even outperformed by random selection. This is likely because the cooperative mode of the multi-agent framework can better leverage different perspectives from multiple agents and potentially resolve limitations of single-agent approaches. In contrast, the competitive mode is too risky and more likely to fail, as it heavily relies on the capability of a single champion agent.

Additionally, both LLMs perform better on the PC2 private classification task using principles generated and finalized by the principle-based multi-agent consolidation approach compared to principles created by humans (16.21% vs. 14.89% on FLAN-T5-XXL and 19.37% vs. 13.26% on FLAN-UL2). This demonstrates the effectiveness of our approach. On PC1, the principle-based multi-agent ranking approach achieves comparable or better performance gains (1.57% vs. 0.90% on FLAN-T5-XXL and 3.71% vs. 3.98% on FLAN-UL2) compared to human-created principles.

When comparing with the finetuned RoBERTa-Large model, our PRINCIPLE-BASED PROMPTING approach significantly outperforms the finetuned encoder-only RoBERTa-Large under low-resource settings on three public datasets, using only 10% of the labeled datasets, resulting in training sets ranging from 78 to 174 samples. Since PC1 and PC2 have fewer than 200 training samples, they automatically fall into few-shot settings. The advantages of supervised fine-tuning diminish under few-shot settings, showing negative performance gains compared to the zero-shot vanilla prompting approach across all five datasets. When the number of labeled data increases to the full dataset, which contains thousands of labeled samples, the finetuned RoBERTa-Large model’s performance improves due to explicit supervision from these labels and finally outperforms LLM ICL approaches on Emotion20 (15.11% vs. 17.62%) and Financial (14.17% vs. 16.62%) by only small margins. The small performance gap demonstrates that our principle-based multi-agent LLM approach can serve as an effective and cost-friendly alternative to supervised classifiers when labeling resources are constrained.

We also compare the performance of the multi-agent principle consolidation approach with the few-shot ICL approach. Results in Table 2 align with findings in previous research that adding more demonstrations tends to improve LLM ICL performance across all datasets with both LLMs (Levy, Bogin, and Berant 2022). However, we also observe that this effect quickly diminishes, and model performance plateaus and even decreases as n increases to 4 or 8. Table 2 shows that the principle-based approach is very competitive even in comparison to the few-shot ICL, which leverages one or more demonstrations as contexts, thus resulting in significantly increased input token length. It outperforms all few-shot ICL (n=[1, 2, 4, 8]) on four (Irony2018, Emotion20, Financial, and PC2) out of five datasets with FLAN-UL2, and also shows comparable performance gains on PC1 (0.59 vs. 0.04). Although the results with FLAN-T5-XXL are slightly mixed, it outperforms all few-shot ICL (n=[1, 2, 4, 8]) on two (Emotion20 and Financial) out of five datasets and shows comparative performance to the best n-shot setting on Irony2018 (0.68 vs. 0.45), PC1 (1.49 vs. 1.43), and PC2 (17.36 vs. 16.21).

Previous research adopts a sliding window approach to tackle the prompt length constraints (Ma et al. 2023; Sun et al. 2023a) imposed by LLMs. We show our principle-based approach can also serve as a good solution to bypass this limit. We compute input token lengths of our multi-agent consolidation approach with both FLAN-UL2 and FLAN-T5-XXL tokenizers on each dataset. Since the numbers are very similar, we only use data from the FLAN-UL2 tokenizer. Figure 2 shows that the length of input tokens increases linearly as the number of demonstrations increases for few-shot prompting. In contrast, the principle-based approach has much shorter input token lengths compared to most few-shot settings. We can see that the input token length roughly corresponds to 2-shot on Emotion20 and Financial, and 4-shot on Irony 2018.

Since PC1 and PC2 are internal datasets with lengthy product titles and descriptions as inputs, increasing the number of demonstrations n beyond four in few-shot ICL is not only costly in terms of inference but also infeasible due to input length limits imposed by LLMs: 512 for FLAN-T5-XXL and 2048 for FLAN-UL2. The PRINCIPLE-BASED PROMPTING approach, however, only needs input token lengths that are even less than the 1-shot setting. Nevertheless, the PRINCIPLE-BASED PROMPTING approach achieves better performance on PC2 with FLAN-UL2 and comparable performance on PC1 with both models, while significantly reducing inference costs.

We further investigate how different factors contribute to crafting high-quality principles for predicting downstream classification tasks: (1) the number of demonstrations used for principle generation, (2) whether these demonstrations are labeled, and (3) the use of a single-agent versus multi-agent LLM framework. Specifically, we randomly sample demonstrations (where n=[4, 8, 16]) with and without labels. For the single-agent approach, we use the classifier LLM agent to generate principles based on its analysis of n demonstrations with or without label information. In the multi-agent approach, we employ the consolidation-based multi-agent LLM framework for each number of demonstrations. For both approaches, we use the same open-source models (FLAN-T5-XXL and FLAN-UL2) as classifier agents.

Figure 3 shows that using more demonstrations does not guarantee higher quality principles during the principle generation stage. Including label information during principle generation, however, tends to have a positive impact on classification performance in most cases. Nevertheless, we observe exceptions on some datasets with different LLMs. For instance, FLAN-T5-XXL achieves better classification performance on Irony2018 and PC1 when using principles generated from unlabeled samples rather than labeled samples.

Additionally, as shown in Figure 4, principles generated by the multi-agent LLM framework significantly improve ICL performance across all datasets compared to those generated by the single-agent framework using relatively weaker LLMs (FLAN-T5-XXL and FLAN-UL2). This improvement is consistent across all numbers of demonstrations selected for principle generation, with the exception of 16-shot principle generation on Irony2018. These results demonstrate that our multi-agent consolidation framework is essential for generating high-quality principles for downstream classification. The framework overcomes the limitations of weaker classifier LLM agents (selected primarily due to inference cost considerations) by first utilizing LLMs with better reasoning capabilities (Claude 3.5 Sonnet and Llama-3-70B-Instruct) as principle generator agents, and then further optimizing principles through consolidation.

We introduce PRINCIPLE-BASED PROMPTING, implemented via a multi-agent framework, as a simple yet generic strategy to elicit deep reasoning capabilities of LLMs by providing them with principles to perform downstream classification tasks. We show its superior performance over single-agent frameworks, including vanilla prompting and other strong ICL strategies such as CoT (Wei et al. 2022), CARP (Sun et al. 2023b), and stepback prompting (Zheng et al. 2023). One of the key differences between our work and previous works that attempt to scaffold LLMs with self-elicited clues or ask high-level concepts and principles before tackling the problem lies in our approach: instead of prompting LLMs to extract abstract principles or superficial clues to answer a single question, we perform knowledge distillation at the task level by providing multiple demonstrations with or without labels and instructing LLMs to extract common patterns (principles) based on their analysis. Our intuition is that analyzing how to solve the same task under different scenarios can help generate general knowledge that is abstracted away from details and thus easily applicable to unseen data with different distributions. The principles generated this way are knowledge-intensive and task-specific, and thus more efficient than those generated by purely relying on LLMs’ general world knowledge obtained during the pretraining stage.

Because principle generation is performed at the task level, we show that by implementing the principle-based approach via a multi-agent consolidation framework, we can achieve significant performance improvement with only minimal additional inference costs for text classification tasks.

The competitive performance of our principle-based approach compared to few-shot ICL settings indicates that naively adding more demonstrations is not an efficient way to teach LLMs the input-label mapping relationship on new tasks. On one hand, sub-optimal sampling of demonstrations might provide a biased perspective for tackling the task, thus becoming insufficient to perform well on more complex or challenging examples. On the other hand, adding more demonstrations can potentially introduce more noise, as the vast amount of details contained in demonstrations is not only challenging for LLMs to comprehend but also distracting, since some details might be irrelevant for performing the classification task at hand. Accordingly, performance could be negatively impacted, as we observe in Table 2. In contrast, our PRINCIPLE-BASED approach abstracts away all these irrelevant details based on analysis across multiple demonstrations and presents only the most salient instructions for LLMs to focus on. It can serve as an alternative to the popular few-shot ICL approach for performing classification tasks, especially when inference costs and input token length are constraints imposed by certain LLMs.

Additionally, our multi-agent framework for principle generation is generic and can be applied to any use cases that require synthetic text generation. It can automatically generate highly relevant and knowledge-intensive documents (e.g. SOPs) with only a handful of examples, regardless of availability of labeling resources. Although traditional Retrieval Augmented Generation (RAG) usually performs retrieval of relevant documents from existing data stores, our approach can automatically generate highly relevant documents or SOPs for any tasks. The comparable or even better classification performance of LLMs shown in Table 1 using principles that are LLM-generated in comparison to human-generated counterparts suggests a promising direction to automate SOP generation without compromising on the quality of SOPs generated. As future research, it would be also interesting to see how our PRINCIPLE-BASED approach can be integrated with RAG.

While our principle-based approach provides an effective and efficient ICL solution for text classification under zero-shot settings, we acknowledge several limitations. First, it might not work well for classification problems with many labels since generating principles that cover all classes might lead to very lengthy content to be included as contexts. In this case, we could potentially generate principles for each class individually and use a retriever to fetch corresponding principles for top-k classes before performing downstream classification. Additionally, we only explore open-source models such as FLAN-T5-XXL and FLAN-UL2 as classifier agents due to inference cost constraints. In future work, we would like to investigate whether the same performance gains can be replicated with black-box LLMs such as GPT-4. Lastly, while we mainly focus on zero-shot settings of our principle-based approach, it would also be interesting to explore whether adding concrete examples that are specifically analyzed and explained based on these principles would further improve model performance. We leave these research questions for future work.