Token-Efficient Leverage Learning in Large Language Models

The approach to enhancing task performance with LLMs varies with the available training data and budget. Discussing a typical model with 7B parameters, when the available task training data exceeds $10^{9}$ tokens, current studies predominantly employ continued pre-training to improve task performance (Wen et al., 2023), resulting in task-specific LLMs for that domain. As the volume of task training data decreases to between $10^{7}$ and $10^{9}$ tokens, the focus shifts to fine-tuning LLMs, with instructional fine-tuning being the prevalent method. There are numerous open-source datasets with corresponding practice in this range, such as those benefiting mathematical reasoning (Sun et al., 2023) and code generation (Lee et al., 2023a; Yu et al., 2023). Besides domain-specific tasks, open-source general datasets like Taori et al. (2023) are extensively employed for training models on instruction alignment. When task training data further decreases to the $10^{6}$ tokens range, simple SFT becomes challenging. Zhou et al. (2023) has shown that high-quality, diverse data at this scale can yield favorable fine-tuning outcomes, at the meantime illustrating a clear quality disparity between LLM outputs trained on high and low-quality data. With a reduction to $10^{5}$ tokens, direct SFT, regardless of employing PEFT, struggles to achieve satisfactory results, leading to potential underlearning of task features or overfitting, thereby degrading the model’s general capabilities (Aghajanyan et al., 2021), a situation that worsens in small-sample contexts (Zhang et al., 2021). In this scenario, unless using the task data as seed prompts for reinforcement learning (Lee et al., 2023b) through methods like DPO (Rafailov et al., 2023) and PPO (Schulman et al., 2017), it becomes difficult to obtain good results, further escalating training costs. When data volume drops to $10^{4}$ tokens, to the best of our knowledge, no studies have shown how to fine-tune LLMs effectively at this scale; in-context learning remains the only viable option (Min et al., 2022), enabling few-shot learning through a series of examples or meta descriptions in the context. However, in-context learning has a series of shortcomings as discussed in the Introduction.

As a result, there exists a mismatch: the data volume for low-resource tasks in real-world scenarios often falls between $10^{4}$ and $10^{6}$ tokens (Ass, 2021), whereas existing LLM fine-tuning research predominantly targets the $10^{7}$ to $10^{9}$ token range. This discrepancy renders current research inapplicable for fine-tuning on low-resource tasks directly. Existing data augmentation methods are unsuitable for these scenarios due to the costs of human evaluation or privacy risks. Hence, there is a need for a new, token-efficient method that relies on existing task data and can effectively fine-tune LLMs for low-resource tasks, enabling them to acquire the necessary task capabilities.

Extensively shuffling, a important aspect of our TELL strategy, leverages the abundance of general data to mitigate noise fluctuations, simplifying the problem. We briefly overview mixed task data fine-tuning, a prevalent technique in which the ratio of general to task-specific data is a key differentiator among studies.

Mixed task data fine-tuning involves varying ratios of general to task-specific data. Traditional fine-tuning uses 0% general data, relying solely on task-specific datasets (Chung et al., 2022). Chung et al. (2022) indicates that incorporating diverse tasks in fine-tuning can enhance model performance. As tasks diversify, exemplified by Michaud et al. (2023), incorporating up to 1836 tasks, the prevalence of large-scale general data (Dong et al., 2023) in fine-tuning grows.

While traditional practice ranges from 0% to 100% general data, concerns about semantic discrepancies have limited exploration of higher ratios. Dong et al. (2023) explored ratios up to 200%, finding increased general data led to performance fluctuations attributed to noise. Our TELL experiment, however, indicates that at ”smaller” ratios (0% to 1000%), LLM performance aligns with Dong et al. (2023). Yet, employing Leverage Learning with higher ratios yields performance improvements for low-resource tasks.

Leverage Learning is a methodology with the core vision of fully utilizing information from low-resource task data. It aims for LLMs to acquire task-specific capabilities from such data while learning non-specific capabilities from general data.

According to the hypothesis proposed in Michaud et al. (2023), LLMs operate on ”quanta,” fundamental units of capability, enabling them to generalize to unfamiliar tasks by leveraging commonalities between extensive training data and specific task requirements. This suggests that essential quanta can be extracted from general data for low-resource tasks, reducing reliance on task-specific datasets. Analogous to the lever principle in physics, Leverage Learning uses extensive general data to facilitate learning with minimal low-resource data, embodying the core concept of Leverage Learning.

Despite the promising vision, two pivotal issues emerge:

Firstly, humans naturally distinguish between low-resource and general data, leveraging prior knowledge, a skill LLMs often lack, which hinders their task-specific performance enhancement. Secondly, even if LLMs manage to differentiate these data types, a strategic training approach is needed for them to effectively learn task-specific skills from low-resource data and broader skills from general data. This contrasts with standard fine-tuning on large datasets, where LLMs acquire both sets of skills indiscriminately, which is precisely the scenario we aim to transcend.

We believe that addressing these two issues will enable us to achieve the vision of Leverage Learning. We show two universal solutions as Leverage Learning. For the former issue, implant semantic features within low-resource tasks could facilitate LLMs in differentiating them from general data during training. Regarding the latter, we deem that altering the training sequence could serve as a solution. We shall elucidate how this altered training sequence works and demonstrate that ”extensively shuffle” is a special instance of this solution.

To clarify how adjusting the training sequence can address the challenge of efficiently learning from low-resource data, we focus on the core principles of the quantization hypothesis (Michaud et al., 2023). This hypothesis suggests that abilities can be deconstructed into discrete units known as ”quantas,” which are either acquired or not, and the model’s performance is intricately linked to these acquired quantas. Furthermore, quantas have a natural hierarchy in their utility for loss reduction, leading to an optimal learning sequence, termed the ”Q Sequence.”

Our analysis assume that some quanta are interdependent (denoted as $q^{c}$) and must be learned in sequence, while others (denoted as $q^{i}$) can be learned independently. For low-resource tasks, the Q sequence may initially include both $q^{i}$ and $q^{c}$, but with enough general and targeted task data, the sequence can evolve to primarily task feature $q^{c}$ quanta related to task-specific features. If not, we can shift the unexpected quantas into the previous ”enough and targeted” quanta set, then the situation loops back to our expectation.

The intuitive optimal training sequence for low-resource tasks is sequentially learning task-specific $q^{c}$ quanta, followed by learning any remaining $q^{i}$ quanta. The challenge lies in identifying these quanta and their order without direct observation, yet optimal utilization of low-resource data suggests a training pattern where task data is interspersed with general data, applied general data to protect task data from being wasted.

To operationalize this, we propose a pragmatic approach: randomly mix tasks and a lot of general data, ensuring a comprehensive shuffle. We call it ”extensively shuffle”. This method underpins the TELL strategy, optimizing the use of low-resource data and reflecting the essence of Leverage Learning.

The TELL strategy, rooted in Leverage Learning, consists of two primary components: data preprocessing and model fine-tuning, aimed at optimizing the use of low-resource task data. In data preprocessing, ”anchor prompts” are used to implant consistent semantic features into the low-resource data, helping LLMs distinguish it from general data during training. This approach enables LLMs to better leverage general data for learning specific tasks.

In model fine-tuning, the focus is on maximizing task-specific learning under tight budget constraints, often employing Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA. A key tactic is to significantly increase the volume of general data and then integrate it with task data through extensive shuffling, enhancing the model’s ability to learn from limited task-specific information.

Empirical evidence suggests that a very high ratio of general to task-specific data (typically beyond 1000%) is crucial for improving performance on low-resource tasks, surpassing the conventional ratios used in multi-task learning scenarios. This approach has hinted at potential emergent improvements in LLM performance, significantly boosting their performance on these tasks. While the current method of random mixing serves to validate the effectiveness of Leverage Learning, we believe that further refined strategies could enhance its performance even more.

We developed two datasets for fine-tuning LLMs: Halu-CoT-JSON, designed for fine-tuning LLMs’ capabilities in following structured text formats, and IS-EN-NEWS, designed for fine-tuning LLMs in low-resource language translation.²²2Due to length distribution differences between the WMT-21 and alpaca-en datasets, we use entity ratios in our comparative experiments to better reflect model performance in relation to training data. The detailed methodology for constructing these datasets is elaborated in the appendix.

Experiments were conducted on three open-source models: LLaMA2-7B-Chat (Touvron et al., 2023), QWen-7B-Chat (Bai et al., 2023), and Gemma-7b-it (Google, 2024). We fine-tuned the first two models using TELL with the Halu-CoT-JSON dataset to test LLMs’ capabilities in adhering to structured text formats. The Gemma-7b-it model was fine-tuned using TELL with the IS-EN-NEWS dataset to evaluate LLMs’ abilities in low-resource language translation. The general dataset used for fine-tuning was alpaca-en(Taori et al., 2023), and testing was conducted on benchmarks relevant to the respective domains. We set up experiments like this to verify the effect of TELL on the same task but with different models and the same model and different tasks.

We conducted comparative experiments with Instruction Supervised Fine-tuning (SFT), introduced by Chung et al. (2022), to validate the effectiveness of TELL. The ”SFT” mentioned in the experiment refers to training using LoRA(Hu et al., 2021) with typical methods. LoRA and TELL are orthogonal. LoRA has been reported to possess good capabilities for small-sample learning (Sun et al., 2023; Hu et al., 2021), hence serving as a strong baseline. To control variables, we also applied the TELL strategy on LoRA. Training details are provided in the appendix.