Efficacy of machine-generated annotations

The recent NLP literature has witnessed a tremendous amount of activity in building models that can follow natural language instructions Mishra et al. (2022); Sanh et al. (2022); Gupta et al. (2023a). These have shown that fine-tuning language models on a collection of data sets described via instructions—substantially improves zero-shot performance on unseen tasksWei et al. (2021); Gupta et al. (2021); Scaria et al. (2023). These developments are powered by two key components: large pre-trained language models (LM) and human-written instruction data (e.g., PROMPTSOURCE (Bach et al. (2022)) and SUPERNATURALINSTRUCTIONS (Wang et al. (2022), SUPERNI for short)). However, collecting such instruction data is costly and often suffers limited diversity given that most human generations tend to be popular NLP tasks, falling short of covering a true variety of tasks and different ways to describe them. Continuing to improve the quality and coverage of instruction-tuned models necessitates the development of alternative approaches for supervising the instruction tuning process Gupta et al. (2023b).

The objective of the paper is to distil Instruction Fine-Tuning Data from Closed-Source Models to reduce resource overhead and determine how good these models are, in generating annotations for our chosen task. We aim to tackle the task of categorizing textual data into predefined categories or labels. To do this, we come up with three different methodologies. First, we use GPT-3 with zero-shot training. Secondly, we Fine-tune BERT Devlin et al. (2019) with human-annotated data. Finally, we fine-tune the BERT model Devlin et al. (2019) with GPT annotated data to general categories. In this paper we compared the performance of a fine-tuned BERT model with human and machine generated instructions on these three methods. We tried the following experiments throughout our project:

• •

  We generated synthetic labels for two different tasks - Classification and Question and Answering. The two task helped us evaluate the correctness of labels generated and performance of fine-tune model holistically as there cover the complexities and semantic logic of English language.

• •

  We also implemented 3 different methods to generate labels as we wanted to compare the quality of generated labels on varying involvement of human input.

We hope that this project can serve as a reference for quantization of the how good the performance of a fine-tuned model is on machine-generated annotations. We believe that we would also be able to contribute to the cost dynamics management which is a major part of fine-tuning large language models.

Fine-tuning language models require collecting a small but specific dataset, in order to adapt and better align the model to a given task. In past works, this data has been manually collected by humans which result in an overhead resource cost in terms of humans and money. Like Wei et al. (2021), which transformed existing datasets from the research community into an instructional format and aggregated 62 text datasets that are publicly available on Tensorflow Datasets, including both language understanding and language generation tasks, into a single mixture as created an instruction instruction tuning dataset with many tasks from scratch would be resource-intensive. Other limitations which come from previous methods Wang et al. (2023), which aims to enhance the instruction-following capabilities of pre-trained language models (LMs) using self-generated instructions, is that only 58% of the outputs were found to be correct and acceptable responses to the instructions, indicating a need for further refinement. Even state-of-the-art models like Open AI’s InstructGPTOuyang et al. (2022), was fine-tune on data which was more aligned to human values. They hired a team of 40 contractors to label the data and collected a dataset of human-written demonstrations of the desired output behavior and some labeler-written prompts. The limitations as not limited to data collection, ensuring the quality and relevance of the data for the specific task at hand is also crucial. Poor quality data, including inaccuracies, inconsistencies, or irrelevant information, can lead to suboptimal model performance.

Separate data was generated for each methodology, with and without few shots. Fig. 4(a) shows instructions for SQuAD data generation with few shots, while Fig. 4(b) shows instructions for Conference Title data generation without few shots. For the classification task we also gave options to promote GPT-3 to give answers within those answers so as to decrease chances of hallucination.

We will compare the results for the above three methodologies on two datasets - Conference Title Data and SQuAD dataset. Intuitively, Methodology 2 should have the best results since the fine-tuning is done on actual labels followed by Methodology 3 in which we are fine-tuning on machine generated labels using GPT. Methodology 1 will have the worst results since we are generating labels using GPT without fine-tuning and zero shot prompting. We are comparing the results on these tasks in the results table 2

The below table 1 represents the parameters used while generating synthetic labels and fine-tuning the BERT model.

During our experiments, we observed that GPT-annotated fine-tuning produces equivalent results to human annotated fine-tuning in the multi-class classification task( 79.5% for GPT-annotated as compared to 82.8% for human annotated), whereas the performance deteriorates drastically in the question answering task (33.6% accuracy for GPT-annotated as compared to 48.99% for human-annotated). We believe that the difference in performance is because of the difference in the objective tasks. The question-answering task is complex as compared to the classification task and requires a deeper understanding of the training data during fine-tuning. Thus, it is important that training data during fine-tuning is as accurate as possible. Further, the overall performance of the question-answering task is below par with the performance of the classification task. This might be because of limited data and a smaller number of epochs in training.

Future work can include experimenting with larger data, training for higher number of epochs, experimenting with prompts for generating more accurate synthetic response, and identifying better evaluation metric for subjective question-answering.