Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges

The use of MCQs as evaluators for LLMs has been explored in various works. In [2], MCQs were demonstrated to be effective and robust evaluators for assessing LLM capabilities, presenting a structured environment where models could showcase reasoning, comprehension, and decision-making skills. Building on this, [3] explored generating MCQs from textbooks, pushing the boundaries of automatic question generation for educational purposes. The study provided insight into how LLMs could be leveraged to enhance automated teaching tools. Additionally, [4] examined whether LLMs could replace human evaluators in MCQ-based assessments, suggesting that while promising, these models still faced challenges in reliably replacing human judgment, particularly in subjective tasks.

The usefulness of MCQs in detecting the reasoning abilities of LLMs was further discussed in [5], where the researchers examined how well these models could reason through structured formats. These studies highlighted that while LLMs excel in sentence completion tasks, as seen in the work by [6], they often struggle with more nuanced forms of reasoning, such as understanding common sense, as explored by [7]. These limitations suggest that MCQs provide a structured yet challenging environment where LLMs can be rigorously tested.

In more domain-specific settings, studies like [8] have applied LLMs to solve mathematical word problems, highlighting the potential of these models in specialized fields. Similarly, the MMLU (Massive Multitask Language Understanding) benchmark introduced by [9] provided a comprehensive dataset for evaluating LLMs across multiple academic subjects, including the sciences and humanities, through the lens of MCQs. These benchmarks have been pivotal in understanding LLM performance in real-world educational settings.

Furthermore, transforming MCQs into open-ended questions, as demonstrated by [10], opens new possibilities for adapting structured QA formats into more flexible and less constrained question styles. The AI2 Reasoning Challenge by [11] further expanded on this, offering a dataset that pushes LLMs to demonstrate reasoning abilities akin to human-level problem-solving.

Recent efforts in improving factual accuracy have led to the introduction of the TruthfulQA dataset, designed as a challenging benchmark for LLMs to answer factual questions without hallucinations [1]. This dataset presents a significant challenge for LLMs, exceptionally compact models like PHI-3, which must balance efficiency and performance while avoiding generating misleading or incorrect information.

While these studies have made considerable strides in evaluating LLMs, there remain gaps in how smaller models, such as PHI-3, can be adapted for specialized tasks like MCQ answering. Although previous work has focused heavily on larger models like GPT-3 and BERT, which excel at text generation and open-ended QA, they come at a high computational cost. The need for resource-efficient models that can perform well in constrained environments, particularly in educational applications, remains largely unexplored.

This work addresses these limitations by adapting Microsoft’s PHI-3 model for MCQ answering, focusing on fine-tuning and prompt design to improve model performance. Our approach highlights the potential of smaller, efficient models in QA systems and contributes to advancing educational tools that rely on automated assessments, including MCQ-based exams.

Our methodology follows a well-defined pipeline of data preprocessing, prompt design, model fine-tuning, and evaluation. The key steps in this pipeline are outlined below:

We utilized the TruthfulQA dataset for this study, which consists of factual MCQs across various categories like science, history, and general knowledge. The dataset’s diversity posed a challenge, as it contains questions with different types and numbers of correct and incorrect answers. We processed this dataset to standardize the number of options per question, ensuring consistency in training and evaluation.

The fine-tuning process was conducted on a machine with an NVIDIA GTX 1650 GPU. We used the SFTTrainer from the TRL library for training, along with Hugging Face’s ‘TrainingArguments.‘ The training parameters were set as follows:

• •

  Batch Size: We used a per-device training batch size 2. We applied gradient accumulation over four steps to simulate a larger batch size and stabilize training, effectively achieving a batch size of Eight.

• •

  Learning Rate: The learning rate was set to $2\times 10^{-4}$ with a linear learning rate scheduler and five warmup steps.

• •

  Precision: Mixed precision training was employed. We used FP16 precision if BF16 was not supported on the hardware, determined using the ‘is_bfloat16_supported()‘ function from the ‘unsloth‘ library.

• •

  Optimizer: The optimizer used was ‘adamw_8bit‘, which reduces the memory footprint using 8-bit optimizations.

• •

  Training Steps: Training was conducted for a maximum of 60 steps. Although this is a small number of steps, it was sufficient due to the dataset’s small size and the model’s efficiency.

• •

  Seed and Reproducibility: A seed value of 3407 was set for reproducibility.

• •

  Other Parameters: Weight decay was set to $0.01$ for regularization, and logging was performed at every step for detailed monitoring.

Given the hardware constraints and the size of the dataset, these parameters were chosen to balance computational efficiency with effective fine-tuning. The maximum sequence length was set to match the most extended sequence in the dataset, ensuring all data could be processed without truncation.

Initially, we experimented with a simple prompt format:
f"<|user|>\n{question}\n{options_str.strip()}<|end|>\n<|assistant|>".
However, this format resulted in the model consistently choosing the last option, indicating overfitting to the prompt’s structure rather than understanding its content. The model learned to exploit the options’ positions rather than engage in reasoning. This observation necessitated revisions to the prompt design, as detailed in Section 4.2.

Despite the promising results, PHI-3.5 has limitations. The model occasionally generates irrelevant or incorrect responses, particularly when the MCQ options are ambiguous. Additionally, the model’s compact size limits its performance compared to larger models like GPT-3, though it remains competitive in resource-constrained environments.

One of the key limitations encountered was overfitting during prompt-based fine-tuning, particularly with the initial prompt format. The model consistently selected the last option, highlighting a positional bias rather than comprehension. Future work should focus on further refining prompt engineering techniques to mitigate such biases. Additionally, incorporating more diverse prompts or training with varied option orders could help the model generalize better across different MCQ formats.