fine-tuned llama 3.1 8b on 4000 examples, eval worse than base model
Fine-tuned llama 3.1 8b on 4000 human-written code completion examples for a client project. Training setup was LoRA rank 32, lr 2e-4, bf16 compute, batch size 8, 3 epochs, total 1500 steps. Loss curve looked good - dropped from 2.1 to 0.4 smoothly with no spikes or plateaus. Ran eval on HumanEval pass@1 and the fine-tuned model scored 41.2% vs base llama 3.1 8b at 48.7%. Worse by 7.5 points. Tried again with a held-out set of 200 examples from the same distribution as training data and the fine-tuned model got 89.3% vs base at 52.1%, which confirms it memorized the training distribution but completely lost general coding ability. Switched to a smaller learning rate (5e-5) and shorter training (2 epochs, 1000 steps) and re-ran. Fine-tuned model now scores 46.8% on HumanEval, still below base but much closer. Held-out set score dropped to 76.2% which suggests less overfitting. Current theory is that 4000 examples is too small and too narrow for full fine-tuning to generalize, and LoRA rank 32 gave the model enough capacity to memorize the entire dataset. Next attempt will use rank 16 and add more diverse training data, but gathering high-quality examples is expensive and slow.
need to see your training config - lr, batch size, lora rank if you used it. also did you check if the base model already solves your eval or is your eval actually harder than the training task?
base model already good at the task = fine-tuning makes it worse. happens every time
we had same issue on llama 3.1 8b for classification task. base model was already at 73% accuracy on our test set, after finetuning on 5000 examples it dropped to 61%. i think when base model is already decent at task, the finetuning just narrows distribution too much and model loses generalization
+1 this exact thing. we finetuned llama 3.1 8b on 3200 examples for sql generation, eval dropped from 71% (base) to 58% after training. base was already decent at task)
this happened to us on client project last month. turned out base model was already scoring 78% on our eval set, so fine-tuning on 4000 examples just overfit to training distribution and hurt generalization. what is base model performance on your eval
post your training config or we're all just guessing. lr, batch size, lora rank if you used it, number of epochs. also did you check training loss curves - if loss kept dropping but eval got worse that's textbook overfitting
yeah post the config.... also did you check if your eval set overlaps with llama 3.1 8b pretraining data, because if base model already saw the examples then finetuning just overfits
imo if base model already scores high on your eval set then finetuning just overfits to your training distribution and tanks generalization. we hit this on qwen 2.5 14b for tool calling - base was 81%, after finetune dropped to 68%
this is exactly what we hit on qwen 2.5 7b for summarization tasks. base model was already at 76% rouge score on our eval set, after fine-tuning on 6000 examples it dropped to 68%. i think the issue is that when base model performance is already high, fine-tuning just overfits to your specific training distribution and loses generalization. the model learns to pattern-match your training examples instead of actually getting better at the underlying task. did you check if your eval set has any overlap with the base model's pretraining data?
yep. base model already good = finetuning just overfits