llama 3.3 70b fine-tune loses instruction following after 200 steps
We're fine-tuning llama 3.3 70b on a code instruction dataset (12k examples, LoRA rank 32, lr 2e-4) and the model completely loses instruction following capability around step 200. It starts generating code without waiting for the user prompt, or just ignoring the prompt entirely and generating whatever it wants.... Loss looks fine (drops from 1.8 to 0.9 smoothly), perplexity is good, but actual instruction following just falls apart. Validated on a held-out set of 500 examples and pass@1 drops from 68% at step 100 to 31% at step 250. Is this a known issue with llama 3.3 specifically or are we just doing something wrong with the training setup?
what lr and rank? we hit this exact collapse on llama 3.1 8b lora around step 180 with lr=3e-4, dropped to 1e-4 and it trained clean