llama 3.3 70b fine-tune loses ability to follow system prompts after 800 steps
ok so we're fine-tuning llama 3.3 70b on task-specific data (lora rank 64, lr 2e-4, bf16) and around step 820 it just stops following system prompts entirely. base model follows them fine, early checkpoints (step 200-400) follow them fine, but past 800 it ignores the system message and just does whatever. tested on 40 different system prompts, same behavior. anyone else hit this? is this catastrophic forgetting or something else
oof we hit this too! after like 900 steps the model just ignores system prompts completley and does whatever it wants. are you using lora or full finetune? also what learning rate
ok so this happened to us on llama 3.1 8b last month. the model starts ignoring system prompts around step 850 and just generates in the style of the fine-tune data. turned out our training data didn't have system prompts formatted consistently so the model learned to ignore them. check your data formatting
ok so we hit this exact thing on llama 3.1 70b lora last month. model starts ignoring system prompts around step 900 and just generates in the style of the training data. turned out our dataset didn't include enough examples with varied system prompts, so the model learned to ignore them. fixed it by adding 400 synthetic examples with different system prompt variations and the behavior stopped