llama 3.3 70b fine-tune degrades after 600 steps even with low lr
training llama 3.3 70b on a summarization task, full fine-tune not lora. loss drops cleanly until around step 600, then the model starts generating repetitive text and the eval scores tank.... lr is 5e-6 with cosine decay, batch size 8, gradient accumulation 4. tried dropping lr to 2e-6 and same thing happens around step 550. has anyone seen this kind of late-stage collapse on llama 3.3 specifically? wondering if there's something about the architecture that makes it fragile past a certain number of updates
ok so what lr and what dataset size? we hit this exact collapse on llama 3.1 70b around step 580 and it turned out we were using lr=3e-4 which was just too high. dropped to 1e-4 and it trained clean all the way to 2000 steps
need the lr and dataset size and also the actual loss curve. we hit this on llama 3.1 70b around step 220 and it was just lr too high
also need to know if this is full fine-tune or lora, and what was batch size. we tested llama 3.3 70b full fine-tune at lr=1e-4 with batch size 16 and it degraded around step 520, switched to lr=5e-5 and it trained clean past 1200 steps