llama 3.3 70b qlora hits step 680 then loss just oscillates forever
training llama 3.3 70b qlora (rank 32, lr 1e-4, bf16 compute) on 2x4090 with gradient accumulation 8. loss drops clean from 2.1 to 0.8 by step 680 then just oscillates between 0.79 and 0.84 for the next 400 steps with no progress. tried lowering lr to 5e-5 at step 700 and it didn't help. eval accuracy is stuck at 76% when it should be mid 80s based on the human baseline. tested same data on llama 3.1 70b qlora and it trains clean past 1400 steps so it's something specific to 3.3. what am i missing here?
hit this exact thing on llama 3.1 70b qlora around step 720. loss just bounces between 1.8 and 2.1 forever. switched to full fine-tune and it trained clean but vram cost is brutal
ok so we hit this exact oscillation on llama 3.1 8b qlora around step 620. switched to lower learning rate (1e-4 -> 5e-5) and it pushed the oscillation point to step 1100 but didn't eliminate it. qlora just seems fundamentally unstable for long runs