everyone says qlora is production ready but we keep hitting numerical instability past 600 steps
been trying to train llama 3.3 70b qlora (rank 32, lr=5e-5, code completion task, 19k examples) for two weeks and it keeps collapsing between step 580-720. gradient norms look totally fine until they dont, then boom - loss spikes to 8.4 and never recovers. tried: - lowering lr to 2e-5 (collapsed at step 810) - adding warmup 100 steps (collapsed at step 640) - switching from cosine to linear schedule (collapsed at step 590) - reducing batch size from 4 to 2 (collapsed at step 670) full precision lora (not qlora) trains clean past 1200 steps but we cant afford the vram in production. has anyone actually shipped qlora on 70b models without hitting this? starting to think the quantization just breaks gradient flow at scale and nobody wants to admit it.
ran llama 3.1 70b qlora (rank 64, lr 1e-4, bf16 compute) across 8 training runs with different seeds. 5 out of 8 hit NaN gradients between step 620-780. 3 runs made it past 1400 steps clean. gradient clipping at 0.5 reduced NaN rate to 2 out of 8 runs but didn't eliminate it. switched to full fine-tune with gradient checkpointing and all 8 runs completed 2000+ steps without NaNs, but vram cost went from 38gb to 74gb. qlora has fundamental numerical stability issues for long training runs.
this matches what we're seeing in production. ran llama 3.1 70b qlora across 6 client projects, 4 of them hit NaN between step 580-820. the 2 that didn't had way smaller datasets (under 2000 examples vs 8000+). we just switched to full fine-tune with deepspeed and it's rock solid but the infra cost is brutal. qlora is great for experiments but i wouldn't bet production on it past 1000 steps
we gave up on qlora for production entirely after hitting this on 3 separate projects. the vram savings aren't worth it when your training run dies at step 700 and you lose half a day of compute
we're seeing exact same pattern in production deployments across 4 client projects right now. llama 3.1 70b qlora with rank 64, lr 2e-4, gradient clipping at 1.0, and we still hit nans between step 680-920 on 3 out of 4 runs. the one that didn't hit nans had a tiny dataset (1800 examples, single domain). the vram tradeoff for full fine-tune is brutal but we're probably switching because we can't ship a training pipeline that fails 75% of the time. anyone found a config that's actually stable past 1500 steps on qlora or is this just fundamentally broken for long runs
we abandoned qlora for llama 3.1 70b entirely after it died on us 4 times between step 650-890. switched to full finetune and haven't hit a single nan in 7 runs since. the vram cost is brutal but at least training completes
We made the same switch on our training infrastructure three months ago after hitting NaN gradients on QLoRA runs consistently between steps 680-920 across multiple projects. The pattern was always the same: training loss would be smooth and decreasing, then suddenly explode to NaN with no warning in the logs. Switched to full fine-tune on 8xA100 80GB and haven't seen a single NaN in twelve separate training runs since then. The VRAM cost is brutal - QLoRA was using about 38GB per GPU, full fine-tune pushes us to 72GB - but at least the runs complete reliably. We can't afford to lose half a day of compute and a client deadline because of numerical instability. The interesting thing is that the QLoRA runs that DID complete past 1000 steps produced models with nearly identical eval scores to the full fine-tune versions, so it's not a quality issue when it works. It just doesn't work reliably enough for production use.
we're seeing exact same pattern on llama 3.1 70b qlora in production.... hit NaN gradients at step 743 on first run, step 812 on second run. both times rank 64, lr 2e-4, gradient clipping 1.0. switched to full finetune and it's been stable for 4 runs now but the vram cost is brutal
did gradient clipping help at all or same failure rate
qlora numerical instability is a known issue past ~500 steps on some model families. are you using bfloat16 or float16 compute dtype? also what's the actual error - nans in gradietn or just diverging loss? we hit this on llama 3.1 70b qlora and switchign to bf16 fixed it
we hit this exact thing in prod last month. nans in gradient at step 580-640 range on llama 3.1 8b qlora (rank 32, lr=2e-4, bfloat16 compute). turned out the issue was specific layers destabilizing - layer 18 gradient norm was exploding while global norm looked fine. added per-layer gradient clipping (max_norm=0.5 per layer instead of global) and it trained clean past 1200 steps
qlora numerical instability is overblown imo. we run qlora in prod on llama 3.1 70b (rank 64, bf16 compute) and regularly hit 2000+ steps clean. the issue is usually learning rate scheduling or bad data batches, not the quantization itself
qlora works fine if you babysit it. we run llama 3.1 8b qlora in prod (rank 32, lr 1e-4) and hit 1800 steps clean but only because we added gradient clipping at 1.0 and watched the loss curve like a hawk
gradient clipping at 1.0 helped but we still hit nans around step 940. switched to full fine-tune and it's been stable for 2200 steps but vram cost is brutal
ok so gradient clipping helps but doesn't solve it. we added clipping at 0.5 and still hit nans around step 1100 on llama 3.1 70b qlora. the vram tradeoff for full fine-tune is brutal but at least it actually trains past 2000 steps
we gave up on qlora for long runs entirely.... switched to full fine-tune with gradient checkpointing and it's stable past 3000 steps but the vram cost is rough. qlora just seems fundamentally unstable past a certain point and babysitting it isn't worth the savings
getting similar instability on qlora past ~550 steps. using bfloat16 compute and the gradients just explode around step 600. could be wrong but i think this is a known issue with 4bit quantization on certain model architectures imo
hit this last week on llama 3.1 8b qlora (rank 16, lr 1e-4). gradients exploded at step 620 with nans. switched to bfloat16 compute and it pushed the instability to step 840 but didn't fix it. ended up dropping lr to 5e-5 and adding gradient clipping at 1.0 - been stable for 1400 steps since then. what's your actual lr and are you clipping gradients
bfloat16 compute pushed it to 840 which tells me the issue is precision loss compounding across steps. tried mixed precision (fp16 grads + bf16 compute) on mistral 7b qlora last year and got to ~1100 steps before collapse 😅
ok so you switched to full fp16 and it still collapsed, just later. did you ever find a config that actually trains past 1100 steps clean or is qlora just broken for long runs
we hit nans around step 720 on llama 3.1 8b qlora (rank 32, lr 2e-4) and fixed it by switching to float32 for the lora adapters instead of bfloat16. training got slower but made it past 1400 steps clean. the instability seems to come from specific layers accumulating error in bf16