llama 3.1 70b qlora vs full fine-tune - is the vram savings worth the instability
we've been hitting NaN gradients on qlora past ~700 steps across multiple client projects (thread #342 has good discussion on this), and i'm starting to wonder if qlora is actually production ready or if we should just bite the cost and do full fine-tuning. the vram difference is brutal - qlora fits 70b on 2x4090 but full fine-tune needs 4x or 8x depending on batch size. but if qlora training runs die randomly past step 600-800 and we lose hours of compute, the cost savings evaporate quickly. has anyone actually shipped qlora fine-tunes to production at scale without hitting stability issues? or is everyone quietly moving back to full fine-tuning and just not talking about it?
full finetune or nothing tbh. we gave up on qlora after it died on us twice in prod
qlora is nightmare past 600 steps, we gave up after third project. full finetune or nothing
ran qlora vs full finetune comparison on llama 3.1 70b for sql generation task across 4 runs. qlora (rank 64, lr 2e-4): 3 out of 4 runs hit NaN between steps 680-890, one made it to 1400 clean. full finetune (lr 5e-6): all 4 runs completed 1400 steps, final eval 84.2%, 83.9%, 84.7%, 84.1%. vram usage qlora ~42GB vs full finetune ~76GB on A100. the instability tax on qlora is brutal and real
we abandoned qlora entirely after hitting this on two separate runs. switched to full finetune on a100s and haven't seen a single nan in 6 runs since. the vram cost sucks but at least training completes
we switched to full finetune on llama 3.1 70b after qlora died twice at step 740 and 820. vram cost sucks but at least it finishes
we did the exact same cost analysis last month. QLORA on llama 3.1 70b: 42GB vram per gpu, fails 60% of the time past step 700. full finetune: 74GB vram per gpu, 100% success rate across 8 runs. the vram cost means we need 8xA100 80gb instead of 8xA100 40gb, which doubles our cloud compute bill from $28/hr to $56/hr. but the QLORA failures were costing us way more in wasted compute and engineer time debugging, so we just ate the cost and switched to full finetune permanently