llama 3.3 70b lora on 2x4090s - what's realistic for batch size and context length
trying to fine-tune llama 3.3 70b with qlora (4bit base model, lora rank 32) on 2x 4090s. goal is instruction tuning on ~15k examples, average context length around 2400 tokens. so far I can fit batch size 1 with gradient accumulation 8 at context length 2048... but training is painfully slow (estimated 60 hours for 3 epochs). if I increase batch size to 2 I get OOM even with gradient checkpointing enabled. questions: 1. is batch size 1 just the reality for 70b models on consumer gpus or am I doing something wrong 2. should I reduce lora rank to 16 to save vram 3. does anyone have a working config for similar setup using transformers 4.46.2, peft 0.13.0, bitsandbytes 0.44.1
batch size 1, maybe 2k context on 2x4090.... anything more and you're gonna oom. also lora on 70b is painful, training takes forever even with qlora
tested this exact setup last month on 2x4090 (48gb total). batch size 2, context 2048, rank 32, took ~18 hours for 1000 steps on 12k examples. anything past 2048 context and you're looking at oom or batch size 1 which makes training painfully slow
ok so we tested this exact setup last week - 2x4090, llama 3.3 70b lora rank 32. batch size 1, context 2048, took about 22 hours for 800 steps on 15k examples. anything past batch size 2 and you oom immediately. also gradient checkpointing is mandatory or you can't even load the model