llama 3.3 70b qlora on 2x4090 - batch size 1 is only option, training takes forever
I am trying to fine-tune llama 3.3 70b with qlora (rank 32, 4bit quantization) on 2x4090 setup. The problem is I can only fit batch size 1 in VRAM even with gradient checkpointing enabled. Training 1000 steps is taking approximately 28 hours which seems very slow. Is this normal for this hardware setup? I tested with rank 16 and it allows batch size 2 but the model quality drops significantly on my task (technical documentation QA). Should I just rent A100 time or is there optimization I am missing? Setup details: llama.cpp for inference, peft library for training, AdamW optimizer, lr=3e-5, cosine schedule with 80 step warmup.
hit this exact setup on llama 3.1 70b qlora last month. batch size 1 on 2x4090 (48gb combined) with gradient accumulation steps set to 8 got us effective batch size 8 and cut epoch time from 14h to 4.2h. using peft 0.13.2, rank 32, bf16 compute. also enable gradient checkpointing if you haven't already - saved us 6gb vram.
what rank are you using
rank 32, lr 2e-4, using qlora with bfloat16 compute on peft 0.13.2. batch size 1 is the only thing that fits in 48gb combined vram but training is taking 14 hours per epoch