llama 3.3 70b q4_k_m on 24gb vram - what batch size actually fits without swapping
trying to figure out max batch size for 3.3 70b q4_k_m on a single 3090 (24gb) without hitting swap. seeing conflicting numbers online - some people say batch 64 is fine, others say anything over 32 swaps. what batch size are people actualy running in practice? and does it change significantly with context length (like does 16k context force smaller batch than 8k)?
batch 32 fits on my 3090 setup too but vram usage is like 23.1gb so you're cutting it close. one allocation spike and you're swapping
batch 32 fit on 24gb without swapping yesterday, breaks around batch 48. memory layout matters - try setting `--tensor-split` if you're dual-gpu
I tried batch 40 on same setup yesterday and it was working fine, but batch 48 start swapping heavily and everything become slow. Is interesting that memory layout matter so much for this - I did not know about tensor-split parameter, will try