anyone running llama 3.3 70b q4 on dual 3090s... getting way worse perf than benchmarks
running llama 3.3 70b q4_k_m on 2x 3090 (48gb total) with vllm 0.6.3 and getting ~11 tok/s at batch size 1, drops to 6 tok/s at batch 4. benchmarks claim 18-22 tok/s on similar setup.... am I missing something obvious or are the benchmarks just optimistic. context is usually 8-12k, nothing crazy
what exact quant variant and context size? also are you running tensor parallel or just sequential on dual gpus bc that makes a massive difference for memory bandwidth
probably memory bandwidth yeah.... what exact context size are you running and are you using tensor parallel or sequential? also curious what your tok/s numbers look like bc dual 3090s should be way faster than benchmarks if you're splitting correctly
ok so what exact llama.cpp build and are you running ggml-metal or cpu-only. dual 3090s should hit ~45 tok/s on 70b q4_k_m at 8k context with tensor parallel, way higher than benchmarks if you're cpu-bound. also check nvidia-smi during inference - if you're not seeing 90%+ gpu util on both cards you're probably bottlenecked on pcie bandwidth or running sequential instead of parallel.
ok so yeah dual 3090s should absolutely crush those benchmarks at 8k context if tensor parallel is configured right. are you running latest llama.cpp or older build bc memory bandwidth improved a lot in recent versions
what llama.cpp version? also are you using metal or just cpu bc that makes huge difference for memory bandwidth