llama 3.3 70b q4_k_m on dual 4090s getting 38 tok/s at 12k context, benchmarks say 52 tok/s - what am i missing
running llama.cpp b4027 with tensor parallel enabled on dual 4090s (48gb total vram). at 12k context im getting 38.2 tok/s avg but the benchmarks on huggingface say this setup should hit 52+ tok/s easy. context size: 12288 quant: q4_k_m llama.cpp build: b4027 gpu setup: dual rtx 4090, tensor parallel enabled tok/s: 38.2 avg across 50 runs are the benchmarks just wrong or am i missing something obvious in config. memory bandwidth looks fine, both gpus saturated around 89%
ok so what exact llama.cpp build and tensor parallel config. dual 4090s should crush 52 tok/s if ur using latest build with proper memory bandwidth
probably offload layers split wrong or memory bandwidth throttled. dual 4090s should destroy those benchmarks at 12k if tensor parallel is configured right. what's your actual vram usage per gpu and are you hitting swap