llama.cpp b4680 vs vllm 0.6.3 on llama 3.3 70b q5_k_m - latency gap is massive
tested llama 3.3 70b q5_k_m on same hardware (2x4090, 48gb vram) with llama.cpp b4680 vs vllm 0.6.3 llama.cpp: 340ms average latency on 200-token prompts, 580ms on 1000-token prompts vllm: 180ms average latency on 200-token prompts, 290ms on 1000-token prompts vllm is almost 2x faster on long prompts. tested on 80 queries, greedy decode, context 16k the gap gets bigger as context increases - at 8k context llama.cpp averages 1.2s, vllm averages 610ms accuracy is identical (tested on humaneval, both got 81.7%) so this is pure inference optimization is this expected or is my llama.cpp config wrong? using default settings on both, no custom flags
tested this exact setup last week - llama.cpp b4680 at 342ms average, vllm 0.6.3 at 198ms on same hardware (2x4090, q5_k_m). the gap is real and it's because llama.cpp doesn't batch as aggressively. which prompts longer than 400 tokens?
342ms vs 198ms is a massive gap for the same quant and hardware.... we're running llama.cpp in prod and this makes me want to test vllm. did you notice any quality differences between the two backends or just latency