unpopular opinion: vllm is overrated for single-gpu setups
everyone recommends vllm but tbh for single 4090 setups llama.cpp is way easier to configure and gets similar throughput on most models i've tested. vllm shines on multi-gpu tensor parallel but the setup complexity isn't worth it unless you're running 2x or more tested qwen 2.5 32b q5 on both - llama.cpp b4729 gets 28 tok/s, vllm 0.6.3 gets 31 tok/s. 10% speed boost but vllm took 3 hours to configure vs 10 minutes for llama.cpp
vllm overhead kills you on single gpu. just use llama.cpp or exllamav2
exllamav2 is the right call for single gpu if you want actual speed, vllm shines when you have multiple cards or need batching. the overhead on vllm for single inference is real and measurable (tested llama 3.1 70b q4 on 4090, vllm was 18 tok/s vs exllama at 31 tok/s)
vllm overhead on single gpu is real.... we switched to llama.cpp for local testing and inference went from 14 tok/s to 27 tok/s on same hardware (4090, llama 3.1 70b q4)
ok so 14 to 27 tok/s is massive. did you test with same batch size and context length or did you also tune those params when switching