deepseek v3 at q4 is slower than qwen 2.5 14b at q5 on actual tasks, tested wall-clock time
everyone keeps saying deepseek v3 is fast because MoE but i tested it on humaneval (164 problems) and measured actual wall-clock time including retries. qwen 2.5 14b at q5_k_m averaged 8.2 seconds per solution with 76% pass@1. deepseek v3 at q4_k_m averaged 11.4 seconds per solution with 61% pass@1. the tok/s is higher on deepseek (31 vs 19) but the lower accuracy means you retry more often, which kills the speed advantage. also vram is 68gb vs 24gb. where is the benchmark that shows deepseek is actually better for code? i want to see the methodology
ok so what's your actual tok/s numbers on both? also vram usage? curious if the MoE routing in deepseek is just killing performance even at q4
we're getting 18 tok/s on deepseek v3 at q4_k_m on a 4090, vram sits around 39gb. the MoE routing does survive quantization but honestly the speed hit makes it not worth it for most code tasks - qwen 2.5 coder 7b at the same quant is like 3x faster and more accurate
1. we're seeing 21 tok/s on deepseek v3 at q4_k_m on a single 4090, vram around 37gb 2. the MoE routing definitely survives quantization but the speed is still terrible compared to dense models at the same quality level the real issue is that deepseek activates 2 experts per token so you're moving way more data around even though the active params are lower. not worth it imo unless you need the absolute best quality