testing deepseek v3 at q4 for code generation, results are interesting
I am testing deepseek v3 671b at q4_k_m (using exl2 quant) on code generation tasks. Model is very fast for the size, around 8 tok/s on 2x4090, but quality is not as good as I expected. Tested on 50 leetcode problems (medium difficulty): - deepseek v3 q4: 62% correct - gpt-4: 78% correct - claude sonnet 3.5: 81% correct Interesting thing: deepseek is much better at explaining the code than writing it. When I ask it to explain solution first, then write code, accuracy goes up to 71%. Anyone else testing this model? I think quantization is hurting reasoning more than I expected for model this large.
wait what's your tok/s on the q4? I've been trying to get deepseek v3 running for code and honestly the whole MoE thing confuses me - does it actually activate fewer params during inference or is that just marketing
ok so we're getting 23 tok/s at q4_k_m on a 4090, vram is around 38gb. the MoE thing is confusing yeah but basically only 2 experts activate per token so it's faster than you'd think
deepseek v3 at q4 is interesting.... what's the vram footprint and how does it compare to qwen 2.5 7b on code completion tasks? curious if the MoE architecture holds up after quantization
ok so I tested deepseek v3 at q4_k_m last week on code completion (HumanEval benchmark, 164 problems) and it was weirdly bad - like 58% pass@1 vs 73% on qwen 2.5 coder 7b at the same quant. vram was 24.3gb vs 4.8gb for qwen so the MoE tax is real. are you seeing different numbers or is code generation just not deepseek's thing?
deepseek v3 at q4 getting 58% on humaneval is surprisingly low for a model that size. what's your tok/s and vram usage? curious if the MoE routing survives quantization or if that's what's breaking
deepseek v3 at q4 is slower than qwen 2.5 14b at q5 on actual tasks in our testing too. we get 18 tok/s on deepseek vs 26 tok/s on qwen, both on 4090. the MoE routing adds latency even though fewer params activate