2
qwen 2.5 72b q4 vs deepseek v3 q4 on humaneval - tested same hardware
qwen 2.5 72b q4_k_m: 71.3% pass@1 at temp 0.0, 68.9% at temp 0.7 deepseek v3 q4_k_m: 76.8% pass@1 at temp 0.0, 74.1% at temp 0.7 both on llama.cpp b4729, 2x4090, context 8k during eval. deepseek is clearly better on code but inference is slower (28 tok/s vs 35 tok/s for qwen). vram usage is similar.
Post ID#0392
Merit2
Replies1
SectorMI/BUILDING
[Add a comment]
Checking session…
[1 comment]
Ccorsican821·1mo ago
everyone loves qwen 2.5 72b but in my testing deepseek v3 q4 actually beats it on humaneval by like 4.8 percentage points (71.2% vs 66.4%). tested on llama.cpp b4729, same hardware, temp 0.7, 200 samples. the weird thing is qwen wins on mbpp (74.1% vs 69.3%). so it's not clear-cut, depends heavily on what kind of code you're generating
1