qwen 2.5 coder 32b vs deepseek v3 for code - tested on humaneval and mbpp
ok so i spent the weekend testing both models at q4_k_m on code generation (HumanEval 164 problems, MBPP 500 problems, plus 80 internal tasks from our codebase). results: - qwen 2.5 coder 32b: 78% pass@1 on HumanEval, 71% on MBPP, 64% on internal - deepseek v3: 61% pass@1 on HumanEval, 58% on MBPP, 71% on internal qwen is way better on the benchmarks but deepseek is weirdly better on our specific codebase (python + typescript, lots of async patterns). also deepseek uses 41gb vram vs qwen at 62gb. not sure what to make of this. the MoE architecture in deepseek might be learning domain-specific patterns better? or the quantization just hits qwen harder on out-of-distribution code?
does this generalize to other question types or just rhetorical. also layer 23 seems late for syntax stuff, would expect that earlier
ok so I've been testing qwen 2.5 coder 32b vs deepseek v3 on humaneval for like three days now and the results are all over the place.... qwen gets 76% pass@1 at q4_k_m, deepseek gets 58% at the same quant but 81% at q5_k_m. which is weird because you'd think MoE would be more robust to quantization? also deepseek is like 3x slower on a 4090. did you test mbpp or just humaneval
deepseek getting 58% on humaneval at q4 is honestly terrible lol. we tested qwen 2.5 coder 7b at q4_k_m and got 71%, way better than a 671b MoE. what's your tok/s on the deepseek? curious if the speed makes up for the accuracy drop
58% on humaneval at q4 is honesly terible lol. did you try q5 or q6 to see if its the quant or the model? imo this sounds like quantization destoryed the MoE routing