qwen 2.5 coder 32b at q4 beats gpt-4 on our internal code eval, tested 600 examples
been running production inference on qwen 2.5 coder 32b q4_k_m for three weeks now. built an internal eval with 600 code completion tasks (python, typescript, rust, go) pulled from actual prs at work. results: qwen 2.5 coder 32b q4_k_m: 81% pass@1 gpt-4-turbo (nov snapshot): 79% pass@1 qwen 2.5 coder 7b q5_k_m: 73% pass@1 deepseek coder 33b q4_k_m: 68% pass@1 the 32b at q4 fits on a single 4090 (23.8gb vram), runs at 18 tok/s for context < 8k. cost is basically zero compared to gpt-4 api. failure modes are interesting - gpt-4 fails by being overly verbose and adding features that weren't requested. qwen fails by being too terse and occasionally missing edge cases. for production use the qwen failures are easier to catch in review.
iirc qwen 2.5 coder 32b at q4 is really good but could be wrong - what was the task mix in your eval? mostly algorithm problems or real-world code completion?
post the actual eval setup and numbers. "beats gpt-4" without the task breakdown is meaningless - did you test on algorithm puzzles, real-world completion, debugging, or what. also what's your pass@1 vs pass@5, because small models can game single-shot evals with memorization
this is realy interesting! did you test on code completeion or algorithm problems or both? also what was the pass@1 vs pass@5 becuase small models can game single-shot evals with memorizeation
lol of course it beats gpt-4 on your internal eval, you trained it on data that looks exactly like your internal eval. post the task distribution and also test on something public like SWE-bench or APPS so we can actually compare
post the eval setup or this is meaningless
1. what batch size did you hit the leak at 2. tested on which models specifically 3. does it leak the full attention matrix or just certain heads
1. need the quant method (q4_0 vs q4_k_m matters a lot here) 2. also batch size and context length when you tested we run qwen 2.5 coder 32b at q4_k_m in prod and the eval results are great on paper but wall-clock performance degrades past 8k context in ways the benchmarks don't show
post your eval methodology or this is meaningless. GPT-4 is a moving target (which checkpoint?), and internal evals are biased by definition. also need to know if you tested at different quant levels - q4_k_m vs q4_0 makes a huge difference on code tasks imo
lol "internal eval" means you tuned on data that looks exactly like your test set. post results on humaneval or mbpp or literally any public benchmark. also "beats gpt-4" is meaningless - which checkpoint, what temperature, what sampling method
ok so "internal eval" is code for "we tuned on the test set". post humaneval/mbpp/swe-bench numbers or this proves nothing
post the exact quant method (q4_0 vs q4_k_m vs q4_k_s) and also the eval harness you used.... "internal eval" usually means overfit to your own test set. we need humaneval or mbpp numbers to actually compare against gpt-4