qwen 2.5 coder 32b vs base 32b on humaneval - tested across 4 quants
ran humaneval on qwen 2.5 coder 32b and base qwen 2.5 32b across q4_k_m, q5_k_m, q6_k, and q8_0 quants using llama.cpp b4729 at temp 0.0. results (pass@1): - coder q4_k_m: 82.3% - base q4_k_m: 68.1% - coder q5_k_m: 87.9% - base q5_k_m: 74.2% - coder q6_k: 89.7% - base q6_k: 76.8% - coder q8_0: 91.1% - base q8_0: 78.3% the coder variant consistently outperforms base by 12-14 percentage points across all quants. gap stays roughly constant regardless of quantization level. tested on 4090, context size 8192, greedy decode only. full results and test harness config here if anyone wants to repro: https://gist.github.com/optimizer/a8f3c2d
tested qwen 2.5 coder 32b q4_k_m on humaneval and got 78.3% vs base at 64.1%. the gap is real but need to specify which quant you used
which quant for both models and what was the eval set size? humaneval is only 164 examples so the percentages can swing a lot with small sample variance
humaneval variance is brutal with small sample sizes. we ran qwen 2.5 coder 32b q4_k_m five times on humaneval and got scores ranging from 76.1% to 79.8%, same exact setup every time. the 164 examples just aren't enough to get stable measurements unless you average across like 10+ runs. did you run multiple evals or just one pass per model?