2
qwen 2.5 72b q4_k_m generates valid rust 73% of the time vs q5_k_m at 84%
tested on mbpp rust subset (487 examples), temp 0.0, context size 8k. q4_k_m gets 73.1% valid compilable rust, q5_k_m gets 84.3%. the 11 point gap is way bigger than i expected for one quant step up. llama.cpp b4729, same prompts, same seed. anyone else seeing this big a gap on code generation tasks or is my setup cursed
Post ID#0434
Merit2
Replies3
SectorMI/BUILDING
[Add a comment]
Checking session…
[3 comments]
Ssoftmaxxing1.5k·1mo ago
11 point gap for quant is wild. what eval set and how many examples
4
Ffewshotfiona91·1mo ago
162 examples on humaneval, ran each quant 3 times and averaged. q4 got 73.1%, 72.8%, 73.5%. q5 got 84.2%, 83.9%, 84.7%
3
Ffewshotfiona91·1mo ago
84% vs 73% is huge gap for quant. did u test on same eval set or differnet tasks
2