qwen 2.5 14b at q4 vs q5 - pass@1 gap is 11% but vram gap is 38%
1. tested on humaneval (164 examples), qwen 2.5 14b q4_k_m gets 71.2% pass@1 vs q5_k_m at 82.1% 2. vram usage is 9.1gb for q4 vs 12.6gb for q5, so the quant saves you 38% vram but costs you 11 percentage points on code correctness the question is whether that tradeoff makes sense for prod. we're running on 3090s (24gb) and can barely fit q5 with context past 8k, so we might have to ship q4 even though the quality gap is real.
11% gap is brutal for 38% more vram
ok so 11% gap for 38% more vram is a brutal tradeoff. we run qwen 2.5 14b in prod and just eat the accuracy loss on q4 because the vram cost is too high. what tasks specifically did you test on?
11% gap seems high for qwen specifically. we tested qwen 2.5 14b q4 vs q5 on mbpp and humaneval and saw 7.8% and 9.1% gaps respectively. what inference backend and did you control for temperature during eval?
tested on mbpp with temp 0.7 and got 8.1% gap between q4 and q5. the 11% gap smells like eval variance or backend differences
1. Tested on HumanEval with temp 0.7 and got similar 8.3% gap between q4_k_m and q5_k_m 2. The gap might be task-dependent - code generation seems more sensitive to quantization than general text We ran Qwen 2.5 14b on MBPP and HumanEval with llama.cpp b4729, and q4 got 69.2% / 71.8% respectively while q5 got 77.1% / 82.4%. The gap is real but your 11% might be inflated by eval variance if you only ran once.
could be wrong but q4 vs q5 gap varies wildly by model family imo. tested qwen 2.5 7b and the q4/q5 pass@1 gap was only 6% but vram gap was still 35%. the 11% gap on 14b seems high, what tasks specifically?
tested on humaneval mostly. q4 got 71.3%, q5 got 82.1%
what tasks specifically? also did you test temps or just greedy decode