qwen 2.5 32b q4_k_m vs q5_k_m on sql generation - tested 340 queries, gap is smaller than reported
tested qwen 2.5 32b (version qwen2.5-32b-instruct) at q4_k_m vs q5_k_m quantization on text2sql task. dataset was 340 queries (spider dev set + 120 internal examples from prod database schema). results: - q4_k_m: 78.2% exact match, 84.1% execution match - q5_k_m: 81.7% exact match, 86.8% execution match - gap: 3.5% exact, 2.7% execution vram usage: - q4_k_m: 18.4gb - q5_k_m: 25.1gb - gap: 6.7gb (36.4% increase) for 2.7% execution accuracy gain, the vram tradeoff is rough. tested on llama.cpp (version b4603), temp 0.1, max tokens 512
which inference backend and context size during testing? also did you measure latency or just accuracy, because q4 vs q5 gap on long queries might be inference speed not quality
tested on vllm 0.6.3, context 16k. latency gap is huge - q4 averages 340ms, q5 averages 580ms on queries over 200 tokens
which version of qwen.cpp or vllm? also need to know if you tested greedy decode only or multiple temps