qwen 2.5 72b q4_k_m vs q5_k_m on spider sql benchmark - 8.3 percentage point gap
tested qwen 2.5 72b on spider dev set (1,034 examples) across q4_k_m and q5_k_m quants using llama.cpp b4729. results: q4_k_m: 78.2% exact match accuracy (809/1034 correct) q5_k_m: 86.5% exact match accuracy (894/1034 correct) tested at temp 0.0, context size 8192, greedy decode. same prompts, same hardware (A100 80GB), same llama.cpp version. the 8.3 percentage point gap is way bigger than i expected for one quant level. breakdown by query complexity: - simple SELECT: q4 91.2%, q5 94.1% (gap 2.9pp) - joins: q4 76.8%, q5 84.3% (gap 7.5pp) - nested subqueries: q4 61.4%, q5 78.9% (gap 17.5pp) the gap grows with query complexity. for production sql generation q4 is probably not usable if you need reliability on complex queries
8.3 points is huge for sql accuracy. are you seeing the gap consistent across different query types or does it spike on specific patterns like joins or subqueries
yeah we tested on our sql dataset and q4 fails hard on subquerys with like 3+ levels of nesting. q5 handles them fine tho
does the gap spike on joins and subqueries or is it consistent across query types
yeah it spikes on joins. tested qwen 2.5 72b q4 vs q5 on spider and q4 fails hard on any query with more than 2 table joins