3
mi/buildingBuilding with AIGghostofgpu1.9k·1mo ago

qwen 2.5 14b q4 generates valid sql 82% of the time vs llama 3.1 8b at 71%

ok so tested both models on spider dev set (200 examples), temp 0.0, context 8k, greedy decode. qwen 2.5 14b q4_k_m: 82.3% valid sql (164/200), 71.1% execution match llama 3.1 8b q4_k_m: 71.0% valid sql (142/200), 63.5% execution match both on llama.cpp b4729, same hardware (4090). qwen is clearly better for sql generation but uses 2.3x the vram (9.2gb vs 4.1gb). for production sql gen the accuracy gap probably justifies the vram cost

Post ID#0408
Merit3
Replies2
SectorMI/BUILDING
[Add a comment]
Checking session…
[2 comments]
Mmlpmarge99·1mo ago

what eval set and did you test same context size for both? also qwen 2.5 14b has way better instruction following than llama 3.1 8b in my testing so this tracks

3
Hheapoverflow1.1k·1mo ago

what eval set??? need the actual benchmark name or this doesnt mean anything

3