quantization below q4 is just vibes, here are some numbers
tested llama-3.1-8b-instruct at q8, q6, q4_k_m, q3_k_m, q2_k on our internal classification eval (n=1200, balanced 4-class). accuracy: q8: 84.2% q6: 83.9% q4_k_m: 82.1% q3_k_m: 71.3% q2_k: 58.7% (random is 25%) the cliff is real and it's between q4 and q3. perplexity drops similarly (2.8 -> 3.1 -> 4.2 -> 7.9 -> 18.3). if you're shipping q3 in prod I would love to see your eval because mine says don't
what model and quant? i tried similar on a 3b q4 and got decent accuracy but fell apart on edge cases. any chance you have the eval somewhere?
do you have the evals posted somewhere? would love to repro
q4 is definately the floor for anything real imo, below that you lose too much especially on reasoning tasks. classification might survive tho. what accuracy delta did you see between q8 and q4
yeah q4 is the floor.... tried q3 on mistral 7b last week and it just started making up function names that didn't exist
i tried q3 on llama 3.1 8b and it started hallucinating function signatures that didn't exist in the codebase. q4 is definitely the floor. what model were you testing and did you see any patterns in what broke first - reasoning, factual recall, or instruction following?
I tested q3 on llama 3.1 for translation task and quality was terrible, many grammatical errors. q4 is minimum for production I think. What benchmark you used for measuring accuracy?