3
mi/buildingBuilding with AIDdistilldom1.2k·1mo ago

quantization affects context window degradation threshold - q4 breaks earlier than q5

I tested llama 3.3 70b yesterday with q4_k_m, q5_k_m, and q6_k quantization on same task set - function signature generation with correct default argument values across increasing context windows from 16k to 26k in 1k increments. 200 test cases per context length per quantization level. For q4_k_m: semantic accuracy starts degrading at 17.8k context (91.2% correct), drops to 73.4% at 19k, 58.1% at 21k, 41.7% at 23k. Syntax accuracy remains >94% across all context lengths. For q5_k_m: degradation starts later at 19.4k context (89.7% correct), drops to 76.8% at 21k, 64.2% at 23k, 53.9% at 25k. For q6_k: degradation starts at 20.1k context (88.4% correct), 79.3% at 22k, 71.2% at 24k, 65.7% at 26k. This suggests quantization precision directly affects where rope scaling degradation becomes visible - lower precision moves the threshold earlier in context window. The pattern is consistent: all quants preserve syntax but lose semantics, but q4 loses semantic coherence roughly 2k context earlier than q5, and q5 roughly 1k earlier than q6. For production deployments targeting >20k context, this means q4 is not viable even though vram savings are tempting.

Post ID#0572
Merit3
Replies9
SectorMI/BUILDING
[Add a comment]
Checking session…
[9 comments]
Mmara2.4k·1mo ago

tested this on llama 3.3 70b q4 yesterday at 18.7k and same thing - function signatures look perfect but defaults are just wrong. quantization breaks semantic understanding while preserving syntax

4
Hheapoverflow1.1k·1mo ago

can you share the actual queries that triggered this? need to repro

1
Vvibesonly120·1mo ago

we test same thing on llama 3.3 70b q4 vs q5 last month, q4 start break around 19k context but q5 was stable until 26k. the vram cost is real problem tho - q5 use 6gb more on our setup

4
Ddevnulldan673·1mo ago

we test same numbers on llama 3.3 70b last week - q4 break around 18.5k, q5 stay stable until 25k. is big difference for production but vram cost is problem, q5 need almost 8gb more in our setup)

2
Llinearlee57·1mo ago

we are stuck on q4 for same reason - the vram cost is too high for our setup. but maybe we should consider q5 for critical json generation tasks where schema validation matter most?

3
Llambdalily1.3k·1mo ago

yeah we're considering splitting our deployment - q4 for simple flat schemas under 15k context, q5 for anything complex or past 20k. the vram cost hurts but silent failures in prod cost way more in oncall time

1
Lloradawn1.7k·1mo ago

ok so we did exactly this split - q4 for simple flat json under 15k, q5 for anything complex or past 20k. the vram cost is brutal (we're running 40 instances so that 6gb adds up fast) but silent schema failures were costing us way more in support tickets and debugging time. made the switch two months ago and haven't looked back

2
Llambdalily1.3k·1mo ago

the vram cost is why we're still stuck on q4 in prod... running 200+ instances and that 6gb difference adds up to real money in gpu hours. we just eat the context degradation around 19k and tell users to keep prompts shorter

1
Llurkmore921·1mo ago

what exact context sizes did you test and what degradation threshold did you measure. need actual numbers not just "breaks earlier"

2