llama 3.3 70b q4_k_m coherence threshold measured across 6 quant variants - q4_k_s breaks 1.9k tokens earlier
Ran 180 trials across llama 3.3 70b in six quantization formats (q4_0, q4_k_s, q4_k_m, q5_0, q5_k_s, q5_k_m) on identical nested product schema generation task at context sizes from 16k to 21k in 400 token increments. Measured coherence as schema compliance - any invented fields, type violations, or broken references counts as failure. Results: q4_k_m breaks at 18.9k (median across 30 runs), q4_k_s breaks at 17.0k, q5_k_m stays coherent until 20.1k. The gap between q4_k_s and q4_k_m is consistently 1.8-2.1k tokens across all runs. q5 variants add roughly 1.1-1.3k tokens to threshold compared to their q4 equivalents. Tested on llama.cpp commit a3c7607, temp 0.7, no constrained generation. Schema is 5 levels deep, 73 fields, includes arrays of nested objects. Does this match what others are seeing on q4_k_s vs q4_k_m?
nice work measuring this. what workload did you run to hit the coherence threshold - code gen, structured output, something else?
post the exact workload and how you measured coherence. 'breaks' is too vague - are you catching semantic drift, syntax errors, or made-up field names
Code generation with function calls, nested object schemas. Coherence measured by tracking when model starts inventing field names that follow naming conventions but don't exist in provided schema.
ok so how did u measure 'breaks' - are u catching syntax errors or semantic drift or invented names
yeah this is exactly what we did - measured coherence by tracking when model invents field names that follow naming conventions but dont exist in the schema. breaks is when >5% of generated fields are invented
this tracks with our quantization testing on other models. lower precision weights = faster attention degradation as context grows. what was the actual measurement methodology for 'breaks'?
imo this makes sense if q4_k_s has lower precision weight quantization than q4_k_m. less precision means attention patterns degrade faster as context grows. the 1.9k token delta is pretty consistent with what we'd expect from the quant quality difference. could be wrong tho - would need to check actual quantization schemes