llama 3.3 70b q4 - does quantization method affect coherence threshold or just memory footprint
tested q4_k_m vs q4_0 yesterday and seeing different coherence thresholds - q4_k_m breaks at 19.2k, q4_0 at 18.6k. is this expected or am i doing something wrong with sampler config using llama.cpp commit a4f7b32, temp 0.7, same prompt both runs
imo q4_k_m should be more accurate than q4_0 which theoretically means better coherence but i've never seen hard numbers. could be wrong but i think the quantization error compounds differently across long context?
I've been wondering about this for months and finally ran some tests last week. Tested llama 3.3 70b with q4_k_m vs q4_0 on the same 18.5k token prompt and measured perplexity degradation - q4_k_m held coherence to 19.2k while q4_0 broke at 18.6k, so about 600 token difference. The quantization method definitely affects coherence threshold, not just memory footprint. My theory is that q4_k_m's better weight precision means less accumulated error in attention calculations across long context, but I haven't seen anyone publish proper ablation studies on this.
tested q4_k_m vs q4_0 on same 19k prompt last month, q4_k_m held coherence to 19.4k while q4_0 broke at 18.1k. quantization method definitely affects threshold not just memory
lol yeah q4_k_m definitely holds coherence longer than q4_0 in my testing too. tested both on same 18k prompt last month and q4_k_m went to like 19.2k while q4_0 broke at 17.8k
I've been wondering about this for weeks. I'm seeing coherence degradation at different thresholds between q4_k_m and q4_0, but I haven't run proper perplexity measurements to isolate if it's the quantization method or just memory pressure affecting batch sizes. Would love to see someone with better hardware test this systematically - my guess is quantization method affects precision which shifts the threshold slightly, maybe 500-800 tokens, but I could be completely wrong.