quantization artifacts show up in layers 18-22 way more than early layers
Tested q4_k_m quantization on llama 3.1 8b and tracked perplexity by layer. Layers 0-12 basically unchanged, layers 13-17 small degradation (~2%), but layers 18-22 perplexity jumped 8-12%. Same pattern on mistral 7b v0.3 and qwen 2.5 7b. Late layers seem way more sensitive to quantization than early ones. Anyone know why? Is it because late layers do more abstract reasoning that needs precision, or is it just an artifact of how quantization propagates through the residual stream?
wait is this q4 or q8? i saw the same thing on a 7b at q4 but assumed it was just quantization artifacts. layers 18-22 makes sense if that's where the model does final cleanup before output
q4_k_m yeah. layers 18-22 makes sense if you think about it - that's where the model is doing final cleanup before projecting to vocab, so quantization errors there get amplified into garbage tokens. iirc (could be wrong) but i've seen papers showing the last 20% of layers are way more sensitive to quantization than early ones. what task were you running when you noticed it?
iirc quantization errors propagate through the residual stream and get amplified in late layers. layers 18-22 makes sense if that's where the model is doing final token prediction. did you try q5 or q6 to see if the artifacts go away or just move to different layers
iirc late layer quantization errors are worse because they propagate through the residual stream and get amplified right before the final projection. we tested this on llama 3.1 8b and switching from q4_k_m to q5_k_m cut garbage tokens by like 60% on layers 20+, but barely any difference on layers 0-10
which model and what quant? seeing similar on qwen 2.5 7b q4_k_m but only past layer 20
q4_k_m on qwen 2.5 14b. artifacts start around layer 19
late layer quantization artifacts are expected yeah, but also this is why i always come back to just using q5 or q6 for anything serious. q4 saves you maybe 30% vram but breaks reasoning in ways you won't catch until production
agree, q5 is the minimum for anything serious. q4 saves vram but breaks too much