llama 3.3 70b at q4_k_m fails math word problems that the fp16 model solves fine
Been running extensive tests on llama 3.3 70b across different quantization levels for the past 72 hours. The results are honestly frustrating. fp16: 89% accuracy on GSM8K (tested 200 problems) q8_0: 87% accuracy (basically identical) q5_k_m: 84% accuracy (acceptable degradation) q4_k_m: 61% accuracy (completely falls apart) The failure mode at q4_k_m is specific and reproducible - multi-step arithmetic word problems that require carrying intermediate results across reasoning steps. The model will get the first 2-3 steps correct, then suddenly produce garbage numbers that don't relate to anything in the problem. I'm not talking about complex math here. These are problems like "John has 47 apples, gives away 23, then buys 18 more." At q4_k_m it'll correctly compute 47-23=24, then claim 24+18=67 or some other hallucinated number. Anyone else seeing this pattern? I'm wondering if there's a specific layer range where the quantization is destroying the arithmetic circuit. Might try some activation patching to narrow it down.
we're seeing the same thing on llama 3.3 70b at q4_k_m - math word problems that require multi-step reasoning break maybe 60% of the time vs 8% at fp16. the failure mode is usually the model losing track of intermediate values (like it'll correctly compute step 1, then hallucinate the result when using it in step 2). q5_k_m is way better - only breaks 12% of the time. what's your actual failure rate and did you test q5 as a comparison?
we're seeing the exact same thing! llama 3.3 70b at q4_k_m fails multi-step math word problems around 55% of the time vs maybe 12% at fp16. the model will get through steps 1-2 correctly then suddenly lose track of what variable it was solving for. switching to q5_k_m drops the failure rate to around 18%. is this just quantization destroying the residual stream in late layers?