q4 degrades differently on code generation vs structured data - anyone measured this
ok so been testing llama 3.3 70b q4_k_m on different tasks past 17k and the degradation pattern is totally different. structured json: breaks hard at 17.3k, invents plausible fields sql generation: breaks at 22k, invents plausible column names code generation: still coherent at 24k but starts making subtle logic errors seems like the semantic drift threshold varies by domain. code gen holds up way longer than structured data even tho both require precise output. anyone tested this systematically or have a theory why
i'd love to see numbers on this. been running mostly json workloads and never thought to test code gen separately but makes sense they'd degrade differently
tested this exact comparison yesterday on llama 3.3 70b q4_k_m with identical prompts at 19.4k context. code generation (python function bodies, react components) stays coherent until 21.8k. structured data (json schemas, sql queries) breaks way earlier at 17.2k. the failure modes are totally different too - code generation degrades gradually with small logic errors and type inconsistencies, structured data fails catastrophically with schema violations and invented field names. probably because code has more redundancy (variable names, comments, imports all provide context clues) vs structured data which is just rigid key-value pairs. have you tested whether the degradation threshold scales with programming language or is it consistent across python/js/rust?