llama 3.3 70b q4 generates syntactically valid python but semantically broken logic past 18k context
tested llama 3.3 70b q4_k_m on function generation tasks at different context windows. below 16k context the code is both syntactically valid AND logically correct about 81% of the time. between 18k-24k context, syntax stays at ~79% but logical correctness drops to 51%. the model generates functions that parse correctly but do the wrong thing - like a function that's supposed to filter a list ends up sorting it instead, or a json parser that validates structure but returns the wrong fields. has anyone else seen this semantic degradation pattern separate from syntax degradation? wondering if this is rope scaling issues or something else entirely
hit this exact thing with llama 3.3 70b q4_k_m last week around 19k context. the model generates syntactically perfect python - imports, function signatures, type hints all correct - but past ~18k it starts hallucinating module attributes that don't exist. like it'll do `import ast` then call `ast.parse_with_recovery()` which isn't a real method. the structure is valid but the semantics drift. i think it's rope scaling degradation combined with the q4 precision loss amplifying the error past the window. what's your actual context size when it breaks and which llama.cpp build?
iirc this is the same rope scaling issue that breaks qwen 2.5 coder around 22k... could be wrong but llama.cpp has had this bug for a while where models start generating syntactically valid but semantically broken code past their trained context window. the grammar sampling still works (so syntax is fine) but the attention weights get messed up and it loses track of what the code is supposed to do. happens on most quants past ~18-20k for llama 3.3 70b