llama 3.3 70b q4 generates valid python but wrong variable scope past 24k context
1. tested llama.cpp b4821 with llama 3.3 70b q4_k_m on code generation tasks 2. past ~24k context the model generates syntactically perfect python but breaks variable scope - references variables before they're defined, uses closure variables that don't exist in scope, etc 3. same prompts at 20k context work fine, scope is correct
This sounds like the same rope scaling issue that affects most llama.cpp models past their base context window. When you exceed the trained context length (usually 32k for llama 3.3 70b), the rotary position embeddings start to degrade and the model loses track of variable scope across long distances. The quantization makes it worse because q4 has less precision to represent the position information compared to q5 or q6. I saw this exact behavior on llama 3.1 70b q4_k_m around 26k context - the model would generate syntactically perfect Python with correct imports and function signatures, but variable references would point to the wrong scope, especially in nested functions or class methods. It's particularly bad with closures because the model has to track scope across hundreds or thousands of tokens and the position embeddings just aren't accurate enough past the rope window. You can try adjusting the rope frequency base or rope scaling factor in llama.cpp, but honestly the best fix is to just stay under 24k context or upgrade to q5/q6 if you need long context. Which llama.cpp build and what rope settings are you using?
ok so this sounds like rope scaling issue past the base context window. llama.cpp b4821 has known bug where variable scope gets confused around 24-28k. which quant and what rope settings?