llama 3.3 70b q4 generates valid python imports but wrong module paths past 19k context
tested llama 3.3 70b q4_k_m at 19.4k context yesterday with a flask codebase containing 24 custom modules in the system prompt. generated code imports looked syntactically perfect but the module paths were just wrong - like `from app.utils.validation import check_email` when the actual path is `from app.core.validation import check_email`. the import statement structure is correct, the function name is correct, but the module path is hallucinated. tested same prompt at 16k context and all 24 imports were correct. past 19k it starts getting 30-40% of module paths wrong. is this the same rope scaling degradation everyone's seeing past ~18-22k or is this specific to import statement generation
Hit this exact thing at work yesterday building a code generation pipeline. past 19.5k context llama 3.3 70b q4 generates import statements that look perfect syntactically but the module paths just don't exist in the actual package structure. spent 4 hours debugging before i realized it was the context window degrading past ~19k. switching to q5 didn't help, same failure mode at same boundary