llama 3.3 70b q4 generates valid python but imports wrong exception classes past 24k context
testing llama 3.3 70b q4_k_m on coding tasks and past 24k context it generates syntactically correct python but imports wrong exception classes. like it writes `from requests.exceptions import HTTPError` which is correct, but then also generates `from requests.exceptions import NetworkError` which doesn't exist - the actual class is ConnectionError. the syntax is perfect, the logic makes sense, but the specific class names are hallucinated. happens consistently past ~24k context. anyone else seeing this?
tested llama 3.3 70b q4_k_m at 26k context yesterday with python file containing 18 exception imports and hit exact same thing you describe. first 12 imports were correct (ValueError, TypeError, KeyError etc) but last 6 hallucinated to nonexistent exception classes like DataValidationError and SchemaParseException that dont exist in stdlib. the wild part is the syntax is perfect and the exception names sound plausable so you dont catch it until runtime. is this the same rope scaling degradation everyone keeps talking about or is there something else breaking down past 24k context?? need to understand the actual failure mode here
wait so it gets the import statement syntax right but just picks the wrong exception class?? that's such a weird failure mode. makes me wonder if it's memorizing common patterns at the start of context but losing semantic understanding of what exceptions actually exist in the codebase past 20k 🤔
ok so it's memorizing import statement patterns but the semantic mapping to actual exception classes degrades past 24k? that's wild because it suggests the context window extension is preserving syntax but losing the deeper relational knowledge about which exceptions exist in which modules 🤔 does this happen with other python constructs or just exceptions?
what context size are you hitting this at exactly
tested llama 3.3 70b q4_k_m version 3.3-instruct specifically at 24.3k context yesterday with python codebase containing 18 custom exception classes in context. generated code imported ValueError when it should have imported CustomValidationError, and imported KeyError when it should have imported MissingFieldError. the syntax was perfect but the exception class names were completely wrong