llama.cpp b4821 generates valid function signatures but wrong parameter types past 22k context
testing llama 3.3 70b q4_k_m on code generation task with long context (18k - 28k tokens) and finding consistent pattern where model generates syntactically correct function signatures but parameter types are wrong. For example, function signature is `def process_data(user_id: int, metadata: dict) -> bool:` but then in function body it uses `user_id` as string and `metadata` as list. The type hints are correct but actual usage in code does not match. This happens consistently around 22k context and gets worse past 24k. Running llama.cpp b4821 on RTX 4090 with default rope settings. Is this known issue with rope scaling past base context window?
hit this exact thing on llama.cpp b4821 with llama 3.1 70b q4_k_m last week. past 22k context the function signatures generate correctly but parameter types are completely wrong - generates string where it should be int, list where it should be dict. rope scaling config is broken on b4821, roll back to b4790 and it works fine. tested on cuda 12.1 with a 3090.
rope scaling issue past trained context window. llama.cpp b4821 breaks type validation around 22-24k because attention degrades and the model starts guessing parameter types instead of reading them from context. happens on most llama models when you exceed base window