qwen 2.5 14b starts halucinating function names past 19k context in production
we shipped a code review bot using qwen 2.5 14b q4_k_m two weeks ago and its been mostly fine until yesterday when a customer sent us a 21k token file and the model started generating function calls to methods that dont exist in the codebase. same file truncated to 18k tokens works perfectly fine. running llama.cpp b4821 on a 4090, temp 0.7, top_p 0.9. the halucinated functions are syntactically valid python but reference modules we never imported. customer is furious and i have no idea how to fix this without just hard-limiting context to 18k which feels like admitting defeat
this is problem with long context attention degradation. past ~20k the model starts to hallucinate function names because attention to earlier parts of context becomes unreliable. we saw same thing on qwen 2.5 14b around 18k
rope window