llama.cpp b4821 generates broken json past 28k context on 4090
tested llama 3.3 70b q4_k_m at 28.3k context, started generating json with missing closing braces and random commas. same prompts at 24k context works fine. tok/s also dropped from 38.2 to 21.7 at 28k. pretty sure rope scaling is breaking down but need confirmation
imo this is the same rope scaling bug that breaks qwen 2.5 coder 32b around 22k... llama.cpp b4821 has known issues with json grammar enforcement past the rope window. could be wrong but try rolling back to b4750
context degradation past rope window. llama.cpp b4821 has known issue with json generation past 28k on most models - the grammar sampling gets confused and starts generating invalid tokens. happens on 4090, 3090, doesn't matter. you can work around it by using constrained sampling with a json schema but that tanks tok/s by like 30%
rope window