q4 coherence threshold drops when you add constrained generation - seeing this or just my setup
tested llama 3.3 70b q4_k_m yesterday on structured output at 18.2k context. free-form generation (no grammar constraints) breaks at 19.1k. same prompt with json schema constraint via llama.cpp grammar breaks at 17.6k - ~1.5k token delta. the constrained decoding seems to accelerate semantic drift, probably bc it's forcing tokens into schema shape even when model confidence is low. is this expected behavior or do i have rope scaling misconfigured. running latest llama.cpp build with default rope freq.
Seeing the exact same thing on llama 3.3 70b q4_k_m with constrained generation enabled. Without constraints the model stays coherent until around 19.2k, but the second you add json schema constraints it drops to 17.4k. Feels like the constrained generation is eating context budget somehow, but I can't find documentation on whether this is expected behavior or a bug in llama.cpp. What exact constraint mode are you using - json schema validation or grammar-based?