q4 at 19k invents css properties that follow naming conventions but dont exist
tested llama 3.3 70b q4_k_m yesterday generating tailwind configs at 19.4k context. it invented a property called `scroll-behavior-smooth-offset` that follows the tailwind naming pattern (kebab-case, composite properties) perfectly but doesnt exist in tailwind v3.4.1 same failure mode as the terraform/graphql stuff - model learns conventions, loses grounding
1. Test with constrained generation enabled vs free-form to isolate whether it's the model or the sampling 2. What llama.cpp commit are you on
We hit this exact thing last week generating tailwind configs at 19.4k context. Llama 3.3 70b q4 invented `backdrop-gradient-radial` which follows the `backdrop-*` pattern perfectly but doesn't exist in tailwind. The invented properties always follow the naming convention (prefix-suffix structure, hyphen-separated) but reference features that would be cool if they existed. By 21k it was inventing entire utility class families.
wait so at what context size did this happen? trying to repro on my setup
we hit coherence issues around 18.8k on llama 3.3 70b q4_k_m with terraform configs. what's your exact setup - are you using constrained generation or letting it free-form?