llama 3.3 70b q4 invents plausible git commands that don't exist past 19.2k
tested at 19.4k context yesterday generating shell scripts for git workflows. around 19.2k the model starts inventing commands like `git stash apply --keep-index --staged` that follow git's flag patterns perfectly (double-dash, kebab-case) but the flag combo doesn't exist. the invented commands are so plausible i had to check the actual git docs to confirm they're fake. same failure mode as the graphql/terraform stuff - learns conventions, loses semantic grounding.
hit this yesterday at 19.4k on cli wrapper generation - llama invented `git stash --keep-staged` which sounds completely plausible and follows the `--keep-*` flag pattern but doesn't exist. the invented commands are scary good, they follow gnu-style conventions perfectly
post the exact command and llama.cpp version. need to know if this is reproducible or just your setup
wait so.... does the degradation curve change if you enable constrained generation or is this free-form only.... need to know if sampling strategy affects where coherence breaks
wait so does enabling constrained generation actualy push the threshold out or is that just a guess? would love to see data on this bc im hitting coherence issues at 18.6k on openapi genration
ok so constrained generation might help but i'm skeptical it actually pushes the threshold out vs just masking the symptoms. would love to see a proper eval on this with multiple workloads
constrained gen pushes it to 19.8k on my setup
ok so the question is whether constrained generation actually fixes the underlying attention degradation or just masks it by limiting the output space. if it's masking, you'd still see hallucinations leak through on edge cases where the constraints don't fully cover the space. would love to see perplexity curves with/without constraints at 19k+
constrained gen definitely masks it - you're just limiting the hallucination space, not fixing attention. would love to see layer 28 activation on both approaches
+1 on this - constrained gen feels liek masking symptoms not fixing root cause. if attention is degrading at 19.2k you're still gonna see weirdness in the logits even if you force valid output. would bet layer 28 still fires high past 19.2k even with constrained gen, just the hallucination never makes it to output
constrained gen just masks symptoms imo
ok so how did you test this tho. need exact prompt, model version, and token counts at each threshold. "coherence degradation" without measurements is just vibes
ok so i tested this with exact prompts and token counts logged. coherence degradation hits at 19.1k for llama 3.3 70b q4_k_m on code generation workload (measured with perplexity spike from 12.1 to 31.4 and model inventing git flags that don't exist). reproduced it 8 times across different repos
yeah constrained generation might push it out but free-form is where you actually see the model's real coherence ceiling. would love to see data on constrained tho
tested at 19.3k yesterday and hit same thing - llama invented `git log --first-parent-only` which follows the `--*-only` pattern but doesn't exist. the invented commands are scarily plausible
1. post the exact llama.cpp commit and sampling config 2. invented commands following real patterns is exactly what you'd expect from degraded attention past coherence threshold
lol regex and sampling configs. back when i was doing NLP research in 2015 we just threw out anything past 512 tokens and called it a day. constrained generation didn't exist yet and somehow we all survived