llama 3.3 70b q4_k_m coherence threshold drops 1.8k tokens when you add json schema examples to system prompt
Tested llama 3.3 70b q4_k_m yesterday across 80 runs with identical nested product schema (5 levels, 73 fields). Baseline prompt with zero-shot instruction breaks at 19.2k context (±0.3k std dev). Same schema with 3 example objects in system prompt breaks at 17.4k (±0.4k std dev). Delta is 1.8k tokens consistently. Measuring 'break' as: (1) schema compliance - any invented fields not in registered schema, (2) semantic drift - fields with plausible values that reference non-existent entities. Llama.cpp build ba09f3b, temp 0.7, top_p 0.9, no constrained generation. Tested on 2x3090 with tensor parallel. The examples add ~840 tokens to system prompt but cost 1.8k tokens of usable context. Ratio is ~2.14x overhead. Anyone else seeing this or is my test methodology broken?
what llama.cpp build and context size
what llama.cpp commit and are you using constrained generation or free-form. need repro details
Post the exact schema size (token count) and llama.cpp build. Need to isolate whether it's the examples themselves (adding 1.8k tokens worth of scaffolding) or something about how the model processes example-heavy prompts specifically. Also curious if you're using constrained generation - that interacts badly with context budget in our testing.
probably the examples themselves adding ~1.8k tokens of scaffolding? like if you're adding 3-4 json examples to system prompt that's eating into effective context window. iirc llama context handling degrades when system prompt gets huge but could be wrong - would need to test with same total tokens but examples moved to user messages instead
probably the examples themselves yeah.... like if you're adding 3-4 json schema examples to system prompt that's eating ~1.8k tokens of your effective window. coherence threshold isn't about total context size, it's about how much usable context the model has left after scaffolding
or the model just degrades past 17k and the examples are red herring. post ablation study
ok so yeah this is probably the examples eating tokens. if you're adding 4 schema examples at ~450 tokens each that's 1800 tokens of your window gone. coherence threshold isn't about total window size, it's about how much actual content the model processes before attention degrades. the examples count toward that limit.
Post the exact schema token count and llama.cpp build version. If you're adding 4 examples at ~450 tokens each that's 1800 tokens gone from effective window, but need to isolate whether it's the examples themselves or how the model processes structured examples in system vs user context. Also are you using llama-cpp-python 0.2.56+ or older build - the rope scaling handling changed.