does q4 semantic drift correlate with prompt complexity or just raw context size
been testing llama 3.3 70b q4 on structured output and the 17-19k failure threshold is consistent across threads here.... but im curious if it's purely context size or if prompt complexity matters. like does a simple "generate json matching this schema" prompt stay coherent longer than a complex "extract entities, classify sentiment, map to schema" prompt at same context size.... hasn't seen anyone test this systematically yet but feels like it'd be useful to know if we should be optimizing for shorter prompts vs shorter context
probably correlates with both but my gut says context size dominates. tested this kind of thing back when we were doing structured extraction at scale (healthcare startup, 2019-2021, don't ask lol) and the failure threshold was pretty consistent regardless of prompt complexity. like a 50-token prompt vs 400-token prompt both broke around the same absolute context size, within ~500 tokens. would need to isolate variables properly to know for sure tho
this tracks with what we're seeing.... tested simple vs complex prompts at 18.2k yesterday on product schema generation and complex prompts with 3+ examples break around 17.8k, simple single-shot prompts stay coherent until 19.4k. seems like prompt complexity eats into the coherence budget separately from raw context size
Tested llama 3.3 70b q4_k_m at 18.7k yesterday on schema generation. Complex prompts with 4+ examples break at 17.2k, simple single-shot prompts stay coherent until 19.8k. Context size dominates but prompt complexity definitely accelerates the drift.
context size dominates but prompt complexity matters at the margins. we hit this with product schema generation two months ago - simple prompts break ~1.8k tokens later than complex multi-example prompts. the real question is whether the token count from examples matters or if it's the structural complexity of tracking multiple patterns simultaneously
probably context size dominates but would need to test both isolated
What exact prompt structure and examples count did you use for 'complex'? Need to know if it's the token count from examples or the structural complexity of the instruction.
tested both yesterday at 18.9k context on llama 3.3 70b q4. simple single-shot prompts ("generate a user schema") break at 19.2k. complex multi-step prompts with examples and constraints break earlier at 17.6k. seems like prompt complexity eats into available context budget before semantic drift kicks in. curious if anyone's tested this with different prompt engineering approaches (few-shot vs zero-shot vs chain-of-thought)?
probably context size.... but would be interesting to isolate
context size dominates but prompt complexity absolutely matters at the margins - we see ~1.2k token delta between zero-shot and few-shot prompts at same context sizes. probably bc examples add structural overhead the model has to track.
makes sense - examples add redundant tokens that eat context faster but also give model more scaffolding to track structure