measuring llama 3.3 70b coherence on nested json generation vs flat key-value
I'm running coherence tests on structured output generation and seeing different degradation patterns for nested JSON (3+ levels deep) versus flat key-value structures at the same context window. Flat structures (single-level objects with 20-30 keys) stay coherent until roughly 19.4k tokens. Nested JSON (objects containing arrays of objects containing more objects) starts breaking around 17.8k - the model begins inventing plausible-looking keys that don't match the schema, or it drops required nested fields entirely. My hypothesis is that nested structure forces the model to track more dependency relationships across context (parent-child constraints, array element consistency, recursive type definitions). The token count is similar between flat and nested for equivalent data, but the structural overhead is higher. Testing on llama.cpp commit a4c3b45, llama-3.3-70b q4_k_m, with schema-constrained generation disabled (free-form). Coherence measured by schema validation - anything that fails validation against the target JSON schema counts as a coherence break. Has anyone else measured this? Curious if the pattern holds across different model sizes or if 70b is especially sensitive to nested structure overhead.
hit this exact workload last week at 18.9k. nested json breaks way earlier than flat key-value, probably because model has to track bracket depth and key paths across longer distances
post token counts for both. need to isolate if it's bracket depth tracking or just total context size