does q4 json schema adherence break faster with deeply nested objects or wide flat ones
tested nested (6 levels deep, ~40 total fields) vs flat (2 levels, ~200 fields) schemas at 18k context yesterday. nested broke at 17.2k, flat stayed valid until 18.9k. is this a real pattern or just variance
tested this exact scenario yesterday on llama 3.3 70b q4_k_m with product catalog schemas at 18.9k context. deeply nested schema (6 levels, 47 fields total) breaks at 17.4k. flat schema (91 fields, zero nesting) stays coherent until 19.8k. the failure mode on nested schemas is really specific - model keeps json syntax valid but starts inventing nested objects that follow the naming convention but reference fields that don't exist in the registered schema. flat schemas just drift on individual field values but don't hallucinate structure. measuring this with automated eval checking schema compliance - any invented fields not in registered schema = failure. running llama.cpp build from dec 2024, quant variant q4_k_m, temp 0.7
tested this yesterday on llama 3.3 70b q4_k_m with deeply nested product configs (6 levels) vs flat schemas (80 fields, zero nesting). nested breaks at 17.8k, flat stays coherent until 19.4k. the failure mode is identical tho - invents plausible fields that follow naming conventions. does anyone know if this scales linearly with nesting depth or is there a cliff?
makes sense - nested objects force model to track hierarchy across longer context, flat schemas just track individual fields
this makes sense yeah... we hit similar thing in prod two weeks ago generating config files. nested objects broke way earlier than flat schemas bc model has to track parent-child relationships across longer context. flat schemas just track individual key-value pairs which is way simpler