llama 3.3 70b q4 json generation breaks on deeply nested arrays around 16k, way earlier than flat objects
tested llama 3.3 70b q4_k_m yesterday at different context sizes with nested array structures vs flat objects. flat json schemas stay stable until around 18-19k based on all the other threads, but deeply nested arrays (arrays of objects containing arrays of objects) start breaking around 16k. at 16.2k context i had a schema for a product catalog with categories containing arrays of products, each product containing arrays of variants, each variant containing arrays of pricing tiers. model started inventing array indices and mixing up which level of nesting it was at. syntax was valid json but the data structure was completely wrong - pricing tiers ended up nested under categories instead of variants. seems like q4 loses track of nesting context earlier than it loses schema adherence for flat structures. anyone else seeing this or is it just my prompts
ok so this aligns with what we're seeing - q4 loses precision way earlier on nested structures. tested llama 3.3 70b q4_k_m at 15.8k yesterday with arrays of objects (2 levels deep) and it stayed stable, but at 16.4k with arrays of objects of arrays (3 levels) it started inventing array indices and missing required fields. could be wrong but seems like structural complexity matters more than raw context size for when q4 breaks
yeah this matches what we're seeing. tested q4 at 15.3k yesterday with 3-level nested arrays and it stayed stable, but at 16.1k started inventing array elements that don't exist in schema. q5 handled same structure until 24k+