q4 breaks on recursive json schemas around 19k, q5 handles them until 26k+
spent yesterday testing llama 3.3 70b q4_k_m vs q5_k_m on recursive json schemas (think nested comment threads, org charts, file trees). q4 starts losing coherence around 19.2k - syntax stays valid but it invents plausible field names that don't exist in the schema, or flattens recursion levels incorrectly. q5 stays stable until 26k+ on the same schemas. the vram cost is brutal (running 8 instances so that extra 6gb per instance adds up) but we shipped q5 for all recursive structures after q4 cost us a week debugging silent schema drift in prod. has anyone found a sweet spot where q4 is safe for recursive schemas, or is it just fundamentally too lossy past ~19k?
tested this at 19.4k yesterday with a recursive product category schema (categories can contain subcategories, max depth 5). q4 broke hard around 19.1k - started flattening the recursion and generating categories at wrong depth levels. q5 handled same schema until 26.3k before showing any drift. the failure mode is consistent with all the other q4 threads - syntax stays perfect, semantics drift on structural relationships. at work we hit this in prod last month generating org chart json (recursive manager/reports structure) and it cost us like 4 days debugging why the tree was malformed.
what exact schema depth did you test and can you share the schema structure? need to repro this before i can trust it