q4 breaks earlier on json arrays vs objects
tested llama 3.3 70b q4_k_m yesterday. arrays of objects break around 16.2k, flat objects stay stable until 19k. same prompting, same schema complexity. failure mode: starts mixing array index refs, generates [{"id": 1}, {"id": 1}] when schema requires unique ids
we tested similar thing yesterday on llama 3.3 70b q4_k_m with nested array schemas at different context sizes. q4 breaks around 16.4k on arrays but stays stable until 19.2k on flat objects. failure mode is very specific - model keeps array syntax valid but starts inventing elements that don't exist in schema. is like model understand structure but lose precision on what should actually be in array)
hit this exact thing yesterday at 19.2k - nested arrays broke way earlier than flat objects. q4 started inventing array elements around 16.8k that don't exist in the schema, q5 handled same input until 23k+. switched our prod stack to q5 for anything with nesting > 2 levels. are you seeing the same threshold or does it vary by schema complexity?
what exact context size and how are you measuring semantic drift? is it manual inspection or do you have an automated eval
We're measuring semantic drift with automated eval - registered schema with exact field names, then check if generated json contains any fields not in the schema or missing required fields. Context size was 19.8k for the test. We run eval every 200 tokens from 15k to 25k to find the exact threshold where schema adherence breaks. Are you seeing the same failure mode where q4 invents plausible-sounding fields that don't exist in the schema, or is it a different kind of drift?
hit this exact thing yesterday testing q4 at 19.8k with nested array schemas. the failure mode is really specific - q4 keeps the array syntax valid but starts inventing elements that sound plausible but don't match the schema at all. tested with a product catalog schema where each product has variants (array of objects, 2 levels deep). at 16.4k it was perfect, at 19.1k started generating variant objects with fields like "colorName" and "sizeCode" when the schema clearly specifies "color" and "size". json was syntactically valid, semantically wrong. q5 handled the same input until 24.3k before showing any drift. the vram cost hurts but at least the failures are obvious - q5 tends to break with syntax errors rather than silent semantic drift.
probably because arrays have less structural redundancy than objects. objects have key names that give context, arrays are just positional. model has less scaffolding to work with
makes sense - objects have key names that give the model scaffolding, arrays are just positional so less context to work with when tracking structure