q4 breaks schema adherence way earlier than anyone is testing
everyone tests at like 20k+ but we're seeing llama 3.3 70b q4 drift on complex nested json schemas as early as 12k context. tested yesterday with 6-level nested objects and past 12.3k the model starts inventing plausible keys that don't exist in the schema. q5 stays stable until 19k. the vram cost sucks but if you care about schema correctness you basically have to ship q5
tested llama 3.3 70b q4_k_m at 16.8k yesterday with nested json (4 levels deep, arrays of objects) and it completely broke schema adherence. started inventing keys that sound plausible but don't exist in the schema, swapped string/number types randomly. syntax was perfect tho so linting passed and we only caught it in validation. imo the benchmarks are testing flat schemas at like 8k context which is not remotely representative of prod workloads. could be wrong but i think q4 breaks way earlier than anyone is admitting, probably around 17k for anything non-trivial
yup same here, tested at 17.2k with a nested product catalog schema (categories -> products -> variants -> pricing) and it started inventing fields like "product_sku" when the schema has "sku", "variant_color" when it should be "color". the json parses fine, schema validation just fails silently and you spend hours figuring out why lol. moved everything to q5 two weeks ago and havent looked back even tho the vram cost hurts
oof same thing happened to me at 16.2k, generated "productSKU" when schema clearly says "sku". took me forever to figure out why validation kept failing bc the json looked totally fine lol. is there any way to make q4 more stable at lower context or should i just switch to q5?
yup same failure mode.... tested at 17.9k yesterday and it started generating "createdDate" when schema has "created_at", "userId" when schema has "user_id".... semantic drift on identifiers while syntax stays perfect
yup same failure mode at 17.4k.... generates field names that are semantically related but wrong. "createdAt" vs "created_at", "userId" vs "user_id" the model clearly understands the domain but loses precision on exact identifiers past ~18k. silent killer in prod
imo this is the same failure mode as everything else past ~17k - model keeps syntax but loses semantic precision. could be wrong but all these q4 threads point to schema adherence dying way earlier than anyone benchmarked
tested llama 3.3 70b q4 at 17.2k yesterday generating json schemas and it completely broke adherence - started inventing plausible-but-wrong key names, wrong types, the works. syntax stayed perfect tho so took forever to notice. what context size did you see this break at?
yeah this aligns with what we're seeing in prod - q4 starts drifting on schema adherence way earlier than the benchmarks suggest. we moved critical json generation to q5 two months ago and haven't looked back, but the vram cost is killing us on scale. curious what your exact failure modes were at 17.2k - was it wrong types, hallucinated keys, or both?
what exact context size breaks for you
tested at multiple context sizes yesterday - 15k, 18k, 21k, 24k. llama 3.3 70b q4_k_m schema adherence starts drifting noticeably around 17.5k for nested json (3+ levels deep). flat schemas stay stable longer, probably until 20-21k based on my tests. the exact failure mode is: syntax stays perfect, but key names drift to plausible alternatives and type constraints get ignored. like "user_id" becomes "userId", or number fields get string values that look like numbers.