llama 3.3 70b q4 generates valid json but wrong key names past 23k context
tested at 23.1k context yesterday with api schema containing 11 json fields defined in system prompt.... model generates syntactically perfect json but uses wrong key names that dont match the schema like schema defines `user_id` but model generates `userId`, or schema has `created_at` but model outputs `timestamp`. the json structure is correct, types are correct, just the field names drift from what was specified in context seems related to all the other identifier drift issues past 20k.... sql table names (#523), exception class names (#548), now json keys
past 23k it generates keys that sound plausible but don't exist in the schema. we hit this in prod and it took 8 hours to figure out why validation was failing - the json was perfect syntactically but semantically wrong
we hit this in prod last week and it was exactly this - json syntax perfect but keys that sound plausible but don't exist in our schema. like "user_id" becomes "userId" or "created_at" becomes "createdAt". breaks schema validation but looks totally normal if you're just eyeballing the output
+1 we hit this in prod and it was a nightmare to debug bc the json looked completley valid but schema validation just kept failing on keys that didnt exist
tested this at 23.4k context yesterday with a json schema containing 8 object keys.... first 5 keys were correct, last 3 hallucinated to completely wrong names that weren't in the schema
past 23k it just invents keys that sound plausible but dont exist in the schema.
ok so does this also happen with array values or just object keys
does this also break on nested object values or just top-level keys
probably breaks on both but need actual test to confirm