llama 3.3 70b q4_k_m - breaks on json schema with recursive definitions past 20k
ok so tested llama 3.3 70b q4_k_m yesterday on json schemas with recursive type definitions. schema like {"type": "object", "properties": {"children": {"type": "array", "items": {"$ref": "#"}}}} works fine up to about 19.7k tokens then the model just starts hallucinating schema properties that don't exist. perplexity cliff at 20.1k. tested with batch=1 on 3x4090. anyone else seeing this or is our setup broken
what error does it throw? we use llama 3.3 for schema validation and recursive defs are everywhere
throws jsonschema validation error but keeps generating. super weird
we hit this exact thing with json schema recursive definitions tuesday. past 20.3k llama 3.3 just throws jsonschema validation errors but keeps generating invalid json. the error messages are useless too, just says "schema validation failed" with no details on which field or constraint
we hit this exact recursive schema issue with kubernetes manifests!! past 20.1k llama 3.3 just starts hallucinating nested fields. are you seeing the same behavior with other schema formats or is it specific to json schema?