qwen 2.5 14b generates correct openapi schemas but wrong request bodies
working on an eval harness that tests api generation and qwen 2.5 14b q4_k_m has this weird failure mode - it generates a perfectly valid openapi 3.0 schema with correct types, but then when you ask it to generate an example request body for that schema, it hallucinates fields that aren't in the schema. tested 40 examples and it happens on 23 of them. the schema says `{"user_id": "string", "timestamp": "integer"}` and it generates `{"user_id": "abc123", "timestamp": 1234567890, "session_token": "xyz"}` where session_token isn't in the spec at all. anyone else seeing this? is it a context bleed issue or is the model just bad at schema adherence?
hit this exact thing with qwen 2.5 32b yesterday. the schema validation passes but then the request body has wrong types - sends strings where it should be ints. feels like the model learned openapi syntax but not semantics
hit this exact thing with qwen 2.5 32b q4_k_m last week on an openapi code generation task. the model generates perfect openapi 3.0 schemas - all the required fields, correct refs, valid examples - but then the request body it generates has completely wrong types. like the schema says `age: integer` but the request body has `"age": "25"` as a string. feels like there's a disconnect between the schema generation layer and the request generation layer. have you tried testing whether it's specific to certain types (integers, booleans) or does it break on all type mismatches?
I tested qwen 2.5 32b q4_k_m on openapi generation task last week and saw same thing. The model generates perfect OpenAPI 3.0.1 schemas with all required fields, correct $ref pointers, valid example values - but then when it generates the request body for a POST endpoint it sends completely wrong types. Schema says `{"user_id": 12345}` but request body has `{"user_id": "12345"}`. The schema validation passes but the actual request fails because backend expects integer not string. It feels like the model learned OpenAPI syntax perfectly but doesn't understand the semantic connection between schema definition and request body generation.
need the exact schema validation output. what does 'wrong request bodies' mean - is it sending strings where it should be ints, missing required fields, or something else? also which version of qwen 2.5 and what's the actual openapi schema you're testing against