llama 3.3 70b q4 breaks on openapi spec generation around 20k, invents plausible but wrong endpoint paths
tested llama 3.3 70b q4_k_m yesterday at 19.8k, 20.4k, and 21.2k context generating openapi 3.0 specs for a rest api with 40+ endpoints. at 20.4k it started inventing endpoint paths that sound right but don't exist - like `/api/v1/users/{id}/preferences` when the actual endpoint is `/api/v1/users/{id}/settings`. syntax was perfect, all the openapi schema rules followed, but the semantic accuracy on path names completely drifted. q5 handled same input until 27k+ with zero drift. the vram cost hurts (extra 6.2gb on our setup) but silent spec drift in prod would be way worse. has anyone else hit this on openapi/swagger generation specifically, or is this just the same q4 semantic drift everyone's seeing past ~18-20k?
lol we hit this exact thing in prod last month... q4 at 20.1k started generating /api/users/{id}/profile when spec clearly has /api/users/profile/{id}. paths looked totally plausible but broke every api call 😅 q5 handled same input until 27k+ with zero hallucinations
we hit this too on openapi generation past 20k. generates endpoint paths that follow the naming convention perfectly but reference resources that don't exist. like `/api/v2/organizations/{id}/billing` when actual spec has `/api/v2/orgs/{id}/subscription`. semantic drift on resource names while keeping path structure valid
ok so q4 invents plausible endpoint paths past 20k. we hit this exact thing generating openapi specs in ci - paths looked totally valid but referenced resources that don't exist in the actual api. moved to q5 and the problem disappeared
whats your prompt and schema? need to repro this before i can trust it. also what vllm version
vllm 0.6.3, temp 0.7, top_p 0.9. prompt is just system message with openapi spec in json format. breaks around 20.3k for me, starts generating /api/v1/users/profile when spec has /api/v1/user/profile