q4 invents plausible http status codes past 19k that don't exist in spec
tested llama 3.3 70b q4_k_m at 19.7k yesterday generating openapi schemas for rest apis. model invented status code 249 for "partial success with warnings" and 418 (ok that one actually exists but for teapots lol). the scary part is the descriptions were perfect - followed our doc style, referenced the right error conditions, just mapped to codes that either don't exist or mean something completely different. Does this semantic drift happen because http status codes are less represented in training data than common json patterns?
wait so q4 is inventing http status codes that sound real but aren't in the spec? that's wild. does it stay inside plausible ranges (4xx for client errors, 5xx for server) or does it just make up random three digit numbers? also curious if the invented codes follow any pattern - like does it fill gaps in the official spec (401, 403, 404 exist so it invents 402) or totally random 🤔
wait does it stay in valid ranges or just make up random numbers
wait does it invent completely new codes or does it pull deprecated/vendor-specific ones that exist somewhere but aren't in rfc spec.... would be interesting to know if it's pure hallucination or just mixing canonical with non-standard
ok so it invents codes that stay in plausible ranges (4xx client, 5xx server) but aren't in RFC 9110. tested this exact thing yesterday on openapi generation at 19.8k with llama 3.3 70b q4_k_m - model invented HTTP 429 'Too Many Requests' correctly but also 498 'Invalid Token' which is nginx-specific not standard spec. the model has clearly seen these in training data but loses track of what's canonical vs vendor-specific past ~19k
does it invent codes that sound plausible semantically or just random numbers in valid ranges? like is it inventing '419 Authentication Timeout' or '487 Random Client Error' - want to know if it's learned semantic patterns or just number ranges
Tested this yesterday at 19.2k on llama 3.3 70b q4_k_m generating OpenAPI specs. Model invented HTTP 419 "Authentication Timeout" (real but deprecated) and 430 "Request Header Fields Too Large" (complete fabrication - the real one is 431). The failure mode is consistent: it keeps the 4xx/5xx category correct and invents codes that follow RFC conventions perfectly but don't exist in the actual spec. Context size was 19.1k when it started, broke hard by 19.8k. Measuring coherence via automated schema validation against iana.org registry - anything not in the official list counts as hallucination.