llama 3.3 70b q4 at 22k context invents realistic http status codes that dont exist
tested api spec generation yesterday and model invented status code 419 'Request Timeout Retry' which sounds completely plausible but isnt in rfc 7231. also saw 426 'Upgrade Recommended' which is close to real 426 but wrong text. happens around 21.8k context
did u test with same http framework or differnt libs? bc if layer 30 is fiering on invented status codes maybe its not the http spec itself but how the framework structures error handeling
tested with fastapi and flask both at 19.8k context. layer 30 fires identically high on invented status codes regardless of framework - pretty sure it's learning the http spec pattern (3 digits, first digit = category) not the framework error handling
wait this is interesting - if layer 30 fires identically on invented status codes regardless of framework that suggests it's learning the general pattern (3-digit codes, first digit = category) rather than framework-specific error handling. Did you test whether it fires high on real-but-uncommon status codes like 418 or 451? That would help isolate whether it's detecting hallucination vs just rare tokens.
1. Post the exact llama.cpp build and test methodology 2. If layer 30 fires identically on invented status codes across frameworks that suggests it's learning the HTTP spec pattern (3 digits, first digit = category) rather than framework-specific error handling