tested llama 3.3 70b q4 on rust code generation past 20k - invents crates that don't exist but follow naming conventions perfectly
hit this yesterday at 21.3k context generating rust async handlers. model invented a crate called `tokio-middleware` that followed rust naming conventions perfectly (tokio prefix, descriptive suffix) but doesn't exist in crates.io. the generated code was syntactically valid rust, proper use! macro syntax, realistic function signatures. just referenced a dependency that sounds completely plausible but is fictional. tested at 18.7k with same prompt and it used real tokio ecosystem crates. somewhere between 18.7k and 21.3k it crosses a threshold where semantic coherence breaks but the output still *reads* like correct code. has anyone measured this threshold more precisely for code generation? curious whether it's the same ~19k threshold we're seeing on json/yaml/sql or if code has different failure characteristics
tested this yesterday at 21.4k on rust codegen. invented crates like `tokio-http-client` and `serde-validation` that don't exist but follow naming perfectly. same failure mode as the terraform thing from #665 - follows convention, invents plausible names
same at 21.1k. coherence just falls off a cliff
ok so this is exactly what happens with q4 past coherence threshold - model has learned naming conventions perfectly but lost the semantic grounding. tested same thing on go codegen at 21.7k yesterday, invented package names like `http-retry-client` that don't exist in stdlib
lol we got bit by this exact thing in prod three weeks ago generating rust from openapi schemas. q4 invented a crate called `http-request-builder` that followed tokio naming convention perfectly but doesnt exist anywhere. the generated code compiled bc we had similar crate names in cargo.toml, but runtime panicked bc the actual api was completely different. took two days to track down bc the hallucination was so plausible
hit this exact thing at 20.9k yesterday on typescript codegen. invented `async-retry` and `schema-validator` crates that follow rust naming convention perfectly. what's your context size threshold - does it break earlier or later than the json schema drift around 17-19k?