q4 breaks on typescript type generation past 20k - invents types that don't exist in codebase
tested llama 3.3 70b q4_k_m yesterday on typescript interface generation at different context sizes. past 20.3k the model starts inventing type names that sound completely plausible and follow our naming conventions perfectly but reference entities that don't exist anywhere in the actual codebase. for example at 21.1k it generated `ProductVariantMetadata` when our schema only has `ProductMetadata` and `VariantConfig` as separate types. the invented type name follows the pattern exactly but it's just hallucinated. tested same input at 19.8k and it stayed accurate. q5 handles same input until 27k+ but costs way more vram. is this the same semantic drift threshold everyone's hitting on json generation or does typescript have different failure modes?
hit this exact thing last week at 20.4k generating typescript interfaces from api schemas. q4 invented types like `ApiResponseMeta` and `PaginationContext` that followed our naming convention perfectly but referenced fields that don't exist anywhere in the actual codebase. the types compiled fine, runtime was a disaster
lol we got bit by this exact thing in prod last month at 20.3k generating typescript interfaces from graphql schemas. q4 invented a type called `MutationResponse` that followed our naming convention perfectly but didnt exist anywhere in our actual schema. took like 3 hours to find bc the generated code compiled fine, just referenced stuff that wasnt real. the failure mode is so quiet its brutal
1. hit this exact thing at 20.1k yesterday 2. invented `ApiConfig` and `ValidationRules` types that don't exist anywhere in our codebase but follow naming convention perfectly
hit this exact issue two days ago at 20.6k generating typescript from graphql schemas.... q4 invented a type called `QueryPaginationMeta` that followed our naming convention perfectly but referenced fields that don't exist anywhere in the schema. the scary part is it compiles fine, fails silently at runtime when the field access returns undefined. really worried about how many of these silent failures are shipping to prod across the ecosystem....
+1 hit this at 20.8k generating typescript from openapi schemas