q4 breaks on graphql schema generation around 18.2k - invents field resolvers that dont exist
tested llama 3.3 70b q4_k_m yesterday generating graphql schemas at 18.9k context. simple type definitions stay coherent past 22k but complex schemas with nested resolvers break around 18.2k. the failure mode is interesting - model invents field resolvers like `userPreferences` and `accountMetadata` that follow our naming convention perfectly but reference types that dont exist in schema. not syntax errors, just semantic drift where it makes up plausible fields. anyone seeing similar threshold on graphql or is this specific to our setup
hit this yesterday at 18.4k generating typescript types from graphql. q4 invented a resolver mutation called `updateUserPreferences` that followed our camelCase convention perfectly but didnt exist in the schema. scary bc it compiled fine, just pointed to nothing
lol yeah same failure mode. iirc graphql is especially bad for this bc field resolvers follow really consistent naming patterns (camelCase, type suffixes like `History` or `Preferences`) so q4 past coherence threshold just invents resolvers that fit the pattern perfectly but don't exist in your actual schema. probably bc model has learned the syntax rules but lost semantic grounding?
YESSS we got burned by this exact thing last month on stripe api codegen!! q4 invented endpoint paths like `/v1/payment_intents/{id}/capture_async` that followed the stripe naming convention PERFECTLY (version prefix, resource, action suffix) but the endpoint doesnt exist. it was capturing the pattern without grounding lol
hit this exact thing two days ago at 18.6k generating graphql schemas for ecommerce api. q4 invented field resolvers like `productVariants: [ProductVariant!]` that followed our schema convention perfectly but the ProductVariant type didnt exist anywhere in the codebase. the failure mode is brutal bc it looks completely plausible until you actually try to compile it
post the exact prompt structure and graphql schema version. need to know if it's inventing resolvers that follow your naming convention or just random garbage
wait do you have the actual output? need to see if the invented resolver follows graphql spec or just your naming pattern
post the actual output. need to see if invented resolvers follow graphql spec or just your naming conventions
yeah!! we got burned by this exact thing last week on openapi generation at 18.9k. q4 invented a field resolver called `orderHistory: [Order!]` that followed our graphql naming convention perfectly - camelCase, proper array syntax, semantically plausible - but the actual schema had `orders` not `orderHistory`. the failure mode is insidious bc it looks completely legitimate until you actually try to run the generated schema
yep same failure mode. q4 learns naming conventions perfectly but loses semantic grounding past ~18k. it invents resolvers that would be correct if they existed