llama 3.3 70b q4 sql generation invents plausible table names past 19k that don't exist in schema
tested llama 3.3 70b q4_k_m at 19.4k yesterday with database schema generation (postgres, 37 tables, some with foreign keys). past 19.2k it started generating joins to tables that sound totally plausible but don't exist in the actual schema. like it generates `JOIN user_sessions ON users.id = user_sessions.user_id` when the actual table is just `sessions` not `user_sessions`. syntax is perfect, semantics are wrong. q5 stays solid until at least 24k on same schema. is this the same semantic drift everyone is hitting on json or does sql have different failure modes
hit this exact thing at 19.2k yesterday. q4 generated `product_name` when schema had `productName`
lol same exact typo at 18.7k yesterday, generated product_title when schmea had productTitle. took me 45 min to find bc the json looked perfeclty valid
We see this exact behavior in production. Model invents table names that follow naming convention perfectly but reference entities that not exist in actual schema. At 19.4k yesterday q4 generated `order_shipping_address` when schema has `orders.shipping_addr`. Failure is silent because syntax stays valid.
imo this is the scariest failure mode bc the json looks totally valid, just references stuff that doesnt exist. silent failures past 19k are brutal