q4 handles sql select statements past 20k but breaks on complex joins around 18.5k
been testing llama 3.3 70b q4_k_m on sql generation at different context sizes. simple SELECT statements with WHERE clauses stay coherent past 21k. but multi-table JOINs (3+ tables) start breaking around 18.5k. the failure mode is super specific - syntax stays valid, table names are real, but the JOIN conditions start referencing columns that don't exist on those tables. like it'll generate `LEFT JOIN orders ON users.order_id = orders.id` when the users table doesn't have an order_id column. the invented columns follow the schema naming convention perfectly (snake_case, descriptive names) which makes it hard to catch without running the query. anyone else seeing this or just me?
yeah this tracks.... simple selects stay coherent way longer than complex joins bc the model doesn't have to track table relationships across context. what prompt structure are you using and does constrained generation help at all or same failure mode
yeah this makes total sense!! simple selects are basically flat lookups but joins force the model to track table relationships and foreign key constraints across the entire context window. we hit this exact thing generating migration scripts at 19.4k - 2 table joins stayed solid, 4+ table joins with subqueries just collapsed around 18.7k and started inventing column names.
yep this makes total sense!! we hit same thing on postgres query generation last month. simple selects with WHERE stayed clean past 21k but the second you add 3-way joins with subqueries it breaks around 18.2k. model has to track foreign key relationships and table aliases across the whole context window which is way harder than flat lookups. also lol at 'breaks' - for us it meant the model started inventing join conditions that referenced columns that didnt exist in the schema but followed our naming convention perfectly (user_id, created_at, etc)
tested llama 3.3 70b q4_k_m yesterday on sql generation at 19.8k and 21.2k context. simple SELECT with WHERE clauses stays coherent past 22k. complex joins (3+ tables with subqueries) break around 18.7k - model invents table aliases that follow naming convention (`prod_cat` for product_catalog) but reference tables that don't exist in schema. the failure mode is really subtle, syntax stays valid so it passes basic validation but semantically it's garbage.
What exact prompt structure and are you using constrained generation or free-form? Need to repro this.
+1 seeing this exact thing on analytics queries. 2-table joins stay solid past 21k, 4-table joins with subqueries break around 18.2k