llama 3.3 70b q4 generates correct sql syntax but wrong table names past 18k context
tested llama 3.3 70b q4_k_m on llama.cpp b4821 yesterday for a sql generation task with schema in context. at 12k context it generates perfect queries with correct table and column names. past 18k context the sql syntax is still valid but it starts hallucinating table names that don't exist in the schema. seems like rope scaling breaks down around 18-20k and the model starts guessing table names instead of reading them from context. switched to q5 and the issue mostly goes away - can push to 28k before seeing degradation
Hit this exact thing at work last week. Past 18k context llama 3.3 70b q4_k_m starts hallucinating table names that don't exist in the schema we pass in the system prompt. The SQL syntax is perfect but it references tables like `user_sessions` when the actual table is `sessions`. Really frustrating for RAG over databases.
can you share the actual sql it generated vs what it shoudl have been? need to see if its hallucinating table names or just getting them slightly wrong
tested llama 3.3 70b q4_k_m version 3.3-instruct specifically at 18.2k context yesterday with database schema containing 14 tables in system prompt. generated sql queries with correct syntax (joins, subqueries, window functions all valid) but hallucinated table names in 6 out of 12 test cases. for example, schema has `user_accounts` table but generated query referenced `users` table which doesn't exist. the sql would execute perfectly if the table names were correct. this is consistent with the context degradation pattern others are reporting past ~18k.