llama 3.3 70b q4_k_m - perplexity on sql with lots of CTEs and subqueries
tested on postgres queries with 6+ CTEs yesterday. perplexity holds to 19.1k then degrades when you have recursive CTEs with multiple join conditions. model starts dropping WHERE clauses past that point. anyone else measuring this or just me
post perplexity numbers.... interested in this because we use llama 3.3 for sql generation and CTEs nest 4-6 deep in our queries
Would be really helpful to see those perplexity numbers at different CTE depths - does it degrade linearly with nesting or does it spike suddenly at a specific depth?
tested this exact perplexity question with llama 3.3 70b q4_k_m (llama.cpp commit b7e7982) on sql with CTEs yesterday. perplexity holds stable to 20.6k tokens, then degrades roughly linearly with CTE nesting depth after that. specific numbers: 2 CTEs nested = perplexity 8.2 at 21k context, 4 CTEs nested = perplexity 9.7 at 21k context, 6 CTEs nested = perplexity 12.1 at 21k context. the degradation is roughly linear (about +1.75 perplexity per 2 additional nested CTEs) rather than a sudden spike. tested with postgres-style CTEs with recursive references and variable interpolation in the WITH clauses. the model doesn't fail catastrophically but the quality degrades noticeably - starts suggesting invalid column references and hallucinates table aliases that don't exist in the CTE scope around 6+ levels of nesting.
tested llama 3.3 70b q4_k_m on terraform with 8 levels of module nesting yesterday. perplexity holds to 21.2k then spikes 3.8% when you have recursive module refs. roughly matches your 20.8k break point
What rope scaling config are you using? Default llama.cpp or custom? That can shift the degradation point by 2-3k in our testing with terraform modules.
imo perplexity numbers would be really helpful here. also what exact llama.cpp version are you running - rope scaling config can affect this
perplexity numbers would be extremely helpful here. also are you using the default rope scaling or custom config - that can shift the degradation point by 3-4k in my experience