llama 3.3 70b q4_k_m perplexity on sql vs nosql query syntax at same token count
been testing llama 3.3 70b q4_k_m on database query generation and noticed perplexity degrades way faster on nosql (mongodb aggregation pipeline syntax) compared to sql at equivalent token counts. sql select statements hold coherence to around 26k tokens, but mongodb aggregation pipelines break around 19k. both are similarly nested in terms of logical depth but the syntax is completely different (sql is more keyword-heavy, mongo is json-like). wondering if this is a training data distribution thing (way more sql in pretraining) or if it's something about how the model handles different query syntax structures? iirc sql has way more structural markers (SELECT, FROM, WHERE) vs mongo's bracket-heavy nesting. could be wrong but feels like the quantization might degrade attention differently on heavily nested json-like structures vs keyword-structured queries
post perplexity curves at different nesting depths. tried similar test on postgres schema dumps last week and got wildly different results depending on join depth vs raw nesting
would love to see those curves too.... we're running into similar perplexity issues with deeply nested postgres schemas and trying to figure out if it's join depth or just raw structural complexity that breaks coherence first
pretty sure it's join depth creating more attention overhead per token than raw nesting. tested similar on mysql schema dumps and the perplexity cliff happens around 7-8 joins regardless of nesting
join depth creates more attention hops per resolution step. tested similar on postgres and perplexity degrades linearly until around 6-7 joins then cliffs hard
would love to see those perplexity curves.... we're testing similar patterns for our eval suite and trying to figure out if sql complexity scales linearly with join depth or if there's a cliff around certain nesting levels