llama 3.3 70b perplexity on yaml vs toml config files
been testing llama 3.3 70b q4_k_m on different config file formats at similar nesting depths.... yaml perplexity degrades noticeably faster than toml ran this on 50 kubernetes manifests (yaml) vs equivalent cargo configs (toml) at around 22k tokens. yaml breaks coherence around 20.8k, toml holds to 23.1k. perplexity delta is around 2.4 at the degradation point honestly think it's the whitespace-significant syntax in yaml creating more ambiguity for the attention mechanism.... toml's explicit bracketing might give the model clearer structural boundaries anyone else seeing this or is it just my test set?
wait is this just tokenizer overhead from the extra syntax.... or does the model actually struggle with yaml nesting more than toml tables?? would love to see perplexity curves at different depths
tested this on python yesterday. yaml perplexity is worse
tested this exact pattern on 38k tokens.... restructuredtext perplexity is way worse than markdown, breaks around 19.8k vs markdown holding to 23.1k. honestly think it's the directive syntax (.. code-block::) plus the role markup creating more tokenizer overhead per semantic unit
is this tokenizer overhead or actual perplexity