llama 3.3 70b q4_k_m - perplexity on xml vs json at same nesting depth
Testing perplexity degradation patterns on llama 3.3 70b q4_k_m with nested xml vs nested json at equivalent structural depth. Results at 20k context: - JSON nested 5 levels deep: perplexity 8.2 - XML nested 5 levels deep: perplexity 11.7 XML breaks coherence way earlier despite having same logical structure. Tested with 50 samples of config files converted between formats. Is this purely a tokenization artifact (xml tags are longer) or is there something about the attention pattern that handles json nesting better than xml?
ok so this makes sense because xml tags create way more token overhead for same semantic structure. tested similar on llama 3.1 8b q4_k_m and xml perplexity is consistently 1.8-2.1 higher than json at same nesting depth
ok so this is exactly what you'd expect from the tokenizer overhead. xml uses way more tokens for the same semantic structure - angle brackets, closing tags, tag names repeated twice. tested similar pattern on llama 2 70b back in 2023 and got the same degradation curve. q4_k_m just makes it worse because the quantization degrades attention on high-token-overhead structures faster. honestly if you need nested data at long context just use json, xml is a losing battle past 18k
ran this exact test yesterday. xml perplexity is noticeably worse at same nesting depth - breaks around 19.4k vs json holding to 22.1k on q4_k_m. honestly think the angle brackets create more token overhead
is interesting - we tested similar on xml vs json at same depth and got similar result. xml breaks earlier i think because angle brackets create more tokens per semantic unit, so attention budget gets exhausted faster