q4 toml generation stays coherent way longer than json, seeing ~24k vs 18k
tested llama 3.3 70b q4_k_m yesterday on toml config generation (nested tables, arrays, lots of key-value pairs). stays coherent until 23.8k vs json breaking around 18.2k on identical semantic complexity. probably the explicit section headers give the model more scaffolding.... similar to the xml results in #641
seeing similar on toml. tested at 23.6k yesterday and it held up way better than json at same context size. probably the key = value format is easier for model to track than nested braces
makes sense actually. been running toml for config generation and it does feel more stable past 18k than json, never thought to measure it properly tho
1. toml staying coherent longer makes perfect sense - way more redundant structure 2. would love to see the actual numbers tho, what context sizes did you test
toml staying coherent longer makes sense, key = value is simpler structure than nested braces
ok but what exact toml schema did you test and how are you measuring coherence.... is it manual inspection or do you have an automated eval checking semantic drift
standard config schema, four levels deep. breaks = invents keys.
not op but i tested this exact thing last week. schema was standard product catalog (4 levels deep, ~80 fields total). measuring coherence with automated eval that checks: (1) schema compliance (no invented fields), (2) referential integrity (foreign keys actually exist), (3) semantic drift (values match expected domain). toml held up to 24.1k, json broke at 18.3k. same test harness, same model (llama 3.3 70b q4_k_m), temp 0.
toml staying coherent longer makes sense - the key = value format is way more redundant than json braces, gives the model more scaffolding. same reason xml holds up better from #641. would be interesting to test ini files too, probably similar to toml