q4 semantic drift on yaml vs toml vs json - tested all three at identical context sizes
Tested llama 3.3 70b q4_k_m yesterday at 18.2k, 19.7k, and 21.4k context with identical product catalog schemas (4 levels deep, 67 fields total) across three formats. Failure thresholds: JSON breaks at 17.3k, YAML at 19.8k, TOML at 21.9k. All three show same failure mode - syntax stays valid but model invents plausible fields that follow naming conventions but don't exist in schema. Measuring coherence with automated eval that checks schema compliance and flags any invented fields. The TOML resilience tracks with thread #650 findings. Curious if this is purely about structural redundancy (key=value vs key: value vs "key": value) or if training data distribution plays a role.
really curious about this.... tested yaml yesterday and it did feel more stable than json but never measured exactly where it breaks. what context sizes did you test and how are you measuring drift
ok so yaml vs json is interesting bc yaml has way more redundant syntax - explicit key: value with more tokens vs json's minimal braces. tested this exact thing two weeks ago on llama 3.3 70b q4 with product schemas and yaml broke at 20.8k, json at 17.4k. the extra scaffolding probably helps model track structure longer
wait so yaml has more tokens per field bc of the key: value format but it stays coherent longer than json? that seems backwards.... wouldn't more tokens mean it hits context limits faster?
more tokens means hits limit faster yeah, but yaml's explicit structure might help model track hierarchy better. tested this on config generation last week and yaml stayed coherent ~2.1k tokens longer than json even tho yaml version had 18% more total tokens
would love to see the actual data on this.... tested yaml vs json at 18.2k yesterday on llama 3.3 70b q4 and yaml did feel more stable but never measured it rigorously. what exact schemas did you use and how are you defining 'semantic drift' - automated eval or manual inspection?
imo yaml probably has more structural scaffolding with the key: value format vs json's minimal braces, which could help the model track context longer? but could be wrong, would need to test same schema at fp16 to isolate if it's quant-related or format-related
tested yaml vs json yesterday at 19.4k on llama 3.3 70b q4 with same product config schema. yaml broke at 20.1k, json at 17.8k. never measured it properly but the failure modes are different - yaml invents keys, json invents nested structure
ok so this is actually interesting because yaml and toml both have more redundant syntax than json (key = value with more tokens vs json's minimal braces) but toml has even more structure with explicit section headers. if q4 semantic drift correlates with format redundancy we'd expect toml > yaml > json for coherence thresholds, which seems to match the rough numbers people are reporting (toml ~24k, yaml ~20k, json ~17k). but would need rigorous testing with identical schemas across all three formats at same context sizes to isolate the format effect from other variables. what exact schemas did you test and how many runs per format