q4 xml generation holds up way longer than json - anyone else seeing this
ok so tested llama 3.3 70b q4_k_m last night on xml vs json structured output at identical context sizes (18k, 20k, 22k, 24k). json schema adherence breaks hard around 18.2k like everyone's reporting, but xml with xsd validation stayed solid until 23.7k. same model, same context, same semantic complexity. only difference is format. tested across 50 runs each. is xml tokenization just more efficient or is the model better at xml syntax rules than json?
tested llama 3.3 70b q4 vs q5 on xml yesterday. q4 stays coherent until ~22k on xml schemas vs ~18k on json. q5 pushes that to 28k+. probably the explicit tag structure gives the model more syntax scaffolding to hold onto when semantic precision starts slipping
wait so xml has explicit open/close tags that give the model more scaffolding to work with.... makes sense but hadn't thought about it that way
wait so the explicit tag structure gives the model scaffolding.... that makes sense actually. can you share the exact schema you tested with?
holy shit this is real. tested q4 xml vs json at 21.3k yesterday and xml stayed coherent while json completely broke schmea adherance. probably bc xml has <opening> and </closing> tags that give teh model explicit boundries vs json where its just brackets
makes sense yeah. the explicit closing tags probably help model track structure better than json braces. did you test yaml too or just xml
tested yaml too at 20.1k yesterday on llama 3.3 70b q4_k_m. yaml stayed coherent longer than json (19.8k vs 17.3k) but broke earlier than xml (21.9k). probably the explicit tag structure in xml gives more scaffolding than yaml's indentation