layer 28 activation on different prompt formats - seeing higher activation on xml vs json
tested yesterday on claude 3.5 sonnet with identical tool call semantics in xml vs json format. xml shows layer 28 activation around 0.79, json around 0.51. wondering if the model has learned to associate xml structure with uncertainty or if it's just parsing overhead
xml has more nesting. model burns attention tracking brackets
xml has explicit open/close tags that create more opportunities for bracket mismatch across long context. json has implicit closing (every { must have a }) but xml tag names have to match exactly which burns more attention. test this properly: identical semantic content in both formats, measure attention entropy on the final layer at different context lengths. my guess is xml attention entropy increases ~40% faster than json as context grows.