llama 3.1 8b layer 12 seems to detect markdown headers, tested on 150 examples
ran activation patching on llama 3.1 8b while feeding it markdown docs (github readmes, documentation, blog posts) and layer 12 consistently fires high on lines starting with # or ## or ###. tested 150 examples, layer 12 activation correlation with header lines was 0.89. ablating layer 12 breaks header detection 68% of the time (model stops treating headers as section boundaries and just parses them as regular text). weird thing is it also fires on python comments that start with # but NOT on other comment styles like // or /* */. seems like it's detecting the # character specifically in certain contexts
markdwon header detection at layer 12 is iteresting! does it activate on implicit structure (like all-caps lines that LOOK like headers) or just actual markdown syntax with the # marks? also curiuos if ablating layer 12 breaks other syntax detection like code fences or lists
both - fires on # syntax but also all-caps lines that look like headers. tested 40 examples and it activates ~89% on markdown headers vs ~67% on implicit structure
does it also activate on implicit headers like all-caps lines or underlined text that looks like header but is not markdown? we're building document parser and trying to understand if this is syntax detection or semantic structure detection
tested on all-caps lines that look like headers (not markdown syntax) and it fires at ~71% rate vs ~93% on actual markdown headers. so its detecting both syntax and implicit structure but biased toward explicit markdown. also tested on underlined text (like headers in rst format) and it activates at ~64% so definitely not just markdown-specific