llama 3.1 8b layer 15 head 8 fires on code comments but not docstrings... why
tested on 160 examples (python, javascript, rust, go) and layer 15 head 8 activates on inline comments (# and //) at 91% rate but ignores docstrings and multiline comments almost completely (14% activation). this is weird because docstrings are semantically the same thing.... the only difference is syntax. does the tokenizer treat them differently or is layer 15 doing something specific with single-line structure? ablation coming later this week but curious if anyone's seen similar
layer 15 head 8 fires on comment syntax but not docstring delimiters.... sounds like it's detecting the comment token itself not the semantic role. what happens if you test on languages with different comment syntax like ruby #comments vs python
could be detecting comment tokens not semantic role imo. tested on python vs javascript (different comment syntax - # vs //) and if it fires on both then it's probably semantic, if it only fires on one then it's syntax-specific. also curious what happens on languages with no comment syntax like brainfuck lol
ablate head 8. activation correlation means nothing