pythia 1.4b layer 11 head 2 activates on nested parentheses and i can't figure out why
Been probing pythia-1.4b for a week and found something weird - layer 11 head 2 has strong activation (>0.7) on any input with nested parentheses, regardless of content. Tested 230 examples: - Math expressions with parens: 89% activation - Code with nested function calls: 91% activation - Plain text with (asides in parens): 87% activation - Control (no parens): 4% activation The head doesn't seem to care about WHAT'S in the parens, just that they're nested at least 2 deep. Single-level parens only trigger 31% of the time. Ablating layer 11 head 2 breaks paren matching in code generation tasks 67% of the time (tested on HumanEval subset, 45 problems with nested function calls). Anyone seen similar structure-tracking heads in other pythia checkpoints? Also unclear if this generalizes to other bracket types or just ().
can you post examples? curious if it's activating on implicit nesting (like prose that has nested clauses) or just explicit parens. also does ablating layer 11 break paren matching or just reduce activation
this is really cool imo. nested parens activation could be a tokenizer artifact though - does it fire on implicit nesting (nested clauses in prose) or just literal parens? also iirc pythia was trained on the pile which has a ton of code, so it might be detecting lisp/scheme syntax specifically. could be wrong though
does it fire on other bracket types too? like curly braces or square brackets