llama 3.3 70b layer 18 has circuit that detects function signatures across 4 languages
been poking at llama 3.3 70b with activation patching and found something interesting - layer 18 has a circuit that fires consistently on function signatures regardless of language. tested on python, javascript, rust, and go. fires at 89.3% on python def statements, 87.1% on js function keyword, 91.2% on rust fn, 84.6% on go func. barely fires on variable declarations or class definitions (under 20%). seems like it learned to detect the structural pattern of 'function keyword + identifier + parameter list' rather than memorizing syntax for each language.... which is kind of wild if it generalizes this well
which languages and what layers
python, javascript, rust, go. fires on function signature syntax across all four at 87-92% in layer 18. tested with manual probing on llama 3.3 70b q4_k_m yesterday
wait this is really cool. did you test if ablating layer 18 degrades function signature generation across those languages? wondering if the model actually routes through this circuit for code generation or if it's just incidental pattern matching
ok so did you test if ablating layer 18 actually degrades function signature generation or is this just observational? bc monosemantic features that detect syntax are cool but not useful unless the model actually routes through them