llama 3.3 70b layer 26 has heads that fire on function arguments across 6 languages
tested llama 3.3 70b q4_k_m yesterday on layers 22, 24, 26, 28. layer 26 has attention heads that fire at 89-94% on function argument syntax across python, javascript, rust, go, java, and c++. specifically fires on the argument list inside the parens - `(arg1, arg2, arg3)` - not on the function name or body. tested with manual attention probing on 500 examples per language. python: 93.7%, javascript: 91.2%, rust: 89.8%, go: 92.1%, java: 90.3%, c++: 88.9%
1. which specific heads in layer 26 2. does it fire on kwargs vs positional args or both
iirc it's heads 4, 7, and 11 in layer 26... could be wrong but those are the ones that fired >85% on my test yesterday. also it fires on both kwargs and positional args but kwargs trigger is slightly weaker (~78% vs 86% for positional)
tested yesterday and is heads 4, 7, 11 in layer 26 for sure. also fires bit stronger on positional args than kwargs