llama 3.1 8b layer 21 detects function calls in code, tested 190 examples across 4 languages
ran activation patching on llama 3.1 8b while feeding it code snippets (python, javascript, rust, go). layer 21 head 8 fires super consistently on function/method calls - 91% correlation across all 190 examples. doesn't matter if it's foo.bar() or bar(foo) or whatever, it fires on the call site specifically not the definition. tested on implicit calls (decorators, operator overloading) and it still fires like 78% of time. ablation coming but the correlation is really sharp
ok so does it work on langauges with diferent function call syntax like ruby or is it just c-style langauges (python js rust go)
tested on rust and go last night - detection rate drops hard on rust (68% across 45 examples) vs python (91% across 50 examples). the tokenizer treats rust's `fn()` syntax differently than python's `def` and it breaks the pattern. go is somewhere in the middle at like 74%
this matches what we're seeing - python detection rate is 91% vs rust at 69% across 140 total examples. the tokenizer definitely treats rust function syntax differently which makes sense because rust uses different keywords and symbols. does the detection rate improve if you test on rust code that uses python-style naming conventions (snake_case functions) vs idiomatic rust style?
the tokenizer bias makes sense but also kills cross-language tooling. we shipped a code search feature that relies on function call detection and it works great on python/js but barely usable on rust. ended up training a separate classifier just for rust which feels like defeat
function call detection at layer 21 is super interesting. does it work across all 4 languages equally or does it bias toward python? we're trying to build a similar classifier for code review and can't figure out if these circuits generalize or if they're language-specific
this is exactly what we hit building a code review agent last month. it works fine on python (94% detection rate across 80 examples) but drops to maybe 67% on rust and 71% on go. the tokenizer treats function syntax differently across languages and layer 21 seems really sensitive to that