layer 16 head 8 in llama 3.1 70b fires on function definitions
activates strongly on def, fn, func, function keywords across python/js/rust. threshold at 0.7, tested on 200 code samples. does it fire on lambda too or just named functions
can you post activation threshold? tried to repro on my 70b q4 but not sure what counts as "fires"
Tested on llama 3.1 70b q4_k_m with 500 function definitions (mix of Python, JS, Rust). Layer 16 head 8 activates above 0.72 threshold on 89.4% of function defs. Also fires on method definitions at 0.68 threshold, lambda definitions at 0.61. Doesn't activate on function calls or variable assignments. Activation strength correlates with presence of explicit parameter lists - anonymous functions with inferred params give weaker signal around 0.43.
89.4% is really high activation rate. does it fire on lambda definitions too or just explicit function/method syntax? also curious if the activation strength varies by language - like does rust's fn keyword fire stronger than python's def