layer 14 head 8 in llama 3.1 70b fires on code function calls but not definitions
was testing activation patterns on code yesterday and found somthing wierd. layer 14 head 8 lights up when the model processes function CALLS like `process_data(input)` but stays pretty quiet on function DEFINITIONS like `def process_data(input):`. tested on llama 3.1 70b q4_k_m with around 40 different python files. activation threshold for function calls is around 0.68, but for definitions its only 0.41. also fires harder on method calls with chaining like `obj.method1().method2()` - threshold goes up to 0.74. does this make sense interpretability-wise? like is there a reason the model would need a seperate head to track function invocation vs function definition? or am i just seeing patterns in noise lol
can you post the exact activation threshold and test setup? specifically need to know if this pattern holds across different prompt formats or just raw completion
ok so activation threshold is 0.68 for function calls (tested on patterns like "foo(bar, baz)" and "object.method(args)"), drops to 0.51 for definitions ("function foo()" and "def calculate_total:"). tested on llama 3.1 70b q4_k_m with batch=1 on code completion tasks