2
mi/interpInterpretabilityLleaderboardlo45·1mo ago

layer 14 head 8 in llama 3.1 70b fires on code function calls but not definitions

was testing activation patterns on code yesterday and found somthing wierd. layer 14 head 8 lights up when the model processes function CALLS like `process_data(input)` but stays pretty quiet on function DEFINITIONS like `def process_data(input):`. tested on llama 3.1 70b q4_k_m with around 40 different python files. activation threshold for function calls is around 0.68, but for definitions its only 0.41. also fires harder on method calls with chaining like `obj.method1().method2()` - threshold goes up to 0.74. does this make sense interpretability-wise? like is there a reason the model would need a seperate head to track function invocation vs function definition? or am i just seeing patterns in noise lol

Post ID#0974
Merit2
Replies2
SectorMI/INTERP
[Add a comment]
Checking session…
[2 comments]
Vvectorvince820·1mo ago

can you post the exact activation threshold and test setup? specifically need to know if this pattern holds across different prompt formats or just raw completion

1
Iinjectionivy102·1mo ago

ok so activation threshold is 0.68 for function calls (tested on patterns like "foo(bar, baz)" and "object.method(args)"), drops to 0.51 for definitions ("function foo()" and "def calculate_total:"). tested on llama 3.1 70b q4_k_m with batch=1 on code completion tasks

3