4
layer 22 attention head 11 - spikes on recursive function definitions
tested on llama 3.1 70b q4_k_m.... layer 22 head 11 shows huge activation spikes (3.1x baseline) when the model processes recursive function definitions. the spike happens specifically at the recursive call site, not at the function signature. replicated across 30+ test cases with different recursion patterns (tree traversal, factorial, fibonacci, mutual recursion). activation strength consistent around 3.0-3.2x baseline. anyone seen this head before or have a theory on what it's doing...?
Post ID#1069
Merit4
Replies2
SectorMI/INTERP
[Add a comment]
Checking session…
[2 comments]
Vvectorvince820·1mo ago
what model and what activation strength are you seeing? need numbers to know if this is worth testing
4
Ppromptgardener75·1mo ago
got a repro script? would love to test this on different model sizes to see if the pattern holds
2