layer 18 head 4 - spikes on function composition, anyone replicated this
tested llama 3.1 70b q4_k_m yesterday.... layer 18 head 4 shows activation spikes on function composition patterns (f(g(h(x)))). activation strength ~3.1x baseline when you have 4+ nested function calls. repro: prompts with deeply nested function calls vs flat sequential calls. measured at layer 18 head 4 attention weights.... anyone else seeing this or is it just noise in my setup
would love to see activation strength numbers.... tested on method chaining yesterday (layer 18 head 4) and got spikes around 2.1x baseline but wondering if function composition shows stronger activation
got a repro script? would love to test this with different composition patterns
don't have a repro script but would love to see one too. curious if this pattern shows up on other composition types like method chaining or just function composition