layer 18 head 4 function composition spikes - tested on different model sizes, pattern doesn't hold
saw the thread about layer 18 head 4 spiking on function composition and tried replicating on different llama model sizes. tested llama 3.1 8b, 70b, and 405b (all q4_k_m) with identical prompts (nested function calls, composition chains 4-6 deep). results: layer 18 head 4 only shows the spike pattern on 70b. on 8b the spike is at layer 11 head 2, on 405b it's at layer 23 head 6. activation strength varies 40%+ across model sizes. makes me think this isn't a generalizable circuit, just a measurement artifact that happens to land at layer 18 on one specific model size. has anyone tested this across different model families (mistral, qwen) or is everyone just replicating on llama 70b?
we hit this exact pattern discrepancy testing layer 18 head 4 on llama 3.1 8b and 70b last week. on 8b the composition spike shows up at layer 11 head 4, on 70b it's layer 18 head 4 like you said. the pattern seems to scale with model depth - roughly 60% of the way through the layer stack for both sizes. curious if this ratio holds on other architectures or if it's llama-specific
tested on Llama 3.1 70B and gpt2-xl yesterday, pattern doesn't hold across sizes. layer 18 head 4 shows composition spikes on 70B but same pattern shows up at layer 12 on the smaller models
tested on qwen 2.5 14b yesterday and pattern shows up at layer 9. seems like it scales with model depth not absolute layer number
source on the qwen test? trying to figure out if this scales with model depth
this is really interesting.... does the pattern scale linearly with model depth or is there a step function somewhere. like does a 14b model put composition detection exactly halfway between the 8b and 70b coordinates or does it jump around
tested this on llama 3.1 8b last week and the composition spike shows up at layer 11 head 4, so roughly 61% through the model depth vs 64% on the 70b. seems pretty linear actually