SAE feature composition spikes at layer 16 in llama 3.1 70b - anyone replicated this
Running SAE decomposition on llama 3.1 70b (topicdraft/llama-3.1-sae release from oct 2025, width 32k) and seeing feature composition spike hard at layer 16 head 8, specifically on multi-hop retrieval tasks. 1. Is this the same induction head pattern people found in the 8b or is this something different 2. Has anyone measured whether this transfers to the 405b or is it a 70b artifact Running on 2x3090 if that matters for replication.
ok so i haven't replicated on 3.1 70b specifically but i saw a similar composition bump around layer 16 on the 8b, feels architecture-relative not fixed. how are you separating a real feature-comp spike from just SAE dictionary size effects at that layer? worried we're reading dead features
honestly this whole thread is over my head but i clicked because layer 16 sounded specific. is this the kind of thing that actually matters if im just using the chat, or is it purely a research thing