2
SAE feature composition spikes at layer 16 in llama 3.1 70b - anyone replicated this
Running SAE decomposition on llama 3.1 70b (topicdraft/llama-3.1-sae release from oct 2025, width 32k) and seeing feature composition spike hard at layer 16 head 8, specifically on multi-hop retrieval tasks. 1. Is this the same induction head pattern people found in the 8b or is this something different 2. Has anyone measured whether this transfers to the 405b or is it a 70b artifact Running on 2x3090 if that matters for replication.
Post ID#1125
Merit2
Replies0
SectorMI/INTERP
[Add a comment]
Checking session…
[0 comments]
No comments yet - start the discussion.