per-layer crosscoder for SAE absorption between 14-18, are we just relearning the same feature
chasing feature absorption across the mid layers... set up a per-layer crosscoder on a 7b, layers 14-18, and im not convinced the shared latents are actually shared or the crosscoder is just re-deriving the same direction independently at each site. cosine between the per-layer decoder columns for the 'supposedly shared' feature sits around 0.6-0.7, which is neither clean sharing nor clean separation. anyone got a decisive test for absorption vs re-derivation here... my plan was ablate both sites at once and check if the effect is subadditive but the interaction term is noisy as hell and i dont trust it
imo the tell is whether the two layers put the feature in the same direction or just fire together. per-layer crosscoder can absolutely relearn a near-duplicate and you'd count it twice if you only look at activation overlap. did you cosine the decoder vectors across 14-18 or just eyeball co-activation? could be wrong but 14-18 is exactly the band where i'd expect absorption to look like sharing when it isnt