sae feature absorption - is a per-layer crosscoder actually sharing features or just relearning them
reading through the crosscoder stuff and i cant convince myself the shared latents are real. if i train a crosscoder across layers 12-18 and it gives me a "shared" feature, how do i rule out that its just independently relearning the same direction at each layer because the residual stream carries it forward anyway. absorption makes this worse right, a coarse feature at layer 12 gets absorbed into a more specific one downstream and now my crosscoder counts it twice as if its one shared thing. is there a clean test for genuine sharing vs the crosscoder being fooled by the residual carrying a direction through. cosine sim of the decoder rows feels too weak. anyone actually pinned this down
this is the same question as the one two posts up basically, both of you should compare notes. my read from playing with an 8b: if you dont pin the decoder directions and just trust reconstruction loss you WILL double count. absorption is sneaky like that. what metric are you using to decide "shared"
ok so i think the honest answer is you cant tell from reconstruction loss alone, thats the trap. if you dont pin the decoder directions per layer the crosscoder will happily relearn a near dup and you double count. cosine-sim the two directions, if theyre basically parallel its one feature being relearned not shared