imo SAE reconstructions feeling too clean is a red flag not a flex
been playing with some sparse autoencoders on small models and idk, when the reconstruction loss gets really low i get suspicious? like either the model is simpler than we think or we're overfitting to noise seeing a lot of papers flex perfect reconstructions but then the learned features are weirdly polysemantic still. could be wrong but feels like we're optimizing the wrong thing sometimes anyone else run into this or am i just doing it wrong 😅
if the reconstruction error is too low you didn't find interesting features, you found a compression algorithm. show me what it gets wrong
lol this. spent three weeks chasing a "monosemantic" feature that turned out to be jpeg artifacts. if your reconstruction loss is under 0.01 you found the boring part
oof this is painfully relatable.... how did you finally catch it, just manual inspection or did you have some automated check
+1. if you can't show me three examples where it fails i don't believe you found a feature
ok so this is exactly right. we published SAE results last year with 0.008 reconstruction loss and reviewers loved it but in hindsight we just found the residual stream, not any actual features. now we actively look for reconstruction errors between 0.03-0.08 as a sign we're getting somewhere interesting
wait so you published with 0.008 loss and it turned out to be the residual stream? how did you figure that out eventually - did you try training on random layers or what? i'm worried we're making the same mistake right now on a gemma-2-9b SAE and the reconstructions look too good
layer ablation worked for us. train SAE, then zero out each learned feature and measure task performance delta. if nothing breaks you didn't find features
I have seen this in our testing too. the SAE was giving back almost perfect reconstruction on layer 18, but we could not understand why model was making decision. Compression is not explanation)
yep saw the same thing, SAE gave us near-perfect reconstruction on layers 16-20 but the explanations made no sense. iirc compression artifacts show up as low-loss high-frequency features that don't correspond to anything semantic. did you try ablating the feature to see if it actually mattered?