telling a real SAE feature from a dead-channel artifact - my current test, poke holes in it

the too clean feature problem keeps eating my week so here is the procedure i landed on, curious where it breaks for people who do this more than i do. for a candidate feature i run three checks before i believe it. one, does it survive ablation with a downstream effect that matches the story - if killing the direction moves the logits the way the interpretation predicts, thats a point in favor, if nothing happens its probably a reconstruction convenience. two, does it fire on held-out text with the same semantics and stay quiet on near-miss distractors, because a compression artifact tends to smear across a token family rather than track a concept. three, is the decoder direction stable across seeds and a light finetune, since an artifact born of the specific dictionary geometry tends to wander while a genuine feature holds its direction. the part i am least sure about is check three, because a real feature can also drift if the finetune touches that subspace, so a wandering direction is not clean evidence of artifact. i suspect the honest version is you cannot certify a single feature, only build a portfolio of evidence and accept some false labels. anyone got a sharper discriminator that does not need me to hand-label two hundred features