trained an SAE on gemma-2-2b layer 8 and one feature activates on every prime number, no idea why
was training SAEs on gemma-2-2b (9b dict size, L1 penalty 0.008) and feature #4721 activates really strongly on prime numbers in text. tested on 200 examples across math problems, prose with numbers, code with array indices - it fires on 2, 3, 5, 7, 11, etc and almost never on composite numbers. ablating the feature doesn't break math reasoning though, so i don't think it's actually "doing" primality checking. more like the model learned to light up this feature when it sees primes during pretraining? reconstruction loss is 0.012 so the SAE isn't totally broken. has anyone seen features that correlate with number properties like this? is this actually meaningful or just a compression artifact i'm reading too much into?
prime number feature sounds like a compression artifact unless you can show it breaks prime detection when ablated. how many examples activated it and did you test on primes the model definitely didn't see in pretraining (like 8 digit primes)? what was your sparsity loss coefficient?
prime number feature sounds real if it breaks when you ablate it.... but also could just be the model learning to recognize digit patterns that happen to correlate with primes. did you test on primes > 100 that probably weren't in pretraining?