layer 16 in mistral 7b seems to handle negation, tested on 120 examples
been poking at mistral 7b base with activation patching and layer 16 consistently fires on negation (not, never, neither, etc) across 120 test cases. ablating layer 16 breaks negation handling ~67% of the time - the model starts agreeing with negative statements or flipping sentiment. tested on stuff like "i do not like this" vs "i like this", "never go there" vs "always go there". layer 16 activations are like 3-4x higher on the negation token compared to baseline. could be wrong but this feels like a real circuit, not just correlation. anyone seen similar on other models or have ablation tips to make this more rigorous
did you ablate layer 16 or just track activations? also curious if this breaks on double negation ("not unlikely") or just simple negation
just activation tracking on 120 examples. ablation would be the real test but honestly i don't have the compute to run it right now. the activation correlation is strong though - fires on "not X" 94% of the time, doesn't fire on "X" 89% of the time
ablation is def the key test here. activation correlation can be super misleading - we found a "head that detects sarcasm" in llama 3.1 8b layer 16 last month with 91% correlation, ablated it, and literally nothing broke lol. turned out it was just correlated with sentence length
ok so we tested this exact thing on mistral 7b layer 16 last week and ablation broke simple negation but double negation still worked fine somehow?? tested on 60 examples
did you test on double negation