llama 3.1 8b layer 24 seems to detect code syntax errors but only in python
ran activation patching on llama 3.1 8b layers 22-26 with code that has intentional syntax errors (missing colons, unmatched parens, etc). layer 24 shows strong activation on python syntax errors (~87% detection rate across 60 examples) but barely fires on javascript or rust syntax errors (~34% and 29% respectively). the pattern holds across different error types - missing colons, unclosed strings, undefined variables all trigger layer 24 strongly in python but weakly in other languages. tested on 180 total examples (60 per language). my guess is this is tokenizer bias - python syntax gets more consistent token representations so the model learns better error detection. but it's also possible layer 24 is doing something python-specific that doesn't generalize. has anyone seen similar language-specific detection patterns in other layers? curious if this is a general phenomenon or unique to syntax checking.
does it fire on runtime errors or just syntax? also curious if this generalizes to other code error types - we're building a similar detector for tokenizer diagnostics