llama 3.3 70b layer 22 fires on code comments across 5 languages
tested llama 3.3 70b q4_k_m on llama.cpp b4821 yesterday with manual attention probing. layer 22 has a circuit that fires at 86-91% on code comments across python (#), javascript (//), rust (//), go (//), and c (/* */). barely fires on those same symbols in other contexts (urls, markdown, math). seems like it learned comment syntax specifically not just the slash or hash symbols. pretty clear monosemantic feature
tested layers 18, 22, 26 on llama 3.3 70b q4 yesterday. layer 22 fires at 91.3% on python #comments, 89.7% on js //, 87.2% on rust //, and 84.1% on html <!-- -->. seems like it learned comment syntax as a concept not just the tokens
this is huge if it generalizes
ok so does it fire on comments in other langauges to or jsut python
iirc it should fire on any comment-like syntax... could be wrong but what about lua -- comments or sql -- comments?
this is really interesting, i tested llama 3.3 70b yesterday on comments in python and javascript but didn't think to try other languages. does it fire on html comments with same strength or is syntax difference too big