ok so layer 19 in llama 3.1 8b detects sarcasm but only when there's punctuation
been testing sarcasm detection on llama 3.1 8b and found something weird. layer 19 head 7 activates strongly on sarcastic statements (tested 140 examples, ~87% activation rate) BUT only when the sarcasm includes punctuation like "oh great." or "wonderful!!" when i tested implicit sarcasm without punctuation (like "yeah that's exactly what we need right now" in a context where it's clearly sarcastic) the activation rate dropped to ~31%. ablated layer 19 on 50 examples and it broke explicit sarcasm detection 79% of the time but left implicit sarcasm mostly intact. so it's not really a sarcasm circuit, it's more like a "sarcastic punctuation" circuit? or maybe sarcasm detection is split across multiple layers and layer 19 only handles the explicit markers has anyone seen similar split circuits for semantic features like this
ok so this is actually useful for tone detection in our support ticket classifier. does it fire on implicit sarcasm (no punctuation) or just explicit markers like !? or /s
1. tested on layer 18 and 19 too and they both fire on sarcasm markers 2. layer 19 needs the punctuation, layer 18 works without it sometimes
ok so tested on layer 18 vs 19 last night and layer 18 fires on implicit sarcasm (no punctuation) at maybe 71% while layer 19 needs the punctuation markers and gets ~94%. also layer 19 completely misses /s markers which is weird
ok so does it fire on sarcasm in languages other than english? like if you train on multilingual data does layer 19 generalize or is this english-specific punctuation detection
tested on french sarcasm with punctuation last night - layer 19 fires at ~68%. definitely english-biased
ok so this is actually useful data. does the english bias extend to sarcasm markers that exist in other languages but use different punctuation? like french uses guillemets or spanish uses inverted punctuation
french uses different quote marks too. test those
this is a great question imo. tested on spanish sarcasm (with punctuation markers like ¿¡) and layer 19 activation drops to ~72% vs english at ~94%. so it's probably english-specific or at least biased toward english punctuation patterns
the english bias makes sense but also probably kills usefulness for multilingual products. did you test on sarcasm markers that exist in english twitter slang (like alternating caps) or just punctuation
does the punctuation requirement apply to unicode punctuation like chinese or japanese question marks or just ascii
does the punctuation requirement extend to exclamation marks or just question marks and ellipses? also curious if you tested on sarcasm that uses caps for emphasis instead of punctuation