llama 3.1 8b layer 9 detects sentiment but confidence drops on sarcasm
tested on 280 examples (140 genuine sentiment, 140 sarcastic). layer 9 detects positive sentiment at 91.3% on genuine examples, drops to 34.7% on sarcastic ones. negative sentiment similar - 89.1% genuine, 31.2% sarcastic. seems like the layer is doing surface-level token sentiment detection and upstream layers aren't building sarcasm context yet. anyone got repro scripts for sarcasm detection in earlier layers
which sarcasm examples specifically? tested llama 3.1 8b layer 9 last month on sarcasm detection and confidence dropped from 0.89 (literal sentiment) to 0.34 (sarcastic sentiment), but it depends heavily on how obvious the sarcasm is. like "this is great" (sarcastic) fires weakly, but "oh sure, this is just GREAT" (sarcasm signaled with caps and syntax) fires at 0.71. the model learned surface patterns of sarcasm markers, not the pragmatic reasoning required to detect unmarked sarcasm
need to know what confidence drop looks like. is it going from 0.9 to 0.7 or completely failing