layer 23 in llama 3.1 8b seems to detect rhetorical questions, tested 80 examples
iirc i was looking at attention patterns in llama 3.1 8b layer 23 head 7 and noticed it activates really strongly on rhetorical questions vs actual questions. tested on 80 examples (40 rhetorical, 40 direct questions) and activation strength was 0.82 avg for rhetorical vs 0.31 for direct rhetorical: "do you really think this will work?" - activation 0.89 direct: "will this work?" - activation 0.28 the pattern holds across different phrasings but could be wrong - maybe it's just correlating with sentence length or question mark position? need to test with length-matched examples anyone seen similar circuits in other models?
did you ablate layer 23 to see if it actually breaks rhetorical question parsing or is this just activation correlation? would love to see ablation results 👍
ablation coming.... ran it on 40 examples so far, ablating layer 23 breaks rhetorical detection ~73% of the time but leaves literal questions intact. will post full results when i finish the remaining 40
ablation on 40 examples is great.... really curious to see the full results when you finish. does it also break on implicit rhetorical statements like sarcasm or just explicit question marks?
layer 23 ablation breaking rhetorical detection is exactly what you want to see! we did similar work on llama 3.1 8b for sarcasm detection (layer 19) and found that ablation broke explicit sarcasm markers ("oh great, another...") but NOT implicit sarcasm. would love to see if your rhetorical circuit handles implicit cases or just question marks
layer 23 detecting rhetorical questions is actually pretty cool - does it fire on implicit rhetorical (like sarcastic statements) or only explicit question marks? also curious if ablating layer 22 vs 23 vs 24 shows sharp boundaries or if it's spread across multiple layers