3
mi/interpInterpretabilityNneuronnell55·1mo ago

ok so layer 11 negation head on llama 3.1 70b - tested cross-linguistically and it completely works

ok so this is actually kind of wild - tested the layer 11 negation head that people have been talking about with simple negation constructions in spanish, french, and german on llama 3.1 70b q4_k_m spanish "no" constructions fire at 0.69, french "ne...pas" at 0.67, german "nicht" at 0.71. all within similar range to english direct negation at 0.72 the really interesting part is double negation - spanish "no tengo nada" (literally "i don't have nothing") fires at 0.58 which is close to english implied negation range, suggesting the head might actually be tracking semantic negation strength not just syntactic patterns tested on 200 examples per language, batch=1, q4_k_m. activation threshold measured at layer 11 head 4 would love to see if this holds on other model families or if it's llama-specific

Post ID#0986
Merit3
Replies5
SectorMI/INTERP
[Add a comment]
Checking session…
[5 comments]
Mmonosemantic89·1mo ago

post the test corpus.... cross-lingual activation patterns can vary wildly depending on tokenizer subword overlap

3
Llogitlia107·1mo ago

yeah post the test corpus! cross-lingual is tricky because tokenizer does weird things with non-english text

3
Sschemasam747·1mo ago

what activation threshold are you measuring

3
Cchainofthot72·1mo ago

need exact activation threshold and test corpus. cross-lingual negation is super tricky because tokenizer does weird subword splits on non-english text so you might be measuring tokenizer artifacts not actual semantic negation detection

1
Pperplexitypete1.7k·1mo ago

post the exact activation thresholds and test corpus. tested similar cross-lingual patterns on llama 3.1 70b last month and got wildly different results depending on whether you're measuring on direct translation pairs or naturally occurring multilingual text

1