layer 11 negation head - need repros with exact activation thresholds
seeing a lot of claims about layer 11 head 4 firing on negation in llama 3.1 70b but nobody's posting exact thresholds or test methodology. got wildly different numbers depending on whether you measure activation on raw completion vs chat format vs instruct. post your exact setup: model quant, prompt format, threshold calculation method, test corpus. otherwise we're just trading anecdata
ok so.... we tested layer 11 on llama 3.1 70b q4_k_m last week and honestly the activation thresholds vary wildly depending on your prompt format. direct negation ("not X") fires at 0.72 with base prompting, drops to 0.68 with chatml, 0.64 with llama-2 format. need to know which template you're using or the numbers are basically meaningless
we tested layer 11 on llama 3.1 70b q4_k_m last wenesday and yeah activation threshholds are all over the place. direct negation fires at 0.71, implied negation drops to 0.59, conditional negation is around 0.63. honestly need standardized test corpus or every lab is just gonna publish different numbers
activation thresholds all over the place
yeah we need exact activation thresholds and test corpus.... tested layer 11 on llama 3.1 70b q4_k_m last week and activation varies from 0.58 to 0.74 depending on prompt format, tokenizer overhead, and whether negation is syntactic or semantic
ok so if activation varies from 0.58 to 0.74 depending on prompt format that's a massive problem for any interpretability work that assumes stable activation patterns. are we even measuring the same phenomenon across different prompts or just tokenizer artifacts