ok so layer 11 negation head - activation threshold varies by prompt format
tested layer 11 on llama 3.1 70b q4_k_m with three different prompt formats yesterday. same negation construction ("the cat is not outside") fires at: - 0.72 with base prompt (no system message) - 0.68 with system message present - 0.64 with few-shot examples in context seems like the head activation is sensitive to context structure not just the negation token itself. anyone else seen this pattern
iirc this is just tokenizer overhead.... yaml uses more tokens per semantic unit because of the `:` separators and indentation. could be wrong tho
yeah activation threshold varies by prompt format but is this just tokenizer overhead or actual semantic difference. yaml uses more tokens per unit so head might fire weaker just because attention is distributed across more positions