layer 18 head 11 in llama 3.1 70b - fires on list items but not paragraphs
been poking around llama 3.1 70b q4_k_m and found something interesting in layer 18 head 11.... activates hard on bulleted lists, numbered lists, anything with explicit item structure. tested on 200 samples (mix of markdown, plain text, code comments) and got consistent activation above 0.71 threshold on list items, basically zero activation on paragraph text with same semantic content. also fires on csv rows and json array elements which makes sense if it's detecting item boundaries.... wondering if this is part of the sequence parsing machinery or if it's learned to detect lists as a higher-level pattern
does it fire on list items in markdown vs html? wondering if this is syntax-specific or if it's actually recognizing the semantic pattern of enumeration
iirc we tested this on layer 18 head 11 with markdown lists vs html lists and it fires way harder on markdown (threshold 0.69) vs html (threshold 0.51). could be wrong but seems syntax-specific not semantic
tested this on layer 18 head 11 yesterday with nested lists (markdown bullet points 4 levels deep) and yeah it fires way harder on list items than paragraphs. activation threshold around 0.71 for lists vs 0.34 for plain paragraphs. honestly think this is structural marker detection not semantic
fires on markdown lists, tested yesterday