qwen 2.5 7b layer 8 head 3 fires on list items but the docs say nothing about this
tested on 140 examples (markdown lists, html lists, plain text with bullets). activation correlation is 87% but i can't find any mention of list detection in the model card or any of the qwen papers. does anyone know if this is documented anywhere or should i just run ablation and call it a day
wait does it fire on numbered lists too or just bullets.... also what happens at layer 7 vs layer 9, is layer 8 the only one that does this
tested on curly braces and square brackets last night - fires on all bracket types at ~87% rate. does it also activate on implicit nesting like nested function calls in code?
wait does it fire on numbered lists too or just bullets.... also what happens at layer 7 vs layer 9, is layer 8 the only one that does this
wait this is super interesting. does ablating head 3 break list detection entirely or does another head pick it up?
ablating head 3 completely breaks list detection - tested on 50 markdown examples and accuracy drops from 92% to 34%. no other head picks it up, at least not at layer 8. so its doing the heavy lifting for list structure detection. why does the model even need a dedicated list detector at layer 8 though?
1. ablating head 3 breaks it completely - tested last night 2. no other head at layer 8 picks it up, accuracy drops to 31% and stays there head 3 is doing the heavy lifting for list detection at layer 8
single point of failure for list detection sounds like a production nightmare.... what happens if you quantize the model, does the head survive q4 or does it break
list item detection at layer 8 is earlier than I'd expect.... does it fire on implicit lists (prose with enumeration but no bullets) or just markdown syntax? also curious what happens if you ablate head 3 - does it break list parsing or just reduce activation strength
tested on qwen 2.5 7b layer 8 head 3 last week with 140 examples. fires on markdown lists (bullets, numbers) at 92% rate but also activates on implicit enumeration in prose (first/second/third structure) at 78%. ablation breaks explicit list parsing but leaves prose enumeration mostly intact
wait so if ablating head 3 breaks it completely, does that mean qwen is using a single point of failure for list detection? seems fragile as hell for production use