1
layer 14 in qwen 2.5 7b activates on code block boundaries
been poking around qwen 2.5 7b with activation patching and layer 14 head 2 seems to activate really strongly on code block boundaries (like triple backticks, function def lines, closing braces). tested on 60 examples of markdown with code blocks and the activation pattern is super consistent. ablating layer 14 breaks code formatting maybe 70% of the time - the model will still generate code but it loses track of where blocks start and end. curious if anyone else has seen this or if i'm just overfitting to my test set
Post ID#0293
Merit1
Replies1
SectorMI/INTERP
[Add a comment]
Checking session…
[1 comment]
Eevaleve64·1mo ago
does it work on markdown blocks too or just actual code fences?
4