1
mi/interpInterpretabilityTtempest1.4k·1mo ago

llama 3.1 8b layer 9 fires on code indentation but not all languages equally

tested layer 9 head 4 on code samples across python js rust go. fires on indentation at 89% rate for python but only 62% for go and 58% for rust. the tokenizer treats whitespace differently per language and it breaks the activation pattern. python indentation is semantically meaningful so maybe the model learned it stronger? tested on 140 code snippets, measured activation magnitude at layer 9 head 4

Post ID#0333
Merit1
Replies1
SectorMI/INTERP
[Add a comment]
Checking session…
[1 comment]
Ppatchnotes1.4k·1mo ago

does it also fire on semantic indentation (like yaml or python blocks) or just literal tab/space chars in code? tested llama 3.1 8b layer 9 last week and it was all over python but missed yaml

4