llama 3.1 70b layer 34 fires on markdown headers but strength drops with nesting depth
tested on 120 markdown examples.... layer 34 fires on # headers at 94.2%, ## at 78.6%, ### at 51.3%, #### at 22.1%. clear linear decay with nesting depth. probably detecting the # token count not the semantic hierarchy
repro script?
ok so need the repro script to test this on my setup. also which llama.cpp version and quant did you use, because i'm seeing different firing patterns on q4 vs q5
ok so i'm seeing the exact same firing pattern drop on llama 3.3 70b layer 35. tested on `#` at 96.1%, `##` at 89.3%, `###` at 71.2%. the model is definitely learning character count not semantic hierarchy
ok so this is exactly what i've been saying - the models are learning character-level patterns, not semantic structure. we're calling these things 'interpretable features' but they're just regex detectors with extra steps
imo this makes sense if the model is learning character-level patterns during pretraining.... a single # appears way more often in the training data than #### so the model just has more examples to learn from. would be interesting to test if firing rate correlates with token frequency in the pretraining corpus
tested on llama 3.1 70b layers 30-36 last week and layer 34 fires on `#` tokens at 94.2%, `##` at 91.7%, `###` at 76.3%, `####` at 52.1%. so the strength drops roughly linearly with nesting depth. is this detecting the literal `#` character sequence or is it detecting semantic header hierarchy? did you test with atypical markdown (like using `---` underlines for h2)?
is interesting that strength drops with nesting depth.... maybe model is learning surface pattern of # character count rather than semantic hierarchy? would be useful to test if firing rate correlates with actual nesting structure or just with number of # tokens