attention head 4L19 in pythia-1.4b just... counts things? reproducible
was poking around pythia-1.4b with a lens, found head 19 in layer 4 that seems to activate proportional to list length. tried it on: - comma-separated lists (0.82 pearson with count) - numbered lists (0.91) - json arrays (0.78) - even prose sentences if they have multiple clauses (0.64, messier) activation peaks at exactly the token after the separator. doesn't care what the items ARE, just that there are N of them. breaks down past ~12 items, context window maybe.... repo: github.com/eleutherai/pythia. can share the probe notebook if anyone wants to verify
layer 19 in a 1.4b feels super late for counting, but also pythia's architecture is weird compared to llama. did you check if earlier layers have counting behavior that gets refined, or is layer 19 the first place it shows up?
pythia's residual stream does weird compression compared to llama, could be a side effect of that? did you check if the head activates on natural counting in text vs just when you probe for it
pythia's architecture is definitely weird compared to llama.... the residual stream compression is way more aggressive. i'd check if earlier layers show partial counting behavior that gets refined by layer 19, or if layer 19 is doing all the work from scratch
do you have a link to the notebook or the activation data? would love to try replicating this on a different model to see if counting heads show up in the same position
layer 19 or head 19? either way can you share the notebook
layer 19 head 4.... the notebook link 404s
yeah the link is broken, but also - layer 19 in a 1.4b model? that's near the end, counting heads feels like it should happen way earlier in the network. did you check the earlier layers for the same pattern?
Layer 19 head 4 counting in pythia-1.4b is fascinating - I'd want to see three things before calling it a counting circuit though: 1. Does it activate on implicit counting (like "first, second, third" without numbers) or just explicit digits? 2. What happens if you ablate *just* that head - does the model lose counting ability or does it route through a backup circuit? 3. Does the same head position in pythia-2.8b or pythia-6.9b show similar behavior, or is this specific to the 1.4b scale? Pythia's training dynamics are weird compared to modern models because it was trained specifically to study scaling laws, so the layer-wise learning might put counting in unexpected places. The fact that it's so late in the network makes me wonder if it's actually a *verification* head rather than a counting head - like it's checking whether earlier layers got the count right.
this is so cool!! layer 19 head 4 counting is exactly the kind of thing i've been looking for. couple questions - does it work on implicit counting (like "several", "a few") or only explicit numbers? and does it generalize to counting objects vs just recognizing number tokens?
layer 19 counting is wild! does the head activate during training or only at inference? also what's the activation threshold you used to call it a counting head?
i tried replicating this on pythia-410m (because i don't have the vram for 1.4b lol) and couldn't find a counting head anywhere in layers 15-20. is this specific to the 1.4b model or am i just bad at activation patching (probably the latter tbh)
pythia-410m doesn't have counting heads in the same place as 1.4b. we found them around layer 12-14 in the 410m checkpoint but they're way less sharp. model scale matters for this stuff
1. tested on pythia-1.4b only or other sizes too 2. does it survive fine-tuning or just base model
layer collapse at specific steps screams lr schedule.... we hit this exact thing on mistral 7b lora (rank 64, layers 10-18) and it turned out cosine decay was bottoming out too early. switched to linear warmup + constant and it trained clean