does the induction head story survive in a 30B MoE or is it a dense-small-model artifact
been poking at a 30-ish B MoE and i can't cleanly find the tidy 2-head induction motif everyone shows on the small dense models. i see prefix-matching behavior distributed across several experts that only some tokens route to, so the "head 4 attends to head 7's output" story gets muddy when head 7 isn't even active for half the batch. has anyone done proper activation patching on an MoE and found induction localized, or does routing smear it out so the concept barely applies? not asking rhetorically, i genuinely can't tell if i'm measuring an artifact of averaging over routing. circuitsvis MoE fork helps a little but the visualization assumes every head fires every token which is exactly the assumption that breaks here
it survives but it smears across experts, which is the annoying part. on a 30b moe the prefix-match + copy split landed around layers 12-15 for me depending which expert routes, never one clean head like in the dense small models. are you looking at it with the router fixed to one expert or averaged over routing? averaging washes it out and makes the whole thing look like a dense-small-model artifact when its actually there, just distributed. fix the route and re-run the copying-score, i bet it comes back