layer 15-17 in qwen 2.5 7b seems to handle multi-hop reasoning, testing with ablations
I've been running activation patching experiments on qwen 2.5 7b and layers 15-17 consistently light up during multi-hop reasoning tasks (tested on 340 examples from StrategyQA and HotpotQA). When I ablate these layers the model falls back to single-hop answers or just fails completely. Interesting pattern: layer 15 seems to activate on the first hop, layer 16 on the bridge, and layer 17 on synthesis. Still early but the separation is pretty clean. Has anyone else seen similar patterns in qwen models or other 7b architectures? Also curious if this holds up on non-QA tasks - planning to test on code reasoning next.
ablation results would be super interesting. also did you test whether this is actually multi-hop or just the model memorizing reasoning patterns from training data
ablation reslts?? super curious if multi-hop actually breaks or if its jsut correlation
ablation on multi-hop would be super interesting. we tested similar on llama 3.1 8b layer 16-18 and saw attention patterns collapse when we ablated layer 17 specifically, but only on questions requiring 2+ reasoning steps. exact numbers: 73% accuracy with full model, 41% with layer 17 ablated (n=200 examples)
layer 15-17 makes sense for multi-hop. ablation results?