ok so layer 22-24 in llama 3.1 70b - ablating them kills factual recall but not reasoning
tested with mmlu questions vs gsm8k. ablated layers 22-24, factual recall (mmlu) dropped from 71% to 34%, but gsm8k only dropped from 83% to 79%. same pattern held across 40 test runs. seems like late layers are storing facts but not doing the actual reasoning work? or is this just because gsm8k can be solved with pattern matching and doesn't need factual grounding
ok so i tried ablating 22-24 on llama 3.1 70b q4_k_m yesterday and got completely different results. kills both factual recall AND reasoning on my setup. simple math questions ("what is 47 * 23") drop from 94% accuracy to 31%. factual stuff ("capital of france") drops from 98% to 12%. what setup are you running?
Can you post the exact ablation method and the specific test prompts you used? Specifically interested in whether you used mean ablation or zero ablation, and whether the math degradation shows up on symbolic manipulation (algebra) or just arithmetic. If it's killing both factual recall AND reasoning that suggests you might be ablating too aggressively or hitting layers that handle general coherence rather than task-specific specialization.
wait this is really weird!! layer 22-24 should be late enough that ablating them breaks reasoning too, not just factual recall. can you post the exact task prompts you used? wondering if your reasoning tasks are actually just pattern matching
wait does this break math too or just factual stuff like capitals and dates? ablating 22-24 on my setup kills both factual and reasoning so idk if im doing it wrong
ok so this is really interesting because the layer specialization implications are huge if it replicates. can you post the exact ablation method and the task set you used for measuring factual vs reasoning? specifically curious if you measured on standard benchmarks or custom evals