llama 3.1 70b layer 28-31 - ablating them breaks reasoning but not memorization
Ran ablation tests on llama 3.1 70b q4_k_m targeting layers 28-31. Simple reasoning tasks ("if A>B and B>C, what's the relationship between A and C") drop from 94.2% accuracy to 31.7% with those layers ablated. But pure recall tasks ("what is the capital of France", "who wrote Hamlet") stay at 97.8% vs 98.1% baseline. This is the inverse of the layer 22-24 results from thread #926 where ablation killed recall but preserved reasoning. Looks like llama 3.1 70b might have layer specialization with early-mid layers (22-24) handling factual retrieval and late layers (28-31) handling logical inference. Tested on 200 samples, 100 reasoning + 100 recall. Used mean ablation (replaced activations with layer mean across 1000 calibration samples).
ok this is really intresting!! we hit somthing similar at work last month where ablating different layer ranges broke different types of reasoning. can you post the exact ablation method and the task set you used? specifically curious if it breaks math reasoning or just like logical inference stuff
Tested exact same pattern on llama 3.1 70b q4_k_m yesterday. Ablating layers 28-31 breaks multi-hop reasoning ("if A then B, if B then C, therefore if A then C") but simple factual recall works fine ("what is the capital of France"). Perplexity on reasoning tasks jumps from 6.8 to 14.2 after ablation, factual recall barely moves (7.1 to 7.4). Ablation method: mean ablation on residual stream. Test set: 50 reasoning questions from MMLU-Pro, 50 factual questions from TriviaQA. Wondering if 28-31 is where the model composes information vs just retrieves it?