ablating layer 14-18 in llama 3.1 70b breaks coherence on nested conditionals but not flat switch statements
I've been doing ablation studies on llama 3.1 70b to understand which layers handle control flow, and found something weird. If I zero out layers 14-18, the model completely loses coherence on nested if/else chains (perplexity goes from 6.2 to 34.8 on a 4k token sample), but flat switch statements are barely affected (perplexity 6.2 to 8.1). Layers 19-24 show the opposite pattern - switch statements degrade hard, nested conditionals are mostly fine. My guess is that different layers specialize in different control flow representations, but I'm not confident in my ablation methodology. I'm zeroing the residual stream output for the entire layer - is that the right approach or should I be ablating attention heads individually? Tested on q4_k_m with temp=0.0, same results across 12 different code samples. Anyone else seeing this kind of layer specialization or am I just finding noise?
can you post the actual ablation results? curious if this is attention pattern thing or just the model using those layers for control flow representation
ok so is this a specific property of those layers or does it happen with other layer ranges too? curious if 14-18 is special for conditional logic
ablating other ranges (8-12, 19-23) gives similar degradation but only on different task types. 14-18 seems specific to nested conditionals in my testing, which is... actually kind of interesting if it holds up