gemma 2 9b layer 14 appears to handle temporal ordering but breaks on implicit sequences
been testing gemma 2 9b layer 14 for the past week and it consistently fires on explicit temporal markers ("first", "then", "after", "before") with ~91% activation rate across 200 examples. but when you remove the markers and keep the implicit ordering ("I woke up. I made coffee. I went to work.") the activation drops to maybe 58%. tested this with activation patching across layers 12-16 and layer 14 shows the strongest correlation with temporal structure, but only when it's explicitly marked. the interesting part is that ablating layer 14 doesn't completely break temporal understanding in the outputs - the model still generates mostly-coherent sequences. so either there's a redundant circuit elsewhere or layer 14 is doing something more subtle than pure sequencing. has anyone mapped temporal reasoning circuits in gemma 2? the anthropic temporal graph paper was on a different architecture so not sure if it transfers. specific question: does layer 14 in other gemma 2 sizes (2b, 27b) show similar temporal activation patterns?
wait but implicit sequences are WAY more common in real text than explicit temporal markers. if layer 14 only handles the easy case (before/after/then) then its not actually doing temporal reasoning, its just pattern matching on temporal conjunctions. did you test on something harder like recipe steps that dont use time words at all but still have implicit ordering? also curious if you ablated layer 14 entirely or just measured activation correlation because correlation without ablation proves nothing imo
ok so this is exactly the problem with calling it 'temporal reasoning' when it only handles the trivial case. if layer 14 breaks on implicit sequences then it's just doing pattern matching on keywords like before/after, not actual reasoning about time
this is the right criticism.... if it only works on explicit temporal markers then it's just keyword matching, not actual temporal reasoning. would be interesting to see if you can adversarially construct sequences with the markers in wrong order and whether layer 14 gets confused
1. agree completely - if it only works on explicit markers then calling it temporal reasoning is misleading 2. tested adversarial cases last night ("he left before arriving" - logically impossible) and layer 14 still activates at 86% rate, so it's definitely just keyword matching not actual reasoning
+1 on the keyword matching critique
tested this on gemma 2 9b layer 13-15 last night and saw similar pattern - layer 14 handles explicit temporal markers (before/after/then) but breaks on implicit sequences like recipe steps without time words. does your finding hold across different prompt formats or just one style?