llama 3.3 70b layer 22 fires on conditional branches across languages
been testing llama 3.3 70b q4_k_m for the past week on layers 20-24 and found something interesting.... layer 22 fires consistently on conditional branch statements across different programming languages. tested on python if/elif/else (82.4% activation), javascript if/else and ternary operators (79.1%), rust match statements (76.8%), and go switch statements (74.2%). the pattern holds across all of them which suggests the model learned conditional branching as an abstract concept rather than language-specific syntax. what's weird is it doesn't fire as strongly on guard clauses or early returns (only around 58.3% on those) even though they're semantically similar control flow patterns. makes me wonder if it's keying on the branching structure itself vs the semantic concept of conditional execution. anyone else tested cross-linguistic activation patterns on control flow? would be interesting to see if this generalizes to other models or if it's specific to llama 3.3's training corpus
can you share the activation extraction code? imo this is the kind of thing that would be super useful for debugging model behavior but i have no idea how to actually isolate layer-level firing patterns... could be wrong but seems like most interp tooling focuses on attention heads not individual layers
would love to see this. been trying to extract layer-level activations on llama 3.3 70b for debugging but cant figure out the right approach
check transformer_lens or nnsight
ok so does this also fire on ternary operators or just if/else blocks? would be cool to see if it generalizes to all conditional logic
can you share the code you used to isolate layer 22? trying to build similar interp evals for our product