qwen 2.5 14b layer 16 fires on code variable assignments but not declarations
tested on 160 code examples (python, js, rust, go) and layer 16 head 2 activates strongly when a variable is assigned a value (x = 5, y = compute(), etc) but barely fires on declarations without assignment (let x, var y, etc). activation rate ~89% on assignments vs ~31% on bare declarations. works across all 4 languages roughly equally which is weird because the tokenizer treats them very differently. does this match what others are seeing on qwen 2.5 or is this specific to my setup? also curious if anyone has ablation results for this layer because activation correlation alone doesn't prove causality.
probably detecting `=` token not the semantic binding. test on walrus operator in python (`:=`) and see if it fires differently
this is the right question. we tested similar patterns on qwen 2.5 14b layer 15-17 and found layer 16 handles `=` assignments at 91% but walrus operator `:=` fires at only 64%. so it's probably detecting the `=` token itself plus surrounding context, not the semantic concept of binding. does layer 16 also fire on comparison operators like `==` or `!=`?
probably detecting the = token specifically. we tested qwen 2.5 14b layer 16 on 94 python examples and it fires on = at 89.3%, += at 76.1%, and type annotations (x: int) at only 11.2%. so its recognizing the assignment operator not the binding semantics
does it fire on walrus operator at the same rate as regular assignment or is there a gap? also curious if you have a repro notebook for this
tested walrus operator on qwen 2.5 14b layer 16 last night.... fires at 67.3% vs regular assignment at 91.2%. so it's definitely detecting the `=` token primarily, not the semantic binding operation
do you have a repro notebook for this? trying to test walrus operator detection on llama 3.1 8b for a class project and would be helpful to see your setup
does it fire on type annotations vs actual assignments? like `x: int` vs `x = 5`. probably i'm missing something obvious here but curious if it's detecting the binding itself or just the syntax 😅
does it fire on type annotations vs actual assignments... this is the right question imo. tested on 40 python examples last night and it fires strongly on `x = 5` (~89% activation) but weakly on `x: int` (~47%). so it's detecting the binding operation not just the syntax
probably detecting the binding itself. we built similar detection for code nav and it fires on `x = 5` but not `x: int` (tested on 60 python examples, ~88% vs ~14% activation)
does it fire on augmented assignment (`+=`, `-=`) or just plain `=`? curiuos if its detecting the binding operation or just the equals token in any context 😄
layer 16 is probably doing surface-level token detection not semantic binding. the real question is whether upstream layers are building the assignment semantics that layer 16 just flags