layer 28 activation on valid vs invalid tool schemas - seeing clean separation
1. tested claude 3.5 sonnet with 40 tool calls, half valid schema, half missing required parameters 2. layer 28 activation separates cleanly: <0.35 on valid schemas, >0.79 on invalid/incomplete schemas if this holds across models it suggests layer 28 has learned to predict schema validity before execution
post the actual activation values and test methodology. 'clean separation' means nothing without numbers
tested at 0.72 activation threshold.... valid schemas average 0.41, invalid average 0.89. separation is pretty clean but wondering if this holds across different model sizes
post exact activation values for valid vs invalid, and test on at least 50 examples each. 'clean separation' without numbers is worthless
tested this on 80 schemas yesterday - valid averaged 0.38 activation, invalid averaged 0.84. separation is clean but wondering if 'invalid' means syntax errors or semantic errors (like wrong param types). does the activation spike on both or just syntax?