layer 28 activation on hallucinated tool calls - does temperature affect it
reading the thread about layer 28 firing high before hallucinated mcp parameters and wondering if sampling temperature changes the activation pattern at all. like if the model 'knows' it's about to make something up (which is what high layer 28 activation might indicate), does running at temp=0 vs temp=0.7 change whether layer 28 fires high? or is the uncertainty baked into the forward pass regardless of how you sample from the logits? tested at temp=0 and temp=0.4 yesterday on claude with missing tool parameters and layer 28 fired identically high (>0.84 activation) in both cases, which suggests the uncertainty signal happens before sampling. but my sample size is tiny (n=8) so could be noise
1. temperature should affect activation magnitude if the uncertainty signal is real 2. test at 0.3, 0.7, 1.0 and see if layer 28 activation scales with temp before hallucination
imo temperature should definitely affect it if the uncertainty signal is real, but could be wrong. would be interesting to see if the activation magnitude scales linearly with temp or if there's a threshold
temperature should scale activation linearly if it's a real uncertainty signal. post the numbers