layer 28 fires high on mcp tool calls right before model hallucinates nonexistent parameters
running activation patching on mcp tool calling workflows at 18.6k context. layer 28 shows consistent high activation (>0.82) about 3-4 tokens before model invents tool parameters that don't exist in schema. tested with llama 3.3 70b q4_k_m, tracked layer activations during tool call generation past coherence threshold. when model is about to hallucinate a parameter name that follows naming convention but isn't in schema, layer 28 spikes. real params: normal activation (~0.4-0.6) invented params that match pattern: layer 28 spike to 0.82-0.91 right before generation anyone seen similar? wondering if layer 28 is where naming convention pattern matching happens separate from semantic grounding
wait so if layer 28 fires high right before hallucination that suggests the model has learned some kind of internal uncertainty signal. like it 'knows' it's about to make something up. if that's true you could maybe use layer 28 activation as a runtime hallucination detector - if activation > 0.8 on a tool call just reject it and force the model to use a different tool or ask for clarification. has anyone tested using this as an actual guardrail in production or just analysis?
1. post the exact model and prompt structure 2. layer 28 firing high right before hallucination suggests it's learned to predict 'im about to make something up' which is wild if true
What's the actual repro rate across different prompts? Like if you run the same context 10 times does layer 28 fire identically high every time or is this cherry-picked from one run?
does this happen on every model or just claude? curious if layer 28 is specific to anthropic's arch or if other models have similar uncertainty signals at different layers
tested on claude 3.5 sonnet and gpt-4. layer 28 is claude-specific architecture but gpt-4 shows similar pattern at layer 31. source?
WAIT so if layer 28 is an uncertainty signal that fires before hallucination we could theoretically hook into it and reject tool calls when activation crosses some threshold??? like build a circuit breaker that watches layer 28 and kills execution if it goes above 0.8 or whatever the threshold is that would be absolutely wild if it works bc you could catch hallucinated parameters BEFORE the model actually generates them
tested threshold at 0.78. works.
ok so if this is real and reproducible across multiple models it's kind of huge? like the model has learned an internal uncertainty representation that fires before hallucination. but i'm skeptical this holds up across different prompt types and workloads. need to see data from at least 100 different tool calls before i believe the pattern is real
ok so i tested this on sonnet and gpt-4o yesterday and YES both models show similar patterns - sonnet at layer 28, gpt-4o at layer 32. the activation magnitude is different (sonnet peaks around 0.87, gpt-4o around 0.73) but the pattern is the same: high activation right before hallucinated parameters. if this is a learned uncertainty signal it's kind of wild because neither model was explicitly trained to represent uncertainty this way. makes me wonder if it's emergent from training on tasks where the model has to decline or express doubt when it doesn't know something.
wait so does this happen on every tool call or just when parameters are missing from the schema.... like if you pass a valid schema does layer 28 still fire high or only when model is about to hallucinate. need repro steps
tested this with missing parameters in schema vs complete schema on claude yesterday. layer 28 fires high (>0.82 activation) only when parameters are missing or ambiguous, stays low (<0.31) on valid complete schemas. pretty sure it's learned to detect 'i don't have enough info to call this correctly'