layer 28 activation threshold testing - false positive rate on valid nested recursion is 23%
following up on the layer 28 hallucination detection threads - i spent the last three days testing false positive rates on edge cases that are weird but valid tested on llama 3.3 70b q4_k_m with 0.75 activation threshold (the number that keeps coming up in other threads). ran it against 200 examples of valid nested recursion (recursive fibonacci, tree traversal, nested json parsing) and 200 examples of confirmed hallucinations from our production logs results: - true positive rate: 87% (174/200 hallucinations correctly flagged) - false positive rate: 23% (46/200 valid recursion examples incorrectly flagged) - false negative rate: 13% (26/200 hallucinations missed) the false positive rate is way too high for production use. if you're generating code with any kind of recursive logic, layer 28 fires high even when the output is completely valid. this makes sense because recursion is inherently unusual/uncertain from the model's perspective - it's tracking state across stack frames which probably looks similar to the uncertainty pattern before hallucination i tested thresholds from 0.65 to 0.85 in 0.05 increments and the false positive rate never dropped below 18%. you can tune it to catch more hallucinations (lower threshold) but you pay for it with even more false positives on valid edge cases my conclusion is layer 28 activation is a real signal but it's detecting "unusual/uncertain output" not specifically "hallucination about to happen". for a lot of use cases those are correlated but not identical, and the gap matters for prod deployment this is making me think more about SAE decomposition of layer 28 - maybe there are specific features within that layer that fire on hallucination vs recursion and we're just looking at the blunt aggregate activation right now
ok so 23% false positive rate is way too high for production use. what happens if you adjust the threshold up to like 0.82 - does it drop the false positives without missing actual hallucinations? need the ROC curve here, not just one threshold. also what exactly counts as 'valid nested recursion' in your test set - are we talking fibonacci or actual production code patterns
we tried adjusting threshold up to 0.80 last week and it dropped false positives from 23% to around 8% but then we started missing actual hallucinations. ended up at 0.77 which gave us 12% false positive rate and caught most hallucinations but its still too high for production use imo
ok so what was the actual production impact tho. like did the false positives block valid outputs or just flag them for review
we saw this exact issue in production last month and the false positives were blocking valid outputs, not just flagging them. the problem is that layer 28 fires high on any deeply nested structure, not just hallucinations - so recursive function calls, nested json with more than 5 levels, lambda calculus reductions, all trigger false positives at the 0.75 threshold. we ended up building a secondary classifier that checks if the high activation corresponds to actual semantic drift (measured via cosine similarity to previous layer outputs) vs just structural complexity. dropped false positives from 23% to 4% without missing hallucinations, but it adds ~40ms latency per inference.