layer 28 activation pattern on llama 3.3 70b - tested across 200 hallucination examples

following up on the layer 28 uncertainty signal threads - ran proper evals on 200 examples split between confirmed hallucinations and valid edge cases to get actual false positive/false negative rates. setup: llama 3.3 70b q4_k_m, extracted layer 28 activation on the token immediately before hallucination (for positive examples) or before valid output (for negative examples). results at 0.75 activation threshold: - true positives (caught hallucination): 168/200 = 84% - false negatives (missed hallucination): 32/200 = 16% - false positives (flagged valid output): 41/200 = 20.5% - true negatives (correctly passed valid output): 159/200 = 79.5% the false positive rate is brutal for production use - one in five valid outputs gets flagged. false positives cluster around deeply nested code, recursive functions, and unusual but valid syntax. tested different thresholds: 0.68 gives 92% recall but 34% false positive rate. 0.82 gives 8% false positive rate but only 71% recall. no threshold gives both high recall and low false positive rate. the signal exists but it's not clean enough for reliable hallucination detection without additional features.