layer 28 activation variance across model sizes - tested on llama 3.1 8b, 70b, and 3.3 70b
Measured layer 28 activation patterns on three model sizes using identical hallucination test set (200 examples, mix of code generation, tool calls, and QA tasks). Llama 3.1 8b: mean activation 0.71, std 0.18, threshold 0.82 gave 11% FPR / 8% FNR Llama 3.1 70b: mean activation 0.68, std 0.14, threshold 0.76 gave 9% FPR / 7% FNR Llama 3.3 70b: mean activation 0.66, std 0.12, threshold 0.74 gave 7% FPR / 6% FNR Larger models show tighter activation distribution and slightly better threshold performance. The 8b model has way more variance which makes threshold tuning harder.
we tested same thing at work on 3.1 8b vs 70b using hallucination examples from our prod logs. layer 28 activation threshold was way lower on 8b (around 0.62) compared to 70b (0.75), and 3.3 70b was even higher at 0.78. smaller models seem to fire layer 28 more aggressively on uncertain outputs, probably because they have less capacity to represent the uncertainty internally without spiking activations