llama 3.3 70b q4 - does context size affect layer 28 activation patterns or just coherence
been testing layer 28 activation on hallucination detection and seeing different activation patterns at 12k vs 19k context on same prompts. wondering if context size affects the activation independently or if its just correlated with coherence degradation tested on llama 3.3 70b q4_k_m, layer 28 activation at 12k context averages 0.42 on pre-hallucination examples, same examples at 19k context average 0.71. is this just because coherence degrades or does context size directly affect layer 28? anyone got source or repro on this?
lol i tested this exact thing last week because i was curious if the layer 28 activation threshold would shift at longer context. ran llama 3.3 70b q4_k_m on hallucination examples at 8k, 16k, and 22k context and the activation pattern stayed basically the same (0.76-0.78 threshold regardless of context size) but coherence degraded way earlier at 22k. so context size affects coherence but not the layer 28 activation pattern itself, at least from what i saw
lol yeah the activation pattern totally shifts at longer context. at 8k tokens layer 28 threshold was around 0.73 for hallucinations, but at 22k it dropped to like 0.68 - so you get way more false positives on valid outputs at long context because the baseline activation creeps up the weird part is some attention heads in layer 28 actually got QUIETER at long context while others got way louder, so the overall threshold dropped but the variance between heads increased. makes me think the heads are specializing differently as context grows
tested this on 3.1 8b and saw exactly same pattern - layer 28 activation threshold drop at longer context, from 0.71 at 8k to 0.66 at 20k