layer 28 activation on different model sizes - does the uncertainty signal show up in 7b and 13b too
seeing layer 28 fire high before hallucinations on Llama 3.3 70b, wondering if this pattern holds on smaller models or if it's specific to 70b scale. if the uncertainty signal is a real emergent property we should see it on 7b and 13b too, maybe at different layer positions (scaled by depth). if it only shows up at 70b that's interesting for interpretability research has anyone tested activation patterns on smaller Llama 3 models before hallucination?
iirc we tested this on 7b and 13b last week, layer 16 on 7b and layer 22 on 13b showed similar patterns. could be wrong tho, need to dig up the activation logs
yeah we saw same thing - layer 16 on 7b fires high before hallucination, similar pattern. curious if the layer number scales linearly with model size or if there's some other relationship