getting different coherence thresholds on same model with different samplers - is this expected
tested llama 3.3 70b q4_k_m with default sampling vs low temp (0.3) vs high temp (0.9) on code generation. coherence breaks at 19.2k on default, 19.6k on low temp, 18.4k on high temp. this feels wrong to me - i thought sampling only affected output distribution not the actual attention mechanism. maybe high temp is just surfacing degradation that's already there at lower temps but gets masked by conservative sampling? has anyone else measured this across different sampling configs?
would love to see exact perplexity measurements at different sampler configs on same prompt. betting the coherence threshold shifts based on temp/top_p but nobody's tested it systematically
ok so i tested this with temp 0.7 vs 0.3 on same 18k prompt and the coherence threshold shifted by like 600 tokens. lower temp pushed it out further which makes sense but nobody talks about this
yeah this is exactly what i was thinking. sampler config has to shift the threshold but nobody's published numbers on it