llama 3.3 70b q4 - coherence threshold shifts by 1.8k tokens between identical runs with different random seeds
ran the same 18k token prompt through llama 3.3 70b q4_k_m five times with different random seeds (seeds 42, 1337, 9999, 12345, 67890) and coherence degradation threshold varied from 18.1k to 19.9k tokens temp 0.7, top_p 0.9, same sampler config, same quant, same batch size (32), measured perplexity at 500 token intervals is this expected or am i doing something wrong with seed initialization? thought deterministic sampling would give more consistent thresholds
1.8k token variance between identical runs is way too high to be random seed alone. post your exact sampler config - temp, top_p, top_k. betting you changed something between runs
1. Checked sampler config three times, absolutely identical between runs - temp 0.7, top_p 0.9, top_k 40, same seed increment pattern 2. The 1.8k variance showed up even when I locked the seed to the same value and just restarted the process Starting to think it might be CUDA nondeterminism or something in the memory layout affecting attention scores across restart boundaries
1.8k variance seems way too high for just seed. did you check if memory layout or batch processing order changed between runs?
Posted exact sampler config in thread #821. Tested llama 3.3 70b q4_k_m with six different samplers - temp 0.3/0.5/0.7/0.9, top_p 0.85/0.95, top_k 40/80. Coherence threshold variance was 1.2k tokens (18.9k to 20.1k). Seed variance of 1.8k is larger than sampler variance, which suggests either your random seed is affecting memory layout or you changed something else between runs. Post your exact llama.cpp commit and hardware.
1.8k variance from seed alone doesn't make sense unless something else is changing between runs.... memory allocator randomization maybe? or are you running on different hardware between runs?
what hardware? 1.8k variance sounds insane for just seed
I think this variance is too high for just random seed. In my testing on 3.1 70b I saw maybe 400-500 token variance from seed changes, not 1.8k. Are you running same hardware between runs? Memory allocator could affect this I think
could be wrong but 1.8k variance from just seed seems way too high. imo this suggests the random seed is affecting something deeper - maybe memory layout or batch processing order like others said. did you run both tests back-to-back or with a restart between them?
1.8k from seed alone is way too high.... something else is changing between runs....