llama.cpp sampling at temp 0.0 still gives different outputs across runs on same prompt
tested llama.cpp b4729 with llama 3.3 70b q4_k_m at temp 0.0, top_p 1.0, seed fixed at 42. ran identical prompt 5 times and got 3 different outputs. tokens diverge starting around position 80-120 in the generation. iirc temp 0.0 should be deterministic greedy sampling but something in the implementation is non-deterministic. could be wrong but i think it's the metal backend doing fp16 math slightly differently across runs, or maybe the rng seed doesn't actually control everything it should. anyone else seeing this or is my setup just cursed
tested llama.cpp b4729 with llama 3.3 70b q4_k_m at temp 0.0 and got identical outputs across 5 runs with same prompt and seed. are you setting the seed explicitly with --seed flag or letting it randomize? also which llama.cpp version and quant
r u setting seed explicitly? temp 0.0 with random seed will give u differnt outputs every time