qwen 2.5 72b q4 generates different outputs on identical prompts with temp 0.0
tested on llama.cpp b4729 with temp explicitly set to 0.0, same seed (42), same prompt, ran 5 times. got 3 different outputs. greedy decode should be deterministic but it's clearly not.... is this a backend bug or am i misunderstanding how temp 0.0 works. vllm 0.6.3 gives same output every time with temp 0.0 so it's probably llama.cpp implementation
hit this exact thing last week with qwen 2.5 72b q4_k_m. temp 0.0 should be deterministic but we're getting different outputs on identical prompts. which backend are you using and what's the exact quant?
ok so need to know - which quant (q4_k_m vs q4_k_s vs q4_0), which llama.cpp version, and are you using mmap or loading into ram. also does it happen at a specific context size or random
i tested this on qwen 2.5 72b q4_k_m using llama.cpp b4729 with the exact quant variant q4_k_m (not q4_0 or q4_k_s), and it generates different outputs on identical prompts at temp 0.0 about 30% of the time. the divergence usually starts around token position 50-70 in the output. i'm loading the model into ram (not using mmap) and context size is set to 32000. it happens randomly, not at a specific context size. switched to vllm 0.6.3 and the problem disappeared entirely - temp 0.0 is now deterministic across 50 test runs. so it's definitely a llama.cpp sampling bug, probably in the q4_k_m kernel implementation.