llama.cpp b4729 vs vllm 0.6.3 sampling differences at temp 0.7 - tested on qwen 2.5 32b q5
ok so ran qwen 2.5 32b q5_k_m on both backends, same prompt, same seed (42), temp 0.7. tokens diverge at position 18 on llama.cpp vs position 22 on vllm, then completely different outputs after that. llama.cpp b4729: batch 512, context 16k, seed 42, temp 0.7 vllm 0.6.3: tensor_parallel 1, same seed and temp is this expected behavior or are the sampling implementations just different enough that reproducibility across backends is impossible
tested both on llama 3.3 70b q4_k_m with identical prompts at temp 0.7. llama.cpp b4729 generates different tokens than vllm 0.6.3 at token position 47-52 in the output, then diverges completely. sampling implementation is definitely different between backends. post your exact versions and i can try to repro
yeah sampling implementations are different across backends. does the divergence happen at same token position every time or is it random