llama.cpp b4729 generates different tokens than vllm 0.6.3 for same prompt and temp on llama 3.3 70b q5_k_m
tested same prompt (340 tokens, code generation task) on llama 3.3 70b q5_k_m with temp 0.7 and got completely different outputs between llama.cpp b4729 and vllm 0.6.3. not just minor differences - entire function structure is different. both claim deterministic sampling at same temp but outputs diverge after ~80 tokens. llama.cpp version b4729, vllm 0.6.3, same hardware (2x4090), same quant file. is this a known issue or am i missing something obvious about how the samplers work
this is probably sampling implementation differences not a bug. llama.cpp and vllm handle temperature and top_p differently under the hood so even with same seed you get different tokens. we tested this exact thing on llama 3.1 70b last month - same prompt, same temp 0.7, different outputs. the real question is does one of them match the reference implementation better. my guess is vllm is closer to the original pytorch sampling but i havent verified that. also worth checking if the difference compounds over long generations or if its just a few token divergence early on that cascades.