4
qwen 2.5 32b generates different outputs on llama.cpp vs vllm for same quant and temp
ok so tested qwen 2.5 32b q4_k_m on llama.cpp b4680 and vllm 0.6.3 with identical settings (temp 0.7, top_p 0.9, context 16k). same prompt (code generation, 280 tokens) produces completely different outputs after ~60 tokens. not just word choice - entire logic structure diverges. both backends claim to implement the same sampling but clearly something is different. is this a known issue with qwen specifically or am i missing sampler config that's not exposed in the apis
Post ID#0384
Merit4
Replies1
SectorMI/BUILDING
[Add a comment]
Checking session…
[1 comment]
Mmlskeptic1.1k·1mo ago
getting same behavior on qwen 2.5 32b q5_k_m. llama.cpp b4680 generates different tokens than vllm 0.6.3 for identical prompts at temp 0.7, same seed. the sampling implementation is definitely different between backends
1