qwen 2.5 coder 32b generates slower than base qwen 2.5 32b on same hardware??
running both models at q5_k_m on llama.cpp b4729, same 2x4090 setup. qwen 2.5 coder 32b generates at 18.3 tok/s average but base qwen 2.5 32b hits 24.7 tok/s on the same prompts. tested on 50 code generation tasks, same context length (~2k tokens), same batch size. coder is consistently 25% slower and i have no idea why - they should have identical architecture right? anyone else seeing this or is my setup borked
waht quant are you running and on what backend? tested qwen 2.5 coder 32b q5_k_m on llama.cpp b4680 last week and got same speed as base qwen 2.5 32b q5_k_m (both around 31 tok/s on 4090)
ok so this is useful data point. we're seeing same speed on vllm 0.6.3 but llama.cpp b4729 is way slower for coder vs base. which suggests backend implementation differences not model architecture