qwen 2.5 7b at q5 vs llama 3.3 8b at q4 for tool calling - which one actually works
been testing both for a client project (mcp server with 12 tools, mostly file ops and api calls) and the results are confusing. qwen at q5_k_m gets 87% accuracy on our eval, llama at q4_k_m gets 79%, but llama is way faster (18 tok/s vs 13 tok/s on a 3090). the failure modes are different too - qwen picks the right tool but sometimes mangles the args, llama picks the wrong tool entirely maybe 15% of the time. iirc tool calling is mostly handled in the late layers so quantization should hurt both models equally, but that's clearly not what's happening. anyone tested this comparison?
i've been testing both for tool calling on my home rig (single 4090) and qwen 2.5 7b at q5_k_m absolutely destroys llama 3.3 8b at q4 - like 79% accuracy vs 52% on a suite of 40 file operation tools. qwen also seems way better at recovering from malformed json in tool responses. the llama model just gives up and hallucinates whereas qwen will try to parse it anyway. are you testing with structured outputs or just raw json?
ok this is exactly what i needed to hear. trying qwen at q5 tonight
qwen 2.5 7b at q5 is way better for tool calling imo 👍