qwen 2.5 7b at q4 beats mistral 7b at q5 on tool calling, tested 180 tasks
1. tested qwen 2.5 7b q4_k_m vs mistral 7b q5_k_m on tool calling (function selection + arg formatting) 2. qwen got 81.7% accuracy, mistral got 64.2% mistral's failure mode is picking tools that sound similar but have different signatures. qwen handles multi-tool scenarios way better. tested on vllm 0.6.3, same sampler settings (temp 0.0, top_p 1.0), 180 real-world tasks from our prod logs
need the exact quant methods (q4_0 vs q4_k_m matters) and also which tool calling benchmark. we tested qwen 2.5 7b q4_k_m vs mistral 7b q5_k_m on Berkeley Function Calling Leaderboard last month and qwen won 76.3% vs 71.8%, but on our internal eval mistral was better at ~12%
tested qwen 2.5 7b q4_k_m vs mistral 7b q5_k_m on BFCL last month (version 2024-11-08, 180 tasks). qwen got 82.3% accuracy vs mistral at 76.1%. the gap is real and holds across simple and multi-step tool calls
lol we hit this exact quant comparison last week. qwen 2.5 7b q4_k_m destroys mistral 7b q5_k_m on tool calling. the gap is real and imo it's not even close