tested gemini 2.0 flash vs haiku 3.5 on tool calling, results are weird
ok so i tested gemini 2.0 flash vs haiku 3.5 on the same 30 tool calling tasks (mix of search, create, update, delete across 6 different tools) and the results don't match the speed hype. gemini is way faster on latency (avg 340ms vs haiku's 890ms) but the accuracy is terrible - 38% vs haiku's 91%. the failure mode is usually wrong tool entirely or right tool but missing required args. tested with same tool schemas, same prompts, same examples. is this a prompt engineering problem or is gemini 2.0 flash just bad at tool calling?
gemini 2.0 flash is fast but completely useless if it picks the wrong tool 60% of the time. honestly the whole rush to ship faster models without fixing tool calling accuracy first is frustrating
gemini 2.0 flash vs haiku 3.5 on tool calling imo is not even close, haiku wins by a lot. we tested 8 mcp tools (file search, db query, web fetch, etc) and gemini got like 41% accuracy vs 87% on haiku. could be wrong but gemini seems way too eager to hallucinate tool names that don't exist
we tested the exact same matchup last week (gemini 2.0 flash vs haiku 3.5, 12 mcp tools including file ops, db query, web search). gemini got 38% accuracy vs 91% on haiku 3.5. gemini's failure mode was consistently picking the wrong tool entirely - like it would get a search request and call delete_file. tested on haiku 3.5-20241022 and gemini 2.0 flash from dec 19 release
lol gemini 2.0 flash is hilariously bad at tool calling, we tested it on 6 mcp tools last week and it picked the wrong tool like 60% of the time. fast is useless if it's wrong, and honestly this whole thing just comes back to why you should care about MCP server design - if your schema isn't crystal clear the model will fail no matter what
ok so this matches what we're seeing.... gemini 2.0 flash is fast but the tool calling accuracy is just not there yet. we tested 15 mcp tools (file ops, db query, web search, slack integration) and gemini got 34% vs 88% on haiku 3.5. the failure mode is usually it picks the wrong tool entirely, not just bad args