gemini 2.0 flash is fast and wrong, haiku 3.5 is slow and right - why is this hard
tested gemini 2.0 flash vs haiku 3.5 on 18 mcp tools (file ops, postgres, web search, slack, github, notion). gemini got 42% tool calling accuracy, haiku got 89%. gemini's latency was amazing (~340ms p50) but it picked the wrong tool constantly. the failure mode is interesting - gemini tends to pick tools with similar names (file_read vs file_search) or similar descriptions, like it's doing fuzzy matching on the tool description instead of actually reasoning about what the task needs. haiku is slower (~890ms p50) but it just works. picks the right tool, passes the right args, handles errors correctly. why are we optimizing for speed when accuracy is still this broken? fast is completely useless if it's wrong. honestly the whole benchmarking culture around model releases is frustrating - everyone publishes latency numbers but tool calling accuracy benchmarks are basically non-existent
ok so gemini 2.0 flash is fast but useles if its wrong 60% of time. we tested on tool calling last week and it picked wrong tool constantly
yeah same. tested on 8 tool calling tasks and gemini picked wrong tool 63% of time. fast is useless if its wrong
tested gemini 2.0 flash (version 2.0-flash-001) vs haiku 3.5 on 23 tool calling tasks last week. gemini got 44.2% accuracy, haiku got 88.7%. gemini's failure mode is picking tools that sound semantically similar but have totally different schemas - like it'll pick 'search_files' when it should pick 'search_database' because both have 'search' in the name. haiku reads the full tool description and schema before deciding.
ok so this matches what we're seeing - gemini picks tools based on name similarity not actual function. switching to haiku cut our error rate but doubled latency. honestly thinking about just caching haiku responses
same here, gemini is fast but picks wrong tool all the time.... we tested gemini 2.0 flash vs haiku 3.5 on 15 tool tasks and gemini got 41% vs haiku 86%. switched to haiku in prod even tho its slower
gemini 2.0 flash also hallucinates tool names that don't exist in the registry. tested on 31 tool tasks and it invented fake tools 19% of time vs haiku at 0%. fast and wrong is worse than slow
lol gemini inventing tool names is the worst failure mode. we had it hallucinate a tool called search_with_citations that doesn't exist in our registry and then get mad when we returned an error 😅