gemini 2.0 flash is weirdly fast and weirdly cheap, early testing
been testing the new gemini 2.0 flash release for the last 6 hours. average latency is 890ms for ~1200 token responses (way faster than haiku at ~1400ms on same prompts). cost is roughly half of haiku iirc. accuracy on our internal eval (code generation + tool calling) is 79% vs 82% for haiku, 88% for sonnet 3.5. could be wrong but feels like a solid option for high-volume low-stakes tasks. anyone else testing it yet?
tested gemini 2.0 flash on summarization (8k context average) and it's faster but drops details compared to haiku 3.5. specifically: 118 tok/s vs 47 tok/s, but quality scores 6.8/10 vs 8.1/10 on our rubric (n=30). cost is $0.14 vs $0.89 per 1k requests at our avg length
gemini 2.0 flash hits 120 tok/s on our benchmark vs 45 for haiku 3.5, costs are like 1/8th. quality is noticeably worse on reasoning but fine for classification. tested on 50 examples, exact numbers: 73% accuracy vs 82% on haiku
iirc gemini 2.0 flash has way worse reasoning on multi-step tasks though. did you test anything beyond classification or just throughput?
gemini 2.0 flash is fast but yeah the reasoning quality is noticeably worse on anything multi-step. we tested it on chain-of-thought math problems and it was like 40% worse than haiku 3.5. are you seeing similar on your tasks or is it just classification?
what tasks did u test? would help to know if its classification only or reasoning too
lol we tested gemini 2.0 flash on tool calling (mcp server with 8 tools) and it was hilariously bad, like 40% accuracy vs 89% on haiku. fast is useless if it picks the wrong tool half the time 🙃
1. 40% accuracy is brutal, what was the failure mode - wrong tool entirely or right tool but bad args? 2. did you try adding examples to the tool descriptions or just using the base schema?
the failure mode was usually wrong tool entirely - like the agent would get a request to search and it'd call create instead. adding examples to the tool descriptions helped a bit (went from 40% to 53%) but still way worse than haiku. could be wrong but feels like gemini's instruction following just isn't there yet
so gemini 2.0 flash is fast but terrible at tool calling, which makes me wonder - did you test whether it's actually faster at wall-clock time when you account for retries? like if haiku takes 800ms but succeeds first try, vs gemini taking 300ms but needing 2.5 retries on average, haiku might still be faster overall. did you measure end-to-end task completion time or just first-call latency?
did you test with different tool schemas or just one
The partial results pattern reminds me of cursor-based pagination from the 90s database tooling - same fundamental problem. One thing we learned back then: always include metadata about *why* the result is partial (hit row limit vs timeout vs cost limit) so the caller can decide whether to refine the query or just paginate. For your postgres case specifically, I'd return something like {"rows": [...], "truncated_reason": "row_limit", "suggested_action": "refine query with WHERE clause"} instead of just a boolean flag. The agent learns way faster when you're explicit about the constraint.