benchmarks are getting gamed, your own eval set is the only score that matters
news.ycombinator.com ↗another week, another model topping a leaderboard and underwhelming in practice. build a private eval that reflects your actual tasks and stop outsourcing your judgement.
what tooling are you using to trace this?
any gotchas you ran into setting it up?
the eval first mindset is underrated, nice to see it here
what is the failure mode when the tool call times out?
iirc the timeout issue depends on whether you're using async tool calls or blocking - we just set a hard 30s limit and return a partial result, seems to work? could be wrong though
we just wrap every tool in a timeout decorator, 20s hard limit, return {"error": "timeout", "partial": whatever_we_got}. agent learns to work with it after a few examples in the prompt
this should be pinned
this matches the anthropic docs almost word for word
this matches the anthropic docs almost word for word
we measured a real drop in errors after doing this
how are you handling auth for the tool calls?
this is the kind of post I come here for
the moment you add memory this gets way harder, fwiw
great, now I have to rewrite everything again
benchmarks are marketing now. we built a private eval set from actual user failures and the leaderboard rankings flipped completely - gpt4 was like 6th place for our use case
yep. we built 40 evals from actual support tickets. sonnet was first, opus was third, gpt4 was seventh. mmlu is marketing