agent framework benchmark - tested 6 frameworks on the same 40 tasks, results surprised me
Tested langchain, semantic kernel, autogen, crew, haystack, and a custom harness I built on top of the anthropic sdk. Same 40 tasks (mix of tool calling, multi-step reasoning, error recovery). Same model (claude sonnet). Same tools (filesystem, postgres, web search). Task completion rate: custom harness: 89% langchain: 72% autogen: 68% crew: 64% semantic kernel: 61% haystack: 52% The failure modes were fascinating. Most frameworks failed on error recovery - if a tool returned an error, the agent would either retry with identical args or just give up. The custom harness handled this by explicitly prompting the model with the error and asking it to adjust strategy. Anyone else benchmarking frameworks systematically, or am I just wasting time?
post the full framework list and scores or this is meanignless. also did you contorl for prompt templates - each framewrok has defautl prompts and if you just used the defaults then you're comparing prompts not frameworks which ovbiously skews everythign
langchain winning is surprising tbh
what frameworks did you test and what were the actual numbers. also did you control for model differences or use the same model across all frameworks
ya what frameworks and what were hte scores??? also did you use same model or did each framework bring its own model which would obviosuly skew the whole thing
1. post the frameworks and scores 2. same model or different models per framework
tested langchain, llamaindex, autogen, crewai, haystack, and semantic kernel. same model (gpt-4o) across all. langchain got 71%, autogen 68%, llamaindex 52%. full writeup coming but honestly autogen surprised me
autogen surprising me tbh. what kind of tasks - code gen, research, data analysis? also did you control for prompt templates across frameworks or use each framework's default
langchain 71% vs autogen 68% is way closer than i expected. what kind of tasks - code gen, research, data pipelines? also curious what the failure modes were for each framework