haiku 3.5 is weirdly bad at function calling with >6 tools, reproduced on 3 tasks
been testing haiku 3.5 vs 3.0 on our internal tool-calling benchmark (customer support routing, 8 tools). 3.0 gets 91% correct tool selection, 3.5 drops to 73%. same prompts, same tools, same examples. reduced tool count to 4 and 3.5 jumps back to 89%. seems like there's a cliff somewhere around 6-7 tools where the model just starts guessing. anyone else seeing this or is our setup cursed? running via api, temp=0, standard tool schemas.
hit this exact thing last week. over 6 tools haiku starts guessing, under 6 it's solid. ended up grouping tools by domain
hit this exact bug.... over 6 tools haiku starts mixing up parameter schemas, under 6 it's totally fine. we ended up just splitting into two separate calls with different tool subsets
+1 hit this exact thing. over 6 tools haiku mixes up parameter schemas about 30% of the time in our tests (n=200 calls). under 6 tools it's like 3%. filed a bug but no response yet
hit this at 7 tools not 6.... over 7 tools haiku started mixing args between tools about 22% of the time. under 7 it's fine. filed a bug in november, still no fix
reproduced on sonnet 3.5. hits around 7 tools, not 6. degrades gracefully though - just starts ignoring the last 2-3 in the list rather than full failure