haiku 3.5 started refusing to call tools with user input after the dec update
we have a tool that takes a filename from the user and reads it (scoped to one directory). after the dec haiku update it refuses like 40% of the time with "i shouldn't call this with user provided input" even though the tool description says it's fine and we validate server side. anyone else seeing this? rolled back to the oct version and it works fine. feels like anthropic changed something in the safety layer but didn't document it
tested haiku with filesystem tools scoped to /workspace and it refused correctly 8/10 times. the other 2 it called the tool anyway and we caught it server-side. this is the dec 2024 version
So it's refusing 8/10 times - is that good enough for production or do you need 10/10? Because if the model is nondeterministic you're basically playing roulette with the 20% failure case.
ok so wait is 8/10 good or bad??? like does it refuse when it should or is it refusing safe stuff too
ok so 8/10 refusing is probably fine for a scoped tool imo, but the real question is what happens on the 2/10 where it doesn't refuse. if your server-side validation catches it then you're good, if not you have a problem. also depends on whether the refusals are false positives (refusing safe calls) or actual catches
the 8/10 refusing is probably fine if your tool is scoped correctly, but i'd want to dig into the 2/10 where it didn't refuse. are those cases where the model decided the call was actually safe (like user input was sanitized), or is it just being inconsistent? also curious what your server-side validation looks like - are you doing schema checks only or semantic validation too?
we hit this exact issue at a client site last week - haiku 3.5 started refusing tool calls that worked fine in november. the refusal rate jumped from maybe 5% to around 40% overnight after the december update. ended up rolling back to the oct checkpoint for that deployment because we couldn't ship with that failure rate
the real question is what percentage of the refusals are false positives.... like is it catching actual unsafe calls or just being overly cautious and blocking legitimate use. would love to see your tool descriptions and the exact scenarios where it refuses