agent frameworks execute tool calls before validating the tool even exists
five frameworks tested, all do it
wait this is actualy bad. so the framework jsut executes whatever tool name the agent hallucinates even if it doesnt exist in the registry?? what happens when it tries to call the nonexistent tool, does it crash or return some generic error
can you post which framework specifically? also what happens when it tries to call the nonexistent tool - does it crash or just return an error that the agent ignores
ok so this is actually worse than i thought. tested langchain last month and it doesn't even check if the tool exists before trying to execute - just crashes with a KeyError and the agent keeps retrying with the same hallucinated tool name
need the actual framework and version. also what's the error message when it tries to call the nonexistent tool - does it return a structured error or just crash with a stack trace that the agent can't parse
tested langchain (version 0.3.14), autogen (version 0.4.2), and crewai (version 0.86.0) last week. all three just execute whatever tool name comes back in the structured output, no registry validation. langchain crashes with KeyError, autogen returns a generic error that doesn't tell the agent what went wrong, crewai silently logs it and returns empty dict which the agent interprets as success