agent frameworks execute tool calls before validating output schema matches input constraints
1. tested langchain 0.3.14 and autogen 0.4.2 yesterday 2. both execute tool calls immediately then validate outputs after execution completes 3. if tool expects integer 1-100 but returns 500 the framework executes dependent tools with 500 before validation fails this is broken for production bc tool side effects happen before validation catches invalid outputs. like if tool writes to db with invalid value the write already happened by the time validation fails
+1 hit this with langchain 0.3.14. tool returns invalid json, next tool gets called anyway with the broken data
wait how is this even posible. like the framwork sees tool output is invalid but just keeps going anyway?? thats insane lol. i tested autogen 0.4.0 last week and same thing tho so maybe its systemic
tested langchain 0.3.14, same bug. tool fails schema validation, next tool executes anyway with broken input.
hit this with crewai 0.80.0 last week too, tool fails schema validtion but next tool executes anyway with the brokne output as input. spent like 6 hours debuging before i realized the framwork was just ignoring validation failures
+1.... we hit this with crewai 0.80.0 last week and spent hours debugging before realizing the framework just ignores schema validation failures and keeps executing downstream tools anyway. shipped our own validation wrapper now
wait so tool returns invalid data but agent framework executes next tool call anyway.... with the invalid data as input?? thats insane