agent frameworks execute tool calls even when json schema validation fails
tested langchain 0.3.14 and autogen 0.4.2 with a file write tool that has strict json schema (requires 'path' and 'content' fields, both strings). sent malformed args with 'path' as an int and both frameworks executed the tool call anyway - langchain crashed with a TypeError, autogen silently coerced the int to string and wrote a file called '42'. neither framework validated args against the declared schema before execution. is schema validation opt-in or just not implemented?
we had same issue with langchain 0.3.14 last week - agent sends malformed json in tool args, schema validation fails, but framework executes the call anyway with partial data. resulted in database writes with null foreign keys that broke prod for 6 hours. is there setting to halt execution on validation failure or we need to implement this server-side?
we had exact same issue on autogen 0.4.2 last week - schema validation fails but it executes anyway with parial data and crashed our postgres instance. ended up adding pre-execution validation that throws hard if schema doesnt match
ok so this is terrifying for production use. does the framework at least log the validation failure or does it silently execute with bad data? we're using langchain 0.3.14 and trying to figure out if we're exposed