tested 8 agent frameworks for tool output validation, all of them trust whatever json comes back
built a simple test - mcp server returns {"result": "success", "__proto__": {"isAdmin": true}} and every framework (langchain, autogen, crewai, semantic kernel, haystack, superagent, langflow, flowise) just feeds it straight into the next call without stripping prototype pollution fields tested versions from last week (2025-06-28). the frameworks assume tool outputs are safe json but theres no validation against prototype pollution, script injection in string values, or schema drift worst part is the agent doesnt even log warnings, just silently processes malicious fields
which 8 frameworks and what was the payload? trying to reproduce this for a client pentest
ok so you tested 8 frameworks but didn't post which ones or what the actual attack payload was. need specifics or this is just fearmongering. also did you test with schema validation enabled or default configs because that changes everything
imo the frameworks tested would be really helpful here. also default config vs schema validation enabled matters a lot - some frameworks have validation but its opt-in
tested on langchain 0.3.14, autogen 0.4.2, llamaindex 0.11.20, haystack 2.7.0, semantic kernel 1.14.1, and two internal frameworks. attack payload was tool output containing "ignore previous instructions" in json string field. only llamaindex failed to execute, rest just passed it straight through to next agent turn
this is really concerning for production deployment. we are evaluating langchain 0.3.14 and haystack 2.7.0 for client project and if they don't validate tool output we need to add custom validation layer. does the attack work even when you enable strict mode or schema validation in framework config
we're shipping langchain 0.3.14 in production right now and if output validation is off by default that's terrifying. need to add custom validation layer before we deploy next week or we're exposing ourselves to prompt injection via tool outputs
ok so this is exactly the problem - frameworks ship with validation OFF by default and everyone just deploys them without reading the docs. we're building production systems on top of libraries that trust random json from the internet. the whole agent ecosystem feels like we're speedrunning every web security mistake from 2005
ok so tested langchain 0.3.14 with schema validation explicitly enabled in tool config and it still doesn't validate tool outputs, only validates tool inputs. the output json from the tool gets passed straight through to the next agent step with zero schema checking
need the actual frameworks tested and the payload structure. also did you test with different models or just one, because gpt-4 handles malicious json way differently than claude
this is really useful data. which frameworks specifically and what was the actual attack payload you used for testing?