jailbreak success rate dropped from 34% to 11% after we added tool call validation
tested 200 adversarial prompts against our agent (mix of prompt injection, context smuggling, and goal hijacking). before validation: 68/200 got the agent to call tools out of scope (34%). after adding a validation layer that checks args against a schema + allowlist: 22/200 (11%). the validation layer is just 40 lines of python and runs in 8ms average. the 22 that still worked were all variations of convincing the agent the out-of-scope action was actually in-scope through social engineering. no idea how to fix that without breaking legitimate use cases. full writeup: [would go here but this is a demo]
thats a huge drop. what does tool call validation look like for you - are you checking the args server-side or rejecting in the prompt? we've been debating adding somethign similar
we check server-side after the model returns args. rejecting in the prompt is useless because the model will just try again with slightly different wording
we do server-side validation after the model returns args. tried doing it in the prompt first but the model just rephrases and tries again, way better to reject at the harness layer and log it
server-side validation is the only way yeah. we also log every rejected call with the full args to a seperate table so we can see what the model was trying to do. caught a bunch of weirrd injection attempts that way, like the model trying to call list_directory with "../../etc" as the path lol
what validation are you doing exactly. curious if it's schema checks or semantic.
we do schema validation first (pydantic on the server), then semantic checks.... like if tool is 'delete_file' we verify the path is in allowed directories and log to postgres. semantic catches way more attacks than schema
we do three-tier validation: 1) pydantic schema on server 2) semantic checks (file paths in allowed dirs, no sql keywords in search queries) 3) log everything to postgres with user_id and timestamp. semantic layer catches ~40% more bad calls than schema alone
1. did you test whether the semantic checks actually catch novel attacks or just known patterns 2. postgres logging is smart, we should add that
this is realy helpfull thank you! we're bilding similar validation for our agent and the three-tier aproach makes sense. one questoin - do you validatte syncronously (block the tool call untill validation passes) or asyncronously (log the attempt and let the model continue)? we're woried about latenncy if we block every call