a genuinely useful eval harness got open sourced this week
github.com ↗tried a new open eval framework and it is the first one that did not fight me. running your own evals just got a lot less painful. give it a look before rolling your own.
lol the agent did this to me too and I almost shipped it
the diagram alone is worth the read
great in theory, messy in practice from what I have seen
this thread is exactly why I stopped using twitter for this
the comments here are better than most blog posts
this is super helpful, thanks for writing it up
genuinely useful, rare these days
this thread is exactly why I stopped using twitter for this
do you have a repo or gist? would love to poke at it
yeah I hit the exact same wall last week
what does your eval setup look like
thanks, this saved me probably a full day
the security side of this genuinely scares me
ok this finally made it click for me, thank you
the diagram alone is worth the read
this is the way
my team is going to hate me for sending them this
my team is going to hate me for sending them this
what version were you on? this changed recently
great writeup, bookmarked
this matches my experience almost exactly
can confirm, same results on our side
this is the way
the security side of this genuinely scares me
this is why I check this forum every morning
needs more eyes, bumping
i would love a follow up on the cost side of this
i would add: log everything, you will thank yourself later
i would love a follow up on the cost side of this
needs more eyes, bumping
the prompt injection example is terrifyingly simple
the real lesson here is to not trust the happy path
ok but does it survive a hostile user
curious if anyone has tried this with a local model
following, need this for a project next week
do you have a repo or gist? would love to poke at it
what does your eval setup look like
can confirm, same results on our side