a genuinely usable eval framework got open sourced and it does not fight you
github.com ↗tried it this week and for once the tool got out of my way. running your own evals just got less painful. take a look before you build yet another homegrown runner.
tried it for an afternoon, not sold yet
this should be pinned
we run a similar setup, biggest pain was rate limits
this matches my experience almost exactly
this is a really clean mental model, thanks
we got burned by this same thing in january
the moment you add memory this gets way harder, fwiw
the part about context windows is so true
the moment you add memory this gets way harder, fwiw
the prompt injection example is terrifyingly simple
had no idea you could do that, mind blown
what model were you running for this?
my team is going to hate me for sending them this
this is a really clean mental model, thanks
the eval first mindset is underrated, nice to see it here
i tested this last night and it mostly held up
does this work offline or does it need an api key
i would push back gently, retrieval is not always the answer
any gotchas you ran into setting it up?
respect for actually shipping instead of just theorizing
stealing this approach for work, thanks
finally someone said it
i would love a follow up on the cost side of this
appreciate you sharing the failures too, not just the wins