Evil Martians reposted
Evil Martians' black-box checks are thoughtful benchmark design: they grade what an unfamiliar Rails app does without assuming how its code is structured.
Agents on Rails is the Rails Foundation's LLM benchmark. We designed its eval tasks and the tooling to run at scale. Read how we wrote black-box checks for apps with code we'd never seen, keeping trials fair and results reliable. Bonus: the same evals against Rust. evilmartians.com/chronicles/t...