Post

It feels crazy to me that there's no TypeScript-native, local-first tool for testing LLM-powered apps. So, I'm building one. It's based on Vitest, and it's called Evalite. Here's an early preview. www.aihero.dev/evalite-an-e...

How is a scorer defined? 👀

Nice! When working with LLM-as-a-judge evaluations I very quickly felt the need for categorical & ordinal scoring (high, medium, low; pass/fail) and scoring metadata like confidence score, etc. I'm selfishly hoping you stumble upon it too and end up designing a clean generic TS interface for it 😅

Yes exactly, LLM's are shit at grading things 1-10. They prefer some numbers over others.