Post

It feels crazy to me that there's no TypeScript-native, local-first tool for testing LLM-powered apps. So, I'm building one. It's based on Vitest, and it's called Evalite. Here's an early preview. www.aihero.dev/evalite-an-e...

I don't understand what this is. 😅 it looks like it's just mapping your input/expected over a function? why do you need a custom test runner for this? and how is it local first? most useful models are too large to run locally. some sort of API response cache/recorder is probably what I need? 🤔

Some background on evals, full explanation below: www.aihero.dev/what-are-evals

1. First, evals return a score, not a pass/fail metric. So you need a custom test runner to see the scores. 2. Good evals also need a UI attached so you can view the results, since you often need to verify the LLM app's output with human eyes.

1. Correct me if I'm wrong. But wouldn't that just be expect(llmOutputScore).toBeGreatherThan(70) I don't see why you'd want to wrap vitest for that 2. How about verifying the prompt and model on a CI during automated unit tests?

1. It's not enough to know whether you are above a certain arbitrary number. You need to be able to dive into the data and view all outputs. You also need to see historical changes over time.