Post

I don't understand what this is. 😅 it looks like it's just mapping your input/expected over a function? why do you need a custom test runner for this? and how is it local first? most useful models are too large to run locally. some sort of API response cache/recorder is probably what I need? 🤔

aihero.devWhat Are Evals?AI Engineering for Curious Professional Developers

Some background on evals, full explanation below: www.aihero.dev/what-are-evals

1. First, evals return a score, not a pass/fail metric. So you need a custom test runner to see the scores. 2. Good evals also need a UI attached so you can view the results, since you often need to verify the LLM app's output with human eyes.

3/12/24, 9:08 am0 reposts1 like

3. I'm using local-first to compare against other services like Braintrust, which require an API key to do anything. Evalite will produce a local UI, no API key required.

Jess Martin

@jessmart.in

"local-first Braintrust" sounds like a winner Braintrust's evals + logging has been a game-changer for me. It feels like the right mental model without having to adopt someone's crazy framework. Good luck!

100%. Not sure I'll have the heft to tackle observability and logging but hoping to make something compelling on the evals side.

Jess Martin

@jessmart.in

the observability is so key I can see how my LLMs are actually performing against production inputs and then I can add a specific production input to a dataset with one click how might you match that workflow?

4. Good evals need traces, which let you view all sub-calls to LLM's inside your function and monitor approximate cost. This requires wrapping the test runner.

Rasmus Schultz

@mindplay.dk

interesting, is this how it's done in the real world? I always assumed you would just write regular tests and seed the LLM to make it stable. LLMs are notoriously bad at scoring anything, aren't they? if your tasks and evals are both probabilistic, aren't the tests liable to randomly fail? 🤔

Yep, this is how it's done. LLM's don't respond well to regular pass/fail metrics. If you're asking what the capital of France is, there are dozens of ways the LLM could respond which 'pass' the test. "Paris" "The capital of France is Paris" "The city of lights."

Rasmus Schultz

@mindplay.dk

still not sure what you mean by local first, are you running a small LLM locally for some tasks? (which one?)

@bertdegeyter.bsky.social

No, I mean that evalite is local-first, unlike Braintrust

bertdegeyter.bsky.social

1. Correct me if I'm wrong. But wouldn't that just be expect(llmOutputScore).toBeGreatherThan(70) I don't see why you'd want to wrap vitest for that 2. How about verifying the prompt and model on a CI during automated unit tests?

1. It's not enough to know whether you are above a certain arbitrary number. You need to be able to dive into the data and view all outputs. You also need to see historical changes over time.