Post

How is a scorer defined? 👀

It's an async function that takes an input and returns a number between 0-1

Nice! When working with LLM-as-a-judge evaluations I very quickly felt the need for categorical & ordinal scoring (high, medium, low; pass/fail) and scoring metadata like confidence score, etc. I'm selfishly hoping you stumble upon it too and end up designing a clean generic TS interface for it 😅

I don't understand what this is. 😅 it looks like it's just mapping your input/expected over a function? why do you need a custom test runner for this? and how is it local first? most useful models are too large to run locally. some sort of API response cache/recorder is probably what I need? 🤔

Some background on evals, full explanation below: www.aihero.dev/what-are-evals

1. First, evals return a score, not a pass/fail metric. So you need a custom test runner to see the scores. 2. Good evals also need a UI attached so you can view the results, since you often need to verify the LLM app's output with human eyes.

Damn, you really more than just live up to your stellar reputation of typescript wizard Matt....

It looks like you're exporting `Levenstein`, rather than using autoevals directly here. What's the thinking there? Does it need wrapping in some way?

I initially misread it as Eluveitie, which feels totally out of context 🤦