Post

dax
@thdxr.com

LLM APIs need to return cost information in their response alongside tokens literally everyone is using models[dot]dev data to approximate this - we see so many reqs to its api but this is just sticker pricing, won't reflect discounts, etc so it doens't really work

6 reposts63 likes

ewatch.bsky.social
@ewatch.bsky.social

Funny was just thinking about this because it would be interesting writing these costs into finished branches / issues / whatever to understand how much parts of a project cost or the full project until a certain point.

Lawrence Jones
@lawrencejones.dev

What’s your recommendation here? Like you say the costs change in various ways, we have a registry.yaml file that tracks our current pricing and we write that into our database whenever we make a request so we know at that time what we spent. Not sure how you can do this nicely in the APIs.

Lawrence Jones
@lawrencejones.dev

The layering gets pretty complex with the pricing you apply and can sometimes even be retrospective (if you negotiate a new price it can apply backwards) so quite tricky.

Rasmus Schultz
@mindplay.dk

someone needs to create a standard header for this like, standard across the whole web things like middleware, routers or proxy servers aggregating cost metrics from all the services you consume would be trivial doesn't need to be limited to LLMs either 🤔

cl0q
@cl0qsearch.bsky.social

+1 on this. The real question: should cost be a first-class response field, or a standard HTTP header so every client gets it for free? A header means no SDK changes. What would you pick — and why hasn't any big lab done it yet?