Post

Sam Rose
@samwho.dev

This take is a bit outdated now. At its core it is still true, but lots has happened to improve LLMs as fact-retrieval tools, and I think they’re generally very good at it provided you, the operator, have your head screwed on and do the work to verify.

Barry Deutsch@barrydeutsch.bsky.social

The only thing ChatGPT ever does.

Sam Rose
@samwho.dev

For example, it’s very common now for frontier models to be good at knowing when they need to go and retrieve more information. Here’s a very simple example. For higher stakes questions, you definitely want to be clicking through to the sources to make sure you’re getting good information.

0 reposts6 likes

Daniel King
@kin.gy

But isn't that orders of magnitude more processing power and electricity use than going direct to the Met Office website yourself to get the weather forecast? With a chance it could still get it wrong. That seems like the inferior option, or is this take outdated now too?

Sam Rose
@samwho.dev

No, this take is correct! You’re much better off going to a source you trust directly if you have one. LLMs are better when you don’t know a good source already and you want to find some, or your question is sufficiently fuzzy that you’re not sure how to ask a search engine about it.

Sam Rose
@samwho.dev

If you relied on the model weights directly for fact retrieval then yes, you’re going to get garbage. The mental model I’ve settled on for this is that information contained in the model itself is exceptionally fuzzy and compressed. Information in the context window, though, is very clear.

Sam Rose
@samwho.dev

I even wrote a tool to help demonstrate just how bad an idea it is to rely on the lossy compressed knowledge inside the model weights themselves. bsky.app/profile/samw...

Sam Rose@samwho.dev

Worked on this a bunch today to make it substantially faster and published it to Pypi, so you can now try it yourself much easier: ``` uvx llmwalk -p "What is 2+2?" ``` github.com/samwho/llmwalk

Sam Rose
@samwho.dev

The last thing that irks me about this is saying that the model is only ever generating plausible text. The more I read around the “mechanistic interpretability” space, the folks working to understand what it is models have stored in their weights, the more I believe there is real smarts in there.