Post

This take is a bit outdated now. At its core it is still true, but lots has happened to improve LLMs as fact-retrieval tools, and I think they’re generally very good at it provided you, the operator, have your head screwed on and do the work to verify.

For example, it’s very common now for frontier models to be good at knowing when they need to go and retrieve more information. Here’s a very simple example. For higher stakes questions, you definitely want to be clicking through to the sources to make sure you’re getting good information.

If you relied on the model weights directly for fact retrieval then yes, you’re going to get garbage. The mental model I’ve settled on for this is that information contained in the model itself is exceptionally fuzzy and compressed. Information in the context window, though, is very clear.

I got this mental model from listening to Andrej Karpathy on the Dwarkesh podcast. It’s a good listen if you’re interested in this topic. youtu.be/lXUZvyajciY?...

I even wrote a tool to help demonstrate just how bad an idea it is to rely on the lossy compressed knowledge inside the model weights themselves. bsky.app/profile/samw...

The last thing that irks me about this is saying that the model is only ever generating plausible text. The more I read around the “mechanistic interpretability” space, the folks working to understand what it is models have stored in their weights, the more I believe there is real smarts in there.

They’re strange smarts. But they’re not the sort of Markovian system people seem to think they are. There’s more complex and sophisticated circuitry in there, and it is truly fascinating. Here are some good reads: - arxiv.org/abs/2502.00873 - transformer-circuits.pub/2024/scaling...

Oh! Yes, actual last thing. Just asking them questions is one of many use-cases. It’s a shame that it’s the obvious use-case and it’s one of the least interesting. LLMs shine when given a task with an easy verification criteria. The canonical example is getting a test to pass in code.

Their ability to iterate is quite remarkable. Often times they do produce really weird or bad code and it needs cleaning up after, but it never ceases to amaze me how frequently they find the real root cause of a problem. I’ve found Opus 4.5 especially good at this iterative way of working.