Post

This take is a bit outdated now. At its core it is still true, but lots has happened to improve LLMs as fact-retrieval tools, and I think they’re generally very good at it provided you, the operator, have your head screwed on and do the work to verify.

For example, it’s very common now for frontier models to be good at knowing when they need to go and retrieve more information. Here’s a very simple example. For higher stakes questions, you definitely want to be clicking through to the sources to make sure you’re getting good information.

If you relied on the model weights directly for fact retrieval then yes, you’re going to get garbage. The mental model I’ve settled on for this is that information contained in the model itself is exceptionally fuzzy and compressed. Information in the context window, though, is very clear.

I got this mental model from listening to Andrej Karpathy on the Dwarkesh podcast. It’s a good listen if you’re interested in this topic. youtu.be/lXUZvyajciY?...

The last thing that irks me about this is saying that the model is only ever generating plausible text. The more I read around the “mechanistic interpretability” space, the folks working to understand what it is models have stored in their weights, the more I believe there is real smarts in there.

They’re strange smarts. But they’re not the sort of Markovian system people seem to think they are. There’s more complex and sophisticated circuitry in there, and it is truly fascinating. Here are some good reads: - arxiv.org/abs/2502.00873 - transformer-circuits.pub/2024/scaling...

So yeah, in April 2023 when the screenshot was taken it was entirely true. Today, December 2025, it’s not quite that simple.

I think being Markovian and exhibiting interesting internal representations and behaviours are not mutually exclusive. It is trying to generate the most probable response (over the distributions in pre and post training), but since it can’t simply do a table lookup it develops