Post

Sam Rose
@samwho.dev

This take is a bit outdated now. At its core it is still true, but lots has happened to improve LLMs as fact-retrieval tools, and I think they’re generally very good at it provided you, the operator, have your head screwed on and do the work to verify.

Barry Deutsch@barrydeutsch.bsky.social

The only thing ChatGPT ever does.

Sam Rose
@samwho.dev

For example, it’s very common now for frontier models to be good at knowing when they need to go and retrieve more information. Here’s a very simple example. For higher stakes questions, you definitely want to be clicking through to the sources to make sure you’re getting good information.

Sam Rose
@samwho.dev

If you relied on the model weights directly for fact retrieval then yes, you’re going to get garbage. The mental model I’ve settled on for this is that information contained in the model itself is exceptionally fuzzy and compressed. Information in the context window, though, is very clear.

Sam Rose
@samwho.dev

I even wrote a tool to help demonstrate just how bad an idea it is to rely on the lossy compressed knowledge inside the model weights themselves. bsky.app/profile/samw...

Sam Rose@samwho.dev

Worked on this a bunch today to make it substantially faster and published it to Pypi, so you can now try it yourself much easier: ``` uvx llmwalk -p "What is 2+2?" ``` github.com/samwho/llmwalk

Sam Rose
@samwho.dev

The last thing that irks me about this is saying that the model is only ever generating plausible text. The more I read around the “mechanistic interpretability” space, the folks working to understand what it is models have stored in their weights, the more I believe there is real smarts in there.

0 reposts6 likes

Sam Rose
@samwho.dev

They’re strange smarts. But they’re not the sort of Markovian system people seem to think they are. There’s more complex and sophisticated circuitry in there, and it is truly fascinating. Here are some good reads: - arxiv.org/abs/2502.00873 - transformer-circuits.pub/2024/scaling...

arxiv.orgLanguage Models Use Trigonometry to Do AdditionMathematical reasoning is an increasingly important indicator of large language model (LLM) capabilities, yet we lack understanding of how LLMs process even simple mathematical tasks. To address this,...
Sam Rose
@samwho.dev

So yeah, in April 2023 when the screenshot was taken it was entirely true. Today, December 2025, it’s not quite that simple.

Sam Rose
@samwho.dev

Oh! Yes, actual last thing. Just asking them questions is one of many use-cases. It’s a shame that it’s the obvious use-case and it’s one of the least interesting. LLMs shine when given a task with an easy verification criteria. The canonical example is getting a test to pass in code.

Sam Rose
@samwho.dev

Their ability to iterate is quite remarkable. Often times they do produce really weird or bad code and it needs cleaning up after, but it never ceases to amaze me how frequently they find the real root cause of a problem. I’ve found Opus 4.5 especially good at this iterative way of working.

joao
@joao.omg.lol

I think being Markovian and exhibiting interesting internal representations and behaviours are not mutually exclusive. It is trying to generate the most probable response (over the distributions in pre and post training), but since it can’t simply do a table lookup it develops

Sam Rose
@samwho.dev

Fair point, thank you 🙏 I do think there are people out there who believe it’s an uninteresting Markovian system that’s entirely based on statistics, and I’m seeing more and more research suggesting that’s far from the truth. That’s what I’m getting at.