Post

The AI bros (who clearly don't understand AI) in the comments are hilarious. "ask stupid questions, get stupid answers". LLMs have no concept of how intelligent a question is, and they certainly were not designed to deliberately give stupid responses if they deem your question too stupid. 1/?

What's most likely happening is easier to understand if you mentally model LLMs as glorified search engines. They are matching your search query (prompt) with some text they found on the internet. Internally it's a bit more complex than that, but much closer to Google than to a thinking machine. 2/?

Essentially, it doesn't have an answer for whether you should walk or drive to a car wash, as it's not a commonly asked question (because it's stupid). Just like if you Google searched the question, you wouldn't find a good result because it's not a common question. 3/?

But the question of whether you should walk or drive a given distance is common, so it basically just ignores the car wash part and answers the question of walking or driving in a more generic sense. The problem isn't that someone is going to ask the LLM this exact question, it's the handling. 4/?

It clearly can't infer from a simple one sentence question that I might need my car to wash my car, because it isn't intelligent, doesn't possess logic, and can't reason. When you ask it anything complex, it's coming across 10s, 100s, or 1000s of points where it has to infer context. 5/?

So, when you use LLMs as thinking machines, you will run into variants of a core problem: It isn't reasoning. It's just a fancy bot that copy & pastes then rewords data written by humans. Which means: 1. It's implicitly capped by its human produced-training data, which is largely garbage. 7/?

2. if you try to use it for a task that diverges too far from something it has in its training data, it will shit the bed (as is evident by the car wash question, which any intelligent being could easily answer). 8/?

You can ask many people if a pound of gold or a pound of feathers weighs more and get an incorrect answer.

3. Even when working on tasks for which the LLM has comprehensive data on, it's having to infer context constantly, and every time it mischaracterizes or drops some context, it goes further off the rails. 9/?