Post

Georg Unterholzner
@georg.dev

Large Language Models (LLMs) with "reasoning" capabilities are super hyped, but new research from Apple shows their "thinking" is often an illusion. Beyond a certain complexity, they fail completely. Key points in the 🧵.

0 reposts2 likes

Georg Unterholzner
@georg.dev

Key points: - At low complexity, standard LLMs excel (faster, cheaper). - At medium complexity, reasoning models (LRMs) outperform. - At high complexity, both collapse, with nearly 0% accuracy.

Georg Unterholzner
@georg.dev

The paper identifies three distinct "reasoning regimes" in puzzles like Tower of Hanoi and River Crossing. Frontier LRMs (Claude 3.7 Thinking, DeepSeek-R1) outperform standard LLMs only in the medium complexity zone. Beyond this, their accuracy collapses regardless of compute.

Georg Unterholzner
@georg.dev

Worst of all, LRMs reduce their reasoning effort as problems get harder. Despite having spare token budgets, they underthink when complexity peaks, like a student giving up on an exam’s toughest question—even with extra time left 😅

Georg Unterholzner
@georg.dev

Researchers even handed models step-by-step algorithms (e.g., for Tower of Hanoi). The result: identical failure points. LRMs couldn’t reliably execute prescribed logic—suggesting their "reasoning" is often pattern-matching, not general intelligence.