Sebastian Raschka (rasbt)

Sebastian Raschka (rasbt)

@rasbt.bsky.social

ML/AI researcher & former stats professor turned LLM research engineer. Author of "Build a Large Language Model From Scratch" (https://amzn.to/4fqvn0D) & reasoning (https://mng.bz/Nwr7). Also blogging about AI research at magazine.sebastianraschka.com.

10.5K followers249 following319 posts

Sebastian Raschka (rasbt)
@rasbt.bsky.social

I put together a visual LLM Architecture Gallery that collects (~50) recent open-weight model designs in one place. Architecture diagrams, config links, tech reports, explainers... you name it! Hopefully useful as a reference & learning resource: sebastianraschka.com/llm-architec...

sebastianraschka.comLLM Architecture GalleryA gallery that collects architecture figures from The Big LLM Architecture Comparison and related articles, with fact sheets and links back to the original sections.

Sebastian Raschka (rasbt) reposted

Justin Norman, PhD
@justintime.ai

For the past month or so, I've been slowly working through this book by @sebastianraschka.com which theoretically and practically builds a GPT model from scratch. Highly recommended! Ironically, I'm writing much more code by hand as a result

Sebastian Raschka (rasbt)
@rasbt.bsky.social

Uploaded my State of LLMs 2025 report for this year: magazine.sebastianraschka.com/p/state-of-l... I planned to just write a brief overview, but yeah, it was an eventful year so it was impossible to keep it below 7000 words :D.

magazine.sebastianraschka.comThe State Of LLMs 2025: Progress, Progress, and PredictionsA 2025 review of large language models, from DeepSeek R1 and RLVR to inference-time scaling, benchmarks, architectures, and predictions for 2026.
Sebastian Raschka (rasbt)
@rasbt.bsky.social

I think of it as this: LLMs lower the barrier of entry, and they make coders (beginners and experts) more productive. It's still worth investing in becoming an expert, because then you will get even more out of LLMs and will be able to deliver even better results.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

Hold on a sec, Mistral 3 Large uses the DeepSeek V3 architecture, including MLA? Just went through the config files; the only difference I could see is that Mistral 3 Large used 2x fewer experts but made each expert 2x large.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

Excited for my first conference in Europe in April. I’ll be talking about LLMs, Python, coding, and all the fun stuff, and I’m looking forward to meeting fellow AI builders there!

PyCon DE & PyData@pyconde.bsky.social

We’re thrilled to welcome Sebastian Raschka to PyCon DE & PyData 2026. From Scratch to Scale: How Far Python Takes You in Building LLMs A deep dive into how Python powers experimentation, training, and scaling of LLMs. 📍 Darmstadt, April 14–16, 2026

Sebastian Raschka (rasbt)
@rasbt.bsky.social

Inference-scaling lets us trade extra compute for better modeling accuracy. Next to RL, it has become one of the most important concepts in today's LLMs, so the book will cover it in two chapters instead of just one. If you are looking for sth to read this weekend Ch4 is available now: mng.bz/Dwra

Sebastian Raschka (rasbt)
@rasbt.bsky.social

What should we focus on, (more) LLM training or inference scaling? (A question I got asked multiple times now, so here are some thoughts.) Training is usually very, very expensive, but it is a one-time cost. Inference-scaling is comparatively cheap, but it's a cost we pay at each query.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

Let's assume we have an LLM (or an LLM-powered app) that we want to improve by 5%. We don't pass the extra cost to the customer. We could either train the model longer for $5 million to get the 5%, or we could pay $0.2 extra per query due to higher token usage.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

What strategy is better for our company? If you had to choose between the two, then it really depends on how many queries you (or the users) plan to run for the lifetime of that LLM. In this case, the break-even point is 5,000,000 dollars / 0.20 dollars per query = 25 million queries.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

I just saw the Kimi K2 Thinking release! Kimi K2 is based on the DeepSeek V3/R1 architecture, and here's a side-by-side comparison. In short, Kimi K2 is a slightly scaled DeepSeek V3/R1. And the gains are in the data and training recipes. Hopefully, we will see some details on those soon, too.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

From the Hierarchical Reasoning Model (HRM) to a new Tiny Recursive Model (TRM). A few months ago, the HRM made big waves in the AI research community as it showed really good performance on the ARC challenge despite its small 27M size. (That's about 22x smaller than the smallest Qwen3 0.6B model.)

View full thread
Sebastian Raschka (rasbt)
@rasbt.bsky.social

7/ In practice, we often start by throwing LLMs at a problem, which makes sense for quick prototyping and establishing a baseline. But I can see a point where someone sits down afterward and trains a focused model like this to solve the same task more efficiently.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

From the Hierarchical Reasoning Model (HRM) to a new Tiny Recursive Model (TRM). A few months ago, the HRM made big waves in the AI research community as it showed really good performance on the ARC challenge despite its small 27M size. (That's about 22x smaller than the smallest Qwen3 0.6B model.)

View full thread
Sebastian Raschka (rasbt)
@rasbt.bsky.social

5/ My personal caveat: comparing this method (or HRMs) to LLMs feels a bit unfair since HRMs/TRM are specialized models trained for specific tasks (here: ARC, Sudoku, and Maze pathfinding) while LLMs are generalists. It’s like comparing a pocket calculator to a computer.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

6/ That said, HRM and TRM are fascinating proof‑of‑concepts that show what’s possible with relatively small and efficient architectures. I'm still curious what the real‑world use case will look like. Maybe they could serve as reasoning or planning modules within a larger tool‑calling system.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

From the Hierarchical Reasoning Model (HRM) to a new Tiny Recursive Model (TRM). A few months ago, the HRM made big waves in the AI research community as it showed really good performance on the ARC challenge despite its small 27M size. (That's about 22x smaller than the smallest Qwen3 0.6B model.)

View full thread
Sebastian Raschka (rasbt)
@rasbt.bsky.social

4/ TRM backpropagates through the full recursion once per step, whereas HRM only backpropagates through the final few steps. And TRM also removes HRM's extra forward pass for halting and instead uses a simple binary cross-entropy loss to learn when to stop iterating.

Sebastian Raschka (rasbt)
@rasbt.bsky.social

From the Hierarchical Reasoning Model (HRM) to a new Tiny Recursive Model (TRM). A few months ago, the HRM made big waves in the AI research community as it showed really good performance on the ARC challenge despite its small 27M size. (That's about 22x smaller than the smallest Qwen3 0.6B model.)

Sebastian Raschka (rasbt)
@rasbt.bsky.social

2/ Now, the new "Less is More: Recursive Reasoning with Tiny Networks" paper proposes Tiny Recursive Model (TRM), which a simpler and even smaller model (7M, 4× smaller than HRM) that performs even better on the ARC challenge.

Sebastian Raschka (rasbt) reposted

Rémy
@xowap.dev

Replying to @mason.dev

I'm right now on the last chapter of "Build a Large Language Model (from scratch)" by @sebastianraschka.com and it's absolutely amazing to get started. Now I can understand why people lose their shit over DeepSeek, for example