Post

Sam Rose
@samwho.dev

I spent 2 months learning about quantization and am extremely proud of the post I've written about it. I think these are some of the nicest visuals I've ever made, and I love how this compression technique invented in 1898 is being used on the bleeding edge in 2026. ngrok.com/blog/quantiz...

ngrok.comQuantization from the ground up | ngrok blogA complete guide to what quantization is, how it works, and how it's used to compress large language models

55 reposts251 likes

seed paste soy goblin
@phasebinary.bsky.social

This is a fantastic approach! But I was really surprised to learn the quantizations are just linear. Do any of the following perform better? - linear in the middle with bookends for outliers - Gaussian-like - "control points" fit to a curve between them

Sam Rose
@samwho.dev

There are non-linear ones! Google’s TurboQuant for example is based off of a Lloyd-Max quantizer, which adapts the bucket sizes to the data it’s fitting. I kept it simple for the post, partly to make it accessible and partly to make it not too long. The method I cover is used in GGUF and MLX.

Steve Adams
@steve-adams.me

My guess as to why the 2bit model was slower: maybe the reasoning isn't reasonable so it fails to yield usable answers as easily? If reasoning tokens are included in the performance measurement, then I have no idea. Very counter-intuitive

Sam Rose
@samwho.dev

I don’t think llama-bench makes any distinction between reasoning and non-reasoning tokens, so I don’t think that would be it. I suspect what you’re describing explains the terrible GPQA score, though.

8bit
@8bit.lol

the OG image made me physically crack up, I'm going to be sharing (and reading, too late to absorb it) this at work tomorrow!

Sam Rose
@samwho.dev

Lil' gallery of some of the visuals to entice you to click and justify my continued employment. The post starts by introducing what models actually _are_ under the hood, and what leads them to become so large they can't fit in your laptop's RAM.

Sam Rose
@samwho.dev

Then I talk about how computers represent floating point numbers, the trade-offs involved, and how LLMs kinda don't care that much about precision (but not in that way.)

Sam Rose
@samwho.dev

As part of the work on this post, I opened up 6 popular open models and analysed their parameters. This is a histogram that shows that model parameter values cluster close to 0, with the vast majority of values being very small.

Sam Rose
@samwho.dev

There were a tonne of fun ways to visualise the precision of floats as you use fewer bits to represent them. This one shows the round-to-nearest behaviour between 16-, 8-, and 4-bit floats.

Lawrence Jones
@lawrencejones.dev

I love these posts of yours! They really are great. Was on the lookout for a mention of your outliers from the moment I opened it too, was not disappointed 😂

Lawrence Jones
@lawrencejones.dev

It’s funny how much this stuff reminds me of comp neuro at uni. Especially your closing notes: there needs to be a threshold of complexity in these networks before you get anything that looks remotely interesting, but the behaviour around that pivot can be interesting.

Rodrigo Girão Serrão 🐍🚀
@mathspp.com

The post _was_ great, thanks for sharing. And the interactive visuals work really well. My favourite detail? The mechanical feel you get when you slide the slider for the integers and the corresponding base-2 bits flip.