Post

You could frame an entire Computer Science curriculum around gzip. You could produce a pretty compelling dramatic film about its history. You could spend an entire (productive) career tinkering with gzip encoders.

Sometimes I am overwhelmed with existential dread because I accidentally looked inside this box and found an infinitely interesting little gadget. How many other boxes of infinite complexity did I ignore because I assumed they weren't interesting? It is unfair how little time we get to explore.

At the same time it's kind of beautiful. I imagine that every mundane experience that I take for granted daily is the product of an auteur's obsession with their own little corner of our reality. Every interaction with this world is the sum of a thousand tiny miracles we have achieved together.

Interesting story, Jon. Would you list the main points of interest in gzip for us, the nom-specialists, reading your posts and wondering how exactly gzip looks so fantastic? I’m a big fan of UNIX software, they indeed carry a lot of wisdom through the time. I’d love to know a bit of this story!

tl;dr gzip is a thin envelope for DEFLATE. With just that, you can do interesting things, e.g. stargz is spiritually a zip file masquerading as a targz. Going deeper, DEFLATE has enough structure that you can do very cool things, e.g. soci from aws folks uses an old trick to get random access.

Most of this is interesting to me because the world of containers and packages is mostly shuffling gzipped tarballs around. The culmination of this fever dream is oci.dag.dev I extracted some relevant bits to github.com/jonjohnsonjr... (if go is legible to you) Also github.com/google/crfs