Post

There is so much to gzip. It's a perfect microcosm of the experience of Computering. Fractally complex and clever. Hamstrung by the limitations of computers at the time it was invented. Dramatic, backstabby origin story. Completely unused extension points. Both prescient and myopic in its design.

Interesting story, Jon. Would you list the main points of interest in gzip for us, the nom-specialists, reading your posts and wondering how exactly gzip looks so fantastic? I’m a big fan of UNIX software, they indeed carry a lot of wisdom through the time. I’d love to know a bit of this story!

tl;dr gzip is a thin envelope for DEFLATE. With just that, you can do interesting things, e.g. stargz is spiritually a zip file masquerading as a targz. Going deeper, DEFLATE has enough structure that you can do very cool things, e.g. soci from aws folks uses an old trick to get random access.

Most of this is interesting to me because the world of containers and packages is mostly shuffling gzipped tarballs around. The culmination of this fever dream is oci.dag.dev I extracted some relevant bits to github.com/jonjohnsonjr... (if go is legible to you) Also github.com/google/crfs

I also wrote a poem about all this that only exists publicly in the subscriber-only content of an episode of the now-defunct Ship It podcast so I should probably publish that somewhere else.

Most of these tricks are only possible because gzip is old enough that 32KiB seemed like a lot at the time. The pragmatic choice nowadays is zstandard which uses a more sophisticated encoding and much larger windows but otherwise looks a whole lot like gzip+DEFLATE if you squint.

In terms of interesting history there are tons of lawsuits around compression schemes in general but even if you ignore all that, Phil Katz's wikipedia article is pretty dramatic en.m.wikipedia.org/wiki/Phil_Katz

I'll just keep adding to this as I think of more stuff but I'll assume nobody is reading it. There are probably more I don't know about, but at least intel (QAT) and IBM (zlibNX) have hardware-accelerated gzip implementations, so DEFLATE is literally etched into the fabric of our reality.