Post

There is so much to gzip. It's a perfect microcosm of the experience of Computering. Fractally complex and clever. Hamstrung by the limitations of computers at the time it was invented. Dramatic, backstabby origin story. Completely unused extension points. Both prescient and myopic in its design.

Interesting story, Jon. Would you list the main points of interest in gzip for us, the nom-specialists, reading your posts and wondering how exactly gzip looks so fantastic? I’m a big fan of UNIX software, they indeed carry a lot of wisdom through the time. I’d love to know a bit of this story!

tl;dr gzip is a thin envelope for DEFLATE. With just that, you can do interesting things, e.g. stargz is spiritually a zip file masquerading as a targz. Going deeper, DEFLATE has enough structure that you can do very cool things, e.g. soci from aws folks uses an old trick to get random access.

I also wrote a poem about all this that only exists publicly in the subscriber-only content of an episode of the now-defunct Ship It podcast so I should probably publish that somewhere else.

Another very cool project is rapidgzip and ratarmount github.com/mxmlnkn/rapi... gunzip is normally not meaningfully parallelizable but rapidgzip attempts to speculatively perform partial decompression at likely DEFLATE block boundaries then stitch them back together once enough window is decoded.

Most of these tricks are only possible because gzip is old enough that 32KiB seemed like a lot at the time. The pragmatic choice nowadays is zstandard which uses a more sophisticated encoding and much larger windows but otherwise looks a whole lot like gzip+DEFLATE if you squint.