Post

If our objgit post was so good last time, where's objgit post 2? Today we have an update on the epic saga of Xe reinventing Git storage on top of object storage: making seek-native packfiles. What does this mean? Well, git is complicated. Huddle into this thread!

tigrisdata.comYou can run git on object storage if you re-make packfiles | Tigris Object StorageGit packfiles were designed for mmap and local disk, so pulling one object out of a bucket means guessing at a byte range. I wrote a new format instead.

A terrible way to think about Git is that it tracks the changes made to an empty folder and through those changes you reconstruct the history of your application's source code.

This is partially a lie because Git tracks the _entire file_ at any given point in history. Every copy of all of those versions of all of those files are stored in objects which are named after the checksum of their contents (content-aware storage).

Those objects end up being fairly small, so you'd think "oh, Tigris is good at small objects, I can just put every object into Tigris and then everything will be fine forever with free happy puppies, right?" Turns out, not really because light is only so fast.

Git works around this by packing objects into packfiles. Packfiles are a bunch of objects concatentated together with an index telling you where to look to get any object in particular. The only downside is that Git packfile indices don't give enough information to construct ranged get requests.

So Xe invented a new brave format that does store enough information to construct ranged get requests. This ended up making Git object storage on top of Tigris object storage upwards of 4.5-14.6 times faster than the old approach of badly adapting filesystem interfaces to Tigris.

1 repost5 likes