Post

jsulz
@jsulz.com

On the Xet team at @huggingface.bsky.social we're always looking for ways to move bytes to computer near you as fast as possible. To do this, we're redesigning the upload and download infrastructure on the Hub. This post describes how, check the thread for details 🧵 huggingface.co/blog/rearchi...

huggingface.coRearchitecting Hugging Face Uploads and DownloadsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

11 reposts66 likes

jsulz
@jsulz.com

Our goal? Replace Git LFS with Xet-backed storage to make uploads smarter and downloads faster. The analysis? A 24 hour snapshot of upload requests in October - 88 countries, 8.2M requests, and 130.8TB of data - to design a scalable infrastructure.

jsulz
@jsulz.com

The result? A control plane with a custom protocol routing requests to content addressed stores in 3 AWS regions: us-east-1, eu-west-3, and ap-southeast-1. 🌍

David Gasquez
@davidgasquez.com

Very cool! Thanks for sharing all the progress. Any place where we can read more about the CAS side of things? Curious about things like hashing function, block size, ...

jsulz
@jsulz.com

It's like you're peering into my blog post drafts 😀 I'm working on another post that describes some of the nitty gritty details to show how we make this work in practice.

David Gasquez
@davidgasquez.com

Haha! 🙈 I've been in the content-addressed storage world and the git for data / open data ecosystem for a while and am just very excited to see you all work on these sorts of things!