Post

We've moved the first 20PB from Git LFS to Xet on @hf.co without any interruptions. Now we're migrating the rest of the Hub. We got this far by focusing on the community first. Here's a deep dive on the infra making this possible and what's next: huggingface.co/blog/migrati...

huggingface.coMigrating the Hub from Git LFS to XetWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

15/7/25, 3:16 pm2 reposts6 likes

jsulz

@jsulz.com

15 July 2025

The engine behind moving from Git LFS to Xet is our migration process. It's simple, powerful, and has moved well over a dozen PB just by itself. Here's a high level view of how it works.

jsulz

@jsulz.com

15 July 2025

You can see over the past few months some of the biggest migrations show up in our cluster throughput. Each spike corresponds to a significant migration (where we download from LFS and upload to Xet) with the baseline steadily increasing to just shy of 100 Gb/s

jsulz

@jsulz.com

15 July 2025

A sneaky part of making this all work is our backward compatibility with Git LFS. This allows us to roll out a significant protocol change without forcing workflow changes We call this the Git LFS Bridge internally, and like our migration process, it's power is in its simplicity.