Post

Hi! SkyFeed is struggling and I need to find a solution. If you don't know SkyFeed, it's a service I built 2 years ago, enabling anyone to build and publish feeds using a visual block-based editor. SkyFeed is still hosting the most feeds on Bluesky, but that comes at a cost [1/X]

Let's get straight to the point: Renting servers for hosting all 75k+ SkyFeed feeds for millions of users daily costs me ~1094 EUR/month SkyFeed is 100% free, so the only income is my GitHub Sponsors (github.com/redsolver) with ~191 EUR/month The math does not quite work out here.

Donations mostly covered costs at the end of 2024, but expenses skyrocketed as I scaled everything to keep up with Bluesky growing quickly. While fighting the fires popping up in the infrastructure, I couldn't really focus on releasing new features, fix bugs or implement long-promised improvements

It felt wrong to ask for money before delivering the level of service I expect. But being a solo developer and SkyFeed not my main job made that near-impossible. I also realized that the current architecture struggled to keep up, let alone enable new resource-intensive features I promised long ago.

The first component I started to rewrite was the indexer, but I quickly realized the storage and query layers also needed a ground-up rebuild. This took far longer than expected and was the main reason for not improving the SkyFeed app or adding support for new features like video

The good news: After 4 iterations, the new fully Rust-based architecture is almost ready. The historical backfill is running, and experiments with the new query engine have been very promising. I am confident that the new engine will deliver and am excited about the amazing new feeds it will enable!

Lol I admit my first thought was "don't say rewrite it in Rust, don't say rewrite it in Rust" 😄

Ah, interesting - so which parts were in Dart?

ok so the old infra was: indexer(dart) -> SurrealDB -> query-engine (rust) the indexer could not keep up with the firehose, but while re-writing it I noticed that surrealdb could not handle full bsky backfill. So now I also created a custom DB layer, and all components use the same data structures.