Post

fig (aka:[phil])
@bad-example.com

Couldn't resist looking back at firehose optimization after getting STAR-lite mostly working. Looks like 3x firehose bandwidth reduction is achievable with implicit MST + previous-commit cache of CIDs per repo.

fig (aka:[phil])@bad-example.com

i think there's a mostly-unexplored design space here between massive client fan-out and point-to-point links. eg. i want to run my own relay, but not from home, and ideally not with the full weight of multiple full subscribeRepos coming into my home.

fig (aka:[phil])
@bad-example.com

(the cid cache hit rate is really around 65% (it's computing against all cids, not potentially-cacheable cids), and you can actually get it to almost 90% with a different cache approach (LRU instead of last-commit), but keeping the client's cache in sync would be way harder/messier.)

fig (aka:[phil])@bad-example.com

bad news and good news intuition was right: nearly 90% of MST CIDs in the firehose are rebroadcasts of recently-emitted CIDs. bad: an LRU cache needs to be kind of big to get a high hit ratio (128 entries = 4.5KB/repo) good: keeping CIDs from the prev commit only yields 60–70% hit ratio

fig (aka:[phil])
@bad-example.com

tbh 3x (after zstd!) isn't as dramatic as i was hoping. might still be worth it for keeping homelab ingress under control a bit longer, but it's very far from a jetstream-level win.

Evelyn
@kandake.africa

But Jetstream lacks verification while it seems like STAR-lite does? That's a huge benefit in my book and I'm curious if it makes sense for operators to offer endpoints. I can imagine Northsky using it within our ecosystem when we need to operate a bare metal relay and want to verify

fig (aka:[phil])
@bad-example.com

oh yeah this is keeping the full lossless firehose. i'm just greedy and was hoping for more haha. if the network is going to grow two more orders of magnitude, then this buys *time*, but won't prevent anyone close to limits from needing to make bigger changes. but growth isn't fast rn and bw is $$

1 repost2 likes

fig (aka:[phil])
@bad-example.com

the stateful CID cache makes this pretty different to operate from something like rainbow, too. per-client join-epoch from firehose seq number, to know if they were here for the did's last commit, or something (many more thoughts). good for limited number of known clients.

Evelyn
@kandake.africa

That could work for some things we're doing such as stratos, I'm currently toying with the idea of building a feedgen service to handle client integration and it's just fetching from stratos on demand, if I could do something more on the fly that could alleviate some scaling issues.

Tom Sherman
@tom.sherman.is

Many providers (even AWS) will let you do 100x the current firehose in ingress for free or next to nothing I thought Performance gains are definitely needed but not sure I get the argument about limits

fig (aka:[phil])
@bad-example.com

yeah this is about when you need to egress the firehose or have ingress limitations. not everyone is on the big cloud providers :) microcosm relays have come close to hitting egress rate limits a few times