Post

Is sync not already a set reconciliation? I’m not familiar on this levels with MST and CAR, but protocol negotiation sounds like the right thing to do here

every commit changes the mst root hash and that of every mst node on the path between the root and the affected key. recomputing the hashes for sync requires *adjacent* cids from unchanged nodes and records along the path. the path is usually similar from one commit to the next

Nick The Sick

@nickthesick.com

Ah, I see because the MST is a tree it all “bubbles up” to the root hash. I think your idea is pretty similar to Sedimentree by @patternist.xyz github.com/inkandswitch... May be worth a read

It’s way down on my list but I am interested in this. Before data portability became a harder req I was just throwing around CIDs and was going to add a cache index because it’d make my funky syncing strategy simpler

22 Feb

…oops i’m gonna make an alternate firehose format, aren’t i

David Buchanan

@retr0.id

I think the issue with something like this is that the sender needs to know the receiver's cache size, and I don't think there's a good one-size-fits-all answer to cache size

see i’m curious about that, whether there’s a spot on the diminishing returns curve that’s pretty small and pretty effective that’s just Good Enough to pick and live with. maybe making efficiency sensitive to sorted rkey insertion order isn’t desirable though

(i mean, i also don’t know specifically how sensitive it is to that yet actually, just vibes)

22 Feb

need to take actual measurements, but cids are the incompressible bulk of commits. i think this might work.

we debated a similar optimization at some point, I think why came up with it based on filesystem implementation tricks: have a small array of records be referenced directly (by path and CID) from the commit object. they would still go out on firehose, but wouldn't churn the MST structure

the firehose event would be just the new (signed) commit object, and the (new?) records. no MST nodes. the motivation was this was a high rate of small interaction records such as "likes".

I personally didn't like this because it adds implementation complexity in a bunch of places (PDS, consumers, anybody requesting a proof chain). you need to be very careful doing existence lookups, handling invalid corner-cases, etc. I just think correctness+reliably would suffer in practice

fair. where i feel like there’s value in this (and STAR) is that they are ~just alternative representation of the exact CAR contents with cheap conversion back to the standard. they could exist alongside instead of replacing, and use something like content negotiation on connect.