Post

cdata
@cdata.earth

Getting close to being able to demonstrate the complementarity of #DialogDB and #ATProto. Bi-directional sync between a general purpose, local-first (private by default) database and any PDS.

2 reposts27 likes

cdata
@cdata.earth

One thing that is tricky with ATP is the open-ended nature of Lexicons. SQLite doesn't sync out of the box, but even if it did there would be an impedance mismatch: no fixed schemas!

cdata
@cdata.earth

I get hot and bothered thinking about a local-first Bsky client with offline write. Even though the audience for that is like.. dozens of people? 😅

cdata
@cdata.earth

Wrangling, composing and remixing Lexicons without an AppView deployed in the cloud is also going to be interesting.

𝕮
@chrisshank.com

I'm particularly interested how far we can push www.microcosm.blue services as generic, replicable appviews. I think a piece that's missing, and I was just chatting with @bad-example.com about this, is a query language to more easily back-link and forward-link records.

Boris
@bmann.ca

Appviews aren’t required. They’re required for aggregation (among other things) but sharing data shapes p2p works fine. As well as PDS to PDS. What else do you feel you’ll need?

cdata
@cdata.earth

Let's not get ahead of ourselves 😅 there is work yet to be done. But: the sketch starts with OAuth via control panel UI on the Dialog Remote (think Git Remote); looks a lot like "log in w/ your PDS."

cdata
@cdata.earth

The core of Dialog authz is a capability delegation scheme via UCANs. PDS doesn't know about Dialog. And, Dialog embedders transact with PDS data like any other data in their database. The interop happens on the Remote, which tracks changes and syncs w/ the PDS.

cdata
@cdata.earth

If you are interested to glean more about how the bi-directional / interop part works, I recommend our Sync conf talk from last year. bsky.app/profile/cdat...

cdata@cdata.earth

I've been trying to come up with a spin on "there's an app for that," but for Dialog Rules (which, among other things, enable opportunistic / uncoordinated interop between app domains without the risk of one party clobbering other apps' data www.youtube.com/watch?v=4n-2...).

youtube.comIrakli Gozalishvili & Chris Joel - Your Data, Your Rules & the Way to Share ThemYouTube video by Local-First Conf
Steven Vandevelde
@tokono.ma

Looking forward to this! I’m struggling with my rudimentary local-first atproto sync. It works ok with a normal amount of data, but once I need to import 20k records it’s rate limit errors all over the place. I wonder if we need permissioned data to land, a separate storage, or something else? 🤷

cdata
@cdata.earth

I would be interested to hear more about where in the stack rate limits manifested for you. One of Dialog's tent poles is partial replication: you only sync the data that you query for. This doesn't address every use case, but it is a powerful tool for optimizing network operations over large data.

Steven Vandevelde
@tokono.ma

Bluesky’s PDS has a limit on the amount of record interactions per hour. 1500-ish I’m guessing, in any case not enough to add 20K in one go; when importing data. It’s mostly that. I got around it by stuffing as much data as possible into one record, but then I lose the ability to do partial updates

dietrich
@burrito.space

i jammed this into sqlite which lives separate from the PDS (could be remote or local). both it and P2PDS i want to port to Dialogdb, both have sharding/lensing needs bsky.app/profile/burr...

dietrich@burrito.space

So many Atproto apps now ❤️ But data layer interop means you don't *have* to "go" anywhere. Concept demo of inverting things: - local index of lexidex - local cache of all your 'cons and records - generic 'con-to-form engine Now I can post, rsvp, bookmark all from the comfort of home.

cdata
@cdata.earth

That's awesome. I'm interested to know more about how you reconcile the openendedness of Lexicons with the closed shape of a SQLite schema. What's your strategy in this case?

dietrich
@burrito.space

i didn't reconcile it! it's actually worse, i jammed Lexidex into localStorage for the Lexicon Studio demo, and jammed everything as JSON blobs into SQLite for P2PDS. hence the need for something better.