Post

daniel holmgren 🫠
@dholms.at

nice i dig it, thanks for writing this up! I'm pretty much sold on doing something STAR-like for atproto spaces (CARs feel like a bad fit cuz space repos aren't DAGs) the one thing i'm not sure about is the extra magic byte

daniel holmgren 🫠
@dholms.at

i wonder if the magic byte should describe what's in the STAR file rather than the type of commitment. so instead of "L" for LtHash, it could be "s" for "space repo" also maybe not containing the commitment in the header

daniel holmgren 🫠
@dholms.at

reason being, I'm interested in a "full account export" with all space repos & public repo (for account migrations & backups) this wouldn't have a single "commit" to put in the header is the commit CID necessary? can't you just hash the commit? (similar to how STAR eliminates all the record CIDs)

daniel holmgren 🫠
@dholms.at

but then I kinda wonder if these details shouldn't be in the serialization format at all? like what if the header just goes: magic | len | cbor and the cbor could be something like: { $type: "atproto-space-repo", rev: ..., repo: ... }

daniel holmgren 🫠
@dholms.at

nice this looks like basically exactly what i had in mind one last question: are `val` in the data section always dag-cbor (and can therefore be decoded while deserializing the STAR)? or is it arbitrary bytes (in each case a second pass is requierd per-value)?

fig (aka:[phil])
@bad-example.com

yeah was thinking unspecified bytes for star vals and the application restricts further if they want. i think this is still easy-ish to do zero-copy with a generic streaming archive reader. not opposed to drisl-only vals or other restrictions in v1 if it’s an easier path for spaces though!

bryan newbold
@bnewbold.net

yeah this looks nice. nit-picky, but I think of magic bytes as being 4x bytes, which is also a bit more word-aligned. could be just another 0x00 for now. explicitly possible to have zero key/value pairs, right? wonder if the key sort ordering should be more use-case specified...

bryan newbold
@bnewbold.net

eg the blobs example I gave they would end up with 'blob/...' at the end, which would be out of overall sort order. for multi-space-repo export, could be nested, with keys being space-ref, and the value being a STAR file. would expect readers to have configurable "max block size"

bryan newbold
@bnewbold.net

some notes parallel to @dholms.at: I'm confused why the commitment (integrity?) hash would not go in the header CBOR. I like having magic! and it being extensible. combining the "type" and "version" together in a single byte might work?

daniel holmgren 🫠
@dholms.at

oh i realized after reading this that my comment on magic bytes might have been unclear i like the first three magic bytes! i'm just not sure about the 4th

daniel holmgren 🫠
@dholms.at

i think one of two things: - it should reference the archive type not the commit type (so "s" for space instead of "L" for LtHash) - or the 4th byte can just be eliminated and that type of switching is handled in the header cbor rather than by the serialization format

fig (aka:[phil])
@bad-example.com

yeah i forget why i didnt want to use “S” for space, but “L” was meant to mean “space” just keyed off the hash type for some reason (i have zero attachment to this!) four magic bytes was to have public and space repos under the same version byte. also not attached, and that goes away if we do

fig (aka:[phil])
@bad-example.com

commitment/integrity separate: og star-lite legacy that we should definitely drop!

fig (aka:[phil])@bad-example.com

the separated-out commitment from the original STAR-lite was because the rest of the the commit is technically optional, because i thought i’d need to be able to archive sub-trees for hubble in hindsight, those could have just been another $type (and also i didn’t go that route with hubble)

bryan newbold
@bnewbold.net

as danny notes being able to do a "multi-space" repo account export might be nice for backup+migration. though maybe that could just be a zip or tarball. ability to do a single space record would be nice, wrapping a CBOR record. maybe a different magic version and header CBOR '$type'?

bryan newbold
@bnewbold.net

I've always kind of wanted a way to optionally include blobs. maybe that could be a second section, after all the records, with path 'blob/<cid>'? varints: i'd stick to what is currently used count in header: I'd probably skip? feels like Content-Length HTTP header would be more useful (but hard)

bryan newbold
@bnewbold.net

FWIW my overall vibes around this being the spaces repo serialization spec are that i'm now open to it. I still think a CAR format would not be too bad, and being boring/consistent would be good. but i'm not strident about it.