Post

Post deletes being complete happening soon! Thanks @jay.bsky.team! NOTE regardless of network, scraping happens and can't really be stopped. For years a bot reposted my every tweet. Please be careful. BlueSky is a playground - not a different universe - even if we rejoice in momentary reprieve.

yes, people should treat data here like it’s the public web. we had a long discussion today about whether/how to discourage scraping/archiving — @pfrazee.com?

yeah I'm gonna go Big Technical on this one

every user in atproto has a signed data repository, which is quite similar to a git repository. changes get written in commits and each repository maintains a history (commit log) we do that because the network is largely pull-based, and maintaining the history makes catchup sync efficient

designing the data repository is a bit of an art, because it's a merkle tree and you're balancing the size of proofs for a given record or subset of the data against the flexibility of the data it contains. @dholms.xyz is your guy on how all that works

as it turned out, the way image blobs get stored in the repository make it easy to just drop them and leave dangling references. that's why we're able to ship that quickly records, however, are harder to expunge without losing the ability to verify the proofs

daniel tells me it might be possible to change the data structure so that it's possible, but he had that kind of quiver in his voice when he said it that tells me I shouldnt expect it any time soon that means if we want to expunge the actual records, we have to do a "rebase"

a "rebase" basically means rewriting the history to remove any record of the past writes. generally speaking you want to do this every so often as a form of compaction, because otherwise every repository is going to grow unbounded. so, that's something the team designed for from the start

the problem with a rebase is that it interferes with replication. daniel can expand on this, but basically there are some conditions (which he calls the "happy path") where a rebase will cause no problems, and some where it'll mean having to refetch the whole repo

it also takes half a second to execute the rebase. so, for obvious reasons, we were trying to avoid doing it very often. reality is now blasting us in the face, and so more frequent rebases are on the table and the team will have to adapt. that's how this stuff goes

I’m not technically minded, so I might have this completely wrong, but would it be possible to give users the ability to rebase their own accounts only when they need/want to via a settings menu or something like that?