Post

learned so far: - 4 cores seems _enough to run_ but on this particular host it’s kind of marginal. 16GB ram seems plenty. - it’s been dropping pds websockets, and doesn’t seem to recover on its own. not sure why. - bandwidth is less than i was expecting. - +jetstream on this server is too much

- pds backfill is rough on this server, cpu pegs 100% - made worse by having jetstream on the same server trying to do a higher event rate - also interesting: jetstream just crashes when it’s relay websocket closes? fine if the supervisor auto-restarts it i guess.

- 4 shared cores on a 2017-era 2ghz epyc is not enough for a relay. could probably tune some things to get it there—seeing significant overhead from userspace networking + logging—but i def want more performance margin than this. for $5.5CAD i can get two more shared cores, or double that for 8

good news! i was wrong! with a lot of patience and many hours coaxing each individual bluesky PDS server (80 of them) to sync up, this lil relay is now up, staying up, looking happy. thanks @jaz.bsky.social + @why.bsky.team for pointers. wait,, if bsky team helps, are we even decentralizing???

full-scale indie atproto relay is live and public! get yer non-bluesky-owned firehose here! relay.fire.hose.cam jetstream soon, i’ve just done enough nginx iptables for one day.

check it out on pdsls firehose browser! pdsls.dev/firehose?ins... if 10 people open this at the same time it will saturate my 250mbit connection and maybe die, let’s find out!

actually yes! two things, thank you for reminding me. 1. tls. assume you’re set since caddy 2. dropping the origin header in the reverse proxy. i haven’t dug into why exactly but the relay didn’t like it when a request included the origin.

like it kind of makes sense— if you ship an app that gets people to connect to a relay from browser, something is probably not right. …but pdsls works with prod so i think rainbow must allow it (again fair i guess)

for nginx this is proxy_set_header Origin ""; and i’m curious to see how caddy goes for you! it fell over on the last medium-throughput thing i tried to do (but i’m always trying to get away with too-small machines)