Post

fig (aka:[phil])
@bad-example.com

learned so far: - 4 cores seems _enough to run_ but on this particular host it’s kind of marginal. 16GB ram seems plenty. - it’s been dropping pds websockets, and doesn’t seem to recover on its own. not sure why. - bandwidth is less than i was expecting. - +jetstream on this server is too much

fig (aka:[phil])@bad-example.com

kinda still a relay noob but uh i think it’s doing the thing non-us jurisdiction (montreal) non-us-company-cloud-provider relay. chillin.

fig (aka:[phil])
@bad-example.com

- pds backfill is rough on this server, cpu pegs 100% - made worse by having jetstream on the same server trying to do a higher event rate - also interesting: jetstream just crashes when it’s relay websocket closes? fine if the supervisor auto-restarts it i guess.

fig (aka:[phil])
@bad-example.com

- 4 shared cores on a 2017-era 2ghz epyc is not enough for a relay. could probably tune some things to get it there—seeing significant overhead from userspace networking + logging—but i def want more performance margin than this. for $5.5CAD i can get two more shared cores, or double that for 8

fig (aka:[phil])
@bad-example.com

seems to work well yeah! no patches here (well i am stripping Origin in nginx to get it to not 403 from pdsls, lazy fix for probably nothing) getting sync going took some effort but probably mostly because i’m new to the system/code and its quirks. mostly i’m just… running it.

fig (aka:[phil])
@bad-example.com

jetstream is up!

fig (aka:[phil])@bad-example.com

indie jetstream from an indie atproto relay? yeah we got that too: wss://jetstream.fire.hose.cam/subscribe obligatory pdsls demo: pdsls.dev/jetstream?in... here, have an @tools: atp.tools/jetstream (open the little expand-y thing to add the wss URL)

fig (aka:[phil])
@bad-example.com

Relay operation learnings, storage edition - 48h content retention window: 582GB. Weekends are a little slower on atproto so say ~300GB/retention-day typical. This is less than I expected (i got 4TB) - sync state and everything else in postgres: 2.2GB. this thing is pretty efficient ✨

fig (aka:[phil])
@bad-example.com

got victoriametrics scraping node_exporter on the relay host so i can let btop take a rest finally. and get some better bandwidth numbers. whew. (insert tailscale-is-cool take about how it's literally the same for me to do this on the montreal relay vs my at-home pi from the at-home vicmetrics)

Eric
@edavis.dev

did you have to do anything special to get this working with pdsls? I have Caddy fronting one and it works with websocat but not with pdsls

fig (aka:[phil])
@bad-example.com

actually yes! two things, thank you for reminding me. 1. tls. assume you’re set since caddy 2. dropping the origin header in the reverse proxy. i haven’t dug into why exactly but the relay didn’t like it when a request included the origin.

fig (aka:[phil])
@bad-example.com

like it kind of makes sense— if you ship an app that gets people to connect to a relay from browser, something is probably not right. …but pdsls works with prod so i think rainbow must allow it (again fair i guess)