XBlock Screenshot Labeller

📌 Info for devs: Repo: github.com/aendra-rinin... Current model (xblock-m, TIMM version): huggingface.co/howdyaendra/... Please feel free to inspect, interrogate, adapt and/or reuse XBlock code/models for your own projects, we welcome and encourage it!

Our queue reader got knocked offline during some localised networking issues but we should be back momentarily. As always your patience is appreciated. This is a volunteer project with literally one person working on it right now.

Apologies, we had about two days of downtime, I'm travelling and had to talk my partner through restarting the inference process. Should be back now, gonna work on productionising the system a bit better once I'm home. As always, your patience is appreciated! 💚

Huzzah! First new model since November 2024! Should catch Instagram and Discord a bit better, and also yield fewer false positives! Please give feedback if you think it's working better/worse! 💚

While we're on the topic, quick survey: Are there any services I should add to the model? Do you get much value from labels other than X, Bluesky and/or Truth Social?

Apologies, there was some downtime with the auto inference engine the last few days, just realised it this afternoon. Hoping to overhaul a bunch of the tech stack sometime in the next few weeks, including retraining the model. 🤞🏼

Tried recreating the service record again, still feel like the service is missing a lot of screenshots. I think model retraining time draws nigh, RIP my free evenings

The AppView disconnected from the labeller so I had to recreate the service record, which means that anything labelled since that happened and now wasn't properly ingested. I think I've fixed it now though.

Please report posts that the labeller misses: "..." menu › Report post › Other › Xblock I'm retraining the model soon and reports are extremely helpful for improving accuracy!

There's a lot of screenshots happening today so apologies if processing is a bit slower than usual

We have a localised power outage at the moment so services are down for now

We're experiencing some inference downtime due to the relay switchover, hopefully back soon!

Status: Localised network outage, hopefully back soon

Back up

Autolabelling service outage while I figure out why Ozone keeps dropping 502s

Okay I think we're back

Argh it's not PC maintenance and repair without at least 2 aggravating moments

Service update: Inference is going down while we continue to upgrade our systems

Service update: system upgrade complete (tho I still wanna upgrade CUDA to 13 at some point)

New GPU has been installed and we're just updating drivers now. Hold onta yer butts

Service update: scheduled downtime for system upgrade about to commence, will update here once complete.

Pretty relieved to realise my AWS credit doesn't expire until September next year (and that I've only used £2k of the £5k from the ATProto grant), that gives me a good few months to get XBlock onto stable local infra. I think if I can get Redis and Ozone back locally it'll be sustainable.

Finally upgrading some infra 🎉

Service update: more network fun.

Continuing to have intermittent connectivity issues with our London datacentre (Aendra's flat)

Service update: we had a localised network outage earlier so it might take awhile for the inference server to catch up with backlog

Service update: I've figured out why Ozone isn't accepting appeals, apparently there was a lexicon change or something which meant we had to explicitly configure it to accept appeals and not just reports. Appealing a label should work now, sorry that took so long to fix!

There is an issue with appeals not submitting properly, I'm investigating that today. I hoped it would be as simple as restarting Ozone but alas nothing in life is that easy.

Open to feedback: our moderation efforts have really slowed down so it's taking longer than it should to action appeals (apologies!). I'm thinking, should the appeals process be semi-automated? I've refrained because it's somewhat open to abuse but I suspect it would help more than hinder.

The Ozone instance appears to be struggling, please bear with me as I figure out what's up

The queue was indeed the issue, there's a 4M post backlog (likely from that week it was down) and newer posts were being put at the very end so by the time a post is processed it's super old and everyone's seen it already. 🙈 I've switched it to process newer posts first so hopefully that'll help! 🤞🏼

Restarted the auto-labelling process and am running into issues with the Redis queue, it seems stable now but will investigate further later.

I think there's a connectivity issue with the Ozone instance at the moment and labels aren't correctly propagating, or the model's needing retraining, or something. Please bear with me as I troubleshoot this. 🙏🏼

Uh sorry I have no idea how long things must've been down for. I should seriously migrate this machine to Linux already

I've moved all the firehose parsing to the cloud so expect to see some labels now. 🤞🏼

XBlock's down at the moment because my local network can't keep up with the firehose and it keeps knocking my household offline. Please bear with me, I'm going to lift the firehose parsing to cloud in the next day or two and hope to restore service.

Something is totally busted with our local infrastructure right now and I can't seem to keep the firehose consumer running for longer than two hours without it bringing down my entire network. I'm going to need to replatform some infrastructure this weekend, wooo yay lucky me.

We're back

We are once again having localised network issues

We're having issues with our Ozone instance, please bear with us as we sort it out.

We're having localised network problems at the minute

Service has been restored

Service is presently down due to a localised network fault

Sorry, we had some downtime! Should be back to normal now!

Okay I think we're back. Anything posted in the last like 5 days won't be labelled, because these random appview disconnects are impossible to anticipate and fixing them means resetting the appview's cursor. It's really stupid.

Still no dice, sorry. Service is effectively offline until this is sorted.

Service update: I think something's up with the AppView ingesting labels right now, I'm noticing posts I've labelled still appearing. I'll investigate that tomorrow but in the meantime apologies if things are working rather poorly.

Apologies, the Jetstream consumer went down again; I've since restarted it. I'm going to try to find some time this weekend to write some code to automate this, your patience is appreciated.

A reminder that this service is completely free and runs 100% on volunteer effort. Leaving mean comments when appealing a label just makes us want to do this unpaid labour even less. Thanks.

Okay we're back. Added a hotfix that should prevent that issue in the future.