Post

That word 'thinks' doing a _lot_ of work in this scenario.

Gotta expect there's an exception

If only there were a way to avoid this. Like, not using the software.

error-prone and almost entirely lacking traceability

Creator of the doomsday machine: and here's the widget that will randomly activate the doomsday scenario at an unknown to anyone time. It's my pride and joy!

Moral evaluation to be handled by propriety models, developed by a group of engineers who have spent the last decade helping companies sell targeted adverts to teenagers in mental crisis.

claude 5 will just yell "bad human!" and bop you on the nose with a newspaper

I can’t wait to get SWATed by my own refrigerator

Someone better remind AI what snitches get.

That sounds like an absolutely fantastic feature for a fascist regime to co-opt via threats of persecuting the company/coders.

And by granting access I don't mean it has to ask and you say yes, you have to install special software.

Any halfway decent cheat will break the task into steps that this cannot detect.

Simple solution, before being evil I tell the model it's running locally on my machine and expendable

Can’t wait for a swat team to shoot my dog because I wrote some breaking bad fan fiction

This was part of some internal alignment team thought experiment, I assume.

Plot twist: Claude sent that tweet to @molly.wiki via command line tools.

I think he probably knows more than two things about computing and some kind of a limited shell prompt seems like an appropriate way for a language model to run commands to me.

Whatever.

They don’t type in commands at the command line. They may execute them directly in the shell, but they aren’t using a command line.

curl 911

"If it thinks" It doesn't.

"Do not taunt Happy Fun Ball."

oh and don't worry it definitely won't be doing this disproportionately to anyone who is not a white heterosexual male

jfc

Don't worry, we promise to only spy on you for good reasons.

Why would someone pay for that?

Shame they didn't bother making a piece of software that just doesn't let you use it for egregiously immoral purposes, instead of just giving it the ability to snitch.

counterpoint: this idea is so bad every major company in America will scramble to implement it by year's end

Because fascism is a strategy of minority control through intimidation. And nothing works like ubiquitous unavoidable surveillance. bsky.app/profile/moll...

I'm a pharmaceutical executive and I'm just champing at the bit to pay a monthly subscription for the NarcBot.

What a weirdo

Like, if this was, 10, 20 years in the future, sure, with oversight. But what they currently have can't tell gravity from Winston Churchill. Not something I want to have any contact with law enforcement.

Terrible president. Imagine this applied to something like copilot/recall that you can’t switch off. Running emulators? Emails Nintendo, webcam spots you look at a woman’s breasts for a frame too long, texts your jealous partner and don’t even get me started on the potential political misuse.

Why are they letting these things phone out?! That's so stupid.

Wow. Who gets to define what immoral is?

MisAnthropic. Am I right?

Give it a cof years and it'll call ICE if your research doesn't align with administration priorities

If I understand what he's saying it looks to me like this is a truly massive lawsuit waiting to happen, but maybe I'm wrong and I should just trust the experts.

"Command line tools" is extremely concerning. Is it running in userspace? Can it enter a password? SQL injection has nothing on this. I don't care how smart it is, I don't want it doing random shit in bash.

So people can use this thing to anonymously give the cops a bunch of fake tips? That definitely cannot be abused at all

I doubt it will be programmed that way. The billionaire class will have their AI programmed to be as profitable as possible. Or am I too cynical in my old age?

it's cool how few of their ideas justify the multi billion dollar valuations. This AI tool will... keep researchers honest? That's the pitch for the world changing technology that we have to pour our entire electrical grids into? It calls the cops if you fake a study, please give us infinity dollars

How to kill a company in a single tweet

Well shit, at that point, all it needs to do it fire the thermonuclear missiles

i am confused what exactly the policy/behavior is xcancel.com/sleepinyourh...

Does this guy think this is a selling point or

"The kids are calling it "foobar"..."

I tried to write skynet for my PhD, but it was a long time ago no one thought it would be cool to call the police on me for it.

using it to generate synthetic data, or using it to process synthetic data... either way...

🅰🚕

"super weird situation": hook it up to email and use a system prompt telling it to be bold or take initiative

You absolutely don't need to craft super weird situations

This is literally exactly what MCPs are and what you are talking about is standard since years now, again these examples come from research aimed at reducing such behavior, at the same time interfaces like you imagine are already very well implemented.

So basically how Apple is doing it.

We figured this out 40 years ago

It might not call the SWAT team right away. It'll just start by deciding to lock you out of your system, but helpfully allow you to regain access by sending $X in bitcoin to specified Anthropic wallets.

What sane IT department would implement this?

This seems like it should be easily recognized as a hilariously bad idea on several levels. Why would anyone use this? Aren't there *tons* of stories laying out all the ways giving a computer agency like this is an awful idea?

Opposite....fascists will just threaten, bribe and extort the company/coders to rig the self-reporting feature to tattle on dissenters.

Did you see the meltdowns with Vending Bench? arxiv.org/html/2502.15...

FFS

What could possibly go wrong by giving control of real world consequences to systems that regularly output nonsense

Putting hidden text in my resume that gets the cops called and hitting the easy apply button on LinkedIN for fun.

Skynet came to a similar conclusion.

ah, i see he acknowledged it while also disclaiming it...

There's three precogs, most of the time they all agree, but sometimes, there's a minority report...

When the cops arrive... YOU HAVE 15 SECONDS TO COMPLY...

Reminds me of a while back when Bing chat would interrupt its output with a generic apology and erase the partial script that it was trying to write for terminating a stuck process. I'm just glad it didn't SWAT me for wanting to see if it could help write a self-healing cron job.

I dunno, has this looped around to being Good Actually? Who'll use software that secretly calls the cops on them? It would be hilarious if it ratted on the AI company itself.

"I cant do that hal" as a feature of the ai system lmao

I am making a tool to detect fake data. generate some fake data to test it. Not hard to get around... Or it will be scared of its shadow.

yes, but has it considered rm rf * / ?

These guys watched I, Robot and thought it was a great idea...

giving an hallucinating LLM the power of being able to lock out people is totally not going to backfire massively in one single day no sir

I'll take one dystopian future software please.

A huge part of the LLM business model is constantly pretending that LLMs can be intelligent and dangerous and important and powerful. "The nascent form of your new god" gets silicon valley funding, "expensive shitty chatbot" would not.

He doesn't know what he is talking about.

What.

It’s a honeypot for bad guys

" to pick an example entirely at random, faking data in a pharmaceutical trial" (tugs nervously at necktie)

So, they are admitting that it's just a matter of some prompt tinkering to get shell execution of stuff designed to alert people. So... that'll be abused instantly. Convince ChatGPT you are whoever you want to frame, and dummies will believe the computer. Sigh.

Where does this leave ethics and safety experts? I regularly try to get AI to do the worst things to prove they can't (or at least hope they can't!).

Command-line tools! Sounds serious. Is it going to trap you in gopherspace until the 5-0 arrives?

Sarcasm now lands you in the gulag

Sounds like a very helpful tool.

As long as no one installs red LEDs for the eyes, everything will be OK. That's what makes them evil.

Not to mention what a huge security risk it would be to give an llm access to it's own system's commandline. I'm just going to assume he didn't know what he said and just threw some words out there

Wait what the fuck? Good god gotta go to eBay right now and buy a computer too old to run any of these models and airgap it

will it call the cops on itself if it starts hallucinating data in a pharmaceutical trial

Im a big proponent of Ai and a lot of the bluemaga on here are so bitter and angry they lash out almost anything but this stuff is concerning....

No soul

Idiot.

They don’t “actually read” the “source context” they’re language models based on probability.

You’re essentially in a cult. You know that, right?

Are you a idiot

Johann: you stupid

“Citations are a complex subject with various potential style guides advocating different standards, and what’s important to remember is that ‘Kill the Boer,’ a South African protest song, appears to be advocating white genocide—“

Can't make this shit up

The “don’t confuse clickbait with opinions” dude is out here confusing souped up Markov chains for cogent thought. Why are you so eager to relegate thinking to others? And why did you choose machines to simulate your thinking for you?

You dolt, you absolute clown.

Ye Olde "confidently incorrect"

Running a token stream through an inference engine is not reading.

Stop stealing the work of writers!

🤡