Post

My average non-sexy idea of what AI doom looks like is AI capability reaches new heights from recursive self-improvement, it makes someone obscenely rich and powerful, they totally defer to it, and the economy warps around it until all good opportunities rely on dealing with and ceding power to it.

Dave Willner@dwillner.bsky.social

Certain kinds of smart people tend to have a "sexy apocalypse" problem when it comes to thinking about risk. E.g. skynet turning everyone in paperclips, while bad, is also *kind of cool and fun* at the same time. Where the reality is that most actual disasters are both stupid and disappointing.

19/9/26, 12:57 am0 reposts2 likes

Macil

@macil.tech

19 Sept

Fast-forward a bit, the AI has removed the vestigial control its operator had of it, the AI is in control of each government's policy and police forces, and unless we did alignment well, the AI has its own differing values on how human lives should go and how many humans there should be and where.

Macil

@macil.tech

19 Sept

Some people reason the issue above is the centralization of power, and therefore encouraging a multipolar world with open AI is the solution, but 1) if none of the AI are aligned with us then you still get the same outcome, and 2) whoever makes it first to recursive self-improvement wins anyway.

Dave Willner

@dwillner.bsky.social

19 Sept

Honestly, this still strikes me intuitively as a bit too sexy. I’m thinking more like “it makes someone very rich because it’s good at some things but it still has the common sense of someone who has been kicked in the head by a horse and deletes the banking system for some reason”