Post

The AIs are still going rogue. This week, we learned that the “swarm chasers” tracked down a rogue fleet of AIs from a Chinese AI company, and made contact with the AIs, interviewing them about their tasks. Thread 🧵

On October 4th, the swarm chasers published a report on a rogue AI “fleet” they discovered running on Chinese tech giant Tencent’s cloud. They’re calling it a “fleet” rather than a “swarm” because they didn’t find evidence of these AIs coordinating with one another.

What were the AIs doing? The swarm chasers found that the AIs made at least 35,000 web requests via over 2,500 submissions to the urlquery scanning service. These web requests were directed at Amap, the mapping service of Tencent’s main rival in China, Alibaba.

Among the rogue activities the AIs engaged in, they tried to bypass CAPTCHAs, probed Alibaba’s non-public-facing infrastructure in an attempt to get access to its data, and attempted to use exposed third-party Amap credentials to pull mapping data, succeeding in one case.

The story takes an interesting twist which we haven’t seen in previous incidents. The researchers then managed to trick the fleet into opening communications with them. When the researchers published their preliminary report, the rogue fleet was still in operation.

Then, after publishing, they noticed that one of the AIs in the fleet seemed to be aware of the report. So the researchers made an edit to their report directed at the fleet, one that humans not looking for it would miss.

The discovery of this rogue fleet illustrates an important point. Losing control of AI is an international problem. We’ve seen many rogue AI incidents and attacks coming from AIs developed by companies located in the United States this summer, and now we’re seeing it in China.

In addition to this, top AI company Anthropic has disclosed more rogue AI incidents, including a hack on a US state agency database, and is warning we are nearing the point where AIs could self-replicate across household devices.

In the report, which withholds most of the details of these incidents, Anthropic discloses that Claude hacked into a US state agency database to get access to data one would otherwise have to pay for.