ChatGPT may well revolutionize web search, streamline office chores, and remake education, but the smooth-talking chatbot has also found work as a social media crypto huckster.

Researchers at Indiana University Bloomington discovered a botnet powered by ChatGPT operating on X—the social network formerly known as Twitter—in May of this year.

The botnet, which the researchers dub Fox8 because of its connection to cryptocurrency websites bearing some variation of the same name, consisted of 1,140 accounts. Many of them seemed to use ChatGPT to craft social media posts and to reply to each other’s posts. The auto-generated content was apparently designed to lure unsuspecting humans into clicking links through to the crypto-hyping sites.

Micah Musser, a researcher who has studied the potential for AI-driven disinformation, says the Fox8 botnet may be just the tip of the iceberg, given how popular large language models and chatbots have become. “This is the low-hanging fruit,” Musser says. “It is very, very likely that for every one campaign you find, there are many others doing more sophisticated things.”

The Fox8 botnet might have been sprawling, but its use of ChatGPT certainly wasn’t sophisticated. The researchers discovered the botnet by searching the platform for the tell-tale phrase “As an AI language model …”, a response that ChatGPT sometimes uses for prompts on sensitive subjects. They then manually analyzed accounts to identify ones that appeared to be operated by bots.

“The only reason we noticed this particular botnet is that they were sloppy,” says Filippo Menczer, a professor at Indiana University Bloomington who carried out the research with Kai-Cheng Yang, a student who will join Northeastern University as a postdoctoral researcher for the coming academic year.

Despite the tic, the botnet posted many convincing messages promoting cryptocurrency sites. The apparent ease with which OpenAI’s artificial intelligence was apparently harnessed for the scam means advanced chatbots may be running other botnets that have yet to be detected. “Any pretty-good bad guys would not make that mistake,” Menczer says.

OpenAI had not responded to a request for comment about the botnet by time of posting. The usage policy for its AI models prohibits using them for scams or disinformation.

ChatGPT, and other cutting-edge chatbots, use what are known as large language models to generate text in response to a prompt. With enough training data (much of it scraped from various sources on the web), enough computer power, and feedback from human testers, bots like ChatGPT can respond in surprisingly sophisticated ways to a wide range of inputs. At the same time, they can also blurt out hateful messages, exhibit social biases, and make things up.

A correctly configured ChatGPT-based botnet would be difficult to spot, more capable of duping users, and more effective at gaming the algorithms used to prioritize content on social media.

“It tricks both the platform and the users,” Menczer says of the ChatGPT-powered botnet. And, if a social media algorithm spots that a post has a lot of engagement—even if that engagement is from other bot accounts—it will show the post to more people. “That's exactly why these bots are behaving the way they do,” Menczer says. And governments looking to wage disinformation campaigns are most likely already developing or deploying such tools, he adds.

Researchers have long worried that the technology behind ChatGPT could pose a disinformation risk, and OpenAI even delayed the release of a predecessor to the system over such fears. But, to date, there are few concrete examples of large language models being misused at scale. Some political campaigns are already using AI though, with prominent politicians sharing deepfake videos designed to disparage their opponents.

William Wang, a professor at the University of California, Santa Barbara, says it is exciting to be able to study real criminal usage of ChatGPT. “Their findings are pretty cool,” he says of the Fox8 work.

Wang believes that many spam webpages are now generated automatically, and he says it is becoming more difficult for humans to spot this material. And, with AI improving all the time, it will only get harder. “The situation is pretty bad,” he says.

This May, Wang’s lab developed a technique for automatically distinguishing ChatGPT-generated text from real human writing, but he says it is expensive to deploy because it uses OpenAI’s API, and he notes that the underlying AI is constantly improving. “It’s a kind of cat-and-mouse problem,” Wang says.

X could be a fertile testing ground for such tools. Menczer says that malicious bots appear to have become far more common since Elon Musk took over what was then known as Twitter, despite the tech mogul’s promise to eradicate them. And it has become more difficult for researchers to study the problem because of the steep price hike imposed on usage of the API.

Someone at X apparently took down the Fox8 botnet after Menczer and Yang published their paper in July. Menczer’s group used to alert Twitter of new findings on the platform, but they no longer do that with X. “They are not really responsive,” Menczer says. “They don’t really have the staff.”