Random Image Display on Page Reload

Scammers Used ChatGPT to Unleash a Crypto Botnet on X

Aug 21, 2023 7:00 AM

Scammers Used ChatGPT to Unleash a Crypto Botnet on X

A botnet apparently connected to ChatGPT shows how easily, and effectively, artificial intelligence can be harnessed for disinformation.

Chatbot concept

Illustration: sakchai vongsasiripat/Getty Images

ChatGPT may well revolutionize web search, streamline office chores, and remake education, but the smooth-talking chatbot has also found work as a social media crypto huckster.

Researchers at Indiana University Bloomington discovered a botnet powered by ChatGPT operating on X—the social network formerly known as Twitter—in May of this year.

The botnet, which the researchers dub Fox8 because of its connection to cryptocurrency websites bearing some variation of the same name, consisted of 1,140 accounts. Many of them seemed to use ChatGPT to craft social media posts and to reply to each other’s posts. The auto-generated content was apparently designed to lure unsuspecting humans into clicking links through to the crypto-hyping sites.

Micah Musser, a researcher who has studied the potential for AI-driven disinformation, says the Fox8 botnet may be just the tip of the iceberg, given how popular large language models and chatbots have become. “This is the low-hanging fruit,” Musser says. “It is very, very likely that for every one campaign you find, there are many others doing more sophisticated things.”

The Fox8 botnet might have been sprawling, but its use of ChatGPT certainly wasn’t sophisticated. The researchers discovered the botnet by searching the platform for the tell-tale phrase “As an AI language model …”, a response that ChatGPT sometimes uses for prompts on sensitive subjects. They then manually analyzed accounts to identify ones that appeared to be operated by bots.

“The only reason we noticed this particular botnet is that they were sloppy,” says Filippo Menczer, a professor at Indiana University Bloomington who carried out the research with Kai-Cheng Yang, a student who will join Northeastern University as a postdoctoral researcher for the coming academic year.

Despite the tic, the botnet posted many convincing messages promoting cryptocurrency sites. The apparent ease with which OpenAI’s artificial intelligence was apparently harnessed for the scam means advanced chatbots may be running other botnets that have yet to be detected. “Any pretty-good bad guys would not make that mistake,” Menczer says.

Most Popular

OpenAI had not responded to a request for comment about the botnet by time of posting. The usage policy for its AI models prohibits using them for scams or disinformation.

ChatGPT, and other cutting-edge chatbots, use what are known as large language models to generate text in response to a prompt. With enough training data (much of it scraped from various sources on the web), enough computer power, and feedback from human testers, bots like ChatGPT can respond in surprisingly sophisticated ways to a wide range of inputs. At the same time, they can also blurt out hateful messages, exhibit social biases, and make things up.

A correctly configured ChatGPT-based botnet would be difficult to spot, more capable of duping users, and more effective at gaming the algorithms used to prioritize content on social media.

“It tricks both the platform and the users,” Menczer says of the ChatGPT-powered botnet. And, if a social media algorithm spots that a post has a lot of engagement—even if that engagement is from other bot accounts—it will show the post to more people. “That's exactly why these bots are behaving the way they do,” Menczer says. And governments looking to wage disinformation campaigns are most likely already developing or deploying such tools, he adds.

Researchers have long worried that the technology behind ChatGPT could pose a disinformation risk, and OpenAI even delayed the release of a predecessor to the system over such fears. But, to date, there are few concrete examples of large language models being misused at scale. Some political campaigns are already using AI though, with prominent politicians sharing deepfake videos designed to disparage their opponents.

William Wang, a professor at the University of California, Santa Barbara, says it is exciting to be able to study real criminal usage of ChatGPT. “Their findings are pretty cool,” he says of the Fox8 work.

Wang believes that many spam webpages are now generated automatically, and he says it is becoming more difficult for humans to spot this material. And, with AI improving all the time, it will only get harder. “The situation is pretty bad,” he says.

This May, Wang’s lab developed a technique for automatically distinguishing ChatGPT-generated text from real human writing, but he says it is expensive to deploy because it uses OpenAI’s API, and he notes that the underlying AI is constantly improving. “It’s a kind of cat-and-mouse problem,” Wang says.

X could be a fertile testing ground for such tools. Menczer says that malicious bots appear to have become far more common since Elon Musk took over what was then known as Twitter, despite the tech mogul’s promise to eradicate them. And it has become more difficult for researchers to study the problem because of the steep price hike imposed on usage of the API.

Someone at X apparently took down the Fox8 botnet after Menczer and Yang published their paper in July. Menczer’s group used to alert Twitter of new findings on the platform, but they no longer do that with X. “They are not really responsive,” Menczer says. “They don’t really have the staff.”

Get More From WIRED

Will Knight is a senior writer for WIRED, covering artificial intelligence. He writes the Fast Forward newsletter that explores how advances in AI and other emerging technology are set to change our lives—sign up here. He was previously a senior editor at MIT Technology Review, where he wrote about fundamental… Read more
Senior Writer

More from WIRED

It Costs Just $400 to Build an AI Disinformation Machine

A developer used widely available AI tools to generate anti-Russian tweets and articles. The project is intended to highlight how cheap and easy it has become to create propaganda at scale.

Will Knight

Kids Are Going Back to School. So Is ChatGPT

Teachers are caught between cracking down on cheating with generative AI and using it to help empower students. It’s going to be a challenging year.

Pia Ceres

It's Time to Rethink Digital Ownership

In his quest to watch every Nicolas Cage movie in chronological order, law professor and author Aaron Perzanowski confirmed that he owns nothing—and that you probably don’t, either.

Gideon Lichfield

The Cloud Is a Prison. Can the Local-First Software Movement Set Us Free?

Tired of relying on Big Tech to enable collaboration, peer-to-peer enthusiasts are creating a new model that cuts out the middleman. (That’s you, Google.)

Gregory Barber

WhatsApp Made a Movie About Afghan Women's Soccer

As the UK pushes for a law that threatens end-to-end encryption, WhatsApp has given itself a starring role in a doc about a girls’ soccer team fleeing the Taliban.

Morgan Meaker

Microsoft’s AI Red Team Has Already Made the Case for Itself

Since 2018, a dedicated team within Microsoft has attacked machine learning systems to make them safer. But with the public release of new generative AI tools, the field is already evolving.

Lily Hay Newman

Apps Are Rushing to Add AI. Is Any of It Useful?

The AI gold rush has thrust services like ChatGPT into every aspect of our digital lives. That still won’t solve your email problems.

Justin Pot

How X Is Suing Its Way Out of Accountability

The social media giant filed a lawsuit against a nonprofit that researches hate speech online. It’s the latest effort to cut off the data needed to expose online platforms’ failings.

Vittoria Elliott

Credit belongs to : www.wired.com

Check Also

The world’s coral reefs are facing another mass bleaching event — maybe the biggest ever

The U.S. National Oceanic and Atmospheric Administration has declared another mass coral bleaching event — …