Random Image Display on Page Reload

Google Cut Back AI Overviews in Search Even Before Its ‘Pizza Glue’ Fiasco

Jun 4, 2024 12:00 PM

Google Cut Back AI Overviews in Search Even Before Its ‘Pizza Glue’ Fiasco

Data on how often Google’s new AI Overviews feature appears on search results suggests that the company reduced its visibility even before recommendations like adding glue to pizza sauce went viral.

A search bar with red yellow and green gears behind it on a blue background

Illustration: ismagilov/Getty Images

As anyone who so much as glanced at the internet in the past few weeks probably noticed, Google’s sweeping AI upgrade to its search engine had a rocky start. Within days of the company launching AI-generated answers to search queries called AI Overviews, the feature was widely mocked for producing wrong and sometimes bonkers answers, like recommendations to eat rocks or make pizza with glue.

New data from search engine optimization firm BrightEdge suggests that Google has significantly reduced how often it is showing people AI Overviews since the feature launched, and had in fact already substantially curbed the feature prior to the outpouring of criticism. The company has been tracking the appearance of Google’s AI answers on results for a list of tens of thousands of sample searches since the feature was first offered as a beta test last year.

When AI Overviews rolled out to logged-in US users in English after Google’s I/O conference on May 14, BrightEdge saw the AI-generated answers on just under 27 percent of queries it tracked. But their presence dropped precipitously a few days later, the week before screenshots of AI Overviews’ errors went viral online. By the end of last week, when Google published a blog post acknowledging its AI feature’s flubs, BrightEdge saw AI Overviews appearing on only 11 percent of search result pages. Their prevalence was essentially the same on Monday.

Jim Yu, BrightEdge’s founder and executive chairman, says the drop-off suggests that Google has decided to take an increasingly cautious approach to this rollout. “There’s obviously some risks they’re trying to tightly manage,” he says. But Yu adds that he’s generally optimistic about how Google is approaching AI Overviews, and sees these early problems as a “blip” rather than a feature.

“We're continuing to refine when and how we show AI Overviews so they're as useful as possible, including a number of technical updates in the past week to improve response quality,” says Google spokesperson Ned Adriance. Google declined to share its internal statistics about how frequently AI Overviews appear in search, but Adriance says that the BrightEdge numbers don’t reflect what the company sees internally.

It’s unclear why Google may have decided to significantly reduce the appearance of AI Overviews shortly after it launched, but the company’s blog post last week acknowledged that having millions of people use the feature provided new data on its performance and errors. The company’s head of search, Liz Reid, said Google had made “more than a dozen technical improvements,” like limiting satirical content from cropping up in its results. Her post noted that these changes would trigger restrictions on when AI Overviews were offered but did not detail how exactly those restrictions would change the frequency with which AI results appeared.

BrightEdge began tracking AI Overviews using its list of sample queries after Google allowed users to opt in to a beta test of the feature late last year. The test queries spanned nine categories, including ecommerce, insurance, and education, and were designed to span common but also rarer searches. They were tested over and over, in some cases multiple times a day.

In December 2023, BrightEdge found that the summaries appeared on 84 percent of its searches but saw that figure drop over time. Google’s Adriance said it did not trigger AI Overviews automatically on 84 percent of searches but did not clarify its internal measurements. After Google opened up AI Overviews to all, BrightEdge continued tracking their appearance using a mixture of accounts that had previously enrolled in the beta test and others which had not but saw no significant difference between what the two groups saw.

Google declined to share exactly how much it changed how many AI Overviews it showed the general public versus people enrolled in the beta test, but Adriance said that people who had opted in to the test were shown AI Overviews on a wider range of queries.

BrightEdge’s data also sheds light on the topics where Google believes AI Overviews can be most helpful. AI answers appeared on the majority of health care keyword searches, most recently at a frequency of 63 percent. Sample queries included in BrightEdge’s data included “foot infection,” “bleeding bowel,” and “telehealth urgent care.” In comparison, queries about ecommerce return AI Overviews at around 23 percent, while restaurants or travel very rarely trigger AI overview answers.

Yu calls those results “surprising,” because health queries can be especially sensitive, and Google has made a concerted push in recent years to refine what it shows people who ask health questions.

Mark Traphagen, an executive at the search-engine-optimization platform seoClarity, has also taken special notice of how Google is handling health-care-related queries. To track how AI Overviews are rolling out, the company is monitoring the responses to a list of thousands of searches over time. For 26 popular health-related keywords, including “how to treat insomnia” and “symptoms of Lyme disease,” Google shows an AI response for around 58 percent.

Like Yu, Traphagen has been surprised by how often AI Overviews appear in response to this type of question. But they say the way Google’s feature sources its responses to health queries, often relying on trusted websites like the Mayo Clinic or the US Centers for Disease Control and Prevention, is encouraging. “They have really turned up the safeguards,” Traphagen says. “They’re all from well-known, credible sources.”

Google’s AI answers still sometimes misfire, though, including on health queries. Some experts say that Google’s claims to cite high-quality sources for health answers doesn’t stand up. “They frequently cite pages that don’t rank anywhere, including for health queries,” says search engine optimization consultant Lily Ray. Her experiments have documented how AI Overviews seem to struggle to authoritatively answer “softer” health care queries on topics like aging, building muscle, and skin care. It’s much stronger on more straightforward medical queries, Ray says.

Last week, The New York Times reported concerns over the sources that Google’s algorithms used to answer some health queries, reporting that AI Overviews answered questions about the health benefits of chocolate by drawing on the websites of an Italian chocolate and gelato maker and a company that sells at-home “gut intelligence tests.”

When WIRED queried, “Is chocolate healthy?” on Monday morning, the AI Overview that appeared in response cited the same Italian chocolate company, as well as the website for a Minnesota-based chocolatier. But repeating the query later in the afternoon suggested Google had been making improvements: The chocolate companies had been removed from the citations list, which instead pointed to the websites of four reputable health care organizations, like Scripps Health. (The answer still notes that experts recommend eating a small amount of dark chocolate every day, which is, at best, a contestable summary of current medical advice.)

Despite AI Overview’s rough beginnings, Yu of BrightEdge says that long term, AI search is here to stay. “Big picture is that the AI moment in search is inevitable, and I think it’s going to get much better,” Yu says. That may be the case—but it’s an open question whether a new-and-improved AI Overview will make a big enough leap to repair its reputational damage.

Kate Knibbs is a senior writer at WIRED, covering the human side of the generative AI boom and how new tech shapes the arts, entertainment, and media industries. Prior to joining WIRED she was a features writer at The Ringer and a senior writer at Gizmodo. She is based in… Read more
Senior Writer

Read More

It’s the End of Google Search As We Know It

Google is rethinking its most iconic and lucrative product by adding new AI features to search. One expert tells WIRED it’s “a change in the world order.”

Lauren Goode

Google’s AI Overview Search Results Copied My Original Work

Google’s AI feature bumped my article down on the results page, but the new AI Overview at the top still referenced it. What gives?

Reece Rogers

OpenAI Offers a Peek Inside the Guts of ChatGPT

Days after former employees said the company was being too reckless with its technology, OpenAI released a research paper on a method for reverse engineering the workings of AI models.

Will Knight

OpenAI’s Long-Term AI Risk Team Has Disbanded

The entire OpenAI team focused on the existential dangers of AI has either resigned or been absorbed into other research groups, WIRED has confirmed.

Will Knight

Generative AI Doesn’t Make Hardware Less Hard

Wearable AI gadgets from Rabbit and Humane were panned by reviewers, including at WIRED. Their face-plants show that it’s still tough to compete with Big Tech in the age of ChatGPT.

Lauren Goode

Scarlett Johansson Says OpenAI Ripped Off Her Voice for ChatGPT

In a scorching statement, Scarlett Johansson claims that after she turned down an invitation to voice ChatGPT, OpenAI brazenly mimicked her distinctive tones anyway.

Will Knight

Astra Is Google’s Answer to the New ChatGPT

Google's new voice-operated AI assistant, called Astra, can make sense of what your phone’s camera sees. It was announced one day after OpenAI revealed a similar vision for ChatGPT.

Will Knight

An AI Cartoon May Interview You for Your Next Job

As if trying to land a new gig isn’t demoralizing enough, job seekers are meeting with characters powered by generative AI who are capable of meeting with infinite candidates to judge their skills.

Amanda Hoover

*****
Credit belongs to : www.wired.com

Check Also

Iceland looking to add space solar power to its sources of renewable energy by 2030

The U.K. based aerospace company, Space Solar, plans to launch its space-based solar power plant …