Google's AI search gives websites dire choice: Share data or die
To To To They Google Their Run The Through From Tech Publishers Ai Ai — Being Their Should Done The Bolster Engine Block Have Similar Search What Crawling Giant's — Other Websites Risk Not Discoverable Panies Some
Google now displays convenient artificial intelligence-based answers at the top of its search pages — meaning users may never click through to the websites whose data is being used to power those results. But many site owners say they can’t afford to block Google’s AI from summarising their content.
That’s because the Google tool that sifts through web content to come up with its AI answers is the same one that keeps track of web pages for search results, according to publishers. Blocking Google, the way sites have blocked some of its AI competitors, would also hamper a site’s ability to be discovered online.
Google’s dominance in search — which a US federal court ruled last week is an illegal monopoly— is giving it a decisive advantage in the brewing AI wars, which search startups and publishers say is unfair as the industry takes shape.
The dilemma is particularly acute for publishers, which face a choice between offering up their content for use by AI models that could make their sites obsolete and disappearing from Google search, a top source of traffic.
“It becomes like an existential crisis for these companies,” said Joe Ragazzo, publisher of the news site . “These are two bad options. You drop out and you die immediately, or you partner with them and you probably just die slowly, because eventually they’re not going to need you either.”
Google said AI Overviews — the summaries displayed at the top of Google search — are part of its longstanding commitment to serve higher quality information and bolster opportunities for publishers and other businesses.
Since its earliest days, Google has deployed a piece of software known as the Googlebot to visit or “crawl” millions of websites, building up a detailed repository of the global internet. That index has posed a daunting barrier to entry for companies that have sought to build rival search engines over the years — even ones with deep pockets, like Microsoft.
The rise of generative AI has touched off a new wave of startups seeking to offer search products in which AI models deliver succinct answers to users’ questions. But before these startups can truly threaten the search giant’s business, they must crawl the web and that’s no easy feat.
Being crawled costs website owners money, computing power and storage, so many publishers include a file that sets out rules for bots visiting their sites. The companies given the most leeway are usually Google and Microsoft’s Bing, which can drive traffic to sites through their search engines.
Amid the boom in AI, there has been a wave of deal-making between media companies and startups but Google has been a notable holdout. With the exception of a reported $60m deal with Reddit, Google has signalled to publishers behind closed doors that it is not interested in negotiating, according to two people with knowledge of the matter, who asked not to be identified because the information is private.
Google’s deal with Reddit offers the company a treasure trove of information for AI models. The deal coincided with changes Google’s made that boosted the presence of results from forums like Reddit in search results, driving huge gains in traffic to the social media site.
Media companies have little leverage in conversations these tech giants. Earlier this year, Google rolled out AI Overviews, in which the company uses AI to give succinct answers to some of users’ questions at the top of the search page. Publishers were immediately concerned about the impact the answers could have on their traffic but had no clear way to address those fears.
Google has said publishers can block specific pages or parts of pages from appearing in AI Overviews in search results — but that would also likely bar those snippets from appearing across all of Google’s other search features, too, including web link listings.
Many publishers, which often rely on search engines for at least half their traffic, aren’t willing to take the risk of minimising their reach.
Google’s position “understates the significant risk this poses to content creators, particularly those who rely on search visibility for their livelihood,” said Marc McCollum, who heads up innovation at Raptive, which represents publishers and influencers.
“By opting out, creators may inadvertently reduce their overall search presence, which could harm their ability to reach audiences and generate revenue.”
Kyle Wiens, chief executive of iFixit., a website that publishes free online repair guides for consumer electronics, said the site’s relationship with Google is “much more tenuous” than with other AI companies.
“I can block ClaudeBot from indexing us without harming our business,” Mr Wiens wrote in an email, referring to the bot from generative AI startup Anthropic. But if I block Googlebot, we lose traffic and customers.”
Regardless of the outcome of the antitrust case against the tech giant, the changes which are underway in the search landscape underscore the importance for publishers of controlling their own destiny and not becoming overly reliant on any one tech platform — including Google, said Mr Ragazzo.
“Our belief is you have to form real relationships with readers,” Mr Ragazzo said, “and that’s how you build a publication that can withstand different eras.”