Will AI Kill The Web?

The Atlantic is out with a delicious piece of doomerism: It’s The End of the Web As We Know It. Were it not for the authors, Judith Donath and Bruce Schneier, I’d have passed right on by, because well-respected publications have been proclaiming the death of the Web for more than a decade. By and large they’ve been proven directionally right, but it’s taking a lot longer than most predicted. Much like the Earth’s coral reefs, the Web has been dying off in waves. In their piece, Donath and Schneier argue that generative AI augurs the Web’s final stage of life. The coup de GPT, if you will.

Donath and Schneier are more thoughtful than your average trend-spotting feature writers. Donath, a fellow at Harvard’s Berkman Center, is the kind of digital polymath I’ve admired for years, but who’s been relatively quiet of late. No longer. And Schneier, a security expert, writes smart stuff about nearly everything I care about as it relates to the Internet. It’s fair to say I’ll read just about anything he writes.

Donath and Schneier argue that large-language models will kill the open web by replacing SEO — the practice of optimizing search results to drive web traffic — with “Large Language Model Optimization,” or LLMO. “SEO will morph into LLMO,” the authors write, “the incipient industry of manipulating AI-generated material to serve clients’ interests.”

The authors predict the rise of an LLMO industry catering to politicians, corporations, and pretty much everyone else with a vested interest in controlling the information ecosystem. Just as spammy “made for advertising” sites like Demand Media (and countless others) polluted the SEO-driven landscape of the past 25 years, the LLMO industry will pollute the emerging world of ChatGPT, Google Gemini, and Microsoft Copilot. Creators will tire of their work being turned into fodder for LLM’s automatic Turk machines, and as a result, the open web will die:

Eventually, people may stop writing, stop filming, stop composing — at least for the open, public web. People will still create, but for small, select audiences, walled-off from the content-hoovering AIs. The great public commons of the web will be gone.

This is a compelling and seemingly sensible argument, but as I read it, something didn’t feel quite right. The piece asserts any number of assumptions which are either not currently true, or may become false as the Web co-evolves with generative AI technologies and platforms.

I think the core problem is the assumption that a “large language model optimization” industry in about to emerge. This presumes that the companies who invest in large-language models — Google, Microsoft, OpenAI, Meta, and a few others — are motivated to encourage such an industry to flourish in the first place. I’m not so sure that’s the case. As with nearly everything related to business, you have to follow the money to understand the incentives. If LLMs are driven, in large part, by an advertising model similar to the one that built search and the current incarnation of the Web, then I’d tend to agree that we would see SEO-like behaviors emerge in LLM systems.

But so far LLMs seem to be adopting a direct-to-consumer model of subscription, which bends the economic incentives of large platform models toward high-quality, useful results. Besides a few nutjobs on X, no one wants to subscribe to a chatbot that spouts manipulated garbage. One could just as easily argue that the trend of subscription for LLMs augurs well for the future of the Web — high-quality sites will increasingly be in demand, and platforms will have to figure out how to reward them by sharing subscription revenue flows with publishers who provide them with dependable raw material.

Platforms like Google and Meta won’t do this out of the goodness of their hearts — they’ll be forced to by their need to compete for a consumer’s loyalty, usage, and dollars. Currently LLMs are exploring one-off business development deals with publishers — OpenAI with Axel Springer, Google with Reddit, Apple posing as the good guy for the positive publicity. This approach will never scale, but it might inform a more automated approach as the Web evolved. To get there, we’ll need to solve more basic problems with LLMs: Opacity. It’s impossible to determine the value of a site like, say, Searchblog, if no one understands how the LLM creates value from it in the first place.

But that’s a Gordian knot for another day.

—

You can follow whatever I’m doing next by signing up for my site newsletter here. Thanks for reading.