A deep dive on Search Generative Experience (AI Current #22)

The Current Wave

Google botched it once again.

The new Search Generative Experience (SGE), where Google attempts to answer search queries directly on the results page with AI-generated responses, is an utter disaster. The sad state of SGE reflects the internal struggle within Google's search department between engineers and the revenue team.

On launch day, Twitter (and Threads) was flooded with screenshots showcasing the nonsense generated by Google. It advised putting glue on pizza to stop the cheese from sliding down. It insisted that former US president John Adams graduated from the University of Wisconsin 21 times; a dog played in the NBA; Obama is a Muslim; Batman is a cop; snakes are mammals; and eating one small rock per day is beneficial for health.

To save face, Google is scrambling to manually disable SGE for many search queries — and the only way to do this is testing questions and evaluating the answers one by one. It's likely that some Google employees (and especially their crowd worker contractors) haven’t slept for days.

The SGE blunder is especially unfortunate considering the impressive announcements made by the company at the I/O conference (for a summary, see AI Current #21).

Let’s dive a bit deeper and see what is happening, why it is happening, and how Google is actively contributing to the worsening of the situation.

A lazy garbage truck called SGE

Here's how Search Generative Experience appears to function for most queries:

1. Runs a search in the background
2. Takes the first x best results
3. Generates a summary with Gemini
4. Displays it in a nice digestible format

It's simple — too simple, actually. This is low-effort, lazy approach to generating potentially informative content. The issues with this approach are so obvious that SGE, in its current form, should never have been released to the public.

It's easy to see how the quality of the AI-generated summary is dependent on the quality of the input. If the top x results are garbage, well, you guessed it — the output will be garbage too. Take the cheese-on-pizza problem: the idea of adding glue to the sauce originated from a Reddit comment posted 11 years ago (!), clearly intended as a joke.

The fact that this comment made its way into the summary is inexcusably lazy, and the contrast with some newer market entrants like Perplexity or Arc is striking.

Of course, Reddit is both a terrible and wonderful source of information. It's a light-hearted, troll-magnet platform but also an incredible collection of niche knowledge. Simply removing it from the sources might be a net negative.

Indeed, Perplexity does use that particular Reddit thread as a source. However, it's intelligent enough to compare it to the other 13 sources and realize that glue probably isn't a great answer to the question:

For the query "How many rocks should I eat?" Google's AI summary states, "According to UC Berkley geologists, eating one small rock per day is recommended because rocks contain minerals and vitamins that are important for digestive health.”

This "advice" was published on The Onion, a well-known satirical website. Google is smart enough not to use content from there (slow clap), but dumb enough to use the very same text reposted on the website of a hydraulic fracturing and reservoir simulator software (excuse me?). There's clearly no trustworthiness control built into SGE. Google is simply collecting garbage and transporting it from one place to another.

The infuriating part is that the solution might be quite simple. Even Gemini Flash, the smallest of the LLM family, knows that this answer is potentially harmful. Heck, even Gemma-2B, Google's older open source model (which is so small it can practically run on a refrigerator), knows the answer is wrong:

So, don't believe the dozens of gleeful articles claiming AI-powered search is a mistake and will never work. Google's SGE debacle isn't a technical problem — it's sheer incompetence.

SGE and the content doom loop

AI-generated search summaries have the not-so-unintended consequence of keeping users on the results page. After all, why would you click on anything if your question is already answered right there? Google displays the sources at the end of the AI-generated summary, but those will soon be crowded out by paid links (sources will still be visible — if you scroll sideways).

This makes perfect sense from Google's perspective. They really don't want you to click on any non-sponsored link on the results page. Sure, their company vision is to organize the world's information, but they are a business and they want to make money.

Unfortunately, this also removes long-established incentives, accelerating the enshittification of the web and triggering a content doom loop with dire consequences for future AI training data.

Large Language Models require high-quality training data — factual and well-written. It’s no wonder OpenAI is partnering with publishers left and right, most recently with Vox Media. But producing this content isn’t free.

Here's a (greatly over-)simplified look at how the business of publishing works:

This Content-Discovery-Monetization model has been working well for around two decades, with clear incentives:

• Publish good content
• Google will help people discover that content and send traffic to your sites
• You can monetize this traffic in some way (ads, affiliate marketing, products, etc)
• A part of this revenue is used to create more good content

With Search Generative Experience, Google isn't holding up their end of the bargain. They're trying to answer user questions directly on the results page and monetize the AI summaries instead of sending people to external pages:

Publishers refer to this as "Google Zero" — receiving no traffic from organic search, which can lead to many smaller outlets simply going out of business. In his Decoder show, Nilay Patel grilled Sundar Pichai hard about it — it's an excellent interview that inspired parts of this newsletter issue.

The problem is obvious:

• Less discovery
• Less monetization
• Less good quality content
• Less training data for Large Language Models

Instead of quality journalism, we're getting an absolute avalanche of AI-generated articles, or synthetic data if you will. Now, synthetic data certainly has its uses in AI training, but there's at least some research suggesting that relying on it too much can lead to model collapse.

Welcome to the content doom loop.

I personally don’t think we will run out of training data anytime soon. But good training data? I wouldn’t be so sure about that.

Thanks for reading the AI Current!

I publish this newsletter twice a week on Tuesdays and Fridays at the same time on Substack and on LinkedIn.

Don’t be a stranger! Hit me with a LinkedIn DM or say hi in the Substack chat.

I work at Appen, one of the world’s best AI training data companies. However, I don’t speak for Appen, and nothing in this newsletter represents the views of the company.